Megatest: Documentation

Road Map
--------

// This file is part of Megatest.
//
// Megatest is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// Megatest is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with Megatest. If not, see <http://www.gnu.org/licenses/>.

TODO / Road Map
---------------

Note: This road-map is a wish list and not a formal plan. Items are in
rough priority but are subject to change. Development is driven by
user requests, developer "itch" and bug reports. Please contact
matt@kiatoa.com with requests or bug reports. Requests from inside
Intel generally take priority.

Dashboard and runs

. Multi-area dashboard view

Tests Support

. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME
. Improve [script], especially indent handling

Scalability

. Overflow database methodology - combine the best of the v1.63
multi-db approach and the current db-in-tmp approach (currently
slowness can be seen when number of tests in a db goes over 50-100k,
with the overflow db it will be able to handle 1000's of runs with
50-100k tests per run). High priority - goal is to complete this by
20Q3.

Mtutils/CI

. Enable mtutil calls from dashboard (for remote control)
. Logs browser (esp. for surfacing mtutil related activities)
. Embed ftfplan for distributed automation, completed activities trigger QA runs which trigger deployment etc.
. Jenkins junit XML support [DONE]
. Add output flushing in teamcity support

Build system

. ./configure => ubuntu, sles11, sles12, rh7 [WIP]
. Switch to using simple runs query everywhere
. Add end_time to runs and add a rollup call that sets state, status and end_time

Code refactoring/quality/performance

. Switch to scsh-process pipeline management for job execution/control
. Use call-with-environment-variables where possible.

Migration to inmem db and or overflow db

. Re-work the dbstruct data structure?
.. [ run-id.db inmemdb last-mod last-read last-sync inuse ]

Some ideas for Megatest 2.0

. Aggressive megatest.config and runconfig.config caching.
.. Cache the configs in $MT_RUNPATH
.. Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed
. If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests
. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.
. Per test copy commands (example is incomplete).
----------------
[testcopy]
%/iind% unison SRC DEST
% cp –r SRC DEST
----------------

Add ability to move runs to other Areas (overlaps with overflow db system)

. allow shrinking megatest.db data by moving runs to an alternate
Megatest area with same keys.
. add param -destination [area|path]. when specified runs are copied to new
area and removed from local db.
. the data move would involve these steps
.. copy the run data to destination area megatest.db
.. mark the run records as deleted, do not remove the run data on disk
. accessing the data would be by running dashboard in the satellite area
. future versions of Megatest dashboard should support displaying areas in a
merged way.
. some new controls would be supported in the config
.. [setup] => allow-runs [no|yes] <== used to disallow runs
.. [setup] => auto-migrate=[areaname|path] <== used to automatically
migrate data to a satellite area.

Eliminate ties to homehost (part of overflow db system)

. Server creates captain pkt
. Create a lock in the db
. Relinquish db when done

Tasks - better management of run manager processes etc.

. adjutant queries tasks table for next action [red]#[Migrate into mtutil]#
.. Task table used for tracking runner process [red]#[Replaced by mtutil]#
.. Task table used for jobs to run [red]#[Replaced by mtutil]#
.. Task table used for queueing runner actions (remove runs,
cleanRunExecute, etc) [red]#[Replaced by mtutil#]
. adjutant (server/task dispatch/execution manager)

Stale propagation

. Mark dependent tests for clean/rerun -rerun-downstream
. On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify
. Fix: refresh of gui sometimes fails on last item (race condition?)

Bin list

. Rerun step and or subsequent steps from gui [DONE?]
. Refresh test area files from gui
. Clean and re-run button
. Clean up STATE and STATUS handling.
.. Dashboard and Test control panel are reverse order - choose and fix
.. Move seldom used states and status to drop down selector
. Access test control panel when clicking on Run Summary tests
. Feature: -generate-index-tree
. Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2

. rest api available for use with Perl, Ruby etc. scripts
. megatest.config setup entries for:
.. run launching (e.g. /bin/sh %CMD% > /dev/null)
.. browser "konqueror %FNAME%

. refdb: Add export of csv, json and sexp
. Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process.
. Re-work text interface wizards. Several bugs on record. Possibly convert to gui based.
. Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test
. Refactor Run Summary view, currently very clumsy
. Add option to show steps in Run Summary view
. Refactor guis for resizeablity
. Add filters to Run Summary view and Run Control view
. Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS...
. Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G
. Tool tips
. Filters on Run Summary, Summary and Run Control panel
. Built in log viewer (partially implemented)
. Refactor the test control panel
Help and documentation
. Complete the user manual (I’ve been working on this lately).
. Online help in the gui
Streamlined install
. Deployed or static build
. Added option to compile IUP (needed for VMs)
. Server side run launching
. Wizards for creating tests, regression areas (current ones are text only and limited).
. Fully functional built in web service (currently you can browse runs but it is very simplistic).
. Gui panels for editing megatest.config and runconfigs.config
. Fully isolated tests (no use of NFS to see regression area files)
. Windows version