The Megatest Users Manual
<matt@kiatoa.com>
-version 1.0, -April 2012 +version 1.5, +June 2020
Table of Contents
This book is organised as three sub-books; getting started, writing tests and reference.
License
-
Copyright 2006-2017, Matthew Welland.
+ Copyright 2006-2020, Matthew Welland.
This document is part of Megatest.
Megatest is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@@ -813,11 +813,11 @@
The Megatest project was started for two reasons, the first was an
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years. I
thought a single open source tool, flexible enough to meet the needs
-of any team doing continuous integrating and or running a complex
+of any team doing continuous integration and or running a complex
suite of tests for release qualification would solve some problems for
me and for others.
-- Matt Welland, original author of the Megatest tool suite.
@@ -959,10 +959,496 @@
+
+Road Map
+
+
+
+
TODO / Road Map
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Note: This road-map is a wish list and not a formal plan. Items are in +rough priority but are subject to change. Development is driven by +user requests, developer "itch" and bug reports. Please contact +matt@kiatoa.com with requests or bug reports. Requests from inside +Intel generally take priority.
Dashboard and runs
-
+
-
+
+Multi-area dashboard view +
+
+
Tests Support
-
+
-
+
+Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +
+
+ -
+
+Improve [script], especially indent handling +
+
+
Scalability
-
+
-
+
+Overflow database methodology - combine the best of the v1.63 + multi-db approach and the current db-in-tmp approach (currently + slowness can be seen when number of tests in a db goes over 50-100k, + with the overflow db it will be able to handle 1000’s of runs with + 50-100k tests per run). High priority - goal is to complete this by + 20Q3. +
+
+
Mtutils/CI
-
+
-
+
+Enable mtutil calls from dashboard (for remote control) +
+
+ -
+
+Logs browser (esp. for surfacing mtutil related activities) +
+
+ -
+
+Embed ftfplan for distributed automation, completed activities trigger QA runs which trigger deployment etc. +
+
+ -
+
+Jenkins junit XML support [DONE] +
+
+ -
+
+Add output flushing in teamcity support +
+
+
Build system
-
+
-
+
+./configure ⇒ ubuntu, sles11, sles12, rh7 [WIP] +
+
+ -
+
+Switch to using simple runs query everywhere +
+
+ -
+
+Add end_time to runs and add a rollup call that sets state, status and end_time +
+
+
Code refactoring/quality/performance
-
+
-
+
+Switch to scsh-process pipeline management for job execution/control +
+
+ -
+
+Use call-with-environment-variables where possible. +
+
+
Migration to inmem db and or overflow db
-
+
-
+
+Re-work the dbstruct data structure? +
++-
+
-
+
+[ run-id.db inmemdb last-mod last-read last-sync inuse ] +
+
+
+ -
+
Some ideas for Megatest 2.0
-
+
-
+
+Aggressive megatest.config and runconfig.config caching. +
++-
+
-
+
+Cache the configs in $MT_RUNPATH +
+
+ -
+
+Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +
+
+
+ -
+
-
+
+If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests +
+
+ -
+
+!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +
+
+ -
+
+Per test copy commands (example is incomplete). +
+
+
+
+
+
[testcopy] +%/iind% unison SRC DEST +% cp –r SRC DEST+
Add ability to move runs to other Areas (overlaps with overflow db system)
-
+
-
+
+allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. +
+
+ -
+
+add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +
+
+ -
+
+the data move would involve these steps +
++-
+
-
+
+copy the run data to destination area megatest.db +
+
+ -
+
+mark the run records as deleted, do not remove the run data on disk +
+
+
+ -
+
-
+
+accessing the data would be by running dashboard in the satellite area +
+
+ -
+
+future versions of Megatest dashboard should support displaying areas in a + merged way. +
+
+ -
+
+some new controls would be supported in the config +
++-
+
-
+
+[setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs +
+
+ -
+
+[setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically + migrate data to a satellite area. +
+
+
+ -
+
Eliminate ties to homehost (part of overflow db system)
-
+
-
+
+Server creates captain pkt +
+
+ -
+
+Create a lock in the db +
+
+ -
+
+Relinquish db when done +
+
+
Tasks - better management of run manager processes etc.
-
+
-
+
+adjutant queries tasks table for next action [Migrate into mtutil] +
++-
+
-
+
+Task table used for tracking runner process [Replaced by mtutil] +
+
+ -
+
+Task table used for jobs to run [Replaced by mtutil] +
+
+ -
+
+Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [Replaced by mtutil] +
+
+
+ -
+
-
+
+adjutant (server/task dispatch/execution manager) +
+
+
Stale propagation
-
+
-
+
+Mark dependent tests for clean/rerun -rerun-downstream +
+
+ -
+
+On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +
+
+ -
+
+Fix: refresh of gui sometimes fails on last item (race condition?) +
+
+
Bin list
-
+
-
+
+Rerun step and or subsequent steps from gui [DONE?] +
+
+ -
+
+Refresh test area files from gui +
+
+ -
+
+Clean and re-run button +
+
+ -
+
+Clean up STATE and STATUS handling. +
++-
+
-
+
+Dashboard and Test control panel are reverse order - choose and fix +
+
+ -
+
+Move seldom used states and status to drop down selector +
+
+
+ -
+
-
+
+Access test control panel when clicking on Run Summary tests +
+
+ -
+
+Feature: -generate-index-tree +
+
+ -
+
+Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +
+
+ -
+
+rest api available for use with Perl, Ruby etc. scripts +
+
+ -
+
+megatest.config setup entries for: +
++-
+
-
+
+run launching (e.g. /bin/sh %CMD% > /dev/null) +
+
+ -
+
+browser "konqueror %FNAME% +
+
+
+ -
+
-
+
+refdb: Add export of csv, json and sexp +
+
+ -
+
+Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +
+
+ -
+
+Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +
+
+ -
+
+Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +
+
+ -
+
+Refactor Run Summary view, currently very clumsy +
+
+ -
+
+Add option to show steps in Run Summary view +
+
+ -
+
+Refactor guis for resizeablity +
+
+ -
+
+Add filters to Run Summary view and Run Control view +
+
+ -
+
+Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS… +
+
+ -
+
+Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme toppath}>1G +
+
+ -
+
+Tool tips +
+
+ -
+
+Filters on Run Summary, Summary and Run Control panel +
+
+ -
+
+Built in log viewer (partially implemented) +
+
+ -
+
+Refactor the test control panel + Help and documentation +
+
+ -
+
+Complete the user manual (I’ve been working on this lately). +
+
+ -
+
+Online help in the gui + Streamlined install +
+
+ -
+
+Deployed or static build +
+
+ -
+
+Added option to compile IUP (needed for VMs) +
+
+ -
+
+Server side run launching +
+
+ -
+
+Wizards for creating tests, regression areas (current ones are text only and limited). +
+
+ -
+
+Fully functional built in web service (currently you can browse runs but it is very simplistic). +
+
+ -
+
+Gui panels for editing megatest.config and runconfigs.config +
+
+ -
+
+Fully isolated tests (no use of NFS to see regression area files) +
+
+ -
+
+Windows version +
+
+
Installation
@@ -969,10 +1455,64 @@
+
+
+
+
+
Dependencies
Chicken scheme and a number of "eggs" are required for building Megatest. See the script installall.sh in the utils directory of the source distribution for an automated way to install everything needed for building Megatest on Linux.
Megatest. In the v1.66 and beyond assistance to create the build +system is built into the Makefile.
+
+Installation steps (overview)
+
+
./configure +make chicken +setup.sh make -j install+
Or install the needed build system manually:
-
+
-
+
+Chicken scheme from http://call-cc.org +
+
+ -
+
+IUP from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+ -
+
+CD from http://webserver2.tecgraf.puc-rio.br/cd/ +
+
+ -
+
+IM from https://webserver2.tecgraf.puc-rio.br/im/ +
+
+ -
+
+ffcall from http://webserver2.tecgraf.puc-rio.br/iup/ +
+
+ -
+
+Nanomsg from https://nanomsg.org/ (NOTE: Plan is to eliminate nanomsg dependency). +
+
+ -
+
+Needed eggs (look at the eggs lists in the Makefile) +
+
+
Then follow these steps:
+
Installation steps (self-built chicken scheme build system)
+
+
./configure +make -j install+
Getting Started
@@ -1797,14 +2337,26 @@ free-space-script check-space.shTrim trailing spaces
+
+
+
+
+ |
+As of Megatest version v1.6548 trim-trailing-spaces defaults to yes. | +
-
[configf:settings trim-trailing-spaces yes]+
[configf:settings trim-trailing-spaces no] +# |<== next line padded with spaces to here +DEFAULT_INDENT +[configf:settings trim-trailing-spaces no]
The variable DEFAULT_INDENT would be a string of 3 spaces
Job Submission Control
Submit jobs to Host Types based on Test Name
@@ -2087,10 +2639,35 @@# A normal waiton waits for the prior tests to be COMPLETED # and PASS, CHECK or WAIVED waiton test1 test2
+
+
+|
+
+ |
+Dynamic waiton lists must be capable of being calculated at the +beginning of a run. This is because Megatest walks the tree of waitons +to create the list of tests to execute. | +
+
+This works
+
+
waiton [system somescript.sh]+
+
+This does NOT work (the full context for the test is not available so #{shell …} is NOT enabled to evaluate.
+
+
waiton #{shell somescript.sh}
+
+
This does NOT work
+
+
waiton [system somescript_that_depends_on_a_prior_test.sh]+
Mode
The default (i.e. if mode is not specified) is normal. All pre-dependent tests must be COMPLETED and PASS, CHECK or WAIVED before the test will start
Controlled waiver propagation
-If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: -If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED
Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined)
If test is FAIL and previous test in run with same MT_TARGET is WAIVED +or if the test/itempath is listed under the matching target in the +waivers roll forward file (see below for file spec) then apply the +following rules from the testconfig: If a waiver check is specified in +the testconfig apply the check and if it passes then set this FAIL to +WAIVED
Waiver check has two parts, 1) a list of waiver, rulename, +filepatterns and 2) the rulename script spec (note that "diff" and +"logpro" are predefined)
###### EXAMPLE FROM testconfig ######### # matching file(s) will be diff'd with previous run and logpro applied # if PASS or WARN result from logpro then WAIVER state is set @@ -2384,10 +2967,36 @@ # diff diff %file1% %file2% # This builtin rule is applied if a <waivername>.logpro file exists # logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html
+
+
Waiver roll-forward files
+To transfer waivers from one Megatest area to another it is possible +to dump waivers into a file and reference that file in another area.
+
+Dumping the waivers
+
+
megatest -list-waivers -runname %-a > mywaivers.dat+
+
+Referencing the saved waivers
+
+
# In megatest.config, all files listed will be loaded - recomended to use +# variables to select directorys to minimize what gets loaded. +[setup] +waivers-dirs /path/to/waiver/files /another/path/to/waiver/files+
+
+Waiver files format
+
+
[the/target/here] +# comments are fine +testname1/itempath A comment about why it was waived +testname2 A comment for a non-itemized test+
Ezsteps
Example ezsteps with logpro rules
@@ -2816,372 +3425,56 @@
+
Test Plan
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tests
+itemwait|33
rerun-downstream-item|20
rerunclean|20
fullrun|18
goodtests|18
kill-rerun|17
items-runconfigvars|16
ro_test|16
runconfig-tests|16
env-pollution|13
itemmap|11
testpatt_envvar|10
toprun|10
chained-waiton|8
skip-on-fileexists|8
killrun_preqfail|7
subrun|6
dependencies|5
itemwait-simple|4
rollup|4
end-of-run|3
killrun|3
listener|3
test2|3
testpatt|3
env-pollution-usecacheno|2
set-values|2 +envvars|1 +listruns-tests|1 +subrun-usecases|1
Megatest Internals
-
Road Map
-
-
-
Note 1: This road-map continues to evolve and subject to change without notice.
-
-
-
-
-Here is a smattering of ideas for Megatest 2.0
--
-
-
-
-Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME -
-
-
-
-
-
[db] -api legacy|new-
-
-
-
-
-One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. -
-
- -
-
-Rigorous megatest.config and runconfig.config caching. -
---
-
-
-
-Cache the configs in $MT_RUNPATH -
-
- -
-
-Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed -
-
-
- -
-
-
-
-If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests -
-
- -
-
-!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. -
-
- -
-
-Per test copy commands (crude example below is not correct). -
-
-
-
-
-
[testcopy] -%/iind% unison SRC DEST -% cp –r SRC DEST-
-
-
-
-
-Test management via pkts (optional?) -
---
-
-
-
-Control pkt types: run, kill, rerunclean, clean, archive, status? -
-
- -
-
-Status pkt types: ack, step, status_change -
-
-
- -
-
-
-
-Add nanomsg as a transport option -. -
-
-
-
-
-
-
-
-
-RFC M01: Add ability to move runs to other Areas
-Purpose: allow shrinking megatest.db data by moving runs to an alternate - Megatest area with same keys.
Method: extend db sync to take a different megatest area as a destination.
Design:
-
-
-
-
-add param -destination [area|path]. when specified runs are copied to new - area and removed from local db. -
-
- -
-
-the data move would involve these steps -
---
-
-
-
-copy the run data to destination area megatest.db -
-
- -
-
-mark the run records as deleted, do not remove the run data on disk -
-
-
- -
-
-
-
-accessing the data would be by running dashboard in the satellite area -
-
- -
-
-future versions of Megatest dashboard should support displaying areas in a - merged way. -
-
- -
-
-some new controls would be supported in the config -
---
-
-
-
-[setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs -
-
- -
-
-[setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically - migrate data to a satellite area. -
-
-
- -
-
Branch: This work is taking place on branch v1.65-reduce-records
-
-
-
-
-
-
-RFC M02: Move data into completed-runs.db
-Purpose: shrink megatest.db data to enable lower load and higher performance.
Method: add a completed-runs.db and automatically move runs data from megatest.db to that db
Design:
-
-
-
-
-completed-runs.db is a full megatest database with complete schema -
-
- -
-
-the data move would involve these steps -
---
-
-
-
-copy the run data to completed-runs.db -
-
- -
-
-remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db -
-
-
- -
-
-
-
-accessing the data would be unchanged for most operations. -
-
- -
-
-a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query -
-
- -
-
-mechanisms for moving runs to/from the megatest.db would be added -
---
-
-
-
--reduce-records ⇒ move runs to completed-runs.db -
-
- -
-
--restore-records ⇒ move runs from completed-runs.db to megatest.db -
-
-
- -
-
Branch: This work is taking place on branch v1.65-reduce-records
-
-
-
-
-
-
-RFC M03: Automatic homehost migrations
-Purpose: Automatically migrate homehost.
Method: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host
Design:
-
-
-
-
-Check that the system is quiescent, i.e. that there are no runs in flight or recently run -
-
- -
-
-Create a lock -
-
- -
-
-Migrate the /tmp cache db to the current host -
-
- -
-
-Update the .homehost file -
-
- -
-
-Remove the lock -
-
-
Branch: This work not yet started
-
-Architecture Refactor
-
-
-
-Goals
--
-
-
-
-Reduce load on the file system. Sqlite3 files on network filesystem can be - a burden. [DONE] -
-
- -
-
-Reduce number of servers and frequency of start/stop. This is mostly an - issue of clutter but also a reduction in "moving parts". [DONE] -
-
- -
-
-Coalesce activities to a single home host where possible. Give the user - feedback that they have started the dashboard on a host other than the - home host. [DONE] -
-
- -
-
-Reduce number of processes involved in managing running tests. -
-
-
-
-
-Changes Needed
--
-
-
-
-ACID compliant db will be on /tmp and synced to megatest.db with a five - second max delay. [DONE] -
-
- -
-
-Read/writes to db for processes on homehost will go direct to /tmp - megatest.db file. [DONE] -
-
- -
-
-Read/wites fron non-homehost processes will go through one server. Bulk - reads (e.g. for dashboard or list-runs) will be cached on the current host - in /tmp and synced from the home megatest.db in the testsuite area. [DONE] -
-
- -
-
-Db syncs rely on the target db file timestame minus some margin. [DONE] -
-
- -
-
-Since bulk reads do not use the server we can switch to simple RPC for the - network transport. [DONE] -
-
- -
-
-Test running manager process extended to manage multiple running tests. -
-
-
-
Current Items
-
-
-
-
ww05 - migrate to inmem-db
--
-
-
-
-Switch to inmem db with fast sync to on disk db’s [DONE] -
-
- -
-
-Server polls tasks table for next action -
---
-
-
-
-Task table used for tracking runner process [Replaced by mtutil] -
-
- -
-
-Task table used for jobs to run [Replaced by mtutil] -
-
- -
-
-Task table used for queueing runner actions (remove runs, - cleanRunExecute, etc) [Replaced by mtutil] -
-
-
- -
-
shifting, note that the preceding blank line is needed.