Index: docs/manual/getting_started.in ================================================================== --- docs/manual/getting_started.in +++ docs/manual/getting_started.in @@ -10,10 +10,12 @@ // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with Megatest. If not, see . +// +// Copyright 2006-2012, Matthew Welland. Getting Started --------------- [partintro] Index: docs/manual/howto.in ================================================================== --- docs/manual/howto.in +++ docs/manual/howto.in @@ -10,10 +10,12 @@ // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with Megatest. If not, see . +// +// Copyright 2006-2012, Matthew Welland. How To Do Things ---------------- Process Runs Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -887,10 +887,95 @@

Road Map

Note 1: This road-map continues to evolve and subject to change without notice.

+
+

Here is a smattering of ideas for Megatest 2.0

+
    +
  1. +

    +Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +

    +
    2. +
+
+
+
[db]
+api legacy|new
+
+
    +
  1. +

    +One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. +

    +
    2. +
  2. +

    +Rigorous megatest.config and runconfig.config caching. +

    +
      +
    1. +

      +Cache the configs in $MT_RUNPATH +

      +
      2. +
    2. +

      +Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +

      +
      3. +
    +
    3. +
  3. +

    +If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests +

    +
    4. +
  4. +

    +!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +

    +
    5. +
  5. +

    +Per test copy commands (crude example below is not correct). +

    +
    6. +
+
+
+
[testcopy]
+%/iind% unison SRC DEST
+% cp –r SRC DEST
+
+
    +
  1. +

    +Test management via pkts (optional?) +

    +
      +
    1. +

      +Control pkt types: run, kill, rerunclean, clean, archive, status? +

      +
      2. +
    2. +

      +Status pkt types: ack, step, status_change +

      +
      3. +
    +
    2. +
  2. +

    +Add nanomsg as a transport option +. +

    +
    3. +
+

RFC M01: Add ability to move runs to other Areas

Purpose: allow shrinking megatest.db data by moving runs to an alternate Megatest area with same keys.

Method: extend db sync to take a different megatest area as a destination.

@@ -1498,18 +1583,79 @@

Reference

+
+

Megatest Use Modes

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Table 2. Base commands
Use case Megatest command mtutil

Start from scratch

-rerun-all

restart

Rerun non-good completed

-rerun-clean

rerunclean

Rerun all non-good and not completed yet

-set-state-status KILLREQ; -rerun-

clean

killrerun

Continue run

-run

resume

Remove run

-remove-runs

clean

Lock run

-lock

lock

Unlock run

-unlock

unlock

killrun

-set-state-status KILLREQ; -kill-run

+

Config File Helpers

Various helpers for more advanced config files.

- + @@ -1778,11 +1924,11 @@

Database settings

Table 2. HelpersTable 3. Helpers
- + @@ -2066,10 +2212,14 @@ Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized.

+
+

itemstable

+

An alternative to defining items is the itemstable section. This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components.

+

Dynamic Flow Dependency Tree

Autogeneration waiton list for dynamic flow dependency trees
@@ -2217,11 +2367,11 @@ fail gracefully if it doesn’t exist.

Table 3. Database config settings in [setup] section of megatest.configTable 4. Database config settings in [setup] section of megatest.config
- + @@ -2415,11 +2565,11 @@ passfail logpro # Optional: logpro ;; if this section exists then logpro is used to determine pass/fail (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) - (expect:fail in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) + (expect:error in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) # Optional: target translator, default is to use the parent target target #{shell somescript.sh} # Optional: runname translator/generator, default is to use the parent runname @@ -2450,11 +2600,11 @@

These routines can be called from the megatest repl.

Table 4. Environment variables visible to the trigger scriptTable 5. Environment variables visible to the trigger script
Variable
- + @@ -2502,10 +2652,10 @@
Index: docs/manual/plan.in ================================================================== --- docs/manual/plan.in +++ docs/manual/plan.in @@ -1,204 +1,232 @@ -Road Map --------- - -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . - -Note 1: This road-map continues to evolve and subject to change without notice. - -RFC M01: Add ability to move runs to other Areas -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: allow shrinking megatest.db data by moving runs to an alternate - Megatest area with same keys. - -*Method*: extend db sync to take a different megatest area as a destination. - -*Design*: - -. add param -destination [area|path]. when specified runs are copied to new - area and removed from local db. -. the data move would involve these steps -.. copy the run data to destination area megatest.db -.. mark the run records as deleted, do not remove the run data on disk -. accessing the data would be by running dashboard in the satellite area -. future versions of Megatest dashboard should support displaying areas in a - merged way. -. some new controls would be supported in the config -.. [setup] => allow-runs [no|yes] <== used to disallow runs -.. [setup] => auto-migrate=[areaname|path] <== used to automatically - migrate data to a satellite area. - -*Branch*: This work is taking place on branch v1.65-reduce-records - -RFC M02: Move data into completed-runs.db -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: shrink megatest.db data to enable lower load and higher performance. - -*Method*: add a completed-runs.db and automatically move runs data from megatest.db to that db - -*Design*: - -. completed-runs.db is a full megatest database with complete schema -. the data move would involve these steps -.. copy the run data to completed-runs.db -.. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db -. accessing the data would be unchanged for most operations. -. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query -. mechanisms for moving runs to/from the megatest.db would be added -.. -reduce-records => move runs to completed-runs.db -.. -restore-records => move runs from completed-runs.db to megatest.db - -*Branch*: This work is taking place on branch v1.65-reduce-records - -RFC M03: Automatic homehost migrations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Purpose*: Automatically migrate homehost. - -*Method*: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host - -*Design*: - -. Check that the system is quiescent, i.e. that there are no runs in flight or recently run -. Create a lock -. Migrate the /tmp cache db to the current host -. Update the .homehost file -. Remove the lock - -*Branch*: This work not yet started - -Architecture Refactor -~~~~~~~~~~~~~~~~~~~~~ - -Goals -^^^^^ - -. Reduce load on the file system. Sqlite3 files on network filesystem can be - a burden. [green]#[DONE]# -. Reduce number of servers and frequency of start/stop. This is mostly an - issue of clutter but also a reduction in "moving parts". [green]#[DONE]# -. Coalesce activities to a single home host where possible. Give the user - feedback that they have started the dashboard on a host other than the - home host. [green]#[DONE]# -. Reduce number of processes involved in managing running tests. - -Changes Needed -^^^^^^^^^^^^^^ - -. ACID compliant db will be on /tmp and synced to megatest.db with a five - second max delay. [green]#[DONE]# -. Read/writes to db for processes on homehost will go direct to /tmp - megatest.db file. [green]#[DONE]# -. Read/wites fron non-homehost processes will go through one server. Bulk - reads (e.g. for dashboard or list-runs) will be cached on the current host - in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# -. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# -. Since bulk reads do not use the server we can switch to simple RPC for the - network transport. [green]#[DONE]# -. Test running manager process extended to manage multiple running tests. - -Current Items -~~~~~~~~~~~~~ - -ww05 - migrate to inmem-db -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# -. Server polls tasks table for next action -.. Task table used for tracking runner process [red]#[Replaced by mtutil]# -.. Task table used for jobs to run [red]#[Replaced by mtutil]# -.. Task table used for queueing runner actions (remove runs, - cleanRunExecute, etc) [red]#[Replaced by mtutil#] - - -// ww32 -// ~~~~ -// -// . Rerun step and or subsequent steps from gui -// . Refresh test area files from gui -// . Clean and re-run button -// . Clean up STATE and STATUS handling. -// .. Dashboard and Test control panel are reverse order - choose and fix -// .. Move seldom used states and status to drop down selector -// . Access test control panel when clicking on Run Summary tests -// . Feature: -generate-index-tree -// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 -// -// ww33 -// ~~~~ -// -// . http api available for use with Perl, Ruby etc. scripts -// . megatest.config setup entries for: -// .. run launching (e.g. /bin/sh %CMD% > /dev/null) -// .. browser "konqueror %FNAME% -// -// ww34 -// ~~~~ -// -// . Mark dependent tests for clean/rerun -rerun-downstream -// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify -// . Fix: refresh of gui sometimes fails on last item (race condition?) -// -// ww35 -// ~~~~ -// -// . refdb: Add export of csv, json and sexp -// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. -// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. -// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test -// . Refactor Run Summary view, currently very clumsy -// . Add option to show steps in Run Summary view -// -// ww36 -// ~~~~ -// -// . Refactor guis for resizeablity -// . Add filters to Run Summary view and Run Control view -// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... -// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G -// -// Bin List -// ~~~~~~~~ -// -// . Quality improvements -// .. Server stutters occasionally -// .. Large number of items or tests still has some issues. -// .. Code refactoring -// .. Replace remote process with true API using json (supports Web app also) -// . Streamline the gui -// .. Everything resizable -// .. Less clutter -// .. Tool tips -// .. Filters on Run Summary, Summary and Run Control panel -// .. Built in log viewer (partially implemented) -// .. Refactor the test control panel -// . Help and documentation -// .. Complete the user manual (I’ve been working on this lately). -// .. Online help in the gui -// . Streamlined install -// .. Deployed version (download a location independent ready to run binary bundle) -// .. Install Makefile (in progress, needed for Mike to install on VMs) -// .. Added option to compile IUP (needed for VMs) -// . Server side run launching -// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). -// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). -// . Wizards for creating tests, regression areas (current ones are text only and limited). -// . Fully functional built in web service (currently you can browse runs but it is very simplistic). -// . Wildcards in runconfigs: e.g. [p1271/9/%/%] -// . Gui panels for editing megatest.config and runconfigs.config -// . Fully isolated tests (no use of NFS to see regression area files) +Road Map +-------- + +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . + +// Copyright 2006-2012, Matthew Welland. + +Note 1: This road-map continues to evolve and subject to change without notice. + +Here is a smattering of ideas for Megatest 2.0 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME +---------------- +[db] +api legacy|new +---------------- +. One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair. +. Rigorous megatest.config and runconfig.config caching. +.. Cache the configs in $MT_RUNPATH +.. Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed +. If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests +. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there. +. Per test copy commands (crude example below is not correct). +---------------- +[testcopy] +%/iind% unison SRC DEST +% cp –r SRC DEST +---------------- +. Test management via pkts (optional?) +.. Control pkt types: run, kill, rerunclean, clean, archive, status? +.. Status pkt types: ack, step, status_change +. Add nanomsg as a transport option +. + +RFC M01: Add ability to move runs to other Areas +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: allow shrinking megatest.db data by moving runs to an alternate + Megatest area with same keys. + +*Method*: extend db sync to take a different megatest area as a destination. + +*Design*: + +. add param -destination [area|path]. when specified runs are copied to new + area and removed from local db. +. the data move would involve these steps +.. copy the run data to destination area megatest.db +.. mark the run records as deleted, do not remove the run data on disk +. accessing the data would be by running dashboard in the satellite area +. future versions of Megatest dashboard should support displaying areas in a + merged way. +. some new controls would be supported in the config +.. [setup] => allow-runs [no|yes] <== used to disallow runs +.. [setup] => auto-migrate=[areaname|path] <== used to automatically + migrate data to a satellite area. + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M02: Move data into completed-runs.db +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: shrink megatest.db data to enable lower load and higher performance. + +*Method*: add a completed-runs.db and automatically move runs data from megatest.db to that db + +*Design*: + +. completed-runs.db is a full megatest database with complete schema +. the data move would involve these steps +.. copy the run data to completed-runs.db +.. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db +. accessing the data would be unchanged for most operations. +. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query +. mechanisms for moving runs to/from the megatest.db would be added +.. -reduce-records => move runs to completed-runs.db +.. -restore-records => move runs from completed-runs.db to megatest.db + +*Branch*: This work is taking place on branch v1.65-reduce-records + +RFC M03: Automatic homehost migrations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +*Purpose*: Automatically migrate homehost. + +*Method*: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host + +*Design*: + +. Check that the system is quiescent, i.e. that there are no runs in flight or recently run +. Create a lock +. Migrate the /tmp cache db to the current host +. Update the .homehost file +. Remove the lock + +*Branch*: This work not yet started + +Architecture Refactor +~~~~~~~~~~~~~~~~~~~~~ + +Goals +^^^^^ + +. Reduce load on the file system. Sqlite3 files on network filesystem can be + a burden. [green]#[DONE]# +. Reduce number of servers and frequency of start/stop. This is mostly an + issue of clutter but also a reduction in "moving parts". [green]#[DONE]# +. Coalesce activities to a single home host where possible. Give the user + feedback that they have started the dashboard on a host other than the + home host. [green]#[DONE]# +. Reduce number of processes involved in managing running tests. + +Changes Needed +^^^^^^^^^^^^^^ + +. ACID compliant db will be on /tmp and synced to megatest.db with a five + second max delay. [green]#[DONE]# +. Read/writes to db for processes on homehost will go direct to /tmp + megatest.db file. [green]#[DONE]# +. Read/wites fron non-homehost processes will go through one server. Bulk + reads (e.g. for dashboard or list-runs) will be cached on the current host + in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# +. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# +. Since bulk reads do not use the server we can switch to simple RPC for the + network transport. [green]#[DONE]# +. Test running manager process extended to manage multiple running tests. + +Current Items +~~~~~~~~~~~~~ + +ww05 - migrate to inmem-db +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# +. Server polls tasks table for next action +.. Task table used for tracking runner process [red]#[Replaced by mtutil]# +.. Task table used for jobs to run [red]#[Replaced by mtutil]# +.. Task table used for queueing runner actions (remove runs, + cleanRunExecute, etc) [red]#[Replaced by mtutil#] + + +// ww32 +// ~~~~ +// +// . Rerun step and or subsequent steps from gui +// . Refresh test area files from gui +// . Clean and re-run button +// . Clean up STATE and STATUS handling. +// .. Dashboard and Test control panel are reverse order - choose and fix +// .. Move seldom used states and status to drop down selector +// . Access test control panel when clicking on Run Summary tests +// . Feature: -generate-index-tree +// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 +// +// ww33 +// ~~~~ +// +// . http api available for use with Perl, Ruby etc. scripts +// . megatest.config setup entries for: +// .. run launching (e.g. /bin/sh %CMD% > /dev/null) +// .. browser "konqueror %FNAME% +// +// ww34 +// ~~~~ +// +// . Mark dependent tests for clean/rerun -rerun-downstream +// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify +// . Fix: refresh of gui sometimes fails on last item (race condition?) +// +// ww35 +// ~~~~ +// +// . refdb: Add export of csv, json and sexp +// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. +// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. +// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test +// . Refactor Run Summary view, currently very clumsy +// . Add option to show steps in Run Summary view +// +// ww36 +// ~~~~ +// +// . Refactor guis for resizeablity +// . Add filters to Run Summary view and Run Control view +// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... +// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G +// +// Bin List +// ~~~~~~~~ +// +// . Quality improvements +// .. Server stutters occasionally +// .. Large number of items or tests still has some issues. +// .. Code refactoring +// .. Replace remote process with true API using json (supports Web app also) +// . Streamline the gui +// .. Everything resizable +// .. Less clutter +// .. Tool tips +// .. Filters on Run Summary, Summary and Run Control panel +// .. Built in log viewer (partially implemented) +// .. Refactor the test control panel +// . Help and documentation +// .. Complete the user manual (I’ve been working on this lately). +// .. Online help in the gui +// . Streamlined install +// .. Deployed version (download a location independent ready to run binary bundle) +// .. Install Makefile (in progress, needed for Mike to install on VMs) +// .. Added option to compile IUP (needed for VMs) +// . Server side run launching +// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). +// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). +// . Wizards for creating tests, regression areas (current ones are text only and limited). +// . Fully functional built in web service (currently you can browse runs but it is very simplistic). +// . Wildcards in runconfigs: e.g. [p1271/9/%/%] +// . Gui panels for editing megatest.config and runconfigs.config +// . Fully isolated tests (no use of NFS to see regression area files) // . Windows version Index: docs/manual/reference.in ================================================================== --- docs/manual/reference.in +++ docs/manual/reference.in @@ -1,813 +1,839 @@ -// This file is part of Megatest. -// -// Megatest is free software: you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation, either version 3 of the License, or -// (at your option) any later version. -// -// Megatest is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with Megatest. If not, see . - -Reference ---------- - -Config File Helpers -~~~~~~~~~~~~~~~~~~~ - -Various helpers for more advanced config files. - -.Helpers -[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Helper | Purpose | Valid values | Comments -| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file -| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program -| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file -| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | -| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | -| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | -| #{rget v} | Replace with variable v from target or default of runconfigs file | | -| #{mtrah} | Replace with the path to the megatest testsuite area | | -|====================== - -Config File Settings -~~~~~~~~~~~~~~~~~~~~ - -Settings in megatest.config - -Config File Additional Features -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Including output from a script as if it was inline to the config file: - -------------------------- -[scriptinc myscript.sh] -------------------------- - -If the script outputs: - -------------------------- -[items] -A a b c -B d e f -------------------------- - -Then the config file would effectively appear to contain an items section -exactly like the output from the script. This is useful when dynamically -creating items, itemstables and other config structures. You can see the -expansion of the call by looking in the cached files (look in your linktree -for megatest.config and runconfigs.config cache files and in your test run -areas for the expanded and cached testconfig). - -Wildcards and regexes in Targets - -------------------------- -[a/2/b] -VAR1 VAL1 - -[a/%/b] -VAR1 VAL2 -------------------------- - -Will result in: - -------------------------- -[a/2/b] -VAR1 VAL2 -------------------------- - -Can use either wildcard of "%" or a regular expression: - -------------------------- -[/abc.*def/] -------------------------- - -Disk Space Checks -^^^^^^^^^^^^^^^^^ - -Some parameters you can put in the [setup] section of megatest.config: - -------------------- -# minimum space required in a run disk -minspace 10000000 - -# minimum space required in dbdir: -dbdir-space-required 100000 - -# script that takes path as parameter and returns number of bytes available: -free-space-script check-space.sh -------------------- - -Trim trailing spaces -^^^^^^^^^^^^^^^^^^^^ - ------------------- -[configf:settings trim-trailing-spaces yes] ------------------- - -Job Submission Control -^^^^^^^^^^^^^^^^^^^^^^ - -Submit jobs to Host Types based on Test Name -++++++++++++++++++++++++++++++++++++++++++++ - -.In megatest.config ------------------------- -[host-types] -general nbfake -remote bsub - -[launchers] -runfirst/sum% remote -% general - -[jobtools] -launcher bsub -# if defined and not "no" flexi-launcher will bypass launcher unless -# there is no host-type match. -flexi-launcher yes ------------------------- - -host-types -++++++++++ - -List of host types and the commandline to run a job on that host type. - -.host-type => launch command ------------- -general nbfake ------------- - -launchers -+++++++++ - -.test/itempath => host-type ------------- -runfirst/sum% remote ------------- - -Miscellaneous Setup Items -+++++++++++++++++++++++++ - -Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. - -.In megatest.config ------------------- -[setup] -reruns 5 ------------------- - -Replace the default blacklisted environment variables with user supplied -list. - -Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES - -.Add a "bad" variable "PROMPT" to the variables that will be commented out -in the megatest.sh and megatest.csh files: ------------------ -[setup] -blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT ------------------ - -Run time limit -++++++++++++++ - ------------------ -[setup] -# this will automatically kill the test if it runs for more than 1h 2m and 3s -runtimelim 1h 2m 3s ------------------ - -Tests browser view -~~~~~~~~~~~~~~~~~~ - -The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. - -. Dot (graphviz) based tree -. No dot, plain listing - -The default is the graphviz based tree but if your tests don't view -well in that mode then use "nodot" to turn it off. - ------------------ -[setup] -nodot ------------------ - -Capturing Test Data -~~~~~~~~~~~~~~~~~~~ - -In a test you can capture arbitrary variables and roll them up in the -megatest database for viewing on the dashboard or web app. - -.In a test as a script ------------------------- -$MT_MEGATEST -load-test-data << EOF -foo,bar, 1.2, 1.9, > -foo,rab, 1.0e9, 10e9, 1e9 -foo,bla, 1.2, 1.9, < -foo,bal, 1.2, 1.2, < , ,Check for overload -foo,alb, 1.2, 1.2, <= , Amps,This is the high power circuit test -foo,abl, 1.2, 1.3, 0.1 -foo,bra, 1.2, pass, silly stuff -faz,bar, 10, 8mA, , ,"this is a comment" -EOF ------------------------- - -Alternatively you can use logpro triggers to capture values and inject them -into megatest using the -set-values mechanism: - -.Megatest help related to -set-values ------------------------- -Test data capture - -set-values : update or set values in the testdata table - :category : set the category field (optional) - :variable : set the variable name (optional) - :value : value measured (required) - :expected : value expected (required) - :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) - :units : name of the units for value, expected_value etc. (optional) ------------------------- - -Dashboard settings -~~~~~~~~~~~~~~~~~~ - -.Runs tab buttons, font and size ------------------- -[dashboard] -btn-height x14 -btn-fontsz 10 -cell-width 60 ------------------- - -Database settings -~~~~~~~~~~~~~~~~~ - -.Database config settings in [setup] section of megatest.config -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Var | Purpose | Valid values | Comments -|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. -|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use -|homehost | Start servers on this host | | Defaults to local host -|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname -|lowport | Start searching for a port at this portnum| 32768 | -|required | Server required | yes\|no or not defined | Default=no, force start of server always -|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 -|timeout | http api timeout | number in hours | Default is 1 minute, do not change -|====================== - -The testconfig File -------------------- - -Setup section -~~~~~~~~~~~~~ - -Header -^^^^^^ - -------------------- -[setup] -------------------- - -The runscript method is a brute force way to run scripts where the -user is responsible for setting STATE and STATUS - -------------------- -runscript main.csh -------------------- - -Requirements section -~~~~~~~~~~~~~~~~~~~~ - -.Header -------------------- -[requirements] -------------------- - -Wait on Other Tests -^^^^^^^^^^^^^^^^^^^ - -------------------- -# A normal waiton waits for the prior tests to be COMPLETED -# and PASS, CHECK or WAIVED -waiton test1 test2 -------------------- - -Mode -^^^^ - -The default (i.e. if mode is not specified) is normal. All pre-dependent tests -must be COMPLETED and PASS, CHECK or WAIVED before the test will start - -------------------- -[requirements] -mode normal -------------------- - -The toplevel mode requires only that the prior tests are COMPLETED. - -------------------- -[requirements] -mode toplevel -------------------- - -A item based waiton will start items in a test when the same-named -item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This -was historically called "itemwait" mode. The terms "itemwait" and -"itemmatch" are synonyms. - -------------------- -[requirements] -mode itemmatch -------------------- - -Overriding Enviroment Variables -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). - --------------------- -[pre-launch-env-vars] -VAR1 value1 - -# Get some generated settings -[include ../generated-vars.config] - -# Use this trick to unset variables -#{scheme (unsetenv "FOOBAR")} --------------------- - -Itemmap Handling -~~~~~~~~~~~~~~~~ - -For cases were the dependent test has a similar but not identical -itempath to the downstream test an itemmap can allow for itemmatch -mode - -.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) -------------------- -[requirements] -mode itemwait -# itemmap -itemmap .*x/ y/ - -------------------- - -.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) -------------------- - -# ## pattern replacement notes -# -# ## Example -# ## Remove everything up to the last / -[requirements] -mode itemwait -# itemmap -itemmap .*/ -------------------- - -.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) -------------------- - -# -# ## Example -# ## Replace foo/ with bar/ -[requirements] -mode itemwait -# itemmap -itemmap foo/ bar/ - -------------------- - -.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ -------------------- -# -# ## Example -# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl -[requirements] -mode itemwait -# itemmap -itemmap foo(\d+)/ num-\1/bar/ - -------------------- - -.example multiple itemmaps -------------------- - -# multi-line; matches are applied in the listed order -# The following would map: -# a123b321 to b321fooa123 then to 321fooa123p -# -[requirements] -itemmap (a\d+)(b\d+) \2foo\1 - b(.*) \1p -------------------- - - -Complex mapping -^^^^^^^^^^^^^^^ -Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) - -Each line in an itemmap section starts with a waiton test name followed by an itemmap expression - -.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested --------------- -[itemmap] -A foo/ bar/ -B stuff/ --------------- - - -Complex mapping example -^^^^^^^^^^^^^^^^^^^^^^^ - - - -// image::itemmap.png[] -image::complex-itemmap.png[] - - -We accomplish this by configuring the testconfigs of our tests C D and E as follows: - -.Testconfig for Test E has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/bb ----------------------- - -.Testconfig for Test D has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/aa ----------------------- - -.Testconfig for Test C has ----------------------- -[requirements] -waiton A B - -[itemmap] -A (\d+)/aa aa/\1 -B (\d+)/bb bb/\1 ----------------------- - -.Testconfigs for Test B and Test A have no waiton or itemmap configured -------------------- -------------------- - -.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): - -. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ -. Full list to be run is now: +D/1/res+ -. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ -. Full list to be run is now: +D/1/res+, +C/1/aa+ -. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ -. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ -. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. - - -Dynamic Flow Dependency Tree -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Autogeneration waiton list for dynamic flow dependency trees -------------------- -[requirements] -# With a toplevel test you may wish to generate your list -# of tests to run dynamically -# -waiton #{shell get-valid-tests-to-run.sh} -------------------- - -Run time limit -^^^^^^^^^^^^^^ - ------------------ -[requirements] -runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s ------------------ - -Skip -^^^^ - -A test with a skip section will conditional skip running. - -.Skip section example ------------------ -[skip] -prevrunning x -# rundelay 30m 15s ------------------ - -Skip on Still-running Tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ------------------ -# NB// If the prevrunning line exists with *any* value the test will -# automatically SKIP if the same-named test is currently RUNNING. The -# "x" can be any string. Comment out the prevrunning line to turn off -# skip. - -[skip] -prevrunning x ------------------ - -Skip if a File Exists -^^^^^^^^^^^^^^^^^^^^^ - ------------------ -[skip] -fileexists /path/to/a/file # skip if /path/to/a/file exists ------------------ - -Skip if test ran more recently than specified time -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Skip if this test has been run in the past fifteen minutes and 15 seconds. ------------------ -[skip] -rundelay 15m 15s ------------------ - -Disks -^^^^^ - -A disks section in testconfig will override the disks section in -megatest.config. This can be used to allocate disks on a per-test or per item -basis. - -Controlled waiver propagation -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: -If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED - -Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) - ------------------ -###### EXAMPLE FROM testconfig ######### -# matching file(s) will be diff'd with previous run and logpro applied -# if PASS or WARN result from logpro then WAIVER state is set -# -[waivers] -# logpro_file rulename input_glob -waiver_1 logpro lookittmp.log - -[waiver_rules] - -# This builtin rule is the default if there is no .logpro file -# diff diff %file1% %file2% - -# This builtin rule is applied if a .logpro file exists -# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html ------------------ - -Ezsteps -~~~~~~~ - -.Example ezsteps with logpro rules ------------------ -[ezsteps] -lookittmp ls /tmp - -[logpro] -lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line - ;; a blank line indicates the end of the block of text - (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) - ------------------ - -To transfer the environment to the next step you can do the following: - ----------------------------- -$MT_MEGATEST -env2file .ezsteps/${stepname} ----------------------------- - -Triggers -~~~~~~~~ - -In your testconfig or megatest.config triggers can be specified - ------------------ -[triggers] - -# Call script running.sh when test goes to state=RUNNING, status=PASS -RUNNING/PASS running.sh - -# Call script running.sh any time state goes to RUNNING -RUNNING/ running.sh - -# Call script onpass.sh any time status goes to PASS -PASS/ onpass.sh ------------------ - -Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. - -HINT - -To start an xterm (useful for debugging), use a command line like the following: - ------------------ -[triggers] -COMPLETED/ xterm -e bash -s -- ------------------ - -NOTE: There is a trailing space after the -- - -There are a number of environment variables available to the trigger script -but since triggers can be called in various contexts not all variables are -available at all times. The trigger script should check for the variable and -fail gracefully if it doesn't exist. - -.Environment variables visible to the trigger script -[width="90%",cols="^,2m",frame="topbot",options="header"] -|====================== -|Variable | Purpose -| MT_TEST_RUN_DIR | The directory where Megatest ran this test -| MT_CMDINFO | Encoded command data for the test -| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest -| MT_RUN_AREA_HOME | Megatest home area -| MT_TESTSUITENAME | The name of this testsuite or area -| MT_TEST_NAME | The name of this test -| MT_ITEM_INFO | The variable and values for the test item -| MT_MEGATEST | Which Megatest binary is being used by this area -| MT_TARGET | The target variable values, separated by '/' -| MT_LINKTREE | The base of the link tree where all run tests can be found -| MT_ITEMPATH | The values of the item path variables, separated by '/' -| MT_RUNNAME | The name of the run -|====================== - - -Override the Toplevel HTML File -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Megatest generates a simple html file summary for top level tests of -iterated tests. The generation can be overridden. NOTE: the output of -the script is captured from stdout to create the html. - - -.For test "runfirst" override the toplevel generation with a script "mysummary.sh" ------------------ -# Override the rollup for specific tests -[testrollup] -runfirst mysummary.sh ------------------ - -Archiving Setup ---------------- - -In megatest.config add the following sections: - -.megatest.config --------------- -[archive] -# where to get bup executable -# bup /path/to/bup - -[archive-disks] - -# Archives will be organised under these paths like this: -# / -# Within the archive the data is structured like this: -# /// -archive0 /mfs/myarchive-data/adisk1 --------------- - -Handling Environment Variables ------------------------------- - -It is often necessary to capture and or manipulate environment -variables. Megatest has some facilities built in to help. - -Capture variables -~~~~~~~~~~~~~~~~~ - -.Commands ------------------------------- -# capture the current enviroment into a db called envdat.db under -# the context "before" -megatest -envcap before - -# capture the current environment into a db called startup.db with -# context "after" -megatest -envcap after startup.db - -# write the diff from before to after -megatest -envdelta before-after -dumpmode bash ------------------------------- - -Dump modes include bash, csh and config. You can include config data -into megatest.config or runconfigs.config. - -.Example of generating and using config data ------------------------------- -megatest -envcap original -# do some stuff here -megatest -envcap munged -megatest -envdelta original-munged -dumpmode ini -o modified.config ------------------------------- - -Then in runconfigs.config - -.Example of using modified.config in a testconfig ------------------------------- -cat testconfig -[pre-launch-env-vars] -[include modified.config] ------------------------------- - -Managing Old Runs ------------------ - -It is often desired to keep some older runs around but this must be balanced with the costs of disk space. - -. Use -remove-keep -. Use -archive (can also be done from the -remove-keep interface) -. use -remove-runs with -keep-records - -.For each target, remove all runs but the most recent 3 if they are over 1 week old ---------------------- -# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. -megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" ---------------------- - -Nested Runs ------------ - -A Megatest test can run a full Megatest run in either the same -Megatest area or in another area. This is a powerful way of chaining -complex suites of tests and or actions. - -If you are not using the current area you can use ezsteps to retrieve -and setup the sub-Megatest run area. - -In the testconfig: ---------------- -[subrun] - -# Required: wait for the run or just launch it -# if no then the run will be an automatic PASS irrespective of the actual result -run-wait yes|no - -# Optional: where to execute the run. Default is the current runarea -run-area /some/path/to/megatest/area - -# Optional: method to use to determine pass/fail status of the run -# auto (default) - roll up the net state/status of the sub-run -# logpro - use the provided logpro rules, happens automatically if there is a logpro section -# passfail auto|logpro -# Example of logpro: -passfail logpro - -# Optional: -logpro ;; if this section exists then logpro is used to determine pass/fail - (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) - (expect:fail in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) - -# Optional: target translator, default is to use the parent target -target #{shell somescript.sh} - -# Optional: runname translator/generator, default is to use the parent runname -run-name #{somescript.sh} - -# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec -test-patt %/item1,test2 - -# Optional: contour spec, use the named contour from the megatest.config contour spec -contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. - -# Optional: mode-patt, use this spec for testpatt from runconfigs -mode-patt TESTPATT - -# Optional: tag-expr, use this tag-expr to select tests -tag-expr quick - -# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent -# test -# Note// default is % for all -propagate remove-runs archive ... - ---------------- - -Programming API ---------------- - -These routines can be called from the megatest repl. - -.API Keys Related Calls -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] -|====================== -|API Call | Purpose comments | Returns | Comments -|(rmt:get-keys run-id) | | ( key1 key2 ... ) | -| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable -|====================== - - +// This file is part of Megatest. +// +// Megatest is free software: you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// Megatest is distributed in the hope that it will be useful, +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU General Public License for more details. +// +// You should have received a copy of the GNU General Public License +// along with Megatest. If not, see . + +Reference +--------- + +Megatest Use Modes +~~~~~~~~~~~~~~~~~~ + +.Base commands +[width="80%",cols="^,2m,2m",frame="topbot",options="header"] +|====================== +|Use case | Megatest command | mtutil +|Start from scratch | -rerun-all | restart +|Rerun non-good completed | -rerun-clean | rerunclean +|Rerun all non-good and not completed yet | -set-state-status KILLREQ; -rerun-|clean | killrerun +|Continue run | -run | resume +|Remove run | -remove-runs | clean +|Lock run | -lock | lock +|Unlock run | -unlock | unlock +|killrun | -set-state-status KILLREQ; -kill-run | killrun +|====================== + +Config File Helpers +~~~~~~~~~~~~~~~~~~~ + +Various helpers for more advanced config files. + +.Helpers +[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Helper | Purpose | Valid values | Comments +| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file +| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program +| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file +| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | +| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | +| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | +| #{rget v} | Replace with variable v from target or default of runconfigs file | | +| #{mtrah} | Replace with the path to the megatest testsuite area | | +|====================== + +Config File Settings +~~~~~~~~~~~~~~~~~~~~ + +Settings in megatest.config + +Config File Additional Features +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Including output from a script as if it was inline to the config file: + +------------------------- +[scriptinc myscript.sh] +------------------------- + +If the script outputs: + +------------------------- +[items] +A a b c +B d e f +------------------------- + +Then the config file would effectively appear to contain an items section +exactly like the output from the script. This is useful when dynamically +creating items, itemstables and other config structures. You can see the +expansion of the call by looking in the cached files (look in your linktree +for megatest.config and runconfigs.config cache files and in your test run +areas for the expanded and cached testconfig). + +Wildcards and regexes in Targets + +------------------------- +[a/2/b] +VAR1 VAL1 + +[a/%/b] +VAR1 VAL2 +------------------------- + +Will result in: + +------------------------- +[a/2/b] +VAR1 VAL2 +------------------------- + +Can use either wildcard of "%" or a regular expression: + +------------------------- +[/abc.*def/] +------------------------- + +Disk Space Checks +^^^^^^^^^^^^^^^^^ + +Some parameters you can put in the [setup] section of megatest.config: + +------------------- +# minimum space required in a run disk +minspace 10000000 + +# minimum space required in dbdir: +dbdir-space-required 100000 + +# script that takes path as parameter and returns number of bytes available: +free-space-script check-space.sh +------------------- + +Trim trailing spaces +^^^^^^^^^^^^^^^^^^^^ + +------------------ +[configf:settings trim-trailing-spaces yes] +------------------ + +Job Submission Control +^^^^^^^^^^^^^^^^^^^^^^ + +Submit jobs to Host Types based on Test Name +++++++++++++++++++++++++++++++++++++++++++++ + +.In megatest.config +------------------------ +[host-types] +general nbfake +remote bsub + +[launchers] +runfirst/sum% remote +% general + +[jobtools] +launcher bsub +# if defined and not "no" flexi-launcher will bypass launcher unless +# there is no host-type match. +flexi-launcher yes +------------------------ + +host-types +++++++++++ + +List of host types and the commandline to run a job on that host type. + +.host-type => launch command +------------ +general nbfake +------------ + +launchers ++++++++++ + +.test/itempath => host-type +------------ +runfirst/sum% remote +------------ + +Miscellaneous Setup Items ++++++++++++++++++++++++++ + +Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. + +.In megatest.config +------------------ +[setup] +reruns 5 +------------------ + +Replace the default blacklisted environment variables with user supplied +list. + +Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES + +.Add a "bad" variable "PROMPT" to the variables that will be commented out +in the megatest.sh and megatest.csh files: +----------------- +[setup] +blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT +----------------- + +Run time limit +++++++++++++++ + +----------------- +[setup] +# this will automatically kill the test if it runs for more than 1h 2m and 3s +runtimelim 1h 2m 3s +----------------- + +Tests browser view +~~~~~~~~~~~~~~~~~~ + +The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. + +. Dot (graphviz) based tree +. No dot, plain listing + +The default is the graphviz based tree but if your tests don't view +well in that mode then use "nodot" to turn it off. + +----------------- +[setup] +nodot +----------------- + +Capturing Test Data +~~~~~~~~~~~~~~~~~~~ + +In a test you can capture arbitrary variables and roll them up in the +megatest database for viewing on the dashboard or web app. + +.In a test as a script +------------------------ +$MT_MEGATEST -load-test-data << EOF +foo,bar, 1.2, 1.9, > +foo,rab, 1.0e9, 10e9, 1e9 +foo,bla, 1.2, 1.9, < +foo,bal, 1.2, 1.2, < , ,Check for overload +foo,alb, 1.2, 1.2, <= , Amps,This is the high power circuit test +foo,abl, 1.2, 1.3, 0.1 +foo,bra, 1.2, pass, silly stuff +faz,bar, 10, 8mA, , ,"this is a comment" +EOF +------------------------ + +Alternatively you can use logpro triggers to capture values and inject them +into megatest using the -set-values mechanism: + +.Megatest help related to -set-values +------------------------ +Test data capture + -set-values : update or set values in the testdata table + :category : set the category field (optional) + :variable : set the variable name (optional) + :value : value measured (required) + :expected : value expected (required) + :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) + :units : name of the units for value, expected_value etc. (optional) +------------------------ + +Dashboard settings +~~~~~~~~~~~~~~~~~~ + +.Runs tab buttons, font and size +------------------ +[dashboard] +btn-height x14 +btn-fontsz 10 +cell-width 60 +------------------ + +Database settings +~~~~~~~~~~~~~~~~~ + +.Database config settings in [setup] section of megatest.config +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] +|====================== +|Var | Purpose | Valid values | Comments +|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. +|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use +|homehost | Start servers on this host | | Defaults to local host +|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname +|lowport | Start searching for a port at this portnum| 32768 | +|required | Server required | yes\|no or not defined | Default=no, force start of server always +|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 +|timeout | http api timeout | number in hours | Default is 1 minute, do not change +|====================== + +The testconfig File +------------------- + +Setup section +~~~~~~~~~~~~~ + +Header +^^^^^^ + +------------------- +[setup] +------------------- + +The runscript method is a brute force way to run scripts where the +user is responsible for setting STATE and STATUS + +------------------- +runscript main.csh +------------------- + +Requirements section +~~~~~~~~~~~~~~~~~~~~ + +.Header +------------------- +[requirements] +------------------- + +Wait on Other Tests +^^^^^^^^^^^^^^^^^^^ + +------------------- +# A normal waiton waits for the prior tests to be COMPLETED +# and PASS, CHECK or WAIVED +waiton test1 test2 +------------------- + +Mode +^^^^ + +The default (i.e. if mode is not specified) is normal. All pre-dependent tests +must be COMPLETED and PASS, CHECK or WAIVED before the test will start + +------------------- +[requirements] +mode normal +------------------- + +The toplevel mode requires only that the prior tests are COMPLETED. + +------------------- +[requirements] +mode toplevel +------------------- + +A item based waiton will start items in a test when the same-named +item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This +was historically called "itemwait" mode. The terms "itemwait" and +"itemmatch" are synonyms. + +------------------- +[requirements] +mode itemmatch +------------------- + +Overriding Enviroment Variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). + +-------------------- +[pre-launch-env-vars] +VAR1 value1 + +# Get some generated settings +[include ../generated-vars.config] + +# Use this trick to unset variables +#{scheme (unsetenv "FOOBAR")} +-------------------- + +Itemmap Handling +~~~~~~~~~~~~~~~~ + +For cases were the dependent test has a similar but not identical +itempath to the downstream test an itemmap can allow for itemmatch +mode + +.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) +------------------- +[requirements] +mode itemwait +# itemmap +itemmap .*x/ y/ + +------------------- + +.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) +------------------- + +# ## pattern replacement notes +# +# ## Example +# ## Remove everything up to the last / +[requirements] +mode itemwait +# itemmap +itemmap .*/ +------------------- + +.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) +------------------- + +# +# ## Example +# ## Replace foo/ with bar/ +[requirements] +mode itemwait +# itemmap +itemmap foo/ bar/ + +------------------- + +.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ +------------------- +# +# ## Example +# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl +[requirements] +mode itemwait +# itemmap +itemmap foo(\d+)/ num-\1/bar/ + +------------------- + +.example multiple itemmaps +------------------- + +# multi-line; matches are applied in the listed order +# The following would map: +# a123b321 to b321fooa123 then to 321fooa123p +# +[requirements] +itemmap (a\d+)(b\d+) \2foo\1 + b(.*) \1p +------------------- + + +Complex mapping +^^^^^^^^^^^^^^^ +Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) + +Each line in an itemmap section starts with a waiton test name followed by an itemmap expression + +.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested +-------------- +[itemmap] +A foo/ bar/ +B stuff/ +-------------- + + +Complex mapping example +^^^^^^^^^^^^^^^^^^^^^^^ + + + +// image::itemmap.png[] +image::complex-itemmap.png[] + + +We accomplish this by configuring the testconfigs of our tests C D and E as follows: + +.Testconfig for Test E has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/bb +---------------------- + +.Testconfig for Test D has +---------------------- +[requirements] +waiton C +itemmap (\d+)/res \1/aa +---------------------- + +.Testconfig for Test C has +---------------------- +[requirements] +waiton A B + +[itemmap] +A (\d+)/aa aa/\1 +B (\d+)/bb bb/\1 +---------------------- + +.Testconfigs for Test B and Test A have no waiton or itemmap configured +------------------- +------------------- + +.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): + +. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ +. Full list to be run is now: +D/1/res+ +. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ +. Full list to be run is now: +D/1/res+, +C/1/aa+ +. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ +. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ +. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. + + + +itemstable +^^^^^^^^^^ +An alternative to defining items is the itemstable section. This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components. + + + + + +Dynamic Flow Dependency Tree +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Autogeneration waiton list for dynamic flow dependency trees +------------------- +[requirements] +# With a toplevel test you may wish to generate your list +# of tests to run dynamically +# +waiton #{shell get-valid-tests-to-run.sh} +------------------- + +Run time limit +^^^^^^^^^^^^^^ + +----------------- +[requirements] +runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s +----------------- + +Skip +^^^^ + +A test with a skip section will conditional skip running. + +.Skip section example +----------------- +[skip] +prevrunning x +# rundelay 30m 15s +----------------- + +Skip on Still-running Tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +----------------- +# NB// If the prevrunning line exists with *any* value the test will +# automatically SKIP if the same-named test is currently RUNNING. The +# "x" can be any string. Comment out the prevrunning line to turn off +# skip. + +[skip] +prevrunning x +----------------- + +Skip if a File Exists +^^^^^^^^^^^^^^^^^^^^^ + +----------------- +[skip] +fileexists /path/to/a/file # skip if /path/to/a/file exists +----------------- + +Skip if test ran more recently than specified time +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.Skip if this test has been run in the past fifteen minutes and 15 seconds. +----------------- +[skip] +rundelay 15m 15s +----------------- + +Disks +^^^^^ + +A disks section in testconfig will override the disks section in +megatest.config. This can be used to allocate disks on a per-test or per item +basis. + +Controlled waiver propagation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: +If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED + +Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) + +----------------- +###### EXAMPLE FROM testconfig ######### +# matching file(s) will be diff'd with previous run and logpro applied +# if PASS or WARN result from logpro then WAIVER state is set +# +[waivers] +# logpro_file rulename input_glob +waiver_1 logpro lookittmp.log + +[waiver_rules] + +# This builtin rule is the default if there is no .logpro file +# diff diff %file1% %file2% + +# This builtin rule is applied if a .logpro file exists +# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html +----------------- + +Ezsteps +~~~~~~~ + +.Example ezsteps with logpro rules +----------------- +[ezsteps] +lookittmp ls /tmp + +[logpro] +lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line + ;; a blank line indicates the end of the block of text + (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) + +----------------- + +To transfer the environment to the next step you can do the following: + +---------------------------- +$MT_MEGATEST -env2file .ezsteps/${stepname} +---------------------------- + +Triggers +~~~~~~~~ + +In your testconfig or megatest.config triggers can be specified + +----------------- +[triggers] + +# Call script running.sh when test goes to state=RUNNING, status=PASS +RUNNING/PASS running.sh + +# Call script running.sh any time state goes to RUNNING +RUNNING/ running.sh + +# Call script onpass.sh any time status goes to PASS +PASS/ onpass.sh +----------------- + +Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. + +HINT + +To start an xterm (useful for debugging), use a command line like the following: + +----------------- +[triggers] +COMPLETED/ xterm -e bash -s -- +----------------- + +NOTE: There is a trailing space after the -- + +There are a number of environment variables available to the trigger script +but since triggers can be called in various contexts not all variables are +available at all times. The trigger script should check for the variable and +fail gracefully if it doesn't exist. + +.Environment variables visible to the trigger script +[width="90%",cols="^,2m",frame="topbot",options="header"] +|====================== +|Variable | Purpose +| MT_TEST_RUN_DIR | The directory where Megatest ran this test +| MT_CMDINFO | Encoded command data for the test +| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest +| MT_RUN_AREA_HOME | Megatest home area +| MT_TESTSUITENAME | The name of this testsuite or area +| MT_TEST_NAME | The name of this test +| MT_ITEM_INFO | The variable and values for the test item +| MT_MEGATEST | Which Megatest binary is being used by this area +| MT_TARGET | The target variable values, separated by '/' +| MT_LINKTREE | The base of the link tree where all run tests can be found +| MT_ITEMPATH | The values of the item path variables, separated by '/' +| MT_RUNNAME | The name of the run +|====================== + + +Override the Toplevel HTML File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Megatest generates a simple html file summary for top level tests of +iterated tests. The generation can be overridden. NOTE: the output of +the script is captured from stdout to create the html. + + +.For test "runfirst" override the toplevel generation with a script "mysummary.sh" +----------------- +# Override the rollup for specific tests +[testrollup] +runfirst mysummary.sh +----------------- + +Archiving Setup +--------------- + +In megatest.config add the following sections: + +.megatest.config +-------------- +[archive] +# where to get bup executable +# bup /path/to/bup + +[archive-disks] + +# Archives will be organised under these paths like this: +# / +# Within the archive the data is structured like this: +# /// +archive0 /mfs/myarchive-data/adisk1 +-------------- + +Handling Environment Variables +------------------------------ + +It is often necessary to capture and or manipulate environment +variables. Megatest has some facilities built in to help. + +Capture variables +~~~~~~~~~~~~~~~~~ + +.Commands +------------------------------ +# capture the current enviroment into a db called envdat.db under +# the context "before" +megatest -envcap before + +# capture the current environment into a db called startup.db with +# context "after" +megatest -envcap after startup.db + +# write the diff from before to after +megatest -envdelta before-after -dumpmode bash +------------------------------ + +Dump modes include bash, csh and config. You can include config data +into megatest.config or runconfigs.config. + +.Example of generating and using config data +------------------------------ +megatest -envcap original +# do some stuff here +megatest -envcap munged +megatest -envdelta original-munged -dumpmode ini -o modified.config +------------------------------ + +Then in runconfigs.config + +.Example of using modified.config in a testconfig +------------------------------ +cat testconfig +[pre-launch-env-vars] +[include modified.config] +------------------------------ + +Managing Old Runs +----------------- + +It is often desired to keep some older runs around but this must be balanced with the costs of disk space. + +. Use -remove-keep +. Use -archive (can also be done from the -remove-keep interface) +. use -remove-runs with -keep-records + +.For each target, remove all runs but the most recent 3 if they are over 1 week old +--------------------- +# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. +megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" +--------------------- + +Nested Runs +----------- + +A Megatest test can run a full Megatest run in either the same +Megatest area or in another area. This is a powerful way of chaining +complex suites of tests and or actions. + +If you are not using the current area you can use ezsteps to retrieve +and setup the sub-Megatest run area. + +In the testconfig: +--------------- +[subrun] + +# Required: wait for the run or just launch it +# if no then the run will be an automatic PASS irrespective of the actual result +run-wait yes|no + +# Optional: where to execute the run. Default is the current runarea +run-area /some/path/to/megatest/area + +# Optional: method to use to determine pass/fail status of the run +# auto (default) - roll up the net state/status of the sub-run +# logpro - use the provided logpro rules, happens automatically if there is a logpro section +# passfail auto|logpro +# Example of logpro: +passfail logpro + +# Optional: +logpro ;; if this section exists then logpro is used to determine pass/fail + (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) + (expect:error in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) + +# Optional: target translator, default is to use the parent target +target #{shell somescript.sh} + +# Optional: runname translator/generator, default is to use the parent runname +run-name #{somescript.sh} + +# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec +test-patt %/item1,test2 + +# Optional: contour spec, use the named contour from the megatest.config contour spec +contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. + +# Optional: mode-patt, use this spec for testpatt from runconfigs +mode-patt TESTPATT + +# Optional: tag-expr, use this tag-expr to select tests +tag-expr quick + +# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent +# test +# Note// default is % for all +propagate remove-runs archive ... + +--------------- + +Programming API +--------------- + +These routines can be called from the megatest repl. + +.API Keys Related Calls +[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] +|====================== +|API Call | Purpose comments | Returns | Comments +|(rmt:get-keys run-id) | | ( key1 key2 ... ) | +| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable +|====================== + + :numbered!:
Table 5. API Keys Related CallsTable 6. API Keys Related Calls