Index: docs/manual/Makefile ================================================================== --- docs/manual/Makefile +++ docs/manual/Makefile @@ -1,10 +1,12 @@ ASCPATH = $(shell which asciidoc) EXEPATH = $(shell readlink -f $(ASCPATH)) BINPATH = $(shell dirname $(EXEPATH)) DISPATH = $(shell dirname $(BINPATH)) +SRCFSL = $(shell fossil info | grep repository: | awk '{print $$2}') +INPAGES = plan.in howto.in reference.in getting_started.in # broad_goals.csv needed_features.csv : tables/*.dat # ./refdb2csv tables # in a makefile recipe, $< denotes the first dependency; $@ the target @@ -13,15 +15,15 @@ # asciidoc -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 design_spec.txt # all : server.ps megatest_manual.html client.ps complex-itemmap.png -megatest_manual.html : megatest_manual.txt getting_started.txt writing_tests.txt reference.txt ../plan.txt howto.txt installation.txt *png +megatest_manual.html : megatest_manual.txt $(INPAGES) writing_tests.txt installation.txt *png asciidoc -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 megatest_manual.txt # dos2unix megatest_manual.html -megatest.pdf : megatest_manual.txt getting_started.txt writing_tests.txt reference.txt ../plan.txt howto.txt *png +megatest.pdf : megatest_manual.txt $(INPAGES) writing_tests.txt *png a2x -a toc -f pdf megatest_manual.txt server.ps : server.dot dot -Tps server.dot > server.ps @@ -30,7 +32,13 @@ complex-itemmap.png : complex-itemmap.dot dot -Tpng complex-itemmap.dot -o complex-itemmap.png dot -Tpdf complex-itemmap.dot -o complex-itemmap.pdf +%.in : $(SRCFSL) + fossil wiki export $* $*.in + +# %.txt : %.in +# cp $*.in $*.txt + clean: rm -f megatest_manual.html DELETED docs/manual/getting_started.txt Index: docs/manual/getting_started.txt ================================================================== --- docs/manual/getting_started.txt +++ /dev/null @@ -1,99 +0,0 @@ - -Getting Started ---------------- - -[partintro] -.Getting started with Megatest --- -Creating a testsuite or flow and your first test or task. --- - -After installing Megatest you can create a flow or testsuite and add some -tests using the helpers. Here is a quickstart sequence to get you up and -running your first automated testsuite. - -Creating a Megatest Area -~~~~~~~~~~~~~~~~~~~~~~~~ - -Choose Target Keys -^^^^^^^^^^^^^^^^^^ - -First choose your "target" keys. These are used to organise your runs in a -way that is meaningful to your project. If you are unsure about what to use -for keys just use a single generic key such as "RUNTYPE". These keys will be -used to hand values to your tests via environment variables so ensure they -are unique. Prefixing them with something such as PROJKEYS_ is a good -strategy. - -Examples of keys: - -.Example keys -[width="60%",options="header"] -|============================================== -| Option | Description -| RELEASE/ITERATION | This example is used by Megatest for its internal QA. -| ARCH/OS/RELEASE | For a software project targeting multiple platforms -| UCTRLR/NODETYPE | Microcontroller project with different controllers -running same software -|============================================== - -Create Area Config Files -^^^^^^^^^^^^^^^^^^^^^^^^ - -You will need to choose locations for your runs (the data generated every -time you run the testsuite) and link tree. For getting started answer the -prompts with "runs" and "links". We use the Unix editor "vi" in the examples -below but you can use any plain text editor. - -.Using the helper to create a Megatest area ------------------- -megatest -create-megatest-area - -# optional: verify that the settings are ok -vi megatest.config -vi runconfigs.config ------------------- - -Creating a Test -~~~~~~~~~~~~~~~ - -Choose the test name for your first test and run the helper. You can edit -the files after the initial creation. You will need to enter names and -scripts for the steps to be run and then edit the -tests//testconfig file and modify the logpro rules to properly -process the log output from your steps. For your first test just hit enter -for the "waiton", "priority" and iteration variable prompts. - -Hint: for geting started make your logpro rules very liberal. expect:error -patterns should match nothing and comment out expect:required rules. - -.Using the helper to create a Megatest test ---------------- -megatest -create-test myfirsttest - -# then edit the generated config -vi tests/myfirsttest/testconfig ---------------- - -Running your test -~~~~~~~~~~~~~~~~~ - -First choose a target and runname. If you have a two-place target such as -RELEASE/ITERATION a target would look like v1.0/aff3 where v1.0 is the -RELEASE and aff3 is the ITERATION. For a run name just use something like -run1. - -.Running all tests (testpatt of "%" matches all tests) ---------------- -megatest -run -target v1.0/aff3 -runname run1 -testpatt % -log run1.log ---------------- - -Viewing the results -~~~~~~~~~~~~~~~~~~~ - -Start the dashboard and browse your run in the "Runs" tab. - -.Starting dashboard ----------------- -dashboard -rows 24 ----------------- DELETED docs/manual/howto.txt Index: docs/manual/howto.txt ================================================================== --- docs/manual/howto.txt +++ /dev/null @@ -1,192 +0,0 @@ - -How To Do Things ----------------- - -Process Runs -~~~~~~~~~~~~ - -Remove Runs -^^^^^^^^^^^ - -From the dashboard click on the button (PASS/FAIL...) for one of the tests. From the test control panel that -comes up push the clean test button. The command field will be prefilled with a template command for removing -that test. You can edit the command, for example change the argument to -testpatt to "%" to remove all tests. - -.Remove the test diskperf and all it's items ----------------- -megatest -remove-runs -target ubuntu/nfs/none -runname ww28.1a -testpatt diskperf/% -v ----------------- - -.Remove all tests for all runs and all targets ----------------- -megatest -remove-runs -target %/%/% -runname % -testpatt % -v ----------------- - -Archive Runs -^^^^^^^^^^^^ - -Megatest supports using the bup backup tool (https://bup.github.io/) to archive your tests for efficient storage -and retrieval. Archived data can be rapidly retrieved if needed. The metadata for the run (PASS/FAIL status, run -durations, time stamps etc.) are all preserved in the megatest database. - -For setup information see the Archiving topic in the reference section of this manual. - -To Archive -++++++++++ - -Hint: use the test control panel to create a template command by pushing the "Archive Tests" button. - -.Archive a full run ----------------- -megatest -target ubuntu/nfs/none -runname ww28.1a -archive save-remove -testpatt % ----------------- - -To Restore -++++++++++ - -.Retrieve a single test ----------------- -megatest -target ubuntu/nfs/none -runname ww28.1a -archive restore -testpatt diskperf/% ----------------- - -Hint: You can browse the archive using bup commands directly. - ----------------- -bup -d /path/to/bup/archive ftp ----------------- - -Submit jobs to Host Types based on Test Name -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.In megatest.config ------------------------- -[host-types] -general ssh #{getbgesthost general} -nbgeneral nbjob run JOBCOMMAND -log $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME.$MT_TESTNAME-$MT_ITEM_PATH.lgo - -[hosts] -general cubian xena - -[launchers] -envsetup general -xor/%/n 4C16G -% nbgeneral - -[jobtools] -launcher bsub -# if defined and not "no" flexi-launcher will bypass launcher unless there is no -# match. -flexi-launcher yes ------------------------- - -Tricks ------- - -This section is a compendium of a various useful tricks for debugging, -configuring and generally getting the most out of Megatest. - -Limiting your running jobs -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously. - -In your testconfig: - ----------------- -[test_meta] -jobgroup group1 ----------------- - -In your megatest.config: - ---------------- -[jobgroups] -group1 10 -custdes 4 ---------------- - -Debugging Tricks ----------------- - -Examining The Environment -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Test Control Panel - xterm -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the -window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run -scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way -to debug your tests. - -During Config File Processing -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -It is often helpful to know the content of variables in various -contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined. - -For example, if an item list is not being generated as expected you -can inject the startup of an xterm as if it were an item: - -.Original items table ------------------ -[items] -CELLNAME [system getcellname.sh] ------------------ - -.Items table modified for debug ------------------ -[items] -DEBUG [system xterm] -CELLNAME [system getcellnames.sh] ------------------ - -When this test is run an xterm will pop up. In that xterm the -environment is exactly that in which the script "getcellnames.sh" -would run. You can now debug the script to find out why it isn't -working as expected. - -Organising Your Tests and Tasks -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The default location "tests" for storing tests can be extended by -adding to your tests-paths section. - ----------------------------- -[misc] -parent #{shell dirname $(readlink -f .)} - -[tests-paths] -1 #{get misc parent}/simplerun/tests ----------------------------- - -The above example shows how you can use addition sections in your -config file to do complex processing. By putting results of relatively -slow operations into variables the processing of your configs can be -kept fast. - -Alternative Method for Running your Job Script -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Directly running job in testconfig -------------------- -[setup] -runscript main.csh -------------------- - -The runscript method is essentially a brute force way to run scripts where the -user is responsible for setting STATE and STATUS and managing the details of running a test. - -Debugging Server Problems -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Some handy Unix commands to track down issues with servers not -communicating with your test manager processes. Please put in tickets -at https://www.kiatoa.com/fossils/megatest if you have problems with -servers getting stuck. - ----------------- -sudo lsof -i -sudo netstat -lptu -sudo netstat -tulpn ----------------- Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -867,10 +867,14 @@

Road Map

Note 1: This road-map is still evolving and subject to change without notice.

+
+

RFC

+

Requests for comment.

+

Architecture Refactor

Goals

    @@ -2301,10 +2305,10 @@
Index: docs/manual/megatest_manual.txt ================================================================== --- docs/manual/megatest_manual.txt +++ docs/manual/megatest_manual.txt @@ -61,24 +61,24 @@ the distributed compute platform in use. A template script is provided which can launch jobs on local and remote Linux hosts. Currently megatest uses the network filesystem to call home to your master sqlite3 database. -include::../plan.txt[] +include::plan.in[] // to allow the getting_started.txt to be a stand-alone document use level shifting, note that the preceding blank line is needed. // :leveloffset: 2 include::installation.txt[] -include::getting_started.txt[] +include::getting_started.in[] :leveloffset: 0 include::writing_tests.txt[] -include::howto.txt[] -include::reference.txt[] +include::howto.in[] +include::reference.in[] Megatest Internals ------------------ ["graphviz", "server.png"] DELETED docs/manual/reference.txt Index: docs/manual/reference.txt ================================================================== --- docs/manual/reference.txt +++ /dev/null @@ -1,766 +0,0 @@ - -Reference ---------- - -Config File Helpers -~~~~~~~~~~~~~~~~~~~ - -Various helpers for more advanced config files. - -.Helpers -[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Helper | Purpose | Valid values | Comments -| #{scheme (scheme code...)} | Execute arbitrary scheme code | Any valid scheme | Value returned from the call is converted to a string and processed as part of the config file -| #{system command} | Execute program, inserts exit code | Any valid Unix command | Discards the output from the program -| #{shell command} or #{sh ...} | Execute program, inserts result from stdout | Any valid Unix command | Value returned from the call is converted to a string and processed as part of the config file -| #{realpath path} or #{rp ...} | Replace with normalized path | Must be a valid path | -| #{getenv VAR} or #{gv VAR} | Replace with content of env variable | Must be a valid var | -| #{get s v} or #{g s v} | Replace with variable v from section s | Variable must be defined before use | -| #{rget v} | Replace with variable v from target or default of runconfigs file | | -| #{mtrah} | Replace with the path to the megatest testsuite area | | -|====================== - -Config File Settings -~~~~~~~~~~~~~~~~~~~~ - -Settings in megatest.config - -Config File Additional Features -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Including output from a script as if it was inline to the config file: - -------------------------- -[scriptinc myscript.sh] -------------------------- - -If the script outputs: - -------------------------- -[items] -A a b c -B d e f -------------------------- - -Then the config file would effectively appear to contain an items section -exactly like the output from the script. This is useful when dynamically -creating items, itemstables and other config structures. You can see the -expansion of the call by looking in the cached files (look in your linktree -for megatest.config and runconfigs.config cache files and in your test run -areas for the expanded and cached testconfig). - -Wildcards and regexes in Targets - -------------------------- -[a/2/b] -VAR1 VAL1 - -[a/%/b] -VAR1 VAL2 -------------------------- - -Will result in: - -------------------------- -[a/2/b] -VAR1 VAL2 -------------------------- - -Can use either wildcard of "%" or a regular expression: - -------------------------- -[/abc.*def/] -------------------------- - -Disk Space Checks -^^^^^^^^^^^^^^^^^ - -Some parameters you can put in the [setup] section of megatest.config: - -------------------- -# minimum space required in a run disk -minspace 10000000 - -# minimum space required in dbdir: -dbdir-space-required 100000 - -# script that takes path as parameter and returns number of bytes available: -free-space-script check-space.sh -------------------- - -Trim trailing spaces -^^^^^^^^^^^^^^^^^^^^ - ------------------- -[configf:settings trim-trailing-spaces yes] ------------------- - -Job Submission Control -^^^^^^^^^^^^^^^^^^^^^^ - -Submit jobs to Host Types based on Test Name -++++++++++++++++++++++++++++++++++++++++++++ - -.In megatest.config ------------------------- -[host-types] -general nbfake -remote bsub - -[launchers] -runfirst/sum% remote -% general - -[jobtools] -launcher bsub -# if defined and not "no" flexi-launcher will bypass launcher unless -# there is no host-type match. -flexi-launcher yes ------------------------- - -host-types -++++++++++ - -List of host types and the commandline to run a job on that host type. - -.host-type => launch command ------------- -general nbfake ------------- - -launchers -+++++++++ - -.test/itempath => host-type ------------- -runfirst/sum% remote ------------- - -Miscellaneous Setup Items -+++++++++++++++++++++++++ - -Attempt to rerun tests in "STUCK/DEAD", "n/a", "ZERO_ITEMS" states. - -.In megatest.config ------------------- -[setup] -reruns 5 ------------------- - -Replace the default blacklisted environment variables with user supplied -list. - -Default list: USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS MAKEF MAKEOVERRIDES - -.Add a "bad" variable "PROMPT" to the variables that will be commented out -in the megatest.sh and megatest.csh files: ------------------ -[setup] -blacklistvars USER HOME DISPLAY LS_COLORS XKEYSYMDB EDITOR MAKEFLAGS PROMPT ------------------ - -Run time limit -++++++++++++++ - ------------------ -[setup] -# this will automatically kill the test if it runs for more than 1h 2m and 3s -runtimelim 1h 2m 3s ------------------ - -Tests browser view -~~~~~~~~~~~~~~~~~~ - -The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. - -. Dot (graphviz) based tree -. No dot, plain listing - -The default is the graphviz based tree but if your tests don't view -well in that mode then use "nodot" to turn it off. - ------------------ -[setup] -nodot ------------------ - -Dashboard settings -~~~~~~~~~~~~~~~~~~ - -.Runs tab buttons, font and size ------------------- -[dashboard] -btn-height x14 -btn-fontsz 10 -cell-width 60 ------------------- - -Database settings -~~~~~~~~~~~~~~~~~ - -.Database config settings in [setup] section of megatest.config -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header"] -|====================== -|Var | Purpose | Valid values | Comments -|delay-on-busy | Prevent concurrent access issues | yes\|no or not defined | Default=no, may help on some network file systems, may slow things down also. -|daemonize | Daemonize the server on start | yes\|no or not defined | Default=no -|faststart | All direct file access to sqlite db files | yes\|no or not defined | Default=yes, suggest no for central automated systems and yes for interactive use -|homehost | Start servers on this host | | Defaults to local host -|hostname | Hostname to bind to | \|- | On multi-homed hosts allows binding to specific hostname -|lowport | Start searching for a port at this portnum| 32768 | -|required | Server required | yes\|no or not defined | Default=no, force start of server always -|server-query-threshold | Start server when queries take longer than this | number in milliseconds | Default=300 -|timeout | http api timeout | number in hours | Default is 1 minute, do not change -|====================== - -The testconfig File -------------------- - -Setup section -~~~~~~~~~~~~~ - -Header -^^^^^^ - -------------------- -[setup] -------------------- - -The runscript method is a brute force way to run scripts where the -user is responsible for setting STATE and STATUS - -------------------- -runscript main.csh -------------------- - -Requirements section -~~~~~~~~~~~~~~~~~~~~ - -.Header -------------------- -[requirements] -------------------- - -Wait on Other Tests -^^^^^^^^^^^^^^^^^^^ - -------------------- -# A normal waiton waits for the prior tests to be COMPLETED -# and PASS, CHECK or WAIVED -waiton test1 test2 -------------------- - -Mode -^^^^ - -The default (i.e. if mode is not specified) is normal. All pre-dependent tests -must be COMPLETED and PASS, CHECK or WAIVED before the test will start - -------------------- -[requirements] -mode normal -------------------- - -The toplevel mode requires only that the prior tests are COMPLETED. - -------------------- -[requirements] -mode toplevel -------------------- - -A item based waiton will start items in a test when the same-named -item is COMPLETED and PASS, CHECK or WAIVED in the prior test. This -was historically called "itemwait" mode. The terms "itemwait" and -"itemmatch" are synonyms. - -------------------- -[requirements] -mode itemmatch -------------------- - -Overriding Enviroment Variables -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Override variables before starting the test. Can include files (perhaps generated by megatest -envdelta or similar). - --------------------- -[pre-launch-env-vars] -VAR1 value1 - -# Get some generated settings -[include ../generated-vars.config] - -# Use this trick to unset variables -#{scheme (unsetenv "FOOBAR")} --------------------- - -Itemmap Handling -~~~~~~~~~~~~~~~~ - -For cases were the dependent test has a similar but not identical -itempath to the downstream test an itemmap can allow for itemmatch -mode - -.example for removing part of itemmap for waiton test (eg: item +foo-x/bar+ depends on waiton's item +y/bar+) -------------------- -[requirements] -mode itemwait -# itemmap -itemmap .*x/ y/ - -------------------- - -.example for removing part of itemmap for waiton test (eg: item +foo/bar/baz+ in this test depends on waiton's item +baz+) -------------------- - -# ## pattern replacement notes -# -# ## Example -# ## Remove everything up to the last / -[requirements] -mode itemwait -# itemmap -itemmap .*/ -------------------- - -.example replacing part of itemmap for (eg: item +foo/1234+ will imply waiton's item +bar/1234+) -------------------- - -# -# ## Example -# ## Replace foo/ with bar/ -[requirements] -mode itemwait -# itemmap -itemmap foo/ bar/ - -------------------- - -.example for backreference (eg: item +foo23/thud+ will imply waiton's item +num-23/bar/thud+ -------------------- -# -# ## Example -# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl -[requirements] -mode itemwait -# itemmap -itemmap foo(\d+)/ num-\1/bar/ - -------------------- - -.example multiple itemmaps -------------------- - -# multi-line; matches are applied in the listed order -# The following would map: -# a123b321 to b321fooa123 then to 321fooa123p -# -[requirements] -itemmap (a\d+)(b\d+) \2foo\1 - b(.*) \1p -------------------- - - -Complex mapping -^^^^^^^^^^^^^^^ -Complex mappings can be handled with a separate [itemmap] section (instead if an itemmap line in the [requirements] section) - -Each line in an itemmap section starts with a waiton test name followed by an itemmap expression - -.eg: The following causes waiton test A item +bar/1234+ to run when our test's +foo/1234+ item is requested as well as causing waiton test B's +blah+ item to run when our test's +stuff/blah+ item is requested --------------- -[itemmap] -A foo/ bar/ -B stuff/ --------------- - - -Complex mapping example -^^^^^^^^^^^^^^^^^^^^^^^ - - - -// image::itemmap.png[] -image::complex-itemmap.png[] - - -We accomplish this by configuring the testconfigs of our tests C D and E as follows: - -.Testconfig for Test E has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/bb ----------------------- - -.Testconfig for Test D has ----------------------- -[requirements] -waiton C -itemmap (\d+)/res \1/aa ----------------------- - -.Testconfig for Test C has ----------------------- -[requirements] -waiton A B - -[itemmap] -A (\d+)/aa aa/\1 -B (\d+)/bb bb/\1 ----------------------- - -.Testconfigs for Test B and Test A have no waiton or itemmap configured -------------------- -------------------- - -.Walk through one item -- we want the following to happen for testpatt +D/1/res+ (see blue boxes in complex itemmaping figure above): - -. eg from command line +megatest -run -testpatt D/1/res -target mytarget -runname myrunname+ -. Full list to be run is now: +D/1/res+ -. Test D has a waiton - test C. Test D's itemmap rule +itemmap (\d+)/res \1/aa+ -> causes +C/1/aa+ to run before +D/1/res+ -. Full list to be run is now: +D/1/res+, +C/1/aa+ -. Test C was a waiton - test A. Test C's rule +A (\d+)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+ -. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+ -. Test A has no waitons. All waitons of all tests in full list have been processed. Full list is finalized. - - -Dynamic Flow Dependency Tree -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Autogeneration waiton list for dynamic flow dependency trees -------------------- -[requirements] -# With a toplevel test you may wish to generate your list -# of tests to run dynamically -# -waiton #{shell get-valid-tests-to-run.sh} -------------------- - -Run time limit -^^^^^^^^^^^^^^ - ------------------ -[requirements] -runtimelim 1h 2m 3s # this will automatically kill the test if it runs for more than 1h 2m and 3s ------------------ - -Skip -^^^^ - -A test with a skip section will conditional skip running. - -.Skip section example ------------------ -[skip] -prevrunning x -# rundelay 30m 15s ------------------ - -Skip on Still-running Tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ------------------ -# NB// If the prevrunning line exists with *any* value the test will -# automatically SKIP if the same-named test is currently RUNNING. The -# "x" can be any string. Comment out the prevrunning line to turn off -# skip. - -[skip] -prevrunning x ------------------ - -Skip if a File Exists -^^^^^^^^^^^^^^^^^^^^^ - ------------------ -[skip] -fileexists /path/to/a/file # skip if /path/to/a/file exists ------------------ - -Skip if test ran more recently than specified time -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.Skip if this test has been run in the past fifteen minutes and 15 seconds. ------------------ -[skip] -rundelay 15m 15s ------------------ - -Disks -^^^^^ - -A disks section in testconfig will override the disks section in -megatest.config. This can be used to allocate disks on a per-test or per item -basis. - -Controlled waiver propagation -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If test is FAIL and previous test in run with same MT_TARGET is WAIVED then apply the following rules from the testconfig: -If a waiver check is specified in the testconfig apply the check and if it passes then set this FAIL to WAIVED - -Waiver check has two parts, 1) a list of waiver, rulename, filepatterns and 2) the rulename script spec (note that "diff" and "logpro" are predefined) - ------------------ -###### EXAMPLE FROM testconfig ######### -# matching file(s) will be diff'd with previous run and logpro applied -# if PASS or WARN result from logpro then WAIVER state is set -# -[waivers] -# logpro_file rulename input_glob -waiver_1 logpro lookittmp.log - -[waiver_rules] - -# This builtin rule is the default if there is no .logpro file -# diff diff %file1% %file2% - -# This builtin rule is applied if a .logpro file exists -# logpro diff %file1% %file2% | logpro %waivername%.logpro %waivername%.html ------------------ - -Ezsteps -~~~~~~~ - -.Example ezsteps with logpro rules ------------------ -[ezsteps] -lookittmp ls /tmp - -[logpro] -lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line - ;; a blank line indicates the end of the block of text - (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/) - ------------------ - -To transfer the environment to the next step you can do the following: - ----------------------------- -$MT_MEGATEST -env2file .ezsteps/${stepname} ----------------------------- - -Triggers -~~~~~~~~ - -In your testconfig or megatest.config triggers can be specified - ------------------ -[triggers] - -# Call script running.sh when test goes to state=RUNNING, status=PASS -RUNNING/PASS running.sh - -# Call script running.sh any time state goes to RUNNING -RUNNING/ running.sh - -# Call script onpass.sh any time status goes to PASS -PASS/ onpass.sh ------------------ - -Scripts called will have; test-id test-rundir trigger test-name item-path state status event-time, added to the commandline. - -HINT - -To start an xterm (useful for debugging), use a command line like the following: - ------------------ -[triggers] -COMPLETED/ xterm -e bash -s -- ------------------ - -NOTE: There is a trailing space after the -- - -There are a number of environment variables available to the trigger script -but since triggers can be called in various contexts not all variables are -available at all times. The trigger script should check for the variable and -fail gracefully if it doesn't exist. - -.Environment variables visible to the trigger script -[width="90%",cols="^,2m",frame="topbot",options="header"] -|====================== -|Variable | Purpose -| MT_TEST_RUN_DIR | The directory where Megatest ran this test -| MT_CMDINFO | Encoded command data for the test -| MT_DEBUG_MODE | Used to pass the debug mode to nested calls to Megatest -| MT_RUN_AREA_HOME | Megatest home area -| MT_TESTSUITENAME | The name of this testsuite or area -| MT_TEST_NAME | The name of this test -| MT_ITEM_INFO | The variable and values for the test item -| MT_MEGATEST | Which Megatest binary is being used by this area -| MT_TARGET | The target variable values, separated by '/' -| MT_LINKTREE | The base of the link tree where all run tests can be found -| MT_ITEMPATH | The values of the item path variables, separated by '/' -| MT_RUNNAME | The name of the run -|====================== - - -Override the Toplevel HTML File -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Megatest generates a simple html file summary for top level tests of -iterated tests. The generation can be overridden. NOTE: the output of -the script is captured from stdout to create the html. - - -.For test "runfirst" override the toplevel generation with a script "mysummary.sh" ------------------ -# Override the rollup for specific tests -[testrollup] -runfirst mysummary.sh ------------------ - -Archiving Setup ---------------- - -In megatest.config add the following sections: - -.megatest.config --------------- -[archive] -# where to get bup executable -# bup /path/to/bup - -[archive-disks] - -# Archives will be organised under these paths like this: -# / -# Within the archive the data is structured like this: -# /// -archive0 /mfs/myarchive-data/adisk1 --------------- - -Handling Environment Variables ------------------------------- - -It is often necessary to capture and or manipulate environment -variables. Megatest has some facilities built in to help. - -Capture variables -~~~~~~~~~~~~~~~~~ - -.Commands ------------------------------- -# capture the current enviroment into a db called envdat.db under -# the context "before" -megatest -envcap before - -# capture the current environment into a db called startup.db with -# context "after" -megatest -envcap after startup.db - -# write the diff from before to after -megatest -envdelta before-after -dumpmode bash ------------------------------- - -Dump modes include bash, csh and config. You can include config data -into megatest.config or runconfigs.config. - -.Example of generating and using config data ------------------------------- -megatest -envcap original -# do some stuff here -megatest -envcap munged -megatest -envdelta original-munged -dumpmode ini -o modified.config ------------------------------- - -Then in runconfigs.config - -.Example of using modified.config in a testconfig ------------------------------- -cat testconfig -[pre-launch-env-vars] -[include modified.config] ------------------------------- - -Managing Old Runs ------------------ - -It is often desired to keep some older runs around but this must be balanced with the costs of disk space. - -. Use -remove-keep -. Use -archive (can also be done from the -remove-keep interface) -. use -remove-runs with -keep-records - -.For each target, remove all runs but the most recent 3 if they are over 1 week old ---------------------- -# use -precmd 'sleep 5;nbfake' to limit overloading the host computer but to allow the removes to run in parallel. -megatest -actions print,remove-runs -remove-keep 3 -target %/%/%/% -runname % -age 1w -precmd 'sleep 5;nbfake'" ---------------------- - -Nested Runs ------------ - -A Megatest test can run a full Megatest run in either the same -Megatest area or in another area. This is a powerful way of chaining -complex suites of tests and or actions. - -If you are not using the current area you can use ezsteps to retrieve -and setup the sub-Megatest run area. - -In the testconfig: ---------------- -[subrun] - -# Required: wait for the run or just launch it -# if no then the run will be an automatic PASS irrespective of the actual result -run-wait yes|no - -# Optional: where to execute the run. Default is the current runarea -run-area /some/path/to/megatest/area - -# Optional: method to use to determine pass/fail status of the run -# auto (default) - roll up the net state/status of the sub-run -# logpro - use the provided logpro rules, happens automatically if there is a logpro section -# passfail auto|logpro -# Example of logpro: -passfail logpro - -# Optional: -logpro ;; if this section exists then logpro is used to determine pass/fail - (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/) - (expect:fail in "LogFileBody" = 0 "No FAILs allowed" #/FAIL/) - -# Optional: target translator, default is to use the parent target -target #{shell somescript.sh} - -# Optional: runname translator/generator, default is to use the parent runname -run-name #{somescript.sh} - -# Optional: testpatt spec, default is to first look for TESTPATT spec from runconfigs unless there is a contour spec -test-patt %/item1,test2 - -# Optional: contour spec, use the named contour from the megatest.config contour spec -contour contourname ### NOTE: Not implemented yet! Let us know if you need this feature. - -# Optional: mode-patt, use this spec for testpatt from runconfigs -mode-patt TESTPATT - -# Optional: tag-expr, use this tag-expr to select tests -tag-expr quick - -# Optional: (not yet implemented, remove-runs is always propagated at this time), propagate these actions from the parent -# test -# Note// default is % for all -propagate remove-runs archive ... - ---------------- - -Programming API ---------------- - -These routines can be called from the megatest repl. - -.API Keys Related Calls -[width="70%",cols="^,2m,2m,2m",frame="topbot",options="header,footer"] -|====================== -|API Call | Purpose comments | Returns | Comments -|(rmt:get-keys run-id) | | ( key1 key2 ... ) | -| (rmt:get-key-val-pairs run-id) | | #t=success/#f=fail | Works only if the server is still reachable -|====================== - - -:numbered!: - DELETED docs/plan.txt Index: docs/plan.txt ================================================================== --- docs/plan.txt +++ /dev/null @@ -1,127 +0,0 @@ -Road Map --------- - -Note 1: This road-map is still evolving and subject to change without notice. - -Architecture Refactor -~~~~~~~~~~~~~~~~~~~~~ - -Goals -^^^^^ - -. Reduce load on the file system. Sqlite3 files on network filesystem can be - a burden. [green]#[DONE]# -. Reduce number of servers and frequency of start/stop. This is mostly an - issue of clutter but also a reduction in "moving parts". [green]#[DONE]# -. Coalesce activities to a single home host where possible. Give the user - feedback that they have started the dashboard on a host other than the - home host. [green]#[DONE]# -. Reduce number of processes involved in managing running tests. - -Changes Needed -^^^^^^^^^^^^^^ - -. ACID compliant db will be on /tmp and synced to megatest.db with a five - second max delay. [green]#[DONE]# -. Read/writes to db for processes on homehost will go direct to /tmp - megatest.db file. [green]#[DONE]# -. Read/wites fron non-homehost processes will go through one server. Bulk - reads (e.g. for dashboard or list-runs) will be cached on the current host - in /tmp and synced from the home megatest.db in the testsuite area. [green]#[DONE]# -. Db syncs rely on the target db file timestame minus some margin. [green]#[DONE]# -. Since bulk reads do not use the server we can switch to simple RPC for the - network transport. [green]#[DONE]# -. Test running manager process extended to manage multiple running tests. - -Current Items -~~~~~~~~~~~~~ - -ww05 - migrate to inmem-db -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -. Switch to inmem db with fast sync to on disk db's [green]#[DONE]# -. Server polls tasks table for next action -.. Task table used for tracking runner process [red]#[Replaced by mtutil]# -.. Task table used for jobs to run [red]#[Replaced by mtutil]# -.. Task table used for queueing runner actions (remove runs, - cleanRunExecute, etc) [red]#[Replaced by mtutil#] - - -// ww32 -// ~~~~ -// -// . Rerun step and or subsequent steps from gui -// . Refresh test area files from gui -// . Clean and re-run button -// . Clean up STATE and STATUS handling. -// .. Dashboard and Test control panel are reverse order - choose and fix -// .. Move seldom used states and status to drop down selector -// . Access test control panel when clicking on Run Summary tests -// . Feature: -generate-index-tree -// . Change specifing of state and status to use STATE1/STATUS1,STATE2/STATUS2 -// -// ww33 -// ~~~~ -// -// . http api available for use with Perl, Ruby etc. scripts -// . megatest.config setup entries for: -// .. run launching (e.g. /bin/sh %CMD% > /dev/null) -// .. browser "konqueror %FNAME% -// -// ww34 -// ~~~~ -// -// . Mark dependent tests for clean/rerun -rerun-downstream -// . On run start check for defunct tests in RUNNING, LAUNCHED or REMOTEHOSTSTART and correct or notify -// . Fix: refresh of gui sometimes fails on last item (race condition?) -// -// ww35 -// ~~~~ -// -// . refdb: Add export of csv, json and sexp -// . Convert to using call-with-environment-variables where possible. Should allow handling of parallel runs in same process. -// . Re-work text interface wizards. Several bugs on record. Possibly convert to gui based. -// . Add to testconfig requirements section; launchlimiter scriptname, calls scriptname to check if ok to launch test -// . Refactor Run Summary view, currently very clumsy -// . Add option to show steps in Run Summary view -// -// ww36 -// ~~~~ -// -// . Refactor guis for resizeablity -// . Add filters to Run Summary view and Run Control view -// . Add to megatest.config or testconfig; rerunok STATE/STATUS,STATE/STATUS... -// . Launch gates for diskspace; /path/one>1G,/path/two>200M,/tmp>5G,#{scheme *toppath*}>1G -// -// Bin List -// ~~~~~~~~ -// -// . Quality improvements -// .. Server stutters occasionally -// .. Large number of items or tests still has some issues. -// .. Code refactoring -// .. Replace remote process with true API using json (supports Web app also) -// . Streamline the gui -// .. Everything resizable -// .. Less clutter -// .. Tool tips -// .. Filters on Run Summary, Summary and Run Control panel -// .. Built in log viewer (partially implemented) -// .. Refactor the test control panel -// . Help and documentation -// .. Complete the user manual (I’ve been working on this lately). -// .. Online help in the gui -// . Streamlined install -// .. Deployed version (download a location independent ready to run binary bundle) -// .. Install Makefile (in progress, needed for Mike to install on VMs) -// .. Added option to compile IUP (needed for VMs) -// . Server side run launching -// . Support for re-running, cleaning etc. of individual steps (ezsteps makes this very easy to implement). -// . Launch process needs built in daemonizing (easy to do, just need to test it thoroughly). -// . Wizards for creating tests, regression areas (current ones are text only and limited). -// . Fully functional built in web service (currently you can browse runs but it is very simplistic). -// . Wildcards in runconfigs: e.g. [p1271/9/%/%] -// . Gui panels for editing megatest.config and runconfigs.config -// . Fully isolated tests (no use of NFS to see regression area files) -// . Windows version -