+Why Megatest?
-
-
The Megatest project was started for two reasons, the first was an -immediate and pressing need for a generalized tool to manage a suite -of regression tests and the second was the fact that I had written or -maintained several such tools at different companies over the years. I -thought a single open source tool, flexible enough to meet the needs -of any team doing continuous integration and or running a complex -suite of tests for release qualification would solve some problems for -me and for others.
-
+
-
-- Matt Welland, original author of the Megatest tool suite.-
Megatest was created to provide a generalized tool for managing suites +of regression tests and to provide a multi-host, distributed +alternative to "make". The EDA world is littered with proprietory, +company-specific tools for this purpose and by going open source and +keeping the tool flexible the hope is that Megatest could be useful to +any team at any company for continuous integration and almost any +other general automation tasks.
Megatest Design Philosophy
@@ -959,15 +954,10 @@
-
Road Map
-
-
TODO / Road Map
Note: This road-map is a wish list and not a formal plan. Items are in @@ -1941,10 +1931,146 @@
This test runs a single step called "stepname1" which runs a script "stepname.sh". Note that although it is common to put the actions needed for a test step into a script it is not necessary.
+
Debugging
+
+
+
+Well Written Tests
+
+
+
+
+
+
+Test Design and Surfacing
+Design your tests to surface errors. Ensure that all logs are processed by logpro and can be reached by a mouse click or two from the test control panel.
using the tool to make life easier
layers - less is often more
Clicks-to-error
+
+
+
+
+
+
+A word on Bisecting
+Bisecting is a debug strategy intended to speed up finding the root +cause.
+
+
+
+
+
+Figure 1. A complex process with a problem found in stage "E"
+It is common to start debugging where the problem was observed and +then work back. However by inspecting the output at stage "C" in the +example above you would potentially save a lot of debug effort, this +is similar to the feature in source control tools like git and fossil +called biseceting.
To know where to look for problems you will want to look at the log files at various stages in the execution process.
+
+
+
+
+
+Figure 2. A simplified diagram of the stages Megatest goes through to run a test.
+| Stage | +How to inspect | +Watch for | +
|---|---|---|
A: post config processing |
+megatest -show-config -target your/target |
+#f (failed var processing) |
+
B: post runconfig |
+megatest -show-runconfig -target your/target |
++ |
C: processing testconfigs |
+inspect output from "megatest -run …" |
+Messages indicating issues process configs, dependency problems. |
+
D: process testconfig for test launch |
+inspect output from megatest runner |
+Zero items (items expansion yielded no items) |
+
E,F: launching test |
+start test xterm, look at mt_launch.log |
+Did your batch system accept the job? Has the job landed on a machine? |
+
G: starting test |
+look at your batch systems logs for the process |
+Did the megatest -execute process start and run? |
+
H,H1,H2: step exectution |
+look at <stepname>.log, <stepname>.html and your own internal logs |
+Do you have sufficiently tight logpro rules? You must always have a "required" rule! |
+
+
+Examining The Test Logs and Environment
+
+
+
+
+
+
+Test Control Panel - xterm
+From the dashboard click on a test PASS/FAIL button. This brings up a +test control panel. Aproximately near the center left of the window +there is a button "Start Xterm". Push this to get an xterm with the +full context and environment loaded for that test. You can run scripts +or ezsteps by copying from the testconfig (hint, load up the +testconfig in a separate text editor window).
With more recent versions of Megatest you can step through your test +from the test control panel. Click on the cell labeled "rerun this +step" to only rerun the step or click on "restart from here" to rerun +that step and downstream steps.
NOTE 1: visual feedback can take some time, give it a few seconds and +you will see the step change color to blue as it starts running.
NOTE 2: steping through only works if you are using ezsteps.
+
+
+
+
+Config File Processing
+As described above it is often helpful to know the content of +variables in various contexts as Megatest works through the actions +needed to run your tests. A handy technique is to force the startup of +an xterm in the context being examined.
For example, if an item list is not being generated as expected you +can inject the startup of an xterm as if it were an item:
+
+Original items table
+
+
[items] +CELLNAME [system getcellname.sh]+
+
+Items table modified for debug
+
+
[items] +DEBUG [system xterm] +CELLNAME [system getcellnames.sh]+
When this test is run an xterm will pop up. In that xterm the +environment is exactly that in which the script "getcellnames.sh" +would run. You can now debug the script to find out why it isn’t +working as expected.
How To Do Things
@@ -2036,14 +2162,14 @@
-
-Tricks
+Tricks and Tips
-
+
-
-This section is a compendium of a various useful tricks for debugging, -configuring and generally getting the most out of Megatest.
This section is a collection of a various useful tricks for that +didn’t quite fit elsewhere.
Limiting your running jobs
The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously.
In your testconfig:
@@ -2056,49 +2182,10 @@
-[jobgroups] group1 10 custdes 4
-
Debugging Tricks
-
-
-
Examining The Environment
-
-
-
-Test Control Panel - xterm
-From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the -window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run -scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way -to debug your tests.
-
-
-
-
During Config File Processing
-It is often helpful to know the content of variables in various -contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined.
For example, if an item list is not being generated as expected you -can inject the startup of an xterm as if it were an item:
-
-Original items table
-
-
[items] -CELLNAME [system getcellname.sh]-
-
-Items table modified for debug
-
-
[items] -DEBUG [system xterm] -CELLNAME [system getcellnames.sh]-
When this test is run an xterm will pop up. In that xterm the -environment is exactly that in which the script "getcellnames.sh" -would run. You can now debug the script to find out why it isn’t -working as expected.
Organising Your Tests and Tasks
The default location "tests" for storing tests can be extended by adding to your tests-paths section.
@@ -2148,11 +2235,11 @@
-
Megatest Use Modes
| Variable | @@ -3476,11 +3563,11 @@
|---|