Megatest

    along with Megatest.  If not, see &lt;http://www.gnu.org/licenses/&gt;.</pre>
</div></div>
</div>
</div>
<div class="sect1">
<h2 id="_why_megatest">Why Megatest?</h2>
<div class="sectionbody">
<div class="paragraph"><p>Megatest was created to provide a generalized tool for managing suites

of regression tests and to provide a multi-host, distributed
alternative to "make". The EDA world is littered with proprietory,
company-specific tools for this purpose and by going open source and
keeping the tool flexible the hope is that Megatest could be useful to
any team at any company for continuous integration and almost any

other general automation tasks.</p></div>




</div>
</div>
<div class="sect1">
<h2 id="_megatest_design_philosophy">Megatest Design Philosophy</h2>
<div class="sectionbody">
<div class="paragraph"><p>Megatest is a distributed system intended to provide the minimum needed
resources to make writing a suite of tests and tasks for implementing

<div class="imageblock">
<div class="content">
<img src="megatest-system-architecture.png" alt="Static">
</div>
</div>
</div>
</div>





</div>
<div class="sect1">
<h2 id="_todo_road_map">TODO / Road Map</h2>
<div class="sectionbody">
<div class="paragraph"><p>Note: This road-map is a wish list and not a formal plan. Items are in
rough priority but are subject to change. Development is driven by
user requests, developer "itch" and bug reports. Please contact

reviewed never</pre>
</div></div>
<div class="paragraph"><p>This test runs a single step called "stepname1" which runs a script
"stepname.sh". Note that although it is common to put the actions
needed for a test step into a script it is not necessary.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_debugging">Debugging</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_well_written_tests">Well Written Tests</h3>
<div class="sect3">
<h4 id="_test_design_and_surfacing_errors">Test Design and Surfacing Errors</h4>
<div class="paragraph"><p>Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.</p></div>
<div class="paragraph"><p>To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:</p></div>
<div class="literalblock">
<div class="title">script1.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script2.sh some parameters &gt; script2.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script2.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script3.sh some more parameters &gt; script3.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script3.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
virtuoso params and switches ...
more code ...</pre>
</div></div>
<div class="paragraph"><p>The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:</p></div>
<div class="literalblock">
<div class="title">testconfig</div>
<div class="content monospaced">
<pre>[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log

[logpro]
step1 ;; some logpro rules
  (expect:required in "LogFileBody" &gt; 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh

[scripts]
script1.sh #!/bin/bash
 code ...

...</pre>
</div></div>
<div class="paragraph"><p>With the above testconfig the logs for every critical part of the
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.</p></div>
<div class="paragraph"><p>An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.</p></div>
<div class="paragraph"><p>The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_examining_the_test_logs_and_environment">Examining The Test Logs and Environment</h3>
<div class="sect3">
<h4 id="_test_control_panel_xterm">Test Control Panel - xterm</h4>
<div class="paragraph"><p>From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).</p></div>
<div class="paragraph"><p>With more recent versions of Megatest you can step through your test
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.</p></div>
<div class="paragraph"><p>NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.</p></div>
<div class="paragraph"><p>NOTE 2: steping through only works if you are using ezsteps.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_a_word_on_bisecting">A word on Bisecting</h3>
<div class="paragraph"><p>Bisecting is a debug strategy intended to speed up finding the root
cause of some bug.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="bisecting.png" alt="bisecting.png">
</div>
<div class="title">Figure 1. A complex process with a problem found in stage "E"</div>
</div>
<div class="paragraph"><p>It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.</p></div>
</div>
<div class="sect2">
<h3 id="_tough_bugs">Tough Bugs</h3>
<div class="paragraph"><p>Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.</p></div>
<div class="paragraph"><p>Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.</p></div>
<div class="paragraph"><p>To isolate bugs like this you may need to look at the log files at
various stages in the execution process of your run and tests.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="megatest-test-stages.png" alt="megatest-test-stages.png">
</div>
<div class="title">Figure 2. A simplified diagram of the stages Megatest goes through to run a test.</div>
</div>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 2. How to check variable values and inspect logs at each stage</caption>
<col style="width:20%;">
<col style="width:40%;">
<col style="width:40%;">
<thead>
<tr>
<th class="tableblock halign-left valign-top" >Stage                      </th>
<th class="tableblock halign-left valign-top" > How to inspect                               </th>
<th class="tableblock halign-left valign-top" > Watch for or try &#8230;</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">A: post config processing</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">megatest -show-config -target your/target</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">#f (failed var processing)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">B: post runconfig</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">megatest -show-runconfig -target your/target</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Add -debug 0,9 to see which file your settings come from</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">C: processing testconfigs</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">inspect output from "megatest -run &#8230;"</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Messages indicating issues process configs, dependency problems</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">D: process testconfig for test launch</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">inspect output from megatest runner</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Zero items (items expansion yielded no items)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">E,F: launching test</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">start test xterm, look at mt_launch.log</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Did your batch system accept the job? Has the job landed on a machine?</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">G: starting test</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">look at your batch systems logs for the process</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Did the megatest -execute process start and run? Extract the "megatest -execute &#8230;" command and run it from your xterm.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">H,H1,H2: step exectution</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">look at &lt;stepname&gt;.log, &lt;stepname&gt;.html and your own internal logs</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Do you have sufficiently tight logpro rules? You must always have a "required" rule!</p></td>
</tr>
</tbody>
</table>
<div class="sect3">
<h4 id="_bisecting_megatest_csh_sh">Bisecting megatest.csh/sh</h4>
<div class="paragraph"><p>Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.</p></div>
<div class="paragraph"><p>Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.</p></div>
<div class="paragraph"><p>This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.</p></div>
</div>
<div class="sect3">
<h4 id="_csh_and_f">csh and -f</h4>
<div class="paragraph"><p>A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.</p></div>
<div class="literalblock">
<div class="title">Not good</div>
<div class="content monospaced">
<pre>#!/bin/tcsh
...</pre>
</div></div>
<div class="literalblock">
<div class="title">Good</div>
<div class="content monospaced">
<pre>#!/bin/tcsh -f
...</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_config_file_processing">Config File Processing</h4>
<div class="paragraph"><p>As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
an xterm in the context being examined.</p></div>
<div class="paragraph"><p>For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:</p></div>
<div class="listingblock">
<div class="title">Original items table</div>
<div class="content monospaced">
<pre>[items]
CELLNAME [system getcellname.sh]</pre>
</div></div>
<div class="listingblock">
<div class="title">Items table modified for debug</div>
<div class="content monospaced">
<pre>[items]
DEBUG [system xterm]
CELLNAME [system getcellnames.sh]</pre>
</div></div>
<div class="paragraph"><p>When this test is run an xterm will pop up. In that xterm the
environment is exactly that in which the script "getcellnames.sh"
would run. You can now debug the script to find out why it isn&#8217;t
working as expected.</p></div>
<div class="paragraph"><p>Similarly in a script just call the xterm.
NOTE: This technique can be very helpful in debugging running of EDA tools in Perl, Ruby, Python or tcl scripts:</p></div>
<div class="literalblock">
<div class="title">Perl example</div>
<div class="content monospaced">
<pre>some_code();
$cmdline="virtuoso -some-switches and params ...";
print "$cmdline"; # print the command line so you can paste it into the xterm that pops up
system("xterm");  # this line is added for the debug and removed when done
system($cmdline);
more_code();</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_misc_other_debugging_hints">Misc Other Debugging Hints</h3>
<div class="sect3">
<h4 id="_annotating_scripts_and_config_files">Annotating scripts and config files</h4>
<div class="paragraph"><p>Run the "env" command to record the environment:</p></div>
<div class="paragraph"><p>env | sort &gt; stagename.log</p></div>
<div class="paragraph"><p>In a config file:</p></div>
<div class="literalblock">
<div class="title">megatest.config, runconfigs.config and testconfig</div>
<div class="content monospaced">
<pre>#{shell env | sort &gt; stagename.log}

# or

[system env | sort &gt; stagename.log]</pre>
</div></div>
<div class="paragraph"><p>In scripts just insert the commands, this example helps you identify
if "some commands &#8230;" changed any environment variables.:</p></div>
<div class="literalblock">
<div class="title">myscript.sh</div>
<div class="content monospaced">
<pre>env | sort &gt; somefile-before.log
some commands ...
env | sort &gt; somefile-after.log</pre>
</div></div>
<div class="literalblock">
<div class="title">Use meld to examine the differences</div>
<div class="content monospaced">
<pre>meld somefile-before.log somefile-after.log</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_oneshot_modifying_a_variable">Oneshot Modifying a Variable</h4>
<div class="paragraph"><p>To try various values for a variable without mutating the current value</p></div>
<div class="literalblock">
<div class="title">within a bash shell</div>
<div class="content monospaced">
<pre>SOMEVAR=123 runcmd.sh</pre>
</div></div>
<div class="literalblock">
<div class="title">within csh</div>
<div class="content monospaced">
<pre>(setenv SOMEVAR 123;runcmd.sh)

# OR

env SOMEVAR=123 runcmd.sh</pre>
</div></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_how_to_do_things">How To Do Things</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_process_runs">Process Runs</h3>
<div class="sect3">

# match.
flexi-launcher yes</pre>
</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_tricks_and_tips">Tricks and Tips</h2>
<div class="sectionbody">
<div class="paragraph"><p>This section is a collection of a various useful tricks for that
didn&#8217;t quite fit elsewhere.</p></div>
<div class="sect2">
<h3 id="_limiting_your_running_jobs">Limiting your running jobs</h3>
<div class="paragraph"><p>The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously.</p></div>
<div class="paragraph"><p>In your testconfig:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[test_meta]
jobgroup group1</pre>
</div></div>
<div class="paragraph"><p>In your megatest.config:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[jobgroups]
group1 10
custdes 4</pre>
</div></div>







































<div class="sect3">
<h4 id="_organising_your_tests_and_tasks">Organising Your Tests and Tasks</h4>
<div class="paragraph"><p>The default location "tests" for storing tests can be extended by
adding to your tests-paths section.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[misc]

<div class="sectionbody">
<div class="sect2">
<h3 id="_megatest_use_modes">Megatest Use Modes</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 3. Base commands</caption>
<col style="width:20%;">
<col style="width:40%;">
<col style="width:40%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Use case                </th>
<th class="tableblock halign-left valign-top" > Megatest command     </th>

<div class="sect2">
<h3 id="_config_file_helpers">Config File Helpers</h3>
<div class="paragraph"><p>Various helpers for more advanced config files.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 4. Helpers</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Helper                      </th>

</div>
<div class="sect2">
<h3 id="_database_settings">Database settings</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 5. Database config settings in [setup] section of megatest.config</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Var                       </th>

[logpro]
lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line
  ;;     a blank line indicates the end of the block of text
  (expect:required in "LogFileBody" &gt; 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/)</pre>
</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
# if your upstream file is csh you can force csh like this
# if your upstream is bash
loadenv     source $REF/ourenviron.sh

<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
</div>
<div class="sect2">

but since triggers can be called in various contexts not all variables are
available at all times. The trigger script should check for the variable and
fail gracefully if it doesn&#8217;t exist.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:90%;
">
<caption class="title">Table 6. Environment variables visible to the trigger script</caption>
<col style="width:50%;">
<col style="width:50%;">
<thead>
<tr>
<th class="tableblock halign-left valign-top" > Variable            </th>
<th class="tableblock halign-left valign-top" > Purpose</th>
</tr>

<h2 id="_programming_api">Programming API</h2>
<div class="sectionbody">
<div class="paragraph"><p>These routines can be called from the megatest repl.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 7. API Keys Related Calls</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >API Call                        </th>