Megatest

Check-in [1d5521b5a2]
Login
☰Home Timeline Files Branches Tags Tickets Wiki Login
Overview
Comment:Merged manual updates on debugging.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | v1.65
Files: files | file ages | folders
SHA1: 1d5521b5a2cae9d82fccda6ef789103f52abdcac
User & Date: mrwellan on 2022-02-17 10:19:40
Other Links: branch diff | manifest | tags
Context
2022-02-17
12:13
Fixed typo (cherrypick) check-in: e106a99b30 user: mrwellan tags: v1.65
10:19
Merged manual updates on debugging. check-in: 1d5521b5a2 user: mrwellan tags: v1.65
2022-02-16
19:24
Changed megatest version to 1.6589 check-in: d82cbc8d90 user: mmgraham tags: v1.65, v1.6589
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Modified docs/manual/Makefile from [ec9633c3d9] to [759e8c25e6].

︙
28
29
30
31
32
33
34
35

36
37
38
39

40
41
42
43
44
45
46

28
29
30
31
32
33
34

35
36
37
38

39
40
41
42
43
44
45
46








-
+



-
+









# design_spec.html : $(SRCFILES) $(CSVFILES)
#         asciidoc -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 design_spec.txt
#

all : server.ps megatest_manual.html client.ps complex-itemmap.png megatest_manual.pdf

megatest_manual.html : megatest_manual.txt  *.txt installation.txt *png
megatest_manual.html : megatest_manual.txt  *.txt installation.txt *png *.dot
	asciidoc  -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 megatest_manual.txt
#	dos2unix megatest_manual.html

megatest_manual.pdf : megatest_manual.txt *.txt *png
megatest_manual.pdf : megatest_manual.txt *.txt *png *.dot
	a2x -a toc -f pdf megatest_manual.txt

server.ps : server.dot
	dot -Tps server.dot > server.ps

client.ps : client.dot
	dot -Tps client.dot > client.ps
︙

Added docs/manual/bisecting.dot version [01396be470]. 

































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

//  Copyright 2021, Matthew Welland.
// 
// This file is part of Megatest.
// 
//     Megatest is free software: you can redistribute it and/or modify
//     it under the terms of the GNU General Public License as published by
//     the Free Software Foundation, either version 3 of the License, or
//     (at your option) any later version.
// 
//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.
//
digraph G {
    rankdir=LR
    subgraph cluster_1 {
        node [style=filled,shape=box];

	B [label="B\nProblem is here"];
	E [label="E\nProblem manifests here"];
	A -> B;
	B -> C;
	C -> D;
	D -> E;
    }

}








Added docs/manual/bisecting.png version [81573d185b].

cannot compute difference between binary files




Added docs/manual/debugging.txt version [731079995f].






























































































































































































































































































































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

//  Copyright 2021, Matthew Welland.
// 
// This file is part of Megatest.
// 
//     Megatest is free software: you can redistribute it and/or modify
//     it under the terms of the GNU General Public License as published by
//     the Free Software Foundation, either version 3 of the License, or
//     (at your option) any later version.
// 
//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

Debugging
---------

Well Written Tests
~~~~~~~~~~~~~~~~~~

Test Design and Surfacing Errors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.

To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:

.script1.sh
..............................
#!/bin/bash
code ...
script2.sh some parameters > script2.log
more code ...
..............................

.script2.sh
..............................
#!/bin/bash
code ...
script3.sh some more parameters > script3.log
more code ...
..............................

.script3.sh
..............................
#!/bin/bash
code ...
virtuoso params and switches ...
more code ...
..............................

The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:

.testconfig
..............................
[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log

[logpro]
step1 ;; some logpro rules
  (expect:required in "LogFileBody" > 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh

[scripts]
script1.sh #!/bin/bash
 code ...

...
..............................

With the above testconfig the logs for every critical part of the
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.

An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.

The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.

Examining The Test Logs and Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^

From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).

With more recent versions of Megatest you can step through your test
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.

NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.

NOTE 2: steping through only works if you are using ezsteps.

A word on Bisecting
~~~~~~~~~~~~~~~~~~~

Bisecting is a debug strategy intended to speed up finding the root
cause of some bug.

.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------

It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.

Tough Bugs
~~~~~~~~~~

Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.

Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.

To isolate bugs like this you may need to look at the log files at
various stages in the execution process of your run and tests.

.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------

.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage	                    | How to inspect                               | Watch for or try ...
|A: post config processing  | megatest -show-config -target your/target	   | #f (failed var processing)
|B: post runconfig          | megatest -show-runconfig -target your/target | Add -debug 0,9 to see which file your settings come from
|C: processing testconfigs  | inspect output from "megatest -run ..."      | Messages indicating issues process configs, dependency problems
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test        | start test xterm, look at mt_launch.log      | Did your batch system accept the job? Has the job landed on a machine? 
|G: starting test           | look at your batch systems logs for the process | Did the megatest -execute process start and run? Extract the "megatest -execute ..." command and run it from your xterm.
|H,H1,H2: step exectution   | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule! 
|======================

Bisecting megatest.csh/sh
^^^^^^^^^^^^^^^^^^^^^^^^^

Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.

Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.

This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.

csh and -f
^^^^^^^^^^

A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.

.Not good
..............................
#!/bin/tcsh
...
..............................

.Good
..............................
#!/bin/tcsh -f
...
..............................


Config File Processing
^^^^^^^^^^^^^^^^^^^^^^

As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
an xterm in the context being examined.

For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:

.Original items table
-----------------
[items]
CELLNAME [system getcellname.sh]
-----------------

.Items table modified for debug
-----------------
[items]
DEBUG [system xterm]
CELLNAME [system getcellnames.sh]
-----------------

When this test is run an xterm will pop up. In that xterm the
environment is exactly that in which the script "getcellnames.sh"
would run. You can now debug the script to find out why it isn't
working as expected.

Similarly in a script just call the xterm.
NOTE: This technique can be very helpful in debugging running of EDA tools in Perl, Ruby, Python or tcl scripts:

.Perl example
..............................
some_code();
$cmdline="virtuoso -some-switches and params ...";
print "$cmdline"; # print the command line so you can paste it into the xterm that pops up
system("xterm");  # this line is added for the debug and removed when done
system($cmdline);
more_code();
..............................

Misc Other Debugging Hints
~~~~~~~~~~~~~~~~~~~~~~~~~~

Annotating scripts and config files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Run the "env" command to record the environment:

env | sort > stagename.log

In a config file:

.megatest.config, runconfigs.config and testconfig
...................
#{shell env | sort > stagename.log}

# or

[system env | sort > stagename.log]
...................

In scripts just insert the commands, this example helps you identify
if "some commands ..." changed any environment variables.:

.myscript.sh
..............................
env | sort > somefile-before.log
some commands ...
env | sort > somefile-after.log
..............................

.Use meld to examine the differences
..............................
meld somefile-before.log somefile-after.log
..............................

Oneshot Modifying a Variable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To try various values for a variable without mutating the current value

.within a bash shell
..............................
SOMEVAR=123 runcmd.sh
..............................

.within csh
..............................
(setenv SOMEVAR 123;runcmd.sh)

# OR

env SOMEVAR=123 runcmd.sh
..............................

Capturing output from a command
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.Use the "script" utility
..............................
script -c "virtuoso -params and switches ..."
..............................











Modified docs/manual/howto.txt
from [5266978039]
to [1b987427fd].




︙


108
109
110
111
112
113
114
115
116


117
118
119


120
121
122
123
124
125
126

108
109
110
111
112
113
114


115
116
117


118
119
120
121
122
123
124
125
126








-
-
+
+

-
-
+
+








[jobtools]
launcher bsub
# if defined and not "no" flexi-launcher will bypass launcher unless there is no
# match.
flexi-launcher yes
------------------------

Tricks
------
Tricks and Tips
---------------

This section is a compendium of a various useful tricks for debugging,
configuring and generally getting the most out of Megatest.
This section is a collection of a various useful tricks for that
didn't quite fit elsewhere.

Limiting your running jobs
~~~~~~~~~~~~~~~~~~~~~~~~~~

The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously.

In your testconfig:




︙


134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187

134
135
136
137
138
139
140








































141
142
143
144
145
146
147








-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-









---------------
[jobgroups]
group1 10
custdes 4
---------------

Debugging Tricks
----------------

Examining The Environment
~~~~~~~~~~~~~~~~~~~~~~~~~

Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^

From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the 
window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run
scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way
to debug your tests.

During Config File Processing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is often helpful to know the content of variables in various
contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined.

For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:

.Original items table
-----------------
[items]
CELLNAME [system getcellname.sh]
-----------------

.Items table modified for debug
-----------------
[items]
DEBUG [system xterm]
CELLNAME [system getcellnames.sh]
-----------------

When this test is run an xterm will pop up. In that xterm the
environment is exactly that in which the script "getcellnames.sh"
would run. You can now debug the script to find out why it isn't
working as expected.

Organising Your Tests and Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The default location "tests" for storing tests can be extended by
adding to your tests-paths section.





︙







Added docs/manual/megatest-test-stages.dot version [41d7d6e43f].














































1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

//  Copyright 2021, Matthew Welland.
// 
// This file is part of Megatest.
// 
//     Megatest is free software: you can redistribute it and/or modify
//     it under the terms of the GNU General Public License as published by
//     the Free Software Foundation, either version 3 of the License, or
//     (at your option) any later version.
// 
//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.
//
digraph G {
    // rankdir=LR
    subgraph cluster_1 {
        node [style=filled,shape=box];
	A [label="A: Process megatest.config"]
	B [label="B: Process runconfig.config"]
	A -> B [label="resolve vars"]
	B2 [label="B2: Resolve variables"]
	B -> B2 -> A
	C [label="C: Process testconfigs (find tests to run)"]
	D [label="D: Process testconfig for test of interest"]
	E [label="E: Set vars for launching test"]
	F [label="F: Launch into batch system, ssh, batch system\nand different hosts can all \nimpact variable values"]
	G [label="G: Process testconfig again with all available variables"]
	H [label="H: Start test"]
	H1 [label="H1: Start step"]
	H2 [label="H2: Execute step script"]
	I [label="I: End step, process logfile with logpro"]
	I -> H [label="Run remaining steps"]
	H-> H1 -> H2 -> I
	B->C->D->E->F->G->H
	I->D
    }

}








Added docs/manual/megatest-test-stages.png version [e53ee8abbe].

cannot compute difference between binary files




Modified docs/manual/megatest_manual.html
from [a02a70016f]
to [e1b85635a3].




︙


806
807
808
809
810
811
812
813

814
815
816
817
818





819
820

821
822
823
824
825
826
827
828
829
830
831

806
807
808
809
810
811
812

813





814
815
816
817
818


819




820
821
822
823
824
825
826








-
+
-
-
-
-
-
+
+
+
+
+
-
-
+
-
-
-
-








    along with Megatest.  If not, see &lt;http://www.gnu.org/licenses/&gt;.</pre>
</div></div>
</div>
</div>
<div class="sect1">
<h2 id="_why_megatest">Why Megatest?</h2>
<div class="sectionbody">
<div class="paragraph"><p>The Megatest project was started for two reasons, the first was an
<div class="paragraph"><p>Megatest was created to provide a generalized tool for managing suites
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years. I
thought a single open source tool, flexible enough to meet the needs
of any team doing continuous integration and or running a complex
of regression tests and to provide a multi-host, distributed
alternative to "make". The EDA world is littered with proprietory,
company-specific tools for this purpose and by going open source and
keeping the tool flexible the hope is that Megatest could be useful to
any team at any company for continuous integration and almost any
suite of tests for release qualification would solve some problems for
me and for others.</p></div>
other general automation tasks.</p></div>
<div class="literalblock">
<div class="content monospaced">
<pre>-- Matt Welland, original author of the Megatest tool suite.</pre>
</div></div>
</div>
</div>
<div class="sect1">
<h2 id="_megatest_design_philosophy">Megatest Design Philosophy</h2>
<div class="sectionbody">
<div class="paragraph"><p>Megatest is a distributed system intended to provide the minimum needed
resources to make writing a suite of tests and tasks for implementing




︙


957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975

952
953
954
955
956
957
958





959
960
961
962
963
964
965








-
-
-
-
-








<div class="imageblock">
<div class="content">
<img src="megatest-system-architecture.png" alt="Static">
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_road_map">Road Map</h2>
<div class="sectionbody">
</div>
</div>
<div class="sect1">
<h2 id="_todo_road_map">TODO / Road Map</h2>
<div class="sectionbody">
<div class="paragraph"><p>Note: This road-map is a wish list and not a formal plan. Items are in
rough priority but are subject to change. Development is driven by
user requests, developer "itch" and bug reports. Please contact




︙


1939
1940
1941
1942
1943
1944
1945












































































































































































































































































































1946
1947
1948
1949
1950
1951
1952

1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236
2237
2238
2239
2240
2241
2242








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








reviewed never</pre>
</div></div>
<div class="paragraph"><p>This test runs a single step called "stepname1" which runs a script
"stepname.sh". Note that although it is common to put the actions
needed for a test step into a script it is not necessary.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_debugging">Debugging</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_well_written_tests">Well Written Tests</h3>
<div class="sect3">
<h4 id="_test_design_and_surfacing_errors">Test Design and Surfacing Errors</h4>
<div class="paragraph"><p>Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.</p></div>
<div class="paragraph"><p>To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:</p></div>
<div class="literalblock">
<div class="title">script1.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script2.sh some parameters &gt; script2.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script2.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script3.sh some more parameters &gt; script3.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script3.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
virtuoso params and switches ...
more code ...</pre>
</div></div>
<div class="paragraph"><p>The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:</p></div>
<div class="literalblock">
<div class="title">testconfig</div>
<div class="content monospaced">
<pre>[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log

[logpro]
step1 ;; some logpro rules
  (expect:required in "LogFileBody" &gt; 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh

[scripts]
script1.sh #!/bin/bash
 code ...

...</pre>
</div></div>
<div class="paragraph"><p>With the above testconfig the logs for every critical part of the
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.</p></div>
<div class="paragraph"><p>An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.</p></div>
<div class="paragraph"><p>The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_examining_the_test_logs_and_environment">Examining The Test Logs and Environment</h3>
<div class="sect3">
<h4 id="_test_control_panel_xterm">Test Control Panel - xterm</h4>
<div class="paragraph"><p>From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).</p></div>
<div class="paragraph"><p>With more recent versions of Megatest you can step through your test
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.</p></div>
<div class="paragraph"><p>NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.</p></div>
<div class="paragraph"><p>NOTE 2: steping through only works if you are using ezsteps.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_a_word_on_bisecting">A word on Bisecting</h3>
<div class="paragraph"><p>Bisecting is a debug strategy intended to speed up finding the root
cause of some bug.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="bisecting.png" alt="bisecting.png">
</div>
<div class="title">Figure 1. A complex process with a problem found in stage "E"</div>
</div>
<div class="paragraph"><p>It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.</p></div>
</div>
<div class="sect2">
<h3 id="_tough_bugs">Tough Bugs</h3>
<div class="paragraph"><p>Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.</p></div>
<div class="paragraph"><p>Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.</p></div>
<div class="paragraph"><p>To isolate bugs like this you may need to look at the log files at
various stages in the execution process of your run and tests.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="megatest-test-stages.png" alt="megatest-test-stages.png">
</div>
<div class="title">Figure 2. A simplified diagram of the stages Megatest goes through to run a test.</div>
</div>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 2. How to check variable values and inspect logs at each stage</caption>
<col style="width:20%;">
<col style="width:40%;">
<col style="width:40%;">
<thead>
<tr>
<th class="tableblock halign-left valign-top" >Stage                      </th>
<th class="tableblock halign-left valign-top" > How to inspect                               </th>
<th class="tableblock halign-left valign-top" > Watch for or try &#8230;</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">A: post config processing</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">megatest -show-config -target your/target</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">#f (failed var processing)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">B: post runconfig</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">megatest -show-runconfig -target your/target</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Add -debug 0,9 to see which file your settings come from</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">C: processing testconfigs</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">inspect output from "megatest -run &#8230;"</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Messages indicating issues process configs, dependency problems</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">D: process testconfig for test launch</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">inspect output from megatest runner</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Zero items (items expansion yielded no items)</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">E,F: launching test</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">start test xterm, look at mt_launch.log</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Did your batch system accept the job? Has the job landed on a machine?</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">G: starting test</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">look at your batch systems logs for the process</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Did the megatest -execute process start and run? Extract the "megatest -execute &#8230;" command and run it from your xterm.</p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">H,H1,H2: step exectution</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">look at &lt;stepname&gt;.log, &lt;stepname&gt;.html and your own internal logs</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Do you have sufficiently tight logpro rules? You must always have a "required" rule!</p></td>
</tr>
</tbody>
</table>
<div class="sect3">
<h4 id="_bisecting_megatest_csh_sh">Bisecting megatest.csh/sh</h4>
<div class="paragraph"><p>Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.</p></div>
<div class="paragraph"><p>Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.</p></div>
<div class="paragraph"><p>This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.</p></div>
</div>
<div class="sect3">
<h4 id="_csh_and_f">csh and -f</h4>
<div class="paragraph"><p>A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.</p></div>
<div class="literalblock">
<div class="title">Not good</div>
<div class="content monospaced">
<pre>#!/bin/tcsh
...</pre>
</div></div>
<div class="literalblock">
<div class="title">Good</div>
<div class="content monospaced">
<pre>#!/bin/tcsh -f
...</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_config_file_processing">Config File Processing</h4>
<div class="paragraph"><p>As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
an xterm in the context being examined.</p></div>
<div class="paragraph"><p>For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:</p></div>
<div class="listingblock">
<div class="title">Original items table</div>
<div class="content monospaced">
<pre>[items]
CELLNAME [system getcellname.sh]</pre>
</div></div>
<div class="listingblock">
<div class="title">Items table modified for debug</div>
<div class="content monospaced">
<pre>[items]
DEBUG [system xterm]
CELLNAME [system getcellnames.sh]</pre>
</div></div>
<div class="paragraph"><p>When this test is run an xterm will pop up. In that xterm the
environment is exactly that in which the script "getcellnames.sh"
would run. You can now debug the script to find out why it isn&#8217;t
working as expected.</p></div>
<div class="paragraph"><p>Similarly in a script just call the xterm.
NOTE: This technique can be very helpful in debugging running of EDA tools in Perl, Ruby, Python or tcl scripts:</p></div>
<div class="literalblock">
<div class="title">Perl example</div>
<div class="content monospaced">
<pre>some_code();
$cmdline="virtuoso -some-switches and params ...";
print "$cmdline"; # print the command line so you can paste it into the xterm that pops up
system("xterm");  # this line is added for the debug and removed when done
system($cmdline);
more_code();</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_misc_other_debugging_hints">Misc Other Debugging Hints</h3>
<div class="sect3">
<h4 id="_annotating_scripts_and_config_files">Annotating scripts and config files</h4>
<div class="paragraph"><p>Run the "env" command to record the environment:</p></div>
<div class="paragraph"><p>env | sort &gt; stagename.log</p></div>
<div class="paragraph"><p>In a config file:</p></div>
<div class="literalblock">
<div class="title">megatest.config, runconfigs.config and testconfig</div>
<div class="content monospaced">
<pre>#{shell env | sort &gt; stagename.log}

# or

[system env | sort &gt; stagename.log]</pre>
</div></div>
<div class="paragraph"><p>In scripts just insert the commands, this example helps you identify
if "some commands &#8230;" changed any environment variables.:</p></div>
<div class="literalblock">
<div class="title">myscript.sh</div>
<div class="content monospaced">
<pre>env | sort &gt; somefile-before.log
some commands ...
env | sort &gt; somefile-after.log</pre>
</div></div>
<div class="literalblock">
<div class="title">Use meld to examine the differences</div>
<div class="content monospaced">
<pre>meld somefile-before.log somefile-after.log</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_oneshot_modifying_a_variable">Oneshot Modifying a Variable</h4>
<div class="paragraph"><p>To try various values for a variable without mutating the current value</p></div>
<div class="literalblock">
<div class="title">within a bash shell</div>
<div class="content monospaced">
<pre>SOMEVAR=123 runcmd.sh</pre>
</div></div>
<div class="literalblock">
<div class="title">within csh</div>
<div class="content monospaced">
<pre>(setenv SOMEVAR 123;runcmd.sh)

# OR

env SOMEVAR=123 runcmd.sh</pre>
</div></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_how_to_do_things">How To Do Things</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_process_runs">Process Runs</h3>
<div class="sect3">




︙


2034
2035
2036
2037
2038
2039
2040
2041

2042
2043
2044


2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106

2324
2325
2326
2327
2328
2329
2330

2331
2332


2333
2334
2335
2336
2337
2338
2339
2340
2341
2342
2343
2344
2345
2346
2347
2348
2349
2350







































2351
2352
2353
2354
2355
2356
2357








-
+

-
-
+
+
















-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-








# match.
flexi-launcher yes</pre>
</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_tricks">Tricks</h2>
<h2 id="_tricks_and_tips">Tricks and Tips</h2>
<div class="sectionbody">
<div class="paragraph"><p>This section is a compendium of a various useful tricks for debugging,
configuring and generally getting the most out of Megatest.</p></div>
<div class="paragraph"><p>This section is a collection of a various useful tricks for that
didn&#8217;t quite fit elsewhere.</p></div>
<div class="sect2">
<h3 id="_limiting_your_running_jobs">Limiting your running jobs</h3>
<div class="paragraph"><p>The following example will limit a test in the jobgroup "group1" to no more than 10 tests simultaneously.</p></div>
<div class="paragraph"><p>In your testconfig:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[test_meta]
jobgroup group1</pre>
</div></div>
<div class="paragraph"><p>In your megatest.config:</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[jobgroups]
group1 10
custdes 4</pre>
</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_debugging_tricks">Debugging Tricks</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_examining_the_environment">Examining The Environment</h3>
<div class="sect3">
<h4 id="_test_control_panel_xterm">Test Control Panel - xterm</h4>
<div class="paragraph"><p>From the dashboard click on a test PASS/FAIL button. This brings up a test control panel. Aproximately near the center left of the
window there is a button "Start Xterm". Push this to get an xterm with the full context and environment loaded for that test. You can run
scripts or ezsteps by copying from the testconfig (hint, load up the testconfig in a separate gvim or emacs window). This is the easiest way
to debug your tests.</p></div>
</div>
<div class="sect3">
<h4 id="_during_config_file_processing">During Config File Processing</h4>
<div class="paragraph"><p>It is often helpful to know the content of variables in various
contexts as Megatest does the actions needed to run your tests. A handy technique is to force the startup of an xterm in the context being examined.</p></div>
<div class="paragraph"><p>For example, if an item list is not being generated as expected you
can inject the startup of an xterm as if it were an item:</p></div>
<div class="listingblock">
<div class="title">Original items table</div>
<div class="content monospaced">
<pre>[items]
CELLNAME [system getcellname.sh]</pre>
</div></div>
<div class="listingblock">
<div class="title">Items table modified for debug</div>
<div class="content monospaced">
<pre>[items]
DEBUG [system xterm]
CELLNAME [system getcellnames.sh]</pre>
</div></div>
<div class="paragraph"><p>When this test is run an xterm will pop up. In that xterm the
environment is exactly that in which the script "getcellnames.sh"
would run. You can now debug the script to find out why it isn&#8217;t
working as expected.</p></div>
</div>
<div class="sect3">
<h4 id="_organising_your_tests_and_tasks">Organising Your Tests and Tasks</h4>
<div class="paragraph"><p>The default location "tests" for storing tests can be extended by
adding to your tests-paths section.</p></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[misc]




︙


2146
2147
2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159
2160

2397
2398
2399
2400
2401
2402
2403

2404
2405
2406
2407
2408
2409
2410
2411








-
+








<div class="sectionbody">
<div class="sect2">
<h3 id="_megatest_use_modes">Megatest Use Modes</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 2. Base commands</caption>
<caption class="title">Table 3. Base commands</caption>
<col style="width:20%;">
<col style="width:40%;">
<col style="width:40%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Use case                </th>
<th class="tableblock halign-left valign-top" > Megatest command     </th>




︙


2208
2209
2210
2211
2212
2213
2214
2215

2216
2217
2218
2219
2220
2221
2222

2459
2460
2461
2462
2463
2464
2465

2466
2467
2468
2469
2470
2471
2472
2473








-
+








<div class="sect2">
<h3 id="_config_file_helpers">Config File Helpers</h3>
<div class="paragraph"><p>Various helpers for more advanced config files.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 3. Helpers</caption>
<caption class="title">Table 4. Helpers</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Helper                      </th>




︙


2493
2494
2495
2496
2497
2498
2499
2500

2501
2502
2503
2504
2505
2506
2507

2744
2745
2746
2747
2748
2749
2750

2751
2752
2753
2754
2755
2756
2757
2758








-
+








</div>
<div class="sect2">
<h3 id="_database_settings">Database settings</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 4. Database config settings in [setup] section of megatest.config</caption>
<caption class="title">Table 5. Database config settings in [setup] section of megatest.config</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Var                       </th>




︙


3006
3007
3008
3009
3010
3011
3012




3013
3014
3015
3016
3017
3018
3019

3257
3258
3259
3260
3261
3262
3263
3264
3265
3266
3267
3268
3269
3270
3271
3272
3273
3274








+
+
+
+









[logpro]
lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line
  ;;     a blank line indicates the end of the block of text
  (expect:required in "LogFileBody" &gt; 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/)</pre>
</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
# if your upstream file is csh you can force csh like this
# if your upstream is bash
loadenv     source $REF/ourenviron.sh

<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
</div>
<div class="sect2">




︙


3158
3159
3160
3161
3162
3163
3164
3165

3166
3167
3168
3169
3170
3171
3172

3413
3414
3415
3416
3417
3418
3419

3420
3421
3422
3423
3424
3425
3426
3427








-
+








but since triggers can be called in various contexts not all variables are
available at all times. The trigger script should check for the variable and
fail gracefully if it doesn&#8217;t exist.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:90%;
">
<caption class="title">Table 5. Environment variables visible to the trigger script</caption>
<caption class="title">Table 6. Environment variables visible to the trigger script</caption>
<col style="width:50%;">
<col style="width:50%;">
<thead>
<tr>
<th class="tableblock halign-left valign-top" > Variable            </th>
<th class="tableblock halign-left valign-top" > Purpose</th>
</tr>




︙


3392
3393
3394
3395
3396
3397
3398
3399

3400
3401
3402
3403
3404
3405
3406

3647
3648
3649
3650
3651
3652
3653

3654
3655
3656
3657
3658
3659
3660
3661








-
+








<h2 id="_programming_api">Programming API</h2>
<div class="sectionbody">
<div class="paragraph"><p>These routines can be called from the megatest repl.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 6. API Keys Related Calls</caption>
<caption class="title">Table 7. API Keys Related Calls</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >API Call                        </th>




︙







Added docs/manual/megatest_manual.pdf version [8fe19d594b].

cannot compute difference between binary files




Modified docs/manual/megatest_manual.txt
from [cb5cc67576]
to [5b54398387].




︙


31
32
33
34
35
36
37
38
39
40
41
42
43






44
45
46

47
48
49
50
51
52
53
54

31
32
33
34
35
36
37






38
39
40
41
42
43



44

45
46
47
48
49
50
51








-
-
-
-
-
-
+
+
+
+
+
+
-
-
-
+
-








    along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

----------------------------

Why Megatest?
-------------

The Megatest project was started for two reasons, the first was an
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years. I
thought a single open source tool, flexible enough to meet the needs
of any team doing continuous integration and or running a complex
Megatest was created to provide a generalized tool for managing suites
of regression tests and to provide a multi-host, distributed
alternative to "make". The EDA world is littered with proprietory,
company-specific tools for this purpose and by going open source and
keeping the tool flexible the hope is that Megatest could be useful to
any team at any company for continuous integration and almost any
suite of tests for release qualification would solve some problems for
me and for others.

other general automation tasks.
 -- Matt Welland, original author of the Megatest tool suite.

Megatest Design Philosophy
--------------------------

Megatest is a distributed system intended to provide the minimum needed
resources to make writing a suite of tests and tasks for implementing
continuous build for software, design engineering or process control (via




︙


96
97
98
99
100
101
102


103
104
105
106
107

108
109
110
111
112
113
114

115
116

117
118
119
120
121
122
123

93
94
95
96
97
98
99
100
101
102
103
104
105

106
107
108
109
110
111
112

113
114

115
116
117
118
119
120
121
122








+
+




-
+






-
+

-
+








Tests are launched using the launching system available for the
distributed compute platform in use. A template script is provided
which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database. Megatest has been used with the Intel Netbatch and
lsf (also known as openlava) batch systems and it should be
straightforward to use it with other similar systems.

// :leveloffset: 0

include::overview.txt[]

include::plan.txt[]

 
include::installation.txt[]

include::getting_started.txt[]

include::study_plan.txt[]

// :leveloffset: 0
include::writing_tests.txt[]

include::writing_tests.txt[]
include::debugging.txt[]

include::howto.txt[]

include::reference.txt[]

include::testplan.txt[]





︙







Modified docs/manual/plan.txt
from [84407a87ee]
to [5f519e210f].



1
2
3
4
5
6
7
8
9
10




1
2
3
4
5
6
7

-
-
-








Road Map
--------

// This file is part of Megatest.
// 
//     Megatest is free software: you can redistribute it and/or modify
//     it under the terms of the GNU General Public License as published by
//     the Free Software Foundation, either version 3 of the License, or
//     (at your option) any later version.
// 




︙







Modified docs/manual/reference.txt
from [6aa04b6eea]
to [7adc3533c3].




︙


203
204
205
206
207
208
209

















210
211
212
213
214
215
216

203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+









-----------------
[setup]
# this will automatically kill the test if it runs for more than 1h 2m and 3s
runtimelim 1h 2m 3s
-----------------

Post Run Hook
+++++++++++++

This runs script to-run.sh after all tests have been completed. It is
not necessary to use -run-wait as each test will check for other
running tests on completion and if there are none it will call the
post run hook.

Note that the output from the script call will be placed in a log file
in the logs directory with a file name derived by replacing / with _
in post-hook-<target>-<runname>.log.

-------------------
[runs]
post-hook /path/to/script/to-run.sh
-------------------

Tests browser view
~~~~~~~~~~~~~~~~~~

The tests browser (see the Run Control tab on the dashboard) has two views for displaying the tests. 

. Dot (graphviz) based tree
. No dot, plain listing




︙


718
719
720
721
722
723
724




725
726
727
728
729
730
731

735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752








+
+
+
+








lookittmp ;; Note: config file format supports multi-line entries where leading whitespace is removed from each line
  ;;     a blank line indicates the end of the block of text 
  (expect:required in "LogFileBody" > 0 "A file name that should never exist!" #/This is a awfully stupid file name that should never be found in the temp dir/)

-----------------

To transfer the environment to the next step you can do the following:
# if your upstream file is csh you can force csh like this
# if your upstream is bash 
loadenv     source $REF/ourenviron.sh


.Propagate environment to next step
----------------------------
$MT_MEGATEST -env2file .ezsteps/${stepname}	  
----------------------------

Scripts




︙







Added docs/megatest-debug-tutorial.odp version [bfd2483cd2].

cannot compute difference between binary files




Modified http-transport.scm
from [2202b22e9f]
to [1dd5b55f0c].




︙


38
39
40
41
42
43
44
45

46
47
48
49
50
51
52

38
39
40
41
42
43
44

45
46
47
48
49
50
51
52








-
+








(declare (uses portlogger))
(declare (uses rmt))

(include "common_records.scm")
(include "db_records.scm")
(include "js-path.scm")

(require-library stml)
(use stml2)
(define (http-transport:make-server-url hostport)
  (if (not hostport)
      #f
      (conc "http://" (car hostport) ":" (cadr hostport))))

(define *server-loop-heart-beat* (current-seconds))





︙







Modified mtut.scm
from [413cf26858]
to [cc88551c71].




︙


33
34
35
36
37
38
39
40

41
42
43
44
45
46
47

33
34
35
36
37
38
39

40
41
42
43
44
45
46
47








-
+








(declare (uses configf))
;; (declare (uses rmt))

(use ducttape-lib)

(include "megatest-fossil-hash.scm")

(require-library stml)
(use stml2)

;; stuff for the mapper and checker functions
;;
(define *target-mappers*  (make-hash-table)) 
(define *runname-mappers* (make-hash-table)) 
(define *area-checkers*   (make-hash-table)) 





︙







Modified tests.scm
from [718cc2994f]
to [4f9f2f84f7].




︙


32
33
34
35
36
37
38
39

40
41
42
43
44
45
46

32
33
34
35
36
37
38

39
40
41
42
43
44
45
46








-
+








(declare (uses runconfig))
;; (declare (uses sdb))
(declare (uses server))
;;(declare (uses stml2))

(use sqlite3 srfi-1 posix regex regex-case srfi-69 dot-locking tcp directory-utils)
(import (prefix sqlite3 sqlite3:))
(require-library stml)
(use stml2)

(include "common_records.scm")
(include "key_records.scm")
(include "db_records.scm")
(include "run_records.scm")
(include "test_records.scm")
(include "js-path.scm")




︙







Modified tests/mintest/megatest.config
from [bfa94e44af]
to [444a72500f].




︙


17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

32
33
34

17
18
19
20
21
22
23

24
25
26
27
28
29

30
31
32
33








-






-
+





[fields]
X TEXT

[setup]
max_concurrent_jobs 50
linktree #{getenv MT_RUN_AREA_HOME}/linktree
transport http

[server]
port 8090

[jobtools]
useshell yes
launcher nbfind
launcher nbfake

[disks]
disk0 #{getenv PWD}/runs








Modified tests/mintest/tests/a/testconfig
from [7f9803b9fe]
to [61be51e47c].




︙


14
15
16
17
18
19
20







21
22
23

14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30








+
+
+
+
+
+
+




# 
#     You should have received a copy of the GNU General Public License
#     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

# Add steps here. Format is "stepname script"
[ezsteps]
step1 echo SUCCESS
# step1 ((xterm&);echo SUCCESS)
# step2 xterm_and_success

[scripts]
xterm_and_success #!/bin/bash
  xterm -T step2 &
  echo SUCCESS

[requirements]
waiton b













This page was generated in about
0.141s by
Fossil 2.24 [452f8e1ffc] 2024-03-26 15:36:00