Megatest

Check-in [43b9e9cf05]
Login
Overview
Comment:Updates to debugging section of manual
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | v1.65-debugging-update-orig
Files: files | file ages | folders
SHA1: 43b9e9cf0579b059b728f0ad2a8e19f943c7f638
User & Date: mrwellan on 2021-09-10 11:41:21
Other Links: branch diff | manifest | tags
Context
2021-09-10
12:54
More tweaks to debugging section check-in: 44a6aef391 user: mrwellan tags: v1.65-debugging-update-orig
11:41
Updates to debugging section of manual check-in: 43b9e9cf05 user: mrwellan tags: v1.65-debugging-update-orig
10:20
stuff check-in: 392f89b8e5 user: mrwellan tags: v1.65-debugging-update-orig
Changes

Modified docs/manual/debugging.txt from [9d2c5b9df9] to [72750bdc63].

17
18
19
20
21
22
23
24
25


26
27
28



29

30



31
32
33
34




35





36
37


38


39

40
41

42
43



44
45
46
47
48
49





50
51
52
53

54
55





56
57





58
59
60
61
62
63






64
65
66
67
68
69
70

71
72
73
74
75
76













77
78
79
80
81
82
83
17
18
19
20
21
22
23


24
25
26


27
28
29
30
31
32
33
34
35




36
37
38
39
40
41
42
43
44
45


46
47
48
49
50

51


52


53
54
55
56





57
58
59
60
61
62



63


64
65
66
67
68
69

70
71
72
73
74
75





76
77
78
79
80
81
82






83






84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103







-
-
+
+

-
-
+
+
+

+

+
+
+
-
-
-
-
+
+
+
+

+
+
+
+
+
-
-
+
+

+
+
-
+
-
-
+
-
-
+
+
+

-
-
-
-
-
+
+
+
+
+

-
-
-
+
-
-
+
+
+
+
+

-
+
+
+
+
+

-
-
-
-
-
+
+
+
+
+
+

-
-
-
-
-
-
+
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+








Debugging
---------

Well Written Tests
~~~~~~~~~~~~~~~~~~

Test Design and Surfacing
^^^^^^^^^^^^^^^^^^^^^^^^^
Test Design and Surfacing Errors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Design your tests to surface errors. Ensure that all logs are processed by logpro and can be reached by a mouse click or two from the test control panel.

Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.

To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:

.script1.sh
..............................
#!/bin/bash
using the tool to make life easier

layers - less is often more

code ...
script2.sh some parameters > script2.log
more code ...
..............................

.script2.sh
..............................
#!/bin/bash
code ...
script3.sh some more parameters > script3.log
Clicks-to-error

more code ...
..............................

.script3.sh
..............................
A word on Bisecting
#!/bin/bash
~~~~~~~~~~~~~~~~~~~

code ...
Bisecting is a debug strategy intended to speed up finding the root
cause.
virtuoso params and switches ...
more code ...
..............................

.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------
The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:

It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
.testconfig
is similar to the feature in source control tools like git and fossil
called biseceting.
..............................
[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log

To know where to look for problems you will want to look at the log files at various stages in the execution process.
[logpro]
step1 ;; some logpro rules
  (expect:required in "LogFileBody" > 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh

.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------
[scripts]
script1.sh #!/bin/bash
 code ...

...
..............................

.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage	                    | How to inspect                               | Watch for
|A: post config processing  | megatest -show-config -target your/target	   | #f (failed var processing)
|B: post runconfig          | megatest -show-runconfig -target your/target | 
With the above testconfig the logs for every critical part of the
|C: processing testconfigs  | inspect output from "megatest -run ..."      | Messages indicating issues process configs, dependency problems.
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test        | start test xterm, look at mt_launch.log      | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test           | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution   | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule! 
|======================
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.

An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.

The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.

Examining The Test Logs and Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Test Control Panel - xterm
^^^^^^^^^^^^^^^^^^^^^^^^^^

94
95
96
97
98
99
100
























































101
102
103
104


105
106


107
108


109
110
111
112
113


114





115







116
117
118
119
120
121
122
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179

180
181
182

183
184
185

186
187
188
189
190
191

192
193
194
195
196
197
198
199

200
201
202
203
204
205
206
207
208
209
210
211
212
213







+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+



-
+
+

-
+
+

-
+
+




-
+
+

+
+
+
+
+
-
+
+
+
+
+
+
+







that step and downstream steps.

NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.

NOTE 2: steping through only works if you are using ezsteps.

A word on Bisecting
~~~~~~~~~~~~~~~~~~~

Bisecting is a debug strategy intended to speed up finding the root
cause of some bug.

.A complex process with a problem found in stage "E"
["graphviz", "bisecting.png"]
----------------------------------------------------------------------
include::bisecting.dot[]
----------------------------------------------------------------------

It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.

Tough Bugs
~~~~~~~~~~

Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.

Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.

To isolate bugs like this you may need to look at the log files at
various stages in the execution process of your run and tests.

.A simplified diagram of the stages Megatest goes through to run a test.
["graphviz", "megatest-test-stages.png"]
----------------------------------------------------------------------
include::megatest-test-stages.dot[]
----------------------------------------------------------------------

.How to check variable values and inspect logs at each stage
[width="80%",cols="<,2m,2m",frame="topbot",options="header"]
|======================
|Stage	                    | How to inspect                               | Watch for
|A: post config processing  | megatest -show-config -target your/target	   | #f (failed var processing)
|B: post runconfig          | megatest -show-runconfig -target your/target | 
|C: processing testconfigs  | inspect output from "megatest -run ..."      | Messages indicating issues process configs, dependency problems.
|D: process testconfig for test launch | inspect output from megatest runner | Zero items (items expansion yielded no items)
|E,F: launching test        | start test xterm, look at mt_launch.log      | Did your batch system accept the job? Has the job landed on a machine?
|G: starting test           | look at your batch systems logs for the process | Did the megatest -execute process start and run?
|H,H1,H2: step exectution   | look at <stepname>.log, <stepname>.html and your own internal logs | Do you have sufficiently tight logpro rules? You must always have a "required" rule! 
|======================

Bisecting megatest.csh/sh
^^^^^^^^^^^^^^^^^^^^^^^^^

Sometimes finding the environment variable that is causing the problem can be very difficult. Bisection can be applied.
Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.

Edit the megatest.csh or megatest.sh file and comment out 50% per round, source in fresh xterm and run the test.
Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.

This idea can also be applied to your .cshrc, .bashrc, .aliases and other similar files.
This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.

csh and -f
^^^^^^^^^^

A common issue when tcsh or csh shells are used for scripting is to forget or choose to not use -f in your #! line
A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.

.Not good
..............................
#!/bin/tcsh
...
..............................
.

.Good
..............................
#!/bin/tcsh -f
...
..............................


Config File Processing
^^^^^^^^^^^^^^^^^^^^^^

As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of

Modified docs/manual/megatest_manual.html from [23e5f0702c] to [5adac2f51c].

1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947





















































































1948
1949
1950
1951
1952
1953

1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964















1965

1966
1967
1968
1969
1970
1971
1972
1936
1937
1938
1939
1940
1941
1942





1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032

2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065
2066
2067







-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+





-
+











+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+







</div>
<div class="sect1">
<h2 id="_debugging">Debugging</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_well_written_tests">Well Written Tests</h3>
<div class="sect3">
<h4 id="_test_design_and_surfacing">Test Design and Surfacing</h4>
<div class="paragraph"><p>Design your tests to surface errors. Ensure that all logs are processed by logpro and can be reached by a mouse click or two from the test control panel.</p></div>
<div class="paragraph"><p>using the tool to make life easier</p></div>
<div class="paragraph"><p>layers - less is often more</p></div>
<div class="paragraph"><p>Clicks-to-error</p></div>
<h4 id="_test_design_and_surfacing_errors">Test Design and Surfacing Errors</h4>
<div class="paragraph"><p>Design your tests to surface errors. Ensure that all logs are
processed by logpro (or a custom log processing tool) and can be
reached by a mouse click or two from the test control panel.</p></div>
<div class="paragraph"><p>To illustrate, here is a set of scripts with nested calls where script1.sh calls script2.sh which calls script3.sh which finally calls the Cadence EDA tool virtuoso:</p></div>
<div class="literalblock">
<div class="title">script1.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script2.sh some parameters &gt; script2.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script2.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
script3.sh some more parameters &gt; script3.log
more code ...</pre>
</div></div>
<div class="literalblock">
<div class="title">script3.sh</div>
<div class="content monospaced">
<pre>#!/bin/bash
code ...
virtuoso params and switches ...
more code ...</pre>
</div></div>
<div class="paragraph"><p>The log files script2.log, script3.log and the log output from
virtuoso are not accessible from the test control panel. It would be
much better for future users of your automation to use steps more
fully. One easy option would be to post process the logs in downstream
additional steps:</p></div>
<div class="literalblock">
<div class="title">testconfig</div>
<div class="content monospaced">
<pre>[ezsteps]
step1 script1.sh
step2 cat script2.log
step3 cat script3.log

[logpro]
step1 ;; some logpro rules
  (expect:required in "LogFileBody" &gt; 0 "Expect this output" #/something expected/)
step2 ;; some logpro rules for script2.sh
step3 ;; some logpro rules for script3.sh

[scripts]
script1.sh #!/bin/bash
 code ...

...</pre>
</div></div>
<div class="paragraph"><p>With the above testconfig the logs for every critical part of the
automation are fully surfaced and rules can be created to flag errors,
warnings, aborts and to ignore false errors. A user of your automation
will be able to see the important error with two mouse clicks from the
runs view.</p></div>
<div class="paragraph"><p>An even better would be to eliminate the nesting if possible. As a
general statement with layers - less is usually more. By flattening
the automation into a sequence of steps you can use the test control
panel to re-run a step with a single click or from the test xterm run
only the errant step from the command line.</p></div>
<div class="paragraph"><p>The message here is make debugging and maintenace easy for future
users (and yourself) by keeping clicks-to-error in mind.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_examining_the_test_logs_and_environment">Examining The Test Logs and Environment</h3>
<div class="sect3">
<h4 id="_test_control_panel_xterm">Test Control Panel - xterm</h4>
<div class="paragraph"><p>From the dashboard click on a test PASS/FAIL button. This brings up a
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).</p></div>
<div class="paragraph"><p>With more recent versions of Megatest you can step through your test
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.</p></div>
<div class="paragraph"><p>NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.</p></div>
<div class="paragraph"><p>NOTE 2: steping through only works if you are using ezsteps.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_a_word_on_bisecting">A word on Bisecting</h3>
<div class="paragraph"><p>Bisecting is a debug strategy intended to speed up finding the root
cause.</p></div>
cause of some bug.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="bisecting.png" alt="bisecting.png">
</div>
<div class="title">Figure 1. A complex process with a problem found in stage "E"</div>
</div>
<div class="paragraph"><p>It is common to start debugging where the problem was observed and
then work back. However by inspecting the output at stage "C" in the
example above you would potentially save a lot of debug effort, this
is similar to the feature in source control tools like git and fossil
called biseceting.</p></div>
</div>
<div class="sect2">
<h3 id="_tough_bugs">Tough Bugs</h3>
<div class="paragraph"><p>Most bugs in Megatest based automation will be in the scripts called
in your test steps and if you utilize the good design practice
described above should be fairly easy for you to reproduce, isolate
and find.</p></div>
<div class="paragraph"><p>Some bugs however will come from subtle and hard to detect
interactions between Megatest and your OS and Unix environment. This
includes things like constructed variables that are legal in one
context (e.g. tcsh) but illegal in another context (e.g. bash),
variables that come from your login scripts and access and permissions
issues (e.g. a script that silently fails due to no access to needed
data). Other bugs might be due to Megatest itself.</p></div>
<div class="paragraph"><p>To isolate bugs like this you may need to look at the log files at
<div class="paragraph"><p>To know where to look for problems you will want to look at the log files at various stages in the execution process.</p></div>
various stages in the execution process of your run and tests.</p></div>
<div class="imageblock graphviz">
<div class="content">
<img src="megatest-test-stages.png" alt="megatest-test-stages.png">
</div>
<div class="title">Figure 2. A simplified diagram of the stages Megatest goes through to run a test.</div>
</div>
<table class="tableblock frame-topbot grid-all"
2018
2019
2020
2021
2022
2023
2024








2025
2026

2027
2028




2029
2030


2031
2032
2033
2034
2035
2036




2037
2038
2039
2040
2041
2042





2043
2044
2045
2046
2047
2048
2049
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128

2129


2130
2131
2132
2133


2134
2135






2136
2137
2138
2139






2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151







+
+
+
+
+
+
+
+

-
+
-
-
+
+
+
+
-
-
+
+
-
-
-
-
-
-
+
+
+
+
-
-
-
-
-
-
+
+
+
+
+







<tr>
<td class="tableblock halign-left valign-top" ><p class="tableblock">H,H1,H2: step exectution</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">look at &lt;stepname&gt;.log, &lt;stepname&gt;.html and your own internal logs</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Do you have sufficiently tight logpro rules? You must always have a "required" rule!</p></td>
</tr>
</tbody>
</table>
<div class="sect3">
<h4 id="_bisecting_megatest_csh_sh">Bisecting megatest.csh/sh</h4>
<div class="paragraph"><p>Sometimes finding the environment variable that is causing the problem
can be very difficult. Bisection can be applied.</p></div>
<div class="paragraph"><p>Edit the megatest.csh or megatest.sh file and comment out 50% per
round, source in fresh xterm and run the test.</p></div>
<div class="paragraph"><p>This idea can also be applied to your .cshrc, .bashrc, .aliases and
other similar files.</p></div>
</div>
<div class="sect2">
<div class="sect3">
<h3 id="_examining_the_test_logs_and_environment">Examining The Test Logs and Environment</h3>
<div class="sect3">
<h4 id="_csh_and_f">csh and -f</h4>
<div class="paragraph"><p>A common issue when tcsh or csh shells are used for scripting is to
forget or choose to not use -f in your #! line.</p></div>
<div class="literalblock">
<h4 id="_test_control_panel_xterm">Test Control Panel - xterm</h4>
<div class="paragraph"><p>From the dashboard click on a test PASS/FAIL button. This brings up a
<div class="title">Not good</div>
<div class="content monospaced">
test control panel. Aproximately near the center left of the window
there is a button "Start Xterm". Push this to get an xterm with the
full context and environment loaded for that test. You can run scripts
or ezsteps by copying from the testconfig (hint, load up the
testconfig in a separate text editor window).</p></div>
<div class="paragraph"><p>With more recent versions of Megatest you can step through your test
<pre>#!/bin/tcsh
...</pre>
</div></div>
<div class="literalblock">
from the test control panel. Click on the cell labeled "rerun this
step" to only rerun the step or click on "restart from here" to rerun
that step and downstream steps.</p></div>
<div class="paragraph"><p>NOTE 1: visual feedback can take some time, give it a few seconds and
you will see the step change color to blue as it starts running.</p></div>
<div class="paragraph"><p>NOTE 2: steping through only works if you are using ezsteps.</p></div>
<div class="title">Good</div>
<div class="content monospaced">
<pre>#!/bin/tcsh -f
...</pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_config_file_processing">Config File Processing</h4>
<div class="paragraph"><p>As described above it is often helpful to know the content of
variables in various contexts as Megatest works through the actions
needed to run your tests. A handy technique is to force the startup of
an xterm in the context being examined.</p></div>
3144
3145
3146
3147
3148
3149
3150

3151



3152
3153
3154
3155
3156
3157
3158
3246
3247
3248
3249
3250
3251
3252
3253
3254
3255
3256
3257
3258
3259
3260
3261
3262
3263
3264







+

+
+
+







<pre>[setup]
ezpropvars sh</pre>
</div></div>
<div class="listingblock">
<div class="title">Write your ezsteps. The loadenv.csh step will use /bin/csh as its shell, other steps will use bash.</div>
<div class="content monospaced">
<pre>[ezsteps]
# if your upstream file is csh you can force csh like this
loadenv.csh source $REF/ourenviron.csh
# if your upstream is bash
loadenv     source $REF/ourenviron.sh

compile make
install make install</pre>
</div></div>
<div class="paragraph"><p>Bash and csh are supported. You can override the shell binary location
from the default /bin/bash and /bin/csh if needed.</p></div>
<div class="listingblock">
<div class="title">Turn on auto propagate for csh</div>

Modified docs/manual/megatest_manual.pdf from [774cf56d82] to [1cdc21f7dd].

cannot compute difference between binary files