Megatest

Check-in [33cf143a50]
Login
☰Home Timeline Files Branches Tags Tickets Wiki Login
Overview
Comment:Docs update
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | v1.65 | v1.6514
Files: files | file ages | folders
SHA1: 33cf143a502e8422991ffd33794c4344e3cfd733
User & Date: mrwellan on 2018-08-24 15:57:06
Other Links: branch diff | manifest | tags
Context
2018-08-27
16:48
Updated version file and made it so that cleanup-db will get rid of old test_dat items check-in: 566569e56f user: jmoon18 tags: v1.65, v1.6514
2018-08-24
15:57
Docs update check-in: 33cf143a50 user: mrwellan tags: v1.65, v1.6514
15:07
Fix cleanup-db to retain DELETED records for some time. default 30d. configurable via key delete-record-age in setup section check-in: 2a51bc4669 user: pjhatwal tags: v1.65
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Modified docs/manual/getting_started.in from [14c83ee5dd] to [7f26703057].

︙
8
9
10
11
12
13
14


15
16
17
18
19
20
21

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23








+
+








//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.
//
// Copyright 2006-2012, Matthew Welland.

Getting Started
---------------

[partintro]
.Getting started with Megatest
--
︙

Modified docs/manual/howto.in from [71fef170bb] to [b66065dad3].

︙
8
9
10
11
12
13
14


15
16
17
18
19
20
21

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23








+
+








//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.
//
// Copyright 2006-2012, Matthew Welland.

How To Do Things
----------------

Process Runs
~~~~~~~~~~~~
︙

Modified docs/manual/megatest_manual.html from [af6b682840] to [9d96457d70].

︙
885
886
887
888
889
890
891





















































































892
893
894
895
896
897
898

885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








sqlite3 database.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_road_map">Road Map</h2>
<div class="sectionbody">
<div class="paragraph"><p>Note 1: This road-map continues to evolve and subject to change without notice.</p></div>
<div class="sect2">
<h3 id="_here_is_a_smattering_of_ideas_for_megatest_2_0">Here is a smattering of ideas for Megatest 2.0</h3>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME
</p>
</li>
</ol></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[db]
api legacy|new</pre>
</div></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair.
</p>
</li>
<li>
<p>
Rigorous megatest.config and runconfig.config caching.
</p>
<div class="olist loweralpha"><ol class="loweralpha">
<li>
<p>
Cache the configs in $MT_RUNPATH
</p>
</li>
<li>
<p>
Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed
</p>
</li>
</ol></div>
</li>
<li>
<p>
If the cached file changes ALL existing tests go from COMPLETED &#8594; STALE, I’m not sure what to do about RUNNING tests
</p>
</li>
<li>
<p>
!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.
</p>
</li>
<li>
<p>
Per test copy commands (crude example below is not correct).
</p>
</li>
</ol></div>
<div class="listingblock">
<div class="content monospaced">
<pre>[testcopy]
%/iind% unison SRC DEST
% cp –r SRC DEST</pre>
</div></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Test management via pkts (optional?)
</p>
<div class="olist loweralpha"><ol class="loweralpha">
<li>
<p>
Control pkt types: run, kill, rerunclean, clean, archive, status?
</p>
</li>
<li>
<p>
Status pkt types: ack, step, status_change
</p>
</li>
</ol></div>
</li>
<li>
<p>
Add nanomsg as a transport option
.
</p>
</li>
</ol></div>
</div>
<div class="sect2">
<h3 id="_rfc_m01_add_ability_to_move_runs_to_other_areas">RFC M01: Add ability to move runs to other Areas</h3>
<div class="paragraph"><p><strong>Purpose</strong>: allow shrinking megatest.db data by moving runs to an alternate
 Megatest area with same keys.</p></div>
<div class="paragraph"><p><strong>Method</strong>: extend db sync to take a different megatest area as a destination.</p></div>
<div class="paragraph"><p><strong>Design</strong>:</p></div>
<div class="olist arabic"><ol class="arabic">
︙
1496
1497
1498
1499
1500
1501
1502
1503





























































1504
1505
1506
1507
1508
1509
1510

1511
1512
1513
1514
1515
1516
1517

1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655

1656
1657
1658
1659
1660
1661
1662
1663









+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+






-
+








</div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_reference">Reference</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_megatest_use_modes">Megatest Use Modes</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 2. Base commands</caption>
<col style="width:20%;">
<col style="width:40%;">
<col style="width:40%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Use case                </th>
<th class="tableblock halign-left valign-top" > Megatest command     </th>
<th class="tableblock halign-left valign-top" >  mtutil</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">Start from scratch</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-rerun-all</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">restart</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">Rerun non-good completed</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-rerun-clean</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">rerunclean</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">Rerun all non-good and not completed yet</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-set-state-status KILLREQ; -rerun-</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">clean</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">killrerun</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Continue run</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-run</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">resume</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Remove run</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-remove-runs</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">clean</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Lock run</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-lock</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">lock</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">Unlock run</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-unlock</p></td>
</tr>
<tr>
<td class="tableblock halign-center valign-top" ><p class="tableblock">unlock</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">killrun</p></td>
<td class="tableblock halign-left valign-top" ><p class="tableblock monospaced">-set-state-status KILLREQ; -kill-run</p></td>
</tr>
</tbody>
</table>
</div>
<div class="sect2">
<h3 id="_config_file_helpers">Config File Helpers</h3>
<div class="paragraph"><p>Various helpers for more advanced config files.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:80%;
">
<caption class="title">Table 2. Helpers</caption>
<caption class="title">Table 3. Helpers</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Helper                      </th>
︙
1776
1777
1778
1779
1780
1781
1782
1783

1784
1785
1786
1787
1788
1789
1790

1922
1923
1924
1925
1926
1927
1928

1929
1930
1931
1932
1933
1934
1935
1936








-
+








</div>
<div class="sect2">
<h3 id="_database_settings">Database settings</h3>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 3. Database config settings in [setup] section of megatest.config</caption>
<caption class="title">Table 4. Database config settings in [setup] section of megatest.config</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Var                       </th>
︙
2064
2065
2066
2067
2068
2069
2070




2071
2072
2073
2074
2075
2076
2077

2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227








+
+
+
+








<li>
<p>
Test A has no waitons.  All waitons of all tests in full list have been processed.  Full list is finalized.
</p>
</li>
</ol></div>
</div>
<div class="sect3">
<h4 id="_itemstable">itemstable</h4>
<div class="paragraph"><p>An alternative to defining items is the itemstable section.  This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components.</p></div>
</div>
<div class="sect3">
<h4 id="_dynamic_flow_dependency_tree">Dynamic Flow Dependency Tree</h4>
<div class="listingblock">
<div class="title">Autogeneration waiton list for dynamic flow dependency trees</div>
<div class="content monospaced">
<pre>[requirements]
# With a toplevel test you may wish to generate your list
︙
2215
2216
2217
2218
2219
2220
2221
2222

2223
2224
2225
2226
2227
2228
2229

2365
2366
2367
2368
2369
2370
2371

2372
2373
2374
2375
2376
2377
2378
2379








-
+








but since triggers can be called in various contexts not all variables are
available at all times. The trigger script should check for the variable and
fail gracefully if it doesn&#8217;t exist.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:90%;
">
<caption class="title">Table 4. Environment variables visible to the trigger script</caption>
<caption class="title">Table 5. Environment variables visible to the trigger script</caption>
<col style="width:33%;">
<col style="width:66%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >Variable             </th>
<th class="tableblock halign-left valign-top" > Purpose</th>
</tr>
︙
2413
2414
2415
2416
2417
2418
2419
2420

2421
2422
2423
2424
2425
2426
2427

2563
2564
2565
2566
2567
2568
2569

2570
2571
2572
2573
2574
2575
2576
2577








-
+








# passfail auto|logpro
# Example of logpro:
passfail logpro

# Optional:
logpro ;; if this section exists then logpro is used to determine pass/fail
  (expect:required in "LogFileBody" &gt;= 1 "At least one pass" #/PASS/)
  (expect:fail     in "LogFileBody"  = 0 "No FAILs allowed"  #/FAIL/)
  (expect:error     in "LogFileBody"  = 0 "No FAILs allowed"  #/FAIL/)

# Optional: target translator, default is to use the parent target
target #{shell somescript.sh}

# Optional: runname translator/generator, default is to use the parent runname
run-name #{somescript.sh}
︙
2448
2449
2450
2451
2452
2453
2454
2455

2456
2457
2458
2459
2460
2461
2462

2598
2599
2600
2601
2602
2603
2604

2605
2606
2607
2608
2609
2610
2611
2612








-
+








<h2 id="_programming_api">Programming API</h2>
<div class="sectionbody">
<div class="paragraph"><p>These routines can be called from the megatest repl.</p></div>
<table class="tableblock frame-topbot grid-all"
style="
width:70%;
">
<caption class="title">Table 5. API Keys Related Calls</caption>
<caption class="title">Table 6. API Keys Related Calls</caption>
<col style="width:14%;">
<col style="width:28%;">
<col style="width:28%;">
<col style="width:28%;">
<thead>
<tr>
<th class="tableblock halign-center valign-top" >API Call                        </th>
︙
2500
2501
2502
2503
2504
2505
2506
2507

2508
2509
2510
2511

2650
2651
2652
2653
2654
2655
2656

2657
2658
2659
2660
2661








-
+





</div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Version 1.0<br>
Last updated 2018-04-02 10:30:50 PDT
Last updated 2018-08-01 09:52:04 PDT
</div>
</div>
</body>
</html>

Modified docs/manual/plan.in from [5f93853f71] to [660a99057c].

︙
11
12
13
14
15
16
17
18


19


























20
21
22
23
24
25
26

11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54









+
+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








//     Megatest is distributed in the hope that it will be useful,
//     but WITHOUT ANY WARRANTY; without even the implied warranty of
//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
//     GNU General Public License for more details.
// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

// Copyright 2006-2012, Matthew Welland.

Note 1: This road-map continues to evolve and subject to change without notice.

Here is a smattering of ideas for Megatest 2.0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME
----------------
[db]
api legacy|new
----------------
. One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair.
.   Rigorous megatest.config and runconfig.config caching.
..  Cache the configs in $MT_RUNPATH
..  Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed
. If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests
. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.
. Per test copy commands (crude example below is not correct).
----------------
[testcopy]
%/iind% unison SRC DEST
% cp –r SRC DEST
----------------
.	Test management via pkts (optional?)
..	Control pkt types: run, kill, rerunclean, clean, archive, status?
..	Status pkt types: ack, step, status_change
.	Add nanomsg as a transport option
.	

RFC M01: Add ability to move runs to other Areas
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*Purpose*: allow shrinking megatest.db data by moving runs to an alternate
 Megatest area with same keys.
︙

Modified docs/manual/reference.in from [68f4b70afa] to [fdffcfb115].

︙
12
13
14
15
16
17
18

















19
20
21
22
23
24
25

12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








// 
//     You should have received a copy of the GNU General Public License
//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

Reference
---------

Megatest Use Modes
~~~~~~~~~~~~~~~~~~

.Base commands
[width="80%",cols="^,2m,2m",frame="topbot",options="header"]
|======================
|Use case	         | Megatest command	|  mtutil
|Start from scratch       |	-rerun-all	|  restart	
|Rerun non-good completed |	-rerun-clean	|  rerunclean	
|Rerun all non-good and not completed yet | -set-state-status KILLREQ; -rerun-|clean | killrerun	
|Continue run             |	-run	        |  resume	
|Remove run	         |      -remove-runs	|  clean	
|Lock run	         |      -lock	        |  lock	
|Unlock run	         |      -unlock	        |  unlock	
|killrun	                 | -set-state-status KILLREQ; -kill-run	| killrun	
|======================

Config File Helpers
~~~~~~~~~~~~~~~~~~~

Various helpers for more advanced config files.

.Helpers
[width="80%",cols="^,2m,2m,2m",frame="topbot",options="header"]
︙
469
470
471
472
473
474
475









476
477
478
479
480
481
482

486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508








+
+
+
+
+
+
+
+
+








. Full list to be run is now: +D/1/res+
. Test D has a waiton - test C.  Test D's itemmap rule +itemmap (\d&plus;)/res \1/aa+ ->  causes +C/1/aa+ to run before +D/1/res+
. Full list to be run is now: +D/1/res+, +C/1/aa+
. Test C was a waiton - test A. Test C's rule +A (\d&plus;)/aa aa/\1+ -> causes +A/aa/1+ to run before +C/1/aa+
. Full list to be run is now: +D/1/res+, +C/1/aa+, +A/aa/1+
. Test A has no waitons.  All waitons of all tests in full list have been processed.  Full list is finalized.



itemstable
^^^^^^^^^^
An alternative to defining items is the itemstable section.  This lets you define the itempath in a table format rather than specifying components and relying on getting all permutations of those components.





Dynamic Flow Dependency Tree
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.Autogeneration waiton list for dynamic flow dependency trees
-------------------
[requirements]
︙
765
766
767
768
769
770
771
772

773
774
775
776
777
778
779

791
792
793
794
795
796
797

798
799
800
801
802
803
804
805








-
+








# passfail auto|logpro
# Example of logpro:
passfail logpro

# Optional: 
logpro ;; if this section exists then logpro is used to determine pass/fail
  (expect:required in "LogFileBody" >= 1 "At least one pass" #/PASS/)
  (expect:fail     in "LogFileBody"  = 0 "No FAILs allowed"  #/FAIL/)
  (expect:error     in "LogFileBody"  = 0 "No FAILs allowed"  #/FAIL/)

# Optional: target translator, default is to use the parent target
target #{shell somescript.sh}

# Optional: runname translator/generator, default is to use the parent runname
run-name #{somescript.sh}
︙

This page was generated in about 0.126s by Fossil 2.24 [452f8e1ffc] 2024-03-26 15:36:00