Megatest

which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database. Megatest has been used with the Intel Netbatch and
lsf (also known as openlava) batch systems and it should be
straightforward to use it with other similar systems.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_overview">Overview</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_stand_alone_megatest_area">Stand-alone Megatest Area</h3>
<div class="paragraph"><p>A single, stand-alone, Megatest based testsuite or "area" is
sufficient for most validation, automation and build problems.</p></div>
<div class="imageblock">
<div class="content">
<img src="megatest-stand-alone-area.png" alt="Static">
</div>
</div>
<div class="paragraph"><p>Megatest is designed as a distributed or decoupled system. This means
you can run the areas stand-alone with no additional
infrastructure. I.e. there are no databases, web servers or other
centralized resources needed. However as your needs grow you can
integrate multiple areas into a bigger system.</p></div>
<div class="sect3">
<h4 id="_component_descriptions">Component Descriptions</h4>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Multi-area dashboard and xterm. A gui (the dashboard) is usually the
  best option for controlling and launching runs but all operations
  can also be done from the commandline. Note: The not yet released
  multi-area dashboard replaces the old dashboard for browsing and
  controlling runs but for managing a single area the old dashboard
  works very well.
</p>
</li>
<li>
<p>
Area/testsuite. This is your testsuite or automation definition and
  consists of the information in megatest.config, runconfigs.config
  and your testconfigs along with any custom scripting that can&#8217;t be
  done with the native Megatest features.
</p>
</li>
<li>
<p>
If your testsuite or build automation is too large to run on a
  single instance you can distribute your jobs into a compute server
  pool. The only current requirements are password-less ssh access and
  a network filesystem.
</p>
</li>
</ol></div>
</div>
</div>
<div class="sect2">
<h3 id="_full_system_architecture">Full System Architecture</h3>
<div class="imageblock">
<div class="content">
<img src="megatest-system-architecture.png" alt="Static">
</div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_installation">Installation</h2>
<div class="sectionbody">
<div class="sect2">
<h3 id="_dependencies">Dependencies</h3>
<div class="paragraph"><p>Chicken scheme and a number of "eggs" are required for building

</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
<div class="listingblock">
<div class="title">Full example with ezsteps, logpro rules, scripts etc.</div>
<div class="content monospaced">
<pre># You can include a common file
#
[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc]

# Use "var" for a scratch pad
#
[var]
dumpsql select * from data;
sepstr .....................................

# NOT IMPLEMENTED YET!
#
[ezsteps-addendum]
prescript something.sh
postscript something2.sh

# Add additional steps here. Format is "stepname script"
[ezsteps]
importdb loaddb prod.db prod.sql
dumpprod dumpdata prod.db "#{get var dumpsql}"
diff (echo "prod#{get var sepstr}test";diff --side-by-side \
     dumpprod.log reference.log ;echo DIFFDONE)

[scripts]
loaddb #!/bin/bash
  sqlite3 $1 &lt;&lt;EOF
  .mode tabs
  .import $2 data
  .q
  EOF

dumpdata #!/bin/bash
  sqlite3 $1 &lt;&lt;EOF
  .separator ,
  $2
  .q
  EOF

# Test requirements are specified here
[requirements]
waiton setup
priority 0

# Iteration for your test is controlled by the items section
# The complicated if is needed to allow processing of the config for the dashboard when there are no actual runs.
[items]
THINGNAME [system generatethings.sh | sort -u]

# Logpro rules for each step can be captured here in the testconfig
# note: The ;; after the stepname and the leading whitespace are required
#
[logpro]
inputdb ;;
  (expect:ignore   in "LogFileBody"  &lt; 99 "Ignore error in comments"      #/^\/\/.*error/)
  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
  (expect:required in "LogFileBody"  &gt; 0 "Some data found"                #/^[a-z]{3,4}[0-9]+_r.*/)

diff ;;
  (expect:ignore   in "LogFileBody"  &lt; 99 "Ignore error in comments"      #/^\/\/.*error/)
  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
  (expect:error    in "LogFileBody"  = 0 "&lt; or &gt; indicate missing entry"  (list #/(&lt;|&gt;)/   #/error/i))
  (expect:error    in "LogFileBody"  = 0 "Difference in data"             (list #/\s+\|\s+/ #/error/i))
  (expect:required in "LogFileBody"  &gt; 0 "DIFFDONE Marker found"          #/DIFFDONE/)
  (expect:required in "LogFileBody"  &gt; 0 "Some things found"              #/^[a-z]{3,4}[0-9]+_r.*/)

# NOT IMPLEMENTED YET!
#
## Also: enhance logpro to take list of command files: file1,file2...
[waivers]
createprod{target=%78/%/%/%} ;;
  (disable:required "DIFFDONE Marker found")
  (disable:error    "Some error")
  (expect:waive  in "LogFileBody" &lt; 99 "Waive if failed due to version" #/\w+3\.6.*/)

# test_meta is a section for storing additional data on your test
[test_meta]
author matt
owner  matt
description Compare things
tags tagone,tagtwo
reviewed never</pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_triggers">Triggers</h3>
<div class="paragraph"><p>In your testconfig or megatest.config triggers can be specified</p></div>
<div class="listingblock">
<div class="title">Triggers spec</div>
<div class="content monospaced">

</div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Version 1.0<br>
Last updated 2020-01-02 13:39:49 PST
</div>
</div>
</body>
</html>

Overview
--------

Stand-alone Megatest Area
~~~~~~~~~~~~~~~~~~~~~~~~~

A single, stand-alone, Megatest based testsuite or "area" is
sufficient for most validation, automation and build problems.

image::megatest-stand-alone-area.png[Static,300]

Megatest is designed as a distributed or decoupled system. This means
you can run the areas stand-alone with no additional
infrastructure. I.e. there are no databases, web servers or other
centralized resources needed. However as your needs grow you can
integrate multiple areas into a bigger system.

Component Descriptions
^^^^^^^^^^^^^^^^^^^^^^

. Multi-area dashboard and xterm. A gui (the dashboard) is usually the
  best option for controlling and launching runs but all operations
  can also be done from the commandline. Note: The not yet released
  multi-area dashboard replaces the old dashboard for browsing and
  controlling runs but for managing a single area the old dashboard
  works very well.
 
. Area/testsuite. This is your testsuite or automation definition and
  consists of the information in megatest.config, runconfigs.config
  and your testconfigs along with any custom scripting that can't be
  done with the native Megatest features.

. If your testsuite or build automation is too large to run on a
  single instance you can distribute your jobs into a compute server
  pool. The only current requirements are password-less ssh access and
  a network filesystem.

Full System Architecture
~~~~~~~~~~~~~~~~~~~~~~~~

image::megatest-system-architecture.png[Static,300]

To transfer the environment to the next step you can do the following:

.Propagate environment to next step
----------------------------
$MT_MEGATEST -env2file .ezsteps/${stepname}	  
----------------------------

.Full example with ezsteps, logpro rules, scripts etc.
-----------------
# You can include a common file
#
[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc]

# Use "var" for a scratch pad
#
[var]
dumpsql select * from data;
sepstr .....................................

# NOT IMPLEMENTED YET!
#
[ezsteps-addendum]
prescript something.sh
postscript something2.sh

# Add additional steps here. Format is "stepname script"
[ezsteps]
importdb loaddb prod.db prod.sql
dumpprod dumpdata prod.db "#{get var dumpsql}"
diff (echo "prod#{get var sepstr}test";diff --side-by-side \
     dumpprod.log reference.log ;echo DIFFDONE)

[scripts]
loaddb #!/bin/bash
  sqlite3 $1 <<EOF
  .mode tabs
  .import $2 data
  .q
  EOF

dumpdata #!/bin/bash
  sqlite3 $1 <<EOF
  .separator ,
  $2
  .q
  EOF

# Test requirements are specified here
[requirements]
waiton setup
priority 0

# Iteration for your test is controlled by the items section
# The complicated if is needed to allow processing of the config for the dashboard when there are no actual runs.
[items]
THINGNAME [system generatethings.sh | sort -u]

# Logpro rules for each step can be captured here in the testconfig
# note: The ;; after the stepname and the leading whitespace are required
#
[logpro]
inputdb ;; 
  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
  (expect:required in "LogFileBody"  > 0 "Some data found"                #/^[a-z]{3,4}[0-9]+_r.*/)

diff ;; 
  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
  (expect:error    in "LogFileBody"  = 0 "< or > indicate missing entry"  (list #/(<|>)/   #/error/i))
  (expect:error    in "LogFileBody"  = 0 "Difference in data"             (list #/\s+\|\s+/ #/error/i))
  (expect:required in "LogFileBody"  > 0 "DIFFDONE Marker found"          #/DIFFDONE/)
  (expect:required in "LogFileBody"  > 0 "Some things found"              #/^[a-z]{3,4}[0-9]+_r.*/)

# NOT IMPLEMENTED YET!
#
## Also: enhance logpro to take list of command files: file1,file2...
[waivers]
createprod{target=%78/%/%/%} ;;
  (disable:required "DIFFDONE Marker found")
  (disable:error    "Some error")
  (expect:waive  in "LogFileBody" < 99 "Waive if failed due to version" #/\w+3\.6.*/)

# test_meta is a section for storing additional data on your test
[test_meta]
author matt
owner  matt
description Compare things
tags tagone,tagtwo
reviewed never
-----------------

Triggers
~~~~~~~~

In your testconfig or megatest.config triggers can be specified 

.Triggers spec

;;     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

;; Always use two or four digit decimal
;; 1.01, 1.02...1.10,1.11,1.1101 ... 1.99,2.00..

(declare (unit megatest-version))

(define megatest-version 1.6545)

	 (let* ((run-id    (db:get-value-by-header run header "id"))
		(run-state (db:get-value-by-header run header "state"))
		(run-name  (db:get-value-by-header run header "runname"))
		(tests     (if (not (equal? run-state "locked"))
			       (proc-get-tests run-id)
			       '()))
		(lasttpath "/does/not/exist/I/hope")
    (lastrealpath "/does/not/exist/I/hope")
		(worker-thread #f))
	   (debug:print-info 4 *default-log-port* "runs:operate-on run=" run ", header=" header)
	   (if (not (null? tests))
	       (begin
		 (case action
                   ((kill-runs)
                    (tasks:kill-runner target run-name "%")

                                              (mt:test-set-state-status-by-id run-id (db:test-get-id test) "KILLREQ" "n/a" #f)
                                              (thread-sleep! 1)))
                                        ;; NOTE: This is suboptimal as the testdata will be used later and the state/status may have changed ...
                                        (if (null? tal)
                                            (loop new-test-dat tal)
                                            (loop (car tal)(append tal (list new-test-dat)))))
                                      (begin
                                        (set! lasttpath (db:test-get-rundir new-test-dat)) ;; remember this path for run removal
                                        (if (file-exists? lasttpath) 
                                          (set! lastrealpath (resolve-pathname lasttpath))
                                          (set! lastrealpath lasttpath)
                                        )
                                        (runs:remove-test-directory new-test-dat mode) ;; 'remove-all)

                                        (if (not (null? tal))
                                            (loop (car tal)(cdr tal)))))))
				(rmt:update-run-stats run-id (rmt:get-raw-run-stats run-id)))
                               ((kill-runs)
                                ;; RUNNING -> KILLREQ
                                ;; LAUNCHED,RUNNING,REMOTEHOSTSTART -> NOT STARTED
                                (cond

		     (if worker-thread (thread-join! worker-thread)))
                   (common:join-backgrounded-threads))))
	   ;; remove the run if zero tests remain
	   (if (eq? action 'remove-runs)
	       (let* ((run-id   (db:get-value-by-header run header "id")) ;; NB// masks run-id from above?
                      (remtests (mt:get-tests-for-run run-id #f '("DELETED") '("n/a") not-in: #t)))
		 (if (null? remtests) ;; no more tests remaining
         ;; Remove the last dir from the path.
         ;; And same for the link-resolved path
		     (let* ((dparts  (string-split lasttpath "/"))
			      (linkspath (conc "/" (string-intersperse (take dparts (- (length dparts) 1)) "/")))
            (real-dparts (string-split lastrealpath "/"))
			      (realpath (conc "/" (string-intersperse (take real-dparts (- (length real-dparts) 1)) "/")))
            )

            (debug:print 1 *default-log-port* "Removing run: " linkspath)
            (if (not keep-records)
               (begin
                 (debug:print 1 *default-log-port* "Removing DB records for the run.")
                 (rmt:delete-run run-id)
                 (rmt:delete-old-deleted-test-records))
            )
	          (debug:print 1 *default-log-port* "Recursively removing links dir " linkspath)
            (runs:recursive-delete-with-error-msg linkspath)




            (debug:print 1 *default-log-port* "Recursively removing real dir " realpath)
            (runs:recursive-delete-with-error-msg realpath)

		       )))))
	 ))
     runs)
    ;; (sqlite3:finalize! (db:delay-if-busy tdbdat))
  )
#t
)

(define (runs:remove-test-directory test mode) ;; remove-data-only)
  (let* ((run-dir       (db:test-get-rundir test))    ;; run dir is from the link tree
	 (real-dir      (if (common:file-exists? run-dir)
			    ;; (resolve-pathname run-dir)
			    (common:nice-path run-dir)
			    #f))

      ((remove-data-only)(mt:test-set-state-status-by-id (db:test-get-run_id test)(db:test-get-id test) "CLEANING" "LOCKED" #f))
      ((remove-all)      (mt:test-set-state-status-by-id (db:test-get-run_id test)(db:test-get-id test) "REMOVING" "LOCKED" #f))
      ((archive-remove)  (mt:test-set-state-status-by-id (db:test-get-run_id test)(db:test-get-id test) "ARCHIVE_REMOVING" #f #f)))
    (debug:print-info 1 *default-log-port* "Attempting to remove " (if real-dir (conc " dir " real-dir " and ") "") " link " run-dir)
    (if (and real-dir 
	     (> (string-length real-dir) 5)
	     (common:file-exists? real-dir)) ;; bad heuristic but should prevent /tmp /home etc.
	(let* ((realpath (resolve-pathname run-dir)))
	  (debug:print-info 1 *default-log-port* "Recursively removing " realpath)
	  (if (common:file-exists? realpath)
	      (runs:safe-delete-test-dir realpath)
	      (debug:print 0 *default-log-port* "WARNING: test dir " realpath " appears to not exist or is not readable")))
	(if real-dir 
	    (debug:print 0 *default-log-port* "WARNING: directory " real-dir " does not exist")
	    (debug:print 0 *default-log-port* "WARNING: no real directory corrosponding to link " run-dir ", nothing done")))
    (if (symbolic-link? run-dir)
	(begin
	  (debug:print-info 1 *default-log-port* "Removing symlink " run-dir)
	  (handle-exceptions