Megatest

The First Part of the Book
==========================
Getting Started
===============

[partintro]
.Optional part introduction title
.Getting started with Megatest
--
Optional part introduction goes here.
How to install Megatest and set it up for running your regressions and continuous integration process.
--

The First Chapter
-----------------
Chapters can be grouped by preceeding them with a level 0 Book Part
title.
Installation
------------

Dependencies
~~~~~~~~~~~~

Chicken scheme and a number of "eggs" are required for building
Megatest. See the script installall.sch in the utils directory of the
distribution for a mostly automated way to install everything needed
for building Megatest on Linux.
Book chapters are at level 1 and can contain sub-sections nested up to

three deep.
footnote:[An example footnote.]
indexterm:[Example index entry]

It's also worth noting that a book part can have it's own preface,

bibliography, glossary and index. Chapters can have their own
bibliography, glossary and index.

And now for something completely different: ((monkeys)), lions and
tigers (Bengal and Siberian) using the alternative syntax index
entries.
(((Big cats,Lions)))
(((Big cats,Tigers,Bengal Tiger)))
(((Big cats,Tigers,Siberian Tiger)))

<div class="sect1">
<h2 id="_thanks">Thanks</h2>
<div class="sectionbody">
<div class="paragraph"><p>To all the great people from the various places I&#8217;ve worked over the
years, thank you. I&#8217;ve benefitted from your patience and insight and I
hope that some of what I&#8217;ve learned from you has been incorporated
into Megatest.</p></div>
<div class="paragraph"><p>Lastly, my thanks to all the wonderful open source creators. These
<div class="paragraph"><p>Also, my thanks to all the wonderful open source creators. These
truly free projects, from Linux, xemacs, chicken scheme, and fossil to
asciidoc and bash are the philosophical and direct underpinnings of
the Megatest project.</p></div>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_example_preface">Example Preface</h2>
<h2 id="_preface">Preface</h2>
<div class="sectionbody">
<div class="paragraph"><p>The optional book preface goes here at section level zero.</p></div>
<div class="paragraph"><p>This book is organised as three sub-books; getting started, writing tests and reference.</p></div>
<div class="sect2">
<h3 id="_preface_sub_section">Preface Sub-section</h3>
<div class="admonitionblock">
<h3 id="_why_megatest">Why Megatest?</h3>
<div class="paragraph"><p>The Megatest project was started for two reasons, the first was an
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years
and it seemed a good thing to have a single open source tool, flexible
enough to meet the needs of any team doing continuous integrating and
or running a complex suite of tests for release qualification.</p></div>
</div>
<div class="sect2">
<table><tr>
<td class="icon">
<div class="title">Note</div>
<h3 id="_megatest_design_philosophy">Megatest Design Philosophy</h3>
<div class="paragraph"><p>Megatest is intended to provide the minimum needed resources to make
writing a suite of tests and implementing continuous build for
software, design engineering or process control (via owlfs for
example) without being specialized for any specific problem
space. Megatest in of itself does not know what constitutes a PASS or
FAIL of a test. In most cases megatest is best used in conjunction
with logpro or a similar tool to parse, analyze and decide on the test
outcome.</p></div>
</div>
<div class="sect2">
</td>
<td class="content">Preface and appendix subsections start out of sequence at level
2 (level 1 is skipped). This only applies to multi-part book
documents.</td>
</tr></table>
<h3 id="_megatest_architecture">Megatest Architecture</h3>
<div class="paragraph"><p>All data to specify the tests and configure the system is stored in
plain text files. All system state is stored in an sqlite3
database. Tests are launched using the launching system available for
the distributed compute platform in use. A template script is provided
which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database.</p></div>
</div>
</div>
</div>
</div>
<h1 id="_the_first_part_of_the_book">The First Part of the Book</h1>
<h1 id="_getting_started">Getting Started</h1>
<div class="openblock">
<div class="title">Optional part introduction title</div>
<div class="title">Getting started with Megatest</div>
<div class="content">
<div class="paragraph"><p>Optional part introduction goes here.</p></div>
<div class="paragraph"><p>How to install Megatest and set it up for running your regressions and continuous integration process.</p></div>
</div></div>
<div class="sect1">
<h2 id="_the_first_chapter">The First Chapter</h2>
<h2 id="_installation">Installation</h2>
<div class="sectionbody">
<div class="paragraph"><p>Chapters can be grouped by preceeding them with a level 0 Book Part
title.</p></div>
<div class="sect2">
<h3 id="_dependencies">Dependencies</h3>
<div class="paragraph"><p>Chicken scheme and a number of "eggs" are required for building
Megatest. See the script installall.sch in the utils directory of the
distribution for a mostly automated way to install everything needed
for building Megatest on Linux.</p></div>
<div class="paragraph"><p>Book chapters are at level 1 and can contain sub-sections nested up to
three deep.
<span class="footnote"><br />[An example footnote.]<br /></span></p></div>
<div class="paragraph"><p><span class="footnote"><br />[An example footnote.]<br /></span></p></div>
<div class="paragraph"><p>It&#8217;s also worth noting that a book part can have it&#8217;s own preface,
bibliography, glossary and index. Chapters can have their own
bibliography, glossary and index.</p></div>
<div class="paragraph"><p>And now for something completely different: monkeys, lions and
tigers (Bengal and Siberian) using the alternative syntax index
entries.



Note that multi-entry terms generate separate index entries.</p></div>

</table>
</div>
<div class="exampleblock">
<div class="title">Example 1. An example example</div>
<div class="content">
<div class="paragraph"><p>Lorum ipum&#8230;</p></div>
</div></div>
</div>
<div class="sect2">
<h3 id="X1">Sub-section with Anchor</h3>
<div class="paragraph"><p>Sub-section at level 2.</p></div>
<div class="sect3">
<h4 id="_chapter_sub_section">Chapter Sub-section</h4>
<div class="paragraph"><p>Sub-section at level 3.</p></div>
<div class="sect4">

</div>
<div class="sect1">
<h2 id="_the_second_chapter">The Second Chapter</h2>
<div class="sectionbody">
<div class="paragraph"><p>An example link to anchor at start of the <a href="#X1">first sub-section</a>.</p></div>
<div class="paragraph"><p>An example link to a bibliography entry <a href="#taoup">[taoup]</a>.</p></div>
</div>
</div>
<h1 id="_writing_tests">Writing Tests</h1>
<div class="sect1">
<h2 id="_the_first_chapter_of_the_second_part">The First Chapter of the Second Part</h2>
<div class="sectionbody">
<div class="paragraph"><p>Chapters grouped into book parts are at level 1 and can contain
sub-sections.</p></div>
</div>
</div>
<h1 id="_reference">Reference</h1>
<div class="sect1">
<h2 id="_the_first_chapter_of_the_second_part_2">The First Chapter of the Second Part</h2>
<h2 id="_the_first_chapter_of_the_second_part">The First Chapter of the Second Part</h2>
<div class="sectionbody">
<div class="paragraph"><p>Chapters grouped into book parts are at level 1 and can contain
sub-sections.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_the_testconfig_file">The testconfig File</h2>

</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Version 1.0<br />
Last updated 2012-05-02 13:56:01 MST
Last updated 2012-05-03 07:53:32 MST
</div>
</div>
</body>
</html>

------

To all the great people from the various places I've worked over the
years, thank you. I've benefitted from your patience and insight and I
hope that some of what I've learned from you has been incorporated
into Megatest.

Lastly, my thanks to all the wonderful open source creators. These
Also, my thanks to all the wonderful open source creators. These
truly free projects, from Linux, xemacs, chicken scheme, and fossil to
asciidoc and bash are the philosophical and direct underpinnings of
the Megatest project.

[preface]
Example Preface
================
The optional book preface goes here at section level zero.
Preface
=======
This book is organised as three sub-books; getting started, writing tests and reference.

Why Megatest?
~~~~~~~~~~~~~

The Megatest project was started for two reasons, the first was an
immediate and pressing need for a generalized tool to manage a suite
of regression tests and the second was the fact that I had written or
maintained several such tools at different companies over the years
and it seemed a good thing to have a single open source tool, flexible
enough to meet the needs of any team doing continuous integrating and
or running a complex suite of tests for release qualification.
Preface Sub-section
~~~~~~~~~~~~~~~~~~~
NOTE: Preface and appendix subsections start out of sequence at level
2 (level 1 is skipped). This only applies to multi-part book

Megatest Design Philosophy
~~~~~~~~~~~~~~~~~~~~~~~~~~

Megatest is intended to provide the minimum needed resources to make
writing a suite of tests and implementing continuous build for
software, design engineering or process control (via owlfs for
example) without being specialized for any specific problem
space. Megatest in of itself does not know what constitutes a PASS or
FAIL of a test. In most cases megatest is best used in conjunction
with logpro or a similar tool to parse, analyze and decide on the test
outcome.

documents.
Megatest Architecture
~~~~~~~~~~~~~~~~~~~~~

All data to specify the tests and configure the system is stored in
plain text files. All system state is stored in an sqlite3
database. Tests are launched using the launching system available for
the distributed compute platform in use. A template script is provided
which can launch jobs on local and remote Linux hosts. Currently
megatest uses the network filesystem to call home to your master
sqlite3 database.

include::getting_started.txt[]
include::getting_started.txt[] include::writing_tests.txt[]
include::writing_tests.txt[]
include::reference.txt[]

[appendix]
Example Appendix
================
One or more optional appendixes go here at section level zero.

	  (let* ((test-record (hash-table-ref test-records hed))
		 (tconfig     (tests:testqueue-get-testconfig test-record))
		 (waitons     (tests:testqueue-get-waitons    test-record))
		 (priority    (tests:testqueue-get-priority   test-record))
		 (itemdat     (tests:testqueue-get-itemdat    test-record)) ;; itemdat can be a string, list or #f
		 (items       (tests:testqueue-get-items      test-record))
		 (item-path   (item-list->path itemdat))
		 (newtal      (append tal (list hed))))
		 (newtal      (append tal (list hed)))
		 (calc-fails  (lambda (prereqs-not-met)
				(filter (lambda (test)
					  (debug:print 9 "test: " test)
					  (and (vector? test) ;; not (string? test))
					       (not (member (db:test-get-status test)
							    '("PASS" "WARN" "CHECK" "WAIVED")))))
					prereqs-not-met))))
	    (debug:print 6
			 "itemdat:     " itemdat
			 "\n  items:     " items
			 "\n  item-path: " item-path)
	    (cond
	     ((not items) ;; when false the test is ok to be handed off to launch (but not before)
	      (let ((have-resources  (runs:can-run-more-tests db test-record)) ;; look at the test jobgroup and tot jobs running
		    (prereqs-not-met (db:get-prereqs-not-met db run-id waitons item-path)))

	      (let* ((have-resources  (runs:can-run-more-tests db test-record)) ;; look at the test jobgroup and tot jobs running
		     (prereqs-not-met (db:get-prereqs-not-met db run-id waitons item-path))
		     (fails           (calc-fails prereqs-not-met)))
		(debug:print 8 "INFO: have-resources: " have-resources " prereqs-not-met: " prereqs-not-met " fails: " fails)
		;; Don't know at this time if the test have been launched at some time in the past
		;; i.e. is this a re-launch?
		(cond
		 ((and have-resources
		       (null? prereqs-not-met))
		  ;; no loop here, just drop though and use the loop at the bottom 
		  (if (patt-list-match item-path item-patts)
		      (run:test db run-id runname keyvallst test-record flags #f)
		      (debug:print 1 "INFO: Skipping " (tests:testqueue-get-testname test-record) " " item-path " as it doesn't match " item-patts))
		  ;; else the run is stuck, temporarily or permanently
		  ;; but should check if it is due to lack of resources vs. prerequisites
		  )
		 ((not have-resources) ;; simply try again after waiting a second
		  (thread-sleep! 1.0)
		  (debug:print 1 "INFO: no resources to run new tests, waiting ...")
		  ;; could have done hed tal here but doing car/cdr of newtal to rotate tests
		  (loop (car newtal)(cdr newtal)))
		 (else ;; must be we have unmet prerequisites
		  (let ((fails  (filter (lambda (test)
					  (and (not (string? test))
					       (not (member (db:test-get-status test)
							    '("PASS" "WARN" "CHECK" "WAIVED")))))
					prereqs-not-met)))
		    (debug:print 4 "FAILS: " fails)
		    ;; If one or more of the prereqs-not-met are FAIL then we can issue
		    ;; a message and drop hed from the items to be processed.
		    (if (null? fails)
			(begin
			  ;; couldn't run, take a breather
			  (debug:print 4 "INFO: Shouldn't really get here, race condition? Unable to launch more tests at this moment, killing time ...")
			  (thread-sleep! 0.1) ;; long sleep here - no resources, may as well be patient
			  ;; we made new tal by sticking hed at the back of the list
			  (loop (car newtal)(cdr newtal)))
			;; the waiton is FAIL so no point in trying to run hed ever again
			(if (not (null? tal))
			    (begin
			      (debug:print 1 "WARN: Dropping test " (db:test-get-test-name hed) "/" (db:test-get-item-path hed)
					   " from the launch list as it has prerequistes that are FAIL")
			      (loop (car tal)(cdr tal))))))))))
			      (loop (car tal)(cdr tal)))))))))
	     
	     ;; case where an items came in as a list been processed
	     ((and (list? items)     ;; thus we know our items are already calculated
		   (not   itemdat)) ;; and not yet expanded into the list of things to be done
	      (if (and (>= *verbosity* 1)
		       (> (length items) 0)
		       (> (length (car items)) 0))

	       items)
	      (if (not (null? tal))
		  (loop (car tal)(cdr tal))))

	     ;; if items is a proc then need to run items:get-items-from-config, get the list and loop 
	     ;;    - but only do that if resources exist to kick off the job
	     ((or (procedure? items)(eq? items 'have-procedure))
	      (let ((can-run-more    (runs:can-run-more-tests db test-record))
		    (prereqs-not-met (db:get-prereqs-not-met db run-id waitons item-path)))
		(if (and can-run-more (null? prereqs-not-met))
		    (let ((test-name (tests:testqueue-get-testname test-record)))
		      (setenv "MT_TEST_NAME" test-name) ;; 
		      (setenv "MT_RUNNAME"   runname)
		      (set-megatest-env-vars db run-id) ;; these may be needed by the launching process
		      (let ((items-list (items:get-items-from-config tconfig)))
			(if (list? items-list)
			    (begin
			      (tests:testqueue-set-items! test-record items-list)
			      (loop hed tal))
			    (begin
			      (debug:print 0 "ERROR: The proc from reading the setup did not yield a list - please report this")
			      (exit 1)))))
		    (if can-run-more
	      (let* ((can-run-more    (runs:can-run-more-tests db test-record))
		     (prereqs-not-met (db:get-prereqs-not-met db run-id waitons item-path))
		     (fails           (calc-fails prereqs-not-met)))
		(debug:print 8 "INFO: can-run-more: " can-run-more
			     " prereqs-not-met:\n  " (intersperse prereqs-not-met "\n")
			     " fails:\n "            (intersperse fails "\n"))
		(cond 
		 ((and can-run-more (null? prereqs-not-met))
		  (let ((test-name (tests:testqueue-get-testname test-record)))
		    (setenv "MT_TEST_NAME" test-name) ;; 
		    (setenv "MT_RUNNAME"   runname)
		    (set-megatest-env-vars db run-id) ;; these may be needed by the launching process
		    (let ((items-list (items:get-items-from-config tconfig)))
		      (if (list? items-list)
			  (begin
			    (tests:testqueue-set-items! test-record items-list)
			    (loop hed tal))
			  (begin
			    (debug:print 0 "ERROR: The proc from reading the setup did not yield a list - please report this")
			    (exit 1))))))
		 ((and can-run-more (null? fails))
			(loop (car newtal)(cdr newtal)) ;; an issue with prereqs not yet met?
			(begin
			  (debug:print 1 "INFO: Can't run more right now, killing a little time ...")
			  (thread-sleep! 1.0)
			  (loop (car newtal)(cdr newtal)))))))
			
		  (debug:print 4 "INFO: Can't run more right now, killing a little time ...")
		  (thread-sleep! 3.0)
		  (loop (car newtal)(cdr newtal))) ;; an issue with prereqs not yet met?
		 (else
		    ;; Can't run anything right now so put the test back on the queue
		    ;; (let ((newtal (append tal (list hed)))
		    ;;       (fails  (filter (lambda (test)  ;; fails is a list of prerequisite tests that are FAIL
		    ;;     		    (not (member (db:test-get-status test)
		    ;;     				 '("PASS" "WARN" "CHECK" "WAIVED"))))
		    ;;     		  prereqs-not-met)))
		    ;;   ;; if can't run more tests, lets take a breather
		    ;;   (debug:print 4 "FAILS: " fails)
		  (debug:print 1 "INFO: test "  hed " has failed prerequisite(s); "
		    ;;   ;; If one or more of the prereqs-not-met are FAIL then we can issue
		    ;;   ;; a message and drop hed from the items to be processed.
		    ;;   (if (null? fails)
			       (string-intersperse (map db:test-get-testname fails) ", ")
		    ;;       (begin
		    ;;         ;; couldn't run, take a breather
		    ;;         (thread-sleep! 0.1) ;; long sleep here - no resources, may as well be patient
		    ;;         (loop (car newtal)(cdr newtal)))
		    ;;       ;; the waiton is FAIL so no point in trying to run hed
		    ;;       (if (not (null? tal))
		    ;;           (loop hed tal)))))))

			       ", removing it from to-do list")
		  (if (not (null? tal))
		      (loop (car tal)(cdr tal)))))))
	     
	     ;; this case should not happen, added to help catch any bugs
	     ((and (list? items) itemdat)
	      (debug:print 0 "ERROR: Should not have a list of items in a test and the itemspath set - please report this")
	      (exit 1))))
	  
	  ;; we get here on "drop through" - loop for next test in queue
	  (if (null? tal)

	    ((equal? (test:get-state testdat) "NOT_STARTED")(set! runflag #t))
	    ;; not -rerun and PASS, WARN or CHECK, do no run
	    ((and (or (not rerun)
		      keepgoing)
		  ;; Require to force re-run for COMPLETED or *anything* + PASS,WARN or CHECK
		  (or (member (test:get-status testdat) '("PASS" "WARN" "CHECK"))
		      (member (test:get-state  testdat) '("COMPLETED")))) 
	     (debug:print 2 "INFO: running test " test-name "/" item-path " suppressed as it is COMPLETED and " (test:get-state testdat))
	     (debug:print 2 "INFO: running test " test-name "/" item-path " suppressed as it is " (test:get-state testdat) " and " (test:get-status testdat))
	     (set! runflag #f))
	    ;; -rerun and status is one of the specifed, run it
	    ((and rerun
		  (let* ((rerunlst   (string-split rerun ","))
			 (must-rerun (member (test:get-status testdat) rerunlst)))
		    (debug:print 3 "INFO: -rerun list: " rerun ", test-status: " (test:get-status testdat)", must-rerun: " must-rerun)
		    must-rerun))