Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -900,10 +900,69 @@ sqlite3 database. Megatest has been used with the Intel Netbatch and lsf (also known as openlava) batch systems and it should be straightforward to use it with other similar systems.

+ +
+

Overview

+
+
+

Stand-alone Megatest Area

+

A single, stand-alone, Megatest based testsuite or "area" is +sufficient for most validation, automation and build problems.

+
+
+Static +
+
+

Megatest is designed as a distributed or decoupled system. This means +you can run the areas stand-alone with no additional +infrastructure. I.e. there are no databases, web servers or other +centralized resources needed. However as your needs grow you can +integrate multiple areas into a bigger system.

+
+

Component Descriptions

+
    +
  1. +

    +Multi-area dashboard and xterm. A gui (the dashboard) is usually the + best option for controlling and launching runs but all operations + can also be done from the commandline. Note: The not yet released + multi-area dashboard replaces the old dashboard for browsing and + controlling runs but for managing a single area the old dashboard + works very well. +

    +
    2. +
  2. +

    +Area/testsuite. This is your testsuite or automation definition and + consists of the information in megatest.config, runconfigs.config + and your testconfigs along with any custom scripting that can’t be + done with the native Megatest features. +

    +
    3. +
  3. +

    +If your testsuite or build automation is too large to run on a + single instance you can distribute your jobs into a compute server + pool. The only current requirements are password-less ssh access and + a network filesystem. +

    +
    4. +
+
+
+
+

Full System Architecture

+
+
+Static +
+
+
+

Installation

@@ -2307,10 +2366,95 @@
Propagate environment to next step
$MT_MEGATEST -env2file .ezsteps/${stepname}
+
+
Full example with ezsteps, logpro rules, scripts etc.
+
+
# You can include a common file
+#
+[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc]
+
+# Use "var" for a scratch pad
+#
+[var]
+dumpsql select * from data;
+sepstr .....................................
+
+# NOT IMPLEMENTED YET!
+#
+[ezsteps-addendum]
+prescript something.sh
+postscript something2.sh
+
+# Add additional steps here. Format is "stepname script"
+[ezsteps]
+importdb loaddb prod.db prod.sql
+dumpprod dumpdata prod.db "#{get var dumpsql}"
+diff (echo "prod#{get var sepstr}test";diff --side-by-side \
+     dumpprod.log reference.log ;echo DIFFDONE)
+
+[scripts]
+loaddb #!/bin/bash
+  sqlite3 $1 <<EOF
+  .mode tabs
+  .import $2 data
+  .q
+  EOF
+
+dumpdata #!/bin/bash
+  sqlite3 $1 <<EOF
+  .separator ,
+  $2
+  .q
+  EOF
+
+# Test requirements are specified here
+[requirements]
+waiton setup
+priority 0
+
+# Iteration for your test is controlled by the items section
+# The complicated if is needed to allow processing of the config for the dashboard when there are no actual runs.
+[items]
+THINGNAME [system generatethings.sh | sort -u]
+
+# Logpro rules for each step can be captured here in the testconfig
+# note: The ;; after the stepname and the leading whitespace are required
+#
+[logpro]
+inputdb ;;
+  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
+  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
+  (expect:required in "LogFileBody"  > 0 "Some data found"                #/^[a-z]{3,4}[0-9]+_r.*/)
+
+diff ;;
+  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
+  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
+  (expect:error    in "LogFileBody"  = 0 "< or > indicate missing entry"  (list #/(<|>)/   #/error/i))
+  (expect:error    in "LogFileBody"  = 0 "Difference in data"             (list #/\s+\|\s+/ #/error/i))
+  (expect:required in "LogFileBody"  > 0 "DIFFDONE Marker found"          #/DIFFDONE/)
+  (expect:required in "LogFileBody"  > 0 "Some things found"              #/^[a-z]{3,4}[0-9]+_r.*/)
+
+# NOT IMPLEMENTED YET!
+#
+## Also: enhance logpro to take list of command files: file1,file2...
+[waivers]
+createprod{target=%78/%/%/%} ;;
+  (disable:required "DIFFDONE Marker found")
+  (disable:error    "Some error")
+  (expect:waive  in "LogFileBody" < 99 "Waive if failed due to version" #/\w+3\.6.*/)
+
+# test_meta is a section for storing additional data on your test
+[test_meta]
+author matt
+owner  matt
+description Compare things
+tags tagone,tagtwo
+reviewed never
+

Triggers

In your testconfig or megatest.config triggers can be specified

@@ -2990,10 +3134,10 @@
ADDED docs/manual/overview.txt Index: docs/manual/overview.txt ================================================================== --- /dev/null +++ docs/manual/overview.txt @@ -0,0 +1,43 @@ + +Overview +-------- + +Stand-alone Megatest Area +~~~~~~~~~~~~~~~~~~~~~~~~~ + +A single, stand-alone, Megatest based testsuite or "area" is +sufficient for most validation, automation and build problems. + +image::megatest-stand-alone-area.png[Static,300] + +Megatest is designed as a distributed or decoupled system. This means +you can run the areas stand-alone with no additional +infrastructure. I.e. there are no databases, web servers or other +centralized resources needed. However as your needs grow you can +integrate multiple areas into a bigger system. + +Component Descriptions +^^^^^^^^^^^^^^^^^^^^^^ + +. Multi-area dashboard and xterm. A gui (the dashboard) is usually the + best option for controlling and launching runs but all operations + can also be done from the commandline. Note: The not yet released + multi-area dashboard replaces the old dashboard for browsing and + controlling runs but for managing a single area the old dashboard + works very well. + +. Area/testsuite. This is your testsuite or automation definition and + consists of the information in megatest.config, runconfigs.config + and your testconfigs along with any custom scripting that can't be + done with the native Megatest features. + +. If your testsuite or build automation is too large to run on a + single instance you can distribute your jobs into a compute server + pool. The only current requirements are password-less ssh access and + a network filesystem. + +Full System Architecture +~~~~~~~~~~~~~~~~~~~~~~~~ + +image::megatest-system-architecture.png[Static,300] + Index: docs/manual/reference.txt ================================================================== --- docs/manual/reference.txt +++ docs/manual/reference.txt @@ -651,10 +651,95 @@ .Propagate environment to next step ---------------------------- $MT_MEGATEST -env2file .ezsteps/${stepname} ---------------------------- + +.Full example with ezsteps, logpro rules, scripts etc. +----------------- +# You can include a common file +# +[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc] + +# Use "var" for a scratch pad +# +[var] +dumpsql select * from data; +sepstr ..................................... + +# NOT IMPLEMENTED YET! +# +[ezsteps-addendum] +prescript something.sh +postscript something2.sh + +# Add additional steps here. Format is "stepname script" +[ezsteps] +importdb loaddb prod.db prod.sql +dumpprod dumpdata prod.db "#{get var dumpsql}" +diff (echo "prod#{get var sepstr}test";diff --side-by-side \ + dumpprod.log reference.log ;echo DIFFDONE) + +[scripts] +loaddb #!/bin/bash + sqlite3 $1 < 0 "Some data found" #/^[a-z]{3,4}[0-9]+_r.*/) + +diff ;; + (expect:ignore in "LogFileBody" < 99 "Ignore error in comments" #/^\/\/.*error/) + (expect:warning in "LogFileBody" = 0 "Any warning" #/warn/) + (expect:error in "LogFileBody" = 0 "< or > indicate missing entry" (list #/(<|>)/ #/error/i)) + (expect:error in "LogFileBody" = 0 "Difference in data" (list #/\s+\|\s+/ #/error/i)) + (expect:required in "LogFileBody" > 0 "DIFFDONE Marker found" #/DIFFDONE/) + (expect:required in "LogFileBody" > 0 "Some things found" #/^[a-z]{3,4}[0-9]+_r.*/) + +# NOT IMPLEMENTED YET! +# +## Also: enhance logpro to take list of command files: file1,file2... +[waivers] +createprod{target=%78/%/%/%} ;; + (disable:required "DIFFDONE Marker found") + (disable:error "Some error") + (expect:waive in "LogFileBody" < 99 "Waive if failed due to version" #/\w+3\.6.*/) + +# test_meta is a section for storing additional data on your test +[test_meta] +author matt +owner matt +description Compare things +tags tagone,tagtwo +reviewed never +----------------- Triggers ~~~~~~~~ In your testconfig or megatest.config triggers can be specified Index: megatest-version.scm ================================================================== --- megatest-version.scm +++ megatest-version.scm @@ -18,6 +18,6 @@ ;; Always use two or four digit decimal ;; 1.01, 1.02...1.10,1.11,1.1101 ... 1.99,2.00.. (declare (unit megatest-version)) -(define megatest-version 1.6543) +(define megatest-version 1.6545) Index: runs.scm ================================================================== --- runs.scm +++ runs.scm @@ -2139,10 +2139,11 @@ (run-name (db:get-value-by-header run header "runname")) (tests (if (not (equal? run-state "locked")) (proc-get-tests run-id) '())) (lasttpath "/does/not/exist/I/hope") + (lastrealpath "/does/not/exist/I/hope") (worker-thread #f)) (debug:print-info 4 *default-log-port* "runs:operate-on run=" run ", header=" header) (if (not (null? tests)) (begin (case action @@ -2315,11 +2316,17 @@ ;; NOTE: This is suboptimal as the testdata will be used later and the state/status may have changed ... (if (null? tal) (loop new-test-dat tal) (loop (car tal)(append tal (list new-test-dat))))) (begin + (set! lasttpath (db:test-get-rundir new-test-dat)) ;; remember this path for run removal + (if (file-exists? lasttpath) + (set! lastrealpath (resolve-pathname lasttpath)) + (set! lastrealpath lasttpath) + ) (runs:remove-test-directory new-test-dat mode) ;; 'remove-all) + (if (not (null? tal)) (loop (car tal)(cdr tal))))))) (rmt:update-run-stats run-id (rmt:get-raw-run-stats run-id))) ((kill-runs) ;; RUNNING -> KILLREQ @@ -2392,31 +2399,38 @@ ;; remove the run if zero tests remain (if (eq? action 'remove-runs) (let* ((run-id (db:get-value-by-header run header "id")) ;; NB// masks run-id from above? (remtests (mt:get-tests-for-run run-id #f '("DELETED") '("n/a") not-in: #t))) (if (null? remtests) ;; no more tests remaining + ;; Remove the last dir from the path. + ;; And same for the link-resolved path (let* ((dparts (string-split lasttpath "/")) - (runpath (conc "/" (string-intersperse - (take dparts (- (length dparts) 1)) - "/")))) - (debug:print 1 *default-log-port* "Removing run: " runkey " " (db:get-value-by-header run header "runname") " and related record") - (if (not keep-records) - (begin - (rmt:delete-run run-id) - (rmt:delete-old-deleted-test-records))) - ;; (rmt:set-var "DELETED_TESTS" (current-seconds)) - ;; need to figure out the path to the run dir and remove it if empty - ;; (if (null? (glob (conc runpath "/*"))) - ;; (begin - ;; (debug:print 1 *default-log-port* "Removing run dir " runpath) - ;; (system (conc "rmdir -p " runpath)))) + (linkspath (conc "/" (string-intersperse (take dparts (- (length dparts) 1)) "/"))) + (real-dparts (string-split lastrealpath "/")) + (realpath (conc "/" (string-intersperse (take real-dparts (- (length real-dparts) 1)) "/"))) + ) + + (debug:print 1 *default-log-port* "Removing run: " linkspath) + (if (not keep-records) + (begin + (debug:print 1 *default-log-port* "Removing DB records for the run.") + (rmt:delete-run run-id) + (rmt:delete-old-deleted-test-records)) + ) + (debug:print 1 *default-log-port* "Recursively removing links dir " linkspath) + (runs:recursive-delete-with-error-msg linkspath) + + (debug:print 1 *default-log-port* "Recursively removing real dir " realpath) + (runs:recursive-delete-with-error-msg realpath) + ))))) )) runs) ;; (sqlite3:finalize! (db:delay-if-busy tdbdat)) - ) - #t) + ) +#t +) (define (runs:remove-test-directory test mode) ;; remove-data-only) (let* ((run-dir (db:test-get-rundir test)) ;; run dir is from the link tree (real-dir (if (common:file-exists? run-dir) ;; (resolve-pathname run-dir) @@ -2444,15 +2458,15 @@ ((archive-remove) (mt:test-set-state-status-by-id (db:test-get-run_id test)(db:test-get-id test) "ARCHIVE_REMOVING" #f #f))) (debug:print-info 1 *default-log-port* "Attempting to remove " (if real-dir (conc " dir " real-dir " and ") "") " link " run-dir) (if (and real-dir (> (string-length real-dir) 5) (common:file-exists? real-dir)) ;; bad heuristic but should prevent /tmp /home etc. - (begin ;; let* ((realpath (resolve-pathname run-dir))) - (debug:print-info 1 *default-log-port* "Recursively removing " real-dir) - (if (common:file-exists? real-dir) - (runs:safe-delete-test-dir real-dir) - (debug:print 0 *default-log-port* "WARNING: test dir " real-dir " appears to not exist or is not readable"))) + (let* ((realpath (resolve-pathname run-dir))) + (debug:print-info 1 *default-log-port* "Recursively removing " realpath) + (if (common:file-exists? realpath) + (runs:safe-delete-test-dir realpath) + (debug:print 0 *default-log-port* "WARNING: test dir " realpath " appears to not exist or is not readable"))) (if real-dir (debug:print 0 *default-log-port* "WARNING: directory " real-dir " does not exist") (debug:print 0 *default-log-port* "WARNING: no real directory corrosponding to link " run-dir ", nothing done"))) (if (symbolic-link? run-dir) (begin