Index: docs/manual/megatest_manual.html ================================================================== --- docs/manual/megatest_manual.html +++ docs/manual/megatest_manual.html @@ -900,10 +900,69 @@ sqlite3 database. Megatest has been used with the Intel Netbatch and lsf (also known as openlava) batch systems and it should be straightforward to use it with other similar systems.

+ +
+

Overview

+
+
+

Stand-alone Megatest Area

+

A single, stand-alone, Megatest based testsuite or "area" is +sufficient for most validation, automation and build problems.

+
+
+Static +
+
+

Megatest is designed as a distributed or decoupled system. This means +you can run the areas stand-alone with no additional +infrastructure. I.e. there are no databases, web servers or other +centralized resources needed. However as your needs grow you can +integrate multiple areas into a bigger system.

+
+

Component Descriptions

+
    +
  1. +

    +Multi-area dashboard and xterm. A gui (the dashboard) is usually the + best option for controlling and launching runs but all operations + can also be done from the commandline. Note: The not yet released + multi-area dashboard replaces the old dashboard for browsing and + controlling runs but for managing a single area the old dashboard + works very well. +

    +
    2. +
  2. +

    +Area/testsuite. This is your testsuite or automation definition and + consists of the information in megatest.config, runconfigs.config + and your testconfigs along with any custom scripting that can’t be + done with the native Megatest features. +

    +
    3. +
  3. +

    +If your testsuite or build automation is too large to run on a + single instance you can distribute your jobs into a compute server + pool. The only current requirements are password-less ssh access and + a network filesystem. +

    +
    4. +
+
+
+
+

Full System Architecture

+
+
+Static +
+
+
+

Installation

@@ -2307,10 +2366,95 @@
Propagate environment to next step
$MT_MEGATEST -env2file .ezsteps/${stepname}
+
+
Full example with ezsteps, logpro rules, scripts etc.
+
+
# You can include a common file
+#
+[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc]
+
+# Use "var" for a scratch pad
+#
+[var]
+dumpsql select * from data;
+sepstr .....................................
+
+# NOT IMPLEMENTED YET!
+#
+[ezsteps-addendum]
+prescript something.sh
+postscript something2.sh
+
+# Add additional steps here. Format is "stepname script"
+[ezsteps]
+importdb loaddb prod.db prod.sql
+dumpprod dumpdata prod.db "#{get var dumpsql}"
+diff (echo "prod#{get var sepstr}test";diff --side-by-side \
+     dumpprod.log reference.log ;echo DIFFDONE)
+
+[scripts]
+loaddb #!/bin/bash
+  sqlite3 $1 <<EOF
+  .mode tabs
+  .import $2 data
+  .q
+  EOF
+
+dumpdata #!/bin/bash
+  sqlite3 $1 <<EOF
+  .separator ,
+  $2
+  .q
+  EOF
+
+# Test requirements are specified here
+[requirements]
+waiton setup
+priority 0
+
+# Iteration for your test is controlled by the items section
+# The complicated if is needed to allow processing of the config for the dashboard when there are no actual runs.
+[items]
+THINGNAME [system generatethings.sh | sort -u]
+
+# Logpro rules for each step can be captured here in the testconfig
+# note: The ;; after the stepname and the leading whitespace are required
+#
+[logpro]
+inputdb ;;
+  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
+  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
+  (expect:required in "LogFileBody"  > 0 "Some data found"                #/^[a-z]{3,4}[0-9]+_r.*/)
+
+diff ;;
+  (expect:ignore   in "LogFileBody"  < 99 "Ignore error in comments"      #/^\/\/.*error/)
+  (expect:warning  in "LogFileBody"  = 0 "Any warning"                    #/warn/)
+  (expect:error    in "LogFileBody"  = 0 "< or > indicate missing entry"  (list #/(<|>)/   #/error/i))
+  (expect:error    in "LogFileBody"  = 0 "Difference in data"             (list #/\s+\|\s+/ #/error/i))
+  (expect:required in "LogFileBody"  > 0 "DIFFDONE Marker found"          #/DIFFDONE/)
+  (expect:required in "LogFileBody"  > 0 "Some things found"              #/^[a-z]{3,4}[0-9]+_r.*/)
+
+# NOT IMPLEMENTED YET!
+#
+## Also: enhance logpro to take list of command files: file1,file2...
+[waivers]
+createprod{target=%78/%/%/%} ;;
+  (disable:required "DIFFDONE Marker found")
+  (disable:error    "Some error")
+  (expect:waive  in "LogFileBody" < 99 "Waive if failed due to version" #/\w+3\.6.*/)
+
+# test_meta is a section for storing additional data on your test
+[test_meta]
+author matt
+owner  matt
+description Compare things
+tags tagone,tagtwo
+reviewed never
+

Triggers

In your testconfig or megatest.config triggers can be specified

@@ -2990,10 +3134,10 @@
ADDED docs/manual/overview.txt Index: docs/manual/overview.txt ================================================================== --- /dev/null +++ docs/manual/overview.txt @@ -0,0 +1,43 @@ + +Overview +-------- + +Stand-alone Megatest Area +~~~~~~~~~~~~~~~~~~~~~~~~~ + +A single, stand-alone, Megatest based testsuite or "area" is +sufficient for most validation, automation and build problems. + +image::megatest-stand-alone-area.png[Static,300] + +Megatest is designed as a distributed or decoupled system. This means +you can run the areas stand-alone with no additional +infrastructure. I.e. there are no databases, web servers or other +centralized resources needed. However as your needs grow you can +integrate multiple areas into a bigger system. + +Component Descriptions +^^^^^^^^^^^^^^^^^^^^^^ + +. Multi-area dashboard and xterm. A gui (the dashboard) is usually the + best option for controlling and launching runs but all operations + can also be done from the commandline. Note: The not yet released + multi-area dashboard replaces the old dashboard for browsing and + controlling runs but for managing a single area the old dashboard + works very well. + +. Area/testsuite. This is your testsuite or automation definition and + consists of the information in megatest.config, runconfigs.config + and your testconfigs along with any custom scripting that can't be + done with the native Megatest features. + +. If your testsuite or build automation is too large to run on a + single instance you can distribute your jobs into a compute server + pool. The only current requirements are password-less ssh access and + a network filesystem. + +Full System Architecture +~~~~~~~~~~~~~~~~~~~~~~~~ + +image::megatest-system-architecture.png[Static,300] + Index: docs/manual/reference.txt ================================================================== --- docs/manual/reference.txt +++ docs/manual/reference.txt @@ -651,10 +651,95 @@ .Propagate environment to next step ---------------------------- $MT_MEGATEST -env2file .ezsteps/${stepname} ---------------------------- + +.Full example with ezsteps, logpro rules, scripts etc. +----------------- +# You can include a common file +# +[include #{getenv MT_RUN_AREA_HOME}/global-testconfig.inc] + +# Use "var" for a scratch pad +# +[var] +dumpsql select * from data; +sepstr ..................................... + +# NOT IMPLEMENTED YET! +# +[ezsteps-addendum] +prescript something.sh +postscript something2.sh + +# Add additional steps here. Format is "stepname script" +[ezsteps] +importdb loaddb prod.db prod.sql +dumpprod dumpdata prod.db "#{get var dumpsql}" +diff (echo "prod#{get var sepstr}test";diff --side-by-side \ + dumpprod.log reference.log ;echo DIFFDONE) + +[scripts] +loaddb #!/bin/bash + sqlite3 $1 < 0 "Some data found" #/^[a-z]{3,4}[0-9]+_r.*/) + +diff ;; + (expect:ignore in "LogFileBody" < 99 "Ignore error in comments" #/^\/\/.*error/) + (expect:warning in "LogFileBody" = 0 "Any warning" #/warn/) + (expect:error in "LogFileBody" = 0 "< or > indicate missing entry" (list #/(<|>)/ #/error/i)) + (expect:error in "LogFileBody" = 0 "Difference in data" (list #/\s+\|\s+/ #/error/i)) + (expect:required in "LogFileBody" > 0 "DIFFDONE Marker found" #/DIFFDONE/) + (expect:required in "LogFileBody" > 0 "Some things found" #/^[a-z]{3,4}[0-9]+_r.*/) + +# NOT IMPLEMENTED YET! +# +## Also: enhance logpro to take list of command files: file1,file2... +[waivers] +createprod{target=%78/%/%/%} ;; + (disable:required "DIFFDONE Marker found") + (disable:error "Some error") + (expect:waive in "LogFileBody" < 99 "Waive if failed due to version" #/\w+3\.6.*/) + +# test_meta is a section for storing additional data on your test +[test_meta] +author matt +owner matt +description Compare things +tags tagone,tagtwo +reviewed never +----------------- Triggers ~~~~~~~~ In your testconfig or megatest.config triggers can be specified