Index: docs/manual/Makefile
==================================================================
--- docs/manual/Makefile
+++ docs/manual/Makefile
@@ -32,10 +32,11 @@
 
 all : server.ps megatest_manual.html client.ps complex-itemmap.png megatest_manual.pdf
 
 megatest_manual.html : megatest_manual.txt  *.txt installation.txt *png *.dot
 	asciidoc  -b html5 -a icons -a iconsdir=$(DISPATH)/images/icons -a toc2 megatest_manual.txt
+	fossil wiki commit Roadmap plan.txt
 #	dos2unix megatest_manual.html
 
 megatest_manual.pdf : megatest_manual.txt *.txt *png *.dot
 	a2x -a toc -f pdf megatest_manual.txt
 

Index: docs/manual/bisecting.png
==================================================================
--- docs/manual/bisecting.png
+++ docs/manual/bisecting.png
cannot compute difference between binary files

Index: docs/manual/megatest-test-stages.png
==================================================================
--- docs/manual/megatest-test-stages.png
+++ docs/manual/megatest-test-stages.png
cannot compute difference between binary files

Index: docs/manual/megatest_manual.html
==================================================================
--- docs/manual/megatest_manual.html
+++ docs/manual/megatest_manual.html
@@ -1,10 +1,10 @@
 <!DOCTYPE html>
 <html lang="en">
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
-<meta name="generator" content="AsciiDoc 9.0.0rc1">
+<meta name="generator" content="AsciiDoc 10.2.0">
 <title>The Megatest Users Manual</title>
 <style type="text/css">
 /* Shared CSS for AsciiDoc xhtml11 and html5 backends */
 
 /* Default font. */
@@ -769,13 +769,13 @@
 <h1>The Megatest Users Manual</h1>
 <span id="author">Matt Welland</span><br>
 <span id="email" class="monospaced">&lt;<a href="mailto:matt@kiatoa.com">matt@kiatoa.com</a>&gt;</span><br>
 <span id="revnumber">version 1.5,</span>
 <span id="revdate">June 2020</span>
-<div id="toc">
-  <div id="toctitle">Table of Contents</div>
-  <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
+<div id="toc">
+  <div id="toctitle">Table of Contents</div>
+  <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
 </div>
 </div>
 <div id="content">
 <div class="sect1">
 <h2 id="_preface">Preface</h2>
@@ -957,12 +957,11 @@
 <h2 id="_todo_road_map">TODO / Road Map</h2>
 <div class="sectionbody">
 <div class="paragraph"><p>Note: This road-map is a wish list and not a formal plan. Items are in
 rough priority but are subject to change. Development is driven by
 user requests, developer "itch" and bug reports. Please contact
-<a href="mailto:matt@kiatoa.com">matt@kiatoa.com</a> with requests or bug reports. Requests from inside
-Intel generally take priority.</p></div>
+<a href="mailto:matt@kiatoa.com">matt@kiatoa.com</a> with requests or bug reports.</p></div>
 <div class="paragraph"><p>Dashboard and runs</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
 <p>
 Multi-area dashboard view
@@ -978,23 +977,10 @@
 </li>
 <li>
 <p>
 Improve [script], especially indent handling
 </p>
-</li>
-</ol></div>
-<div class="paragraph"><p>Scalability</p></div>
-<div class="olist arabic"><ol class="arabic">
-<li>
-<p>
-Overflow database methodology - combine the best of the v1.63
-  multi-db approach and the current db-in-tmp approach (currently
-  slowness can be seen when number of tests in a db goes over 50-100k,
-  with the overflow db it will be able to handle 1000&#8217;s of runs with
-  50-100k tests per run). High priority - goal is to complete this by
-  20Q3.
-</p>
 </li>
 </ol></div>
 <div class="paragraph"><p>Mtutils/CI</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
@@ -1028,20 +1014,10 @@
 <li>
 <p>
 ./configure &#8658; ubuntu, sles11, sles12, rh7 [WIP]
 </p>
 </li>
-<li>
-<p>
-Switch to using simple runs query everywhere
-</p>
-</li>
-<li>
-<p>
-Add end_time to runs and add a rollup call that sets state, status and end_time
-</p>
-</li>
 </ol></div>
 <div class="paragraph"><p>Code refactoring/quality/performance</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
 <p>
@@ -1050,10 +1026,20 @@
 </li>
 <li>
 <p>
 Use call-with-environment-variables where possible.
 </p>
+</li>
+<li>
+<p>
+Switch to using simple runs query everywhere
+</p>
+</li>
+<li>
+<p>
+Add end_time to runs and add a rollup call that sets state, status and end_time
+</p>
 </li>
 </ol></div>
 <div class="paragraph"><p>Migration to inmem db and or overflow db</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
@@ -1071,11 +1057,11 @@
 </ol></div>
 <div class="paragraph"><p>Some ideas for Megatest 2.0</p></div>
 <div class="olist arabic"><ol class="arabic">
 <li>
 <p>
-Aggressive megatest.config and runconfig.config caching.
+Generate cache files of megatest.config and runconfig.config and use them consistently. No magic.
 </p>
 <div class="olist loweralpha"><ol class="loweralpha">
 <li>
 <p>
 Cache the configs in $MT_RUNPATH
@@ -1090,15 +1076,10 @@
 </li>
 <li>
 <p>
 If the cached file changes ALL existing tests go from COMPLETED &#8594; STALE, I’m not sure what to do about RUNNING tests
 </p>
-</li>
-<li>
-<p>
-!VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.
-</p>
 </li>
 <li>
 <p>
 Per test copy commands (example is incomplete).
 </p>
@@ -1428,10 +1409,63 @@
 </li>
 <li>
 <p>
 Windows version
 </p>
+</li>
+</ol></div>
+<div class="paragraph"><p>WW14
+. Streamline compilation - DONE, all non-official egg modules are now bundled.</p></div>
+<div class="paragraph"><p>WW15
+. syscheck; touch file in home, tmp, runs, links and start xterm
+. pull in ftfplan (not integrated, just code pulled in)
+. fill newview matrix with data, filter pipeline gui elements
+. improve [script], especially indent handling</p></div>
+<div class="paragraph"><p>WW16
+. split db into megatest.db (runs etc.) db/&lt;something&gt;.db
+. release basic newview implementation
+. add section "archive" to testconfig with save yes to save the test to archive at end of test run.
+. add -getdata and -savedata; writes a path to archive using provided target, runname, test and item_patt and path</p></div>
+<div class="paragraph"><p>WW18
+. release split db implementation
+. mtutil calls from dashboard (for remote control)
+. logs browser (esp. for surfacing mtutil related activities)</p></div>
+<div class="paragraph"><p>WW19
+.</p></div>
+<div class="paragraph"><p>WW20
+. Switch to using simple runs query everywhere
+. Add end_time to runs and add a rollup call that sets state, status and end_time</p></div>
+<div class="paragraph"><p>Future
+. Switch to scsh-process pipeline management for job execution/control
+. Use call-with-environment-variables more.</p></div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_migration_to_inmem_db_plus_per_run_db">Migration to inmem db plus per run db</h2>
+<div class="sectionbody">
+<div class="olist arabic"><ol class="arabic">
+<li>
+<p>
+Re-work the dbstruct data structure?
+</p>
+<div class="olist loweralpha"><ol class="loweralpha">
+<li>
+<p>
+Move main.db to global?
+</p>
+</li>
+<li>
+<p>
+[ run-id.db inmemdb last-mod last-read last-sync inuse ]
+</p>
+</li>
+</ol></div>
+</li>
+<li>
+<p>
+Open main.db directly in calls to -runtests etc. No need to talk remote?
+</p>
 </li>
 </ol></div>
 </div>
 </div>
 <div class="sect1">
@@ -3016,11 +3050,11 @@
 <div class="listingblock">
 <div class="title">example for backreference (eg: item <span class="monospaced">foo23/thud</span> will imply waiton&#8217;s item <span class="monospaced">num-23/bar/thud</span></div>
 <div class="content monospaced">
 <pre>#
 # ## Example
-# ## can use \{number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl
+# ## can use {number} in replacement pattern to backreference a (capture) from matching pattern similar to sed or perl
 [requirements]
 mode itemwait
 # itemmap &lt;item pattern for this test&gt;  &lt;item replacement pattern for waiton test&gt;
 itemmap foo(\d+)/ num-\1/bar/</pre>
 </div></div>
@@ -3758,10 +3792,10 @@
 <div id="footnotes"><hr></div>
 <div id="footer">
 <div id="footer-text">
 Version 1.5<br>
 Last updated
- 2023-01-23 11:18:29 EST
+ 2024-06-20 22:56:52 EDT
 </div>
 </div>
 </body>
 </html>

Index: docs/manual/megatest_manual.pdf
==================================================================
--- docs/manual/megatest_manual.pdf
+++ docs/manual/megatest_manual.pdf
cannot compute difference between binary files

Index: docs/manual/plan.txt
==================================================================
--- docs/manual/plan.txt
+++ docs/manual/plan.txt
@@ -1,30 +1,12 @@
-// This file is part of Megatest.
-// 
-//     Megatest is free software: you can redistribute it and/or modify
-//     it under the terms of the GNU General Public License as published by
-//     the Free Software Foundation, either version 3 of the License, or
-//     (at your option) any later version.
-// 
-//     Megatest is distributed in the hope that it will be useful,
-//     but WITHOUT ANY WARRANTY; without even the implied warranty of
-//     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-//     GNU General Public License for more details.
-// 
-//     You should have received a copy of the GNU General Public License
-//     along with Megatest.  If not, see <http://www.gnu.org/licenses/>.
-
-// Copyright 2006-2020, Matthew Welland.
-
 TODO / Road Map
 ---------------
 
 Note: This road-map is a wish list and not a formal plan. Items are in
 rough priority but are subject to change. Development is driven by
 user requests, developer "itch" and bug reports. Please contact
-matt@kiatoa.com with requests or bug reports. Requests from inside
-Intel generally take priority.
+matt@kiatoa.com with requests or bug reports.
 
 Dashboard and runs
 
 . Multi-area dashboard view
 
@@ -31,19 +13,10 @@
 Tests Support
 
 . Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME
 . Improve [script], especially indent handling
 
-Scalability
-
-. Overflow database methodology - combine the best of the v1.63
-  multi-db approach and the current db-in-tmp approach (currently
-  slowness can be seen when number of tests in a db goes over 50-100k,
-  with the overflow db it will be able to handle 1000's of runs with
-  50-100k tests per run). High priority - goal is to complete this by
-  20Q3.
-
 Mtutils/CI
 
 . Enable mtutil calls from dashboard (for remote control)
 . Logs browser (esp. for surfacing mtutil related activities)
 . Embed ftfplan for distributed automation, completed activities trigger QA runs which trigger deployment etc.
@@ -51,30 +24,29 @@
 . Add output flushing in teamcity support
 
 Build system
 
 . ./configure => ubuntu, sles11, sles12, rh7 [WIP]
-. Switch to using simple runs query everywhere
-. Add end_time to runs and add a rollup call that sets state, status and end_time
 
 Code refactoring/quality/performance
 
 . Switch to scsh-process pipeline management for job execution/control
 . Use call-with-environment-variables where possible.
+. Switch to using simple runs query everywhere
+. Add end_time to runs and add a rollup call that sets state, status and end_time
 
 Migration to inmem db and or overflow db
 
 . Re-work the dbstruct data structure?
 .. [ run-id.db inmemdb last-mod last-read last-sync inuse ]
 
 Some ideas for Megatest 2.0
 
-.   Aggressive megatest.config and runconfig.config caching.
+.   Generate cache files of megatest.config and runconfig.config and use them consistently. No magic.
 ..  Cache the configs in $MT_RUNPATH
 ..  Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed
 . If the cached file changes ALL existing tests go from COMPLETED -> STALE, I’m not sure what to do about RUNNING tests
-. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.
 . Per test copy commands (example is incomplete).
 ----------------
 [testcopy]
 %/iind% unison SRC DEST
 % cp –r SRC DEST
@@ -159,5 +131,47 @@
  . Wizards for creating tests, regression areas (current ones are text only and limited).
  . Fully functional built in web service (currently you can browse runs but it is very simplistic).
  . Gui panels for editing megatest.config and runconfigs.config
  . Fully isolated tests (no use of NFS to see regression area files)
  . Windows version
+
+WW14
+. Streamline compilation - DONE, all non-official egg modules are now bundled.
+
+WW15
+. syscheck; touch file in home, tmp, runs, links and start xterm
+. pull in ftfplan (not integrated, just code pulled in)
+. fill newview matrix with data, filter pipeline gui elements
+. improve [script], especially indent handling
+
+WW16
+. split db into megatest.db (runs etc.) db/<something>.db
+. release basic newview implementation
+. add section "archive" to testconfig with save yes to save the test to archive at end of test run.
+. add -getdata and -savedata; writes a path to archive using provided target, runname, test and item_patt and path
+
+WW18
+. release split db implementation
+. mtutil calls from dashboard (for remote control)
+. logs browser (esp. for surfacing mtutil related activities)
+
+WW19
+. 
+
+WW20
+. Switch to using simple runs query everywhere
+. Add end_time to runs and add a rollup call that sets state, status and end_time
+
+Future
+. Switch to scsh-process pipeline management for job execution/control
+. Use call-with-environment-variables more.
+
+
+Migration to inmem db plus per run db
+-------------------------------------
+
+. Re-work the dbstruct data structure?
+.. Move main.db to global?
+.. [ run-id.db inmemdb last-mod last-read last-sync inuse ]
+. Open main.db directly in calls to -runtests etc. No need to talk remote?
+
+

Index: docs/manual/server.png
==================================================================
--- docs/manual/server.png
+++ docs/manual/server.png
cannot compute difference between binary files

