Megatest

Check-in [21d4a4c30b]
Login
☰Home Timeline Files Branches Tags Tickets Wiki Login
Overview
Comment:Added auto propagation of env. vars in steps
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | v1.65
Files: files | file ages | folders
SHA1: 21d4a4c30b7d4a9e2dd50e5d35b8e231dd37b128
User & Date: mrwellan on 2020-10-26 22:12:21
Other Links: branch diff | manifest | tags
Context
2020-10-27
22:43
Updated ezsteps propagation and documentation check-in: b8cc18503a user: matt tags: v1.65
2020-10-26
22:12
Added auto propagation of env. vars in steps check-in: 21d4a4c30b user: mrwellan tags: v1.65
2020-10-25
22:54
Removed gratuitious call to get-count-tests-running-for-run-id and some indenting fixes. check-in: 8085b19fd6 user: matt tags: v1.65
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Modified docs/manual/megatest_manual.html from [00d3df112f] to [70c989fffc].

︙
3026
3027
3028
3029
3030
3031
3032




















3033
3034
3035
3036
3037
3038
3039

3026
3027
3028
3029
3030
3031
3032
3033
3034
3035
3036
3037
3038
3039
3040
3041
3042
3043
3044
3045
3046
3047
3048
3049
3050
3051
3052
3053
3054
3055
3056
3057
3058
3059








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








</div></div>
<div class="paragraph"><p>To transfer the environment to the next step you can do the following:</p></div>
<div class="listingblock">
<div class="title">Propagate environment to next step</div>
<div class="content monospaced">
<pre>$MT_MEGATEST -env2file .ezsteps/${stepname}</pre>
</div></div>
<div class="sect3">
<h4 id="_automatic_environment_propagation_with_ezsteps">Automatic environment propagation with Ezsteps</h4>
<div class="paragraph"><p>Simply give your step name an extension of .sh or .csh. The environment variables will be propagated.</p></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img src="/nfs/pdx/disks/ice.disk.002/icfadm/pkgs/asciidoc/8.6.7/images/icons/note.png" alt="Note">
</td>
<td class="content">aliases and variables with strange whitespace or characters will not propagate correctly. Put in a ticket on the <a href="http://www.kiatoa.com/fossils/megatest">http://www.kiatoa.com/fossils/megatest</a> site if you need support for a specific strange character combination.</td>
</tr></table>
</div>
<div class="listingblock">
<div class="title">Example of auto propagation</div>
<div class="content monospaced">
<pre>[ezsteps]
step1.sh export SOMEVAR=$(ps -def | wc -l);ls /tmp
# The next step will get the value of $SOMEVAR from step1.sh
step2.sh echo $SOMEVAR</pre>
</div></div>
</div>
</div>
<div class="sect2">
<h3 id="_scripts">Scripts</h3>
<div class="listingblock">
<div class="title">Specifying scripts inline (best used for only simple scripts)</div>
<div class="content monospaced">
<pre>[scripts]
︙
3497
3498
3499
3500
3501
3502
3503
3504

3505
3506
3507
3508

3517
3518
3519
3520
3521
3522
3523

3524
3525
3526
3527
3528








-
+





</div>
</div>
</div>
<div id="footnotes"><hr></div>
<div id="footer">
<div id="footer-text">
Version 1.5<br>
Last updated 2020-10-12 20:12:01 PDT
Last updated 2020-10-13 21:24:33 PDT
</div>
</div>
</body>
</html>

Modified docs/manual/reference.txt from [530f6e150c] to [a5ee87815b].

︙
740
741
742
743
744
745
746















747
748
749
750
751
752
753

740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+









To transfer the environment to the next step you can do the following:

.Propagate environment to next step
----------------------------
$MT_MEGATEST -env2file .ezsteps/${stepname}	  
----------------------------

Automatic environment propagation with Ezsteps
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Simply give your step name an extension of .sh or .csh. The environment variables will be propagated.

NOTE: aliases and variables with strange whitespace or characters will not propagate correctly. Put in a ticket on the http://www.kiatoa.com/fossils/megatest site if you need support for a specific strange character combination.

.Example of auto propagation
---------------------------
[ezsteps]
step1.sh export SOMEVAR=$(ps -def | wc -l);ls /tmp
# The next step will get the value of $SOMEVAR from step1.sh
step2.sh echo $SOMEVAR
---------------------------

Scripts
~~~~~~~

.Specifying scripts inline (best used for only simple scripts)
----------------------------
[scripts]
︙

Modified env.scm from [028e47144f] to [c7d61e935d].

︙
19
20
21
22
23
24
25


26

27
28
29
30
31
32
33
34
35






36
37
38
39
40
41
42

19
20
21
22
23
24
25
26
27

28
29
30
31






32
33
34
35
36
37
38
39
40
41
42
43
44








+
+
-
+



-
-
-
-
-
-
+
+
+
+
+
+








;;======================================================================

(declare (unit env))

(use sql-de-lite) ;; srfi-1 posix regex regex-case srfi-69 srfi-18 call-with-environment-variables)

(define (env:open-db fname)
  (let* ((db-exists (if (equal? fname ":memory:")
			#f
  (let* ((db-exists (common:file-exists? fname))
			(common:file-exists? fname)))
	 (db        (open-database fname)))
    (if (not db-exists)
	(begin
	  (exec (sql db "CREATE TABLE envvars (
                    id INTEGER PRIMARY KEY,
                    context TEXT NOT NULL,
                    var TEXT NOT NULL,
                    val TEXT NOT NULL,
                       CONSTRAINT envvars_constraint UNIQUE (context,var))"))))
	  (exec (sql db "CREATE TABLE IF NOT EXISTS envvars (
                            id INTEGER PRIMARY KEY,
                            context TEXT NOT NULL,
                            var TEXT NOT NULL,
                            val TEXT NOT NULL,
                               CONSTRAINT envvars_constraint UNIQUE (context,var))"))))
    (set-busy-handler! db (busy-timeout 10000))
    db))

;; save vars in given context, this is NOT incremental by default
;;
(define (env:save-env-vars db context #!key (incremental #f)(vardat #f))
  (with-transaction
︙
75
76
77
78
79
80
81























82
83
84
85
86
87
88

77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113








+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+








				  (let ((sep (cadr (assoc var paths))))
				    (env:merge-path-envvar sep (hash-table-ref result var) val))
				  val)))))
	(sql db "SELECT var,val FROM envvars WHERE context=?")
	context))
     contexts)
    result))

;; envdelta: a-b (start=a, end=b, get the delta)
;; ofile:    #f = write to stdout, else write to file with string name
;;
(define (env:envdelta db envdelta ofile)
  (let ((match (string-split envdelta "-")));; (string-match "([a-z0-9_]+)=([a-z0-9_\\-,]+)" envdelta)))
    (if (not (null? match))
	(let* ((parts     match) ;; (string-split equn "-"))
	       (minuend   (car parts))
	       (subtraend (cadr parts))
	       (added     (env:get-added   db minuend subtraend))
	       (removed   (env:get-removed db minuend subtraend))
	       (changed   (env:get-changed db minuend subtraend)))
	  ;; (pp (hash-table->alist added))
	  ;; (pp (hash-table->alist removed))
	  ;; (pp (hash-table->alist changed))
	  (if (args:get-arg "-o")
	      (with-output-to-file
		  (args:get-arg "-o")
		(lambda ()
		  (env:print added removed changed)))
	      (env:print added removed changed)))
	#f)))

;;  get list of removed variables between two contexts
;;
(define (env:get-removed db contexta contextb)
  (let ((result (make-hash-table)))
    (query
     (for-each-row
︙

Modified ezsteps.scm from [ef12da0318] to [d79baadfc3].

︙
31
32
33
34
35
36
37




38
















39








40
41
42
43


44

45
46
47
48
49
50
51

31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

59
60
61
62
63
64
65
66
67



68
69
70
71
72
73
74
75
76
77
78








+
+
+
+

+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
+
+
+
+
+
+
+

-
-
-
+
+

+








;; (declare (uses filedb))

(include "common_records.scm")
(include "key_records.scm")
(include "db_records.scm")
(include "run_records.scm")

(define (ezsteps:step-name->mode stepname)
  (match (string-search "\\.([^\\.]+)$" stepname)
    ((_ ext) (string->symbol ext))
    (else    #f)))

(define (ezsteps:create-step-script envdbf stepname prevstepname mode cmd)
  (let* ((shebang (case mode
		    ((sh)  "/bin/sh")
		    ((csh) "/bin/csh")
		    (else  "/bin/bash")))
	 (sourcef (conc ".ezsteps/" prevstepname))
	 (scriptn stepname)) ;; remember the name already has an extension .sh, .csh etc.
    (with-output-to-file scriptn
      (lambda ()
	;; the shebang line
	(print "#!" shebang "\n")
	;; first setup the source of the previous changes
	(if (file-exists? sourcef)
	    (print "source " sourcef))
	;; save the env at start
	(print "megatest -envcap "stepname"_start "envdbf)
;;(rmt:get-test-info-by-id run-id test-id) -> testdat
	;; run the command
	(print cmd "\necode=$?")
	;; save the env at end
	(print "megatest -envcap "stepname"_end "envdbf)
	;; write the delta
	(print "megatest -envdelta "stepname"_start-"stepname"_end -dumpmode bash -o .ezsteps/"stepname" "envdbf)
	(print "exit $ecode")))
    (system (conc "chmod a+x " scriptn))))

;; TODO: deprecate me in favor of ezsteps.scm
;;
(define (launch:runstep ezstep run-id test-id exit-info m tal testconfig all-steps-dat)
;; 
(define (launch:runstep ezstep run-id test-id exit-info m tal testconfig all-steps-dat prevstepname envdbf)
  (let* ((stepname       (car ezstep))  ;; do stuff to run the step
	 (stepmode       (ezsteps:step-name->mode stepname))
	 (stepinfo       (cadr ezstep))
	;; (let ((info (cadr ezstep)))
	;; 		   (if (proc? info) "" info)))
	;; (stepproc       (let ((info (cadr ezstep)))
	;; 		   (if (proc? info) info #f)))
	 (stepparts      (string-match (regexp "^(\\{([^\\}\\{]*)\\}\\s*|)(.*)$") stepinfo))
	 (stepparams     (if (and (list? stepparts)
︙
78
79
80
81
82
83
84
85


86
87



88
89
90
91
92
93
94
95
96
97
98
99
100
101
102



103

104
105
106
107
108
109
110

105
106
107
108
109
110
111

112
113
114

115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135

136
137
138
139
140
141
142
143








-
+
+

-
+
+
+















+
+
+
-
+








	    (lambda ()
	      (print ";; logpro file extracted from testconfig\n"
		     ";;")
	      (print tconfig-logpro)))
	  (set! logpro-used #t)))
    
    ;; NB// can safely assume we are in test-area directory
    (debug:print 4 *default-log-port* "ezsteps:\n stepname: " stepname " stepinfo: " stepinfo " stepparts: " stepparts
    (debug:print 4 *default-log-port* "ezsteps:\n stepname: " stepname " stepinfo: " stepinfo
		 " stepparts: " stepparts
		 " stepparams: " stepparams " stepcmd: " stepcmd)
    

    (if stepmode (ezsteps:create-step-script envdbf stepname prevstepname stepmode stepcmd))

    ;; ;; first source the previous environment
    ;; (let ((prev-env (conc ".ezsteps/" prevstep (if (string-search (regexp "csh") 
    ;;      							 (get-environment-variable "SHELL")) ".csh" ".sh"))))
    ;;   (if (and prevstep (common:file-exists? prev-env))
    ;;       (set! script (conc script "source " prev-env))))
    
    ;; call the command using mt_ezstep
    ;; (set! script (conc "mt_ezstep " stepname " " (if prevstep prevstep "x") " " stepcmd))
    
    (debug:print 4 *default-log-port* "script: " script)
    (rmt:teststep-set-status! run-id test-id stepname "start" "-" #f #f)
    ;; now launch the actual process
    (call-with-environment-variables 
     (list (cons "PATH" (conc (get-environment-variable "PATH") ":.")))
     (lambda () ;; (process-run "/bin/bash" "-c" "exec ls -l /tmp/foobar > /tmp/delme-more.log 2>&1")
       (let* ((cmd (conc (if stepmode
			     stepname
			     stepcmd)
       (let* ((cmd (conc stepcmd " > " stepname ".log 2>&1")) ;; >outfile 2>&1 
			 " > " stepname ".log 2>&1")) ;; >outfile 2>&1 
	      (pid #f))
	 (let ((proc (lambda ()
		       (set! pid (process-run "/bin/bash" (list "-c" cmd))))))
	   (if subrun
               (begin
                 (debug:print-info 0 *default-log-port* "Running without MT_.* environment variables.")
                 (common:propogate-mt-vars-to-subrun proc '("MT_TARGET" "MT_LINKTREE" "MT_RUNNAME")))
︙

Modified launch.scm from [2c7bba3a8e] to [740cbb909d].

︙
166
167
168
169
170
171
172

173



174
175
176
177
178
179
180
181
182
183
184


185
186
187
188
189
190
191

166
167
168
169
170
171
172
173

174
175
176
177
178
179
180
181
182
183
184
185
186

187
188
189
190
191
192
193
194
195








+
-
+
+
+










-
+
+








              (set! ezsteps #t) ;; set the needed flag
	      (set! ezstepslst
                    (append (or ezstepslst '())
                            (list (list "subrun" (conc "{subrun=true} " mt-cmd)))))))

	;; process the ezsteps
	(if ezsteps
	    (let* ((envdbf        (conc "/tmp/."(current-user-name)"-"(current-process-id)"-"run-id"-"test-id".db"))
	    (let* ((all-steps-dat (make-hash-table))) ;; keep all the info around as stepname ==> alist; where  'params is the params list (add other stuff as needed)
		   (all-steps-dat (make-hash-table))) ;; keep all the info around as stepname ==> alist;
	                                              ;;; where  'params is the params list (add other
	                                              ;;; stuff as needed)
	      (if (not (common:file-exists? ".ezsteps"))(create-directory ".ezsteps"))
	      ;; if ezsteps was defined then we are sure to have at least one step but check anyway
	      (if (not (> (length ezstepslst) 0))
		  (debug:print-error 0 *default-log-port* "ezsteps defined but ezstepslst is zero length")
		  (let loop ((ezstep (car ezstepslst))
			     (tal    (cdr ezstepslst))
			     (prevstep #f))
                    (debug:print-info 0 *default-log-port* "Processing ezstep \"" (string-intersperse ezstep " ") "\"")
		    ;; check exit-info (vector-ref exit-info 1)
		    (if (launch:einf-exit-status exit-info) ;; (vector-ref exit-info 1)
			(let* ((logpro-used (launch:runstep ezstep run-id test-id exit-info m tal testconfig all-steps-dat))
			(let* ((logpro-used (launch:runstep ezstep run-id test-id exit-info m
							    tal testconfig all-steps-dat prevstep envdbf))
			       (stepname    (car ezstep))
			       (stepparms   (hash-table-ref all-steps-dat stepname)))
			  (setenv "MT_STEP_NAME" stepname)
			  (pp (hash-table->alist all-steps-dat))
			  ;; if logpro-used read in the stepname.dat file
			  (if (and logpro-used (common:file-exists? (conc stepname ".dat")))
			      (launch:load-logpro-dat run-id test-id stepname))
︙

Modified megatest.scm from [9752b7fe00] to [5ccef97439].

︙
866
867
868
869
870
871
872
873




874
875
876
877
878
879
880

866
867
868
869
870
871
872

873
874
875
876
877
878
879
880
881
882
883








-
+
+
+
+








(let ((envcap (args:get-arg "-envcap")))
  (if envcap
      (let* ((db      (env:open-db (if (null? remargs) "envdat.db" (car remargs)))))
	(env:save-env-vars db envcap)
	(env:close-database db)
	(set! *didsomething* #t))))

;; delta "language" will eventually be res=a+b-c but for now it is just res=a-b 
;; delta "language" will eventually be res=a+b-c but for now it is just res=a-b
;;
;; db file can be stuck on the end of the command line:
;;   megatest -envdelta start-end -dumpmode bash -o .ezsteps/step5.sh /tmp/myfile.db 
;;
(let ((envdelta (args:get-arg "-envdelta")))
  (if envdelta
      (let ((match (string-split envdelta "-")));; (string-match "([a-z0-9_]+)=([a-z0-9_\\-,]+)" envdelta)))
	(if (not (null? match))
	    (let* ((db        (env:open-db (if (null? remargs) "envdat.db" (car remargs))))
		   ;; (resctx    (cadr match))
︙

This page was generated in about 0.137s by Fossil 2.24 [452f8e1ffc] 2024-03-26 15:36:00