Megatest

Check-in [268a3235ad]
Login
☰Home Timeline Files Branches Tags Tickets Wiki Login
Overview
Comment:Merged in fixes from v1.55, fixed couple queries related to jobgroups
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | inmem-per-run-db
Files: files | file ages | folders
SHA1: 268a3235ad4f6234b44fbe42b0f33ded49b559a7
User & Date: matt on 2013-12-09 22:57:57
Other Links: branch diff | manifest | tags
Context
2014-01-23
23:07
Merged v1.55 into inmem-per-run-db check-in: 120fb16c21 user: matt tags: inmem-per-run-db
2013-12-09
22:57
Merged in fixes from v1.55, fixed couple queries related to jobgroups check-in: 268a3235ad user: matt tags: inmem-per-run-db
21:30
Fixed couple more bugs in jobgroups check-in: 8b39e9feb6 user: matt tags: v1.55
08:54
fixed typo check-in: f6c7f13b18 user: mrwellan tags: inmem-per-run-db
Changes
Hide Diffs Side-by-Side Diffs Ignore Whitespace Patch

Modified db.scm from [4a10ed63ed] to [e6985c5610].

︙
1347
1348
1349
1350
1351
1352
1353
1354


1355







1356
1357
1358
1359
1360
1361







1362
1363
1364
1365
1366
1367
1368

1347
1348
1349
1350
1351
1352
1353

1354
1355
1356
1357
1358
1359
1360
1361
1362
1363






1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377








-
+
+

+
+
+
+
+
+
+
-
-
-
-
-
-
+
+
+
+
+
+
+








     (db:get-db dbstruct run-id)
     "SELECT count(id) FROM tests WHERE state in ('RUNNING','LAUNCHED','REMOTEHOSTSTART') AND run_id=?;" run-id)
    res))

(define (db:get-count-tests-running-in-jobgroup dbstruct run-id jobgroup)
  (if (not jobgroup)
      0 ;; 
      (let ((res 0))
      (let ((res        0)
	    (testnames '()))
	(sqlite3:for-each-row
	 (lambda (testname)
	   (set! testnames (cons testname testnames)))
	 (db:get-db dbstruct #f)
	 "SELECT testname FROM test_meta WHERE jobgroup=?;" 
	 jobgroup)
	(if (not (null? testnames))
	    (sqlite3:for-each-row
	 (lambda (count)
	   (set! res count))
	 (db:get-db dbstruct run-id)
	 "SELECT count(id) FROM tests WHERE state = 'RUNNING' OR state = 'LAUNCHED' OR state = 'REMOTEHOSTSTART'
             AND testname in (SELECT testname FROM test_meta WHERE jobgroup=?);"
	 jobgroup)
	     (lambda (count)
	       (set! res count))
	     (db:get-db dbstruct run-id)
	     (conc "SELECT count(id) FROM tests WHERE state in ('RUNNING','LAUNCHED','REMOTEHOSTSTART')"
		   " AND testname in ('"
		   (string-intersperse testnames "','")
		   "');")))
	res)))

;; done with run when:
;;   0 tests in LAUNCHED, NOT_STARTED, REMOTEHOSTSTART, RUNNING
(define (db:estimated-tests-remaining dbstruct run-id)
  (let ((res 0))
    (sqlite3:for-each-row
︙
1895
1896
1897
1898
1899
1900
1901
1902
1903


1904
1905

1906
1907
1908
1909
1910
1911
1912

1904
1905
1906
1907
1908
1909
1910


1911
1912
1913

1914
1915
1916
1917
1918
1919
1920
1921








-
-
+
+

-
+








;; Tests meta data
;;======================================================================

;; read the record given a testname
(define (db:testmeta-get-record dbstruct testname)
  (let ((res #f))
    (sqlite3:for-each-row
     (lambda (id testname author owner description reviewed iterated avg_runtime avg_disk tags)
       (set! res (vector id testname author owner description reviewed iterated avg_runtime avg_disk tags)))
     (lambda (id testname author owner description reviewed iterated avg_runtime avg_disk tags jobgroup)
       (set! res (vector id testname author owner description reviewed iterated avg_runtime avg_disk tags jobgroup)))
     (db:get-db dbstruct #f)
     "SELECT id,testname,author,owner,description,reviewed,iterated,avg_runtime,avg_disk,tags FROM test_meta WHERE testname=?;"
     "SELECT id,testname,author,owner,description,reviewed,iterated,avg_runtime,avg_disk,tags,jobgroup FROM test_meta WHERE testname=?;"
     testname)
    res))

;; create a new record for a given testname
(define (db:testmeta-add-record dbstruct testname)
  (sqlite3:execute (db:get-db dbstruct #f) "INSERT OR IGNORE INTO test_meta (testname,author,owner,description,reviewed,iterated,avg_runtime,avg_disk,tags) VALUES (?,'','','','','','','','');" testname))
︙

Modified runs.scm from [ace0f064f9] to [c47d5b284a].

︙
163
164
165
166
167
168
169
170




171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190

191
192


193
194
195
196
197
198
199

163
164
165
166
167
168
169

170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194


195
196
197
198
199
200
201
202
203








-
+
+
+
+




















+
-
-
+
+









(define (runs:can-run-more-tests run-id jobgroup max-concurrent-jobs)
  (thread-sleep! (cond
		  ((> *runs:can-run-more-tests-count* 20) 2);; obviously haven't had any work to do for a while
		  (else 0)))
  (let* ((num-running             (rmt:get-count-tests-running run-id))
	 (num-running-in-jobgroup (rmt:get-count-tests-running-in-jobgroup run-id jobgroup))
	 (job-group-limit         (config-lookup *configdat* "jobgroups" jobgroup)))
	 (job-group-limit         (let ((jobg-count (config-lookup *configdat* "jobgroups" jobgroup)))
				    (if (string? jobg-count)
					(string->number jobg-count)
					jobg-count))))
    (if (> (+ num-running num-running-in-jobgroup) 0)
	(set! *runs:can-run-more-tests-count* (+ *runs:can-run-more-tests-count* 1)))
    (if (not (eq? *last-num-running-tests* num-running))
	(begin
	  (debug:print 2 "max-concurrent-jobs: " max-concurrent-jobs ", num-running: " num-running)
	  (set! *last-num-running-tests* num-running)))
    (if (not (eq? 0 *globalexitstatus*))
	(list #f num-running num-running-in-jobgroup max-concurrent-jobs job-group-limit)
	(let ((can-not-run-more (cond
				 ;; if max-concurrent-jobs is set and the number running is greater 
				 ;; than it than cannot run more jobs
				 ((and max-concurrent-jobs (>= num-running max-concurrent-jobs))
				  (if (runs:lownoise "mcj msg" 60)
				      (debug:print 0 "WARNING: Max running jobs exceeded, current number running: " num-running 
						   ", max_concurrent_jobs: " max-concurrent-jobs))
				  #t)
				 ;; if job-group-limit is set and number of jobs in the group is greater
				 ;; than the limit then cannot run more jobs of this kind
				 ((and job-group-limit
				       (>= num-running-in-jobgroup job-group-limit))
				  (if (runs:lownoise (conc "maxjobgroup " jobgroup) 60)
				  (debug:print 1 "WARNING: number of jobs " num-running-in-jobgroup 
					       " in " jobgroup " exceeded, will not run " (tests:testqueue-get-testname test-record))
				      (debug:print 1 "WARNING: number of jobs " num-running-in-jobgroup 
						   " in jobgroup \"" jobgroup "\" exceeds limit of " job-group-limit))
				  #t)
				 (else #f))))
	  (list (not can-not-run-more) num-running num-running-in-jobgroup max-concurrent-jobs job-group-limit)))))

;;  test-names: Comma separated patterns same as test-patts but used in selection 
;;              of tests to run. The item portions are not respected.
;;              FIXME: error out if /patt specified
︙
788
789
790
791
792
793
794
795

796
797
798
799
800
801
802

792
793
794
795
796
797
798

799
800
801
802
803
804
805
806








-
+








      ;;       (set! last-time-incomplete (current-seconds))
      ;;       (cdb:remote-run db:find-and-mark-incomplete #f)))

      ;; (print "Top of loop, hed=" hed ", tal=" tal " ,reruns=" reruns)
      (let* ((test-record (hash-table-ref test-records hed))
	     (test-name   (tests:testqueue-get-testname test-record))
	     (tconfig     (tests:testqueue-get-testconfig test-record))
	     (jobgroup    (config-lookup tconfig "requirements" "jobgroup"))
	     (jobgroup    (config-lookup tconfig "test_meta" "jobgroup"))
	     (testmode    (let ((m (config-lookup tconfig "requirements" "mode")))
			    (if m (string->symbol m) 'normal)))
	     (waitons     (tests:testqueue-get-waitons    test-record))
	     (priority    (tests:testqueue-get-priority   test-record))
	     (itemdat     (tests:testqueue-get-itemdat    test-record)) ;; itemdat can be a string, list or #f
	     (items       (tests:testqueue-get-items      test-record))
	     (item-path   (item-list->path itemdat))
︙
1455
1456
1457
1458
1459
1460
1461
1462

1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474

1475
1476
1477
1478
1479
1480
1481

1459
1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477

1478
1479
1480
1481
1482
1483
1484
1485








-
+











-
+








;;======================================================================

;; Update the test_meta table for this test
(define (runs:update-test_meta test-name test-conf)
  (let ((currrecord (rmt:testmeta-get-record test-name)))
    (if (not currrecord)
	(begin
	  (set! currrecord (make-vector 10 #f))
	  (set! currrecord (make-vector 11 #f))
	  (rmt:testmeta-add-record test-name)))
    (for-each 
     (lambda (key)
       (let* ((idx (cadr key))
	      (fld (car  key))
	      (val (config-lookup test-conf "test_meta" fld)))
	 ;; (debug:print 5 "idx: " idx " fld: " fld " val: " val)
	 (if (and val (not (equal? (vector-ref currrecord idx) val)))
	     (begin
	       (print "Updating " test-name " " fld " to " val)
	       (rmt:testmeta-update-field test-name fld val)))))
     '(("author" 2)("owner" 3)("description" 4)("reviewed" 5)("tags" 9)))))
     '(("author" 2)("owner" 3)("description" 4)("reviewed" 5)("tags" 9)("jobgroup" 10)))))

;; Update test_meta for all tests
(define (runs:update-all-test_meta db)
  (let ((test-names (tests:get-all))) ;; (tests:get-valid-tests)))
    (for-each 
     (lambda (test-name)
       (let* ((test-conf    (mt:lazy-read-test-config test-name)))
︙

Modified tests/fullrun/megatest.config from [eb4e1b22b2] to [991ad5a94e].

︙
119
120
121
122
123
124
125





119
120
121
122
123
124
125
126
127
128
129








+
+
+
+

## -or-
## name /path/to/area
[disks]
disk0 /foobarbazz
[include config/mt_include_2.config]

[include #{getenv USER}_testing.config]

[jobgroups]
sqlite3 3
blockz  5

Modified tests/fullrun/tests/blocktestxz/testconfig from [ea79db52b2] to [689bce3544].

︙
14
15
16
17
18
19
20



14
15
16
17
18
19
20
21
22








+
+

description This test will fail causing the dependent test "testxz"\
 to never run. This triggers the code that must determine\
 that a test will never be run and thus remove it from\
 the queue of tests to be run.

tags first,single
reviewed 1/1/1965

jobgroup blockz

Modified tests/fullrun/tests/priority_1/testconfig from [a4d944cb23] to [737fc2258c]. 

1
2
3
4
5
6
7

8
9
10
11
12
13

1
2
3
4
5
6
7
8
9
10
11
12
13
14








+







[setup]
runscript main.sh

[requirements]
priority 1

[test_meta]
jobgroup sqlite3
author matt
owner  bob
description This test checks that a multi-lineitem test with mix of pass and non-fail rolls up a PASS

tags first,single
reviewed 09/10/2011, by Matt

Modified tests/fullrun/tests/priority_2/testconfig from [6cbcfb8c99] to [7fc8d055ec]. 

1
2
3
4
5
6
7
8
9

10
11
12
13
14
15

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16










+







[setup]
runscript main.sh

[requirements]
priority 2
# runtimelim 1d 1h 1m 10s
runtimelim 20s

[test_meta]
jobgroup sqlite3
author matt
owner  bob
description This test checks that a multi-lineitem test with mix of pass and non-fail rolls up a PASS

tags first,single
reviewed 09/10/2011, by Matt

Modified tests/fullrun/tests/priority_3/testconfig from [3693d6b2ed] to [392fa56879]. 

1
2
3
4
5
6
7
8

9
10
11
12
13
14

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15









+







[setup]
runscript main.sh

[requirements]
priority 3


[test_meta]
jobgroup sqlite3
author matt
owner  bob
description This test checks that a multi-lineitem test with mix of pass and non-fail rolls up a PASS

tags first,single
reviewed 09/10/2011, by Matt

Modified tests/fullrun/tests/priority_4/testconfig from [331e061c45] to [0f3ea908bb]. 

1
2
3
4
5
6
7

8
9
10
11
12
13

1
2
3
4
5
6
7
8
9
10
11
12
13
14








+







[setup]
runscript main.sh

[requirements]
priority 4

[test_meta]
jobgroup sqlite3
author matt
owner  bob
description This test checks that a multi-lineitem test with mix of pass and non-fail rolls up a PASS

tags first,single
reviewed 09/10/2011, by Matt

Modified tests/fullrun/tests/sqlitespeed/testconfig from [a64305aaab] to [d7b60872d2]. 

1
2
3
4
5
6
7
8
9
10
11



1
2
3
4
5
6
7
8
9
10
11
12
13












+
+

[setup]
runscript runscript.rb
tags non important,dumb junk

[requirements]
waiton    runfirst

[items]
MANYITEMS [system (env > envfile.txt;echo aa ab ac ad ae af ag ah ai)]
# BORKED  

[test_meta]
jobgroup sqlite3

This page was generated in about 0.138s by Fossil 2.24 [452f8e1ffc] 2024-03-26 15:36:00