Overview
Comment: | Rollup to test_data completed. Rebuild db reworked |
---|---|
Downloads: | Tarball | ZIP archive | SQL archive |
Timelines: | family | ancestors | descendants | both | trunk | v1.24 |
Files: | files | file ages | folders |
SHA1: |
d406fee8c4c01b11c38efdf878fa830b |
User & Date: | matt on 2011-09-12 00:05:43 |
Other Links: | manifest | tags |
Context
2011-09-12
| ||
06:51 | Partial implementation of test steps as text box check-in: ea880f01b9 user: matt tags: trunk | |
00:05 | Rollup to test_data completed. Rebuild db reworked check-in: d406fee8c4 user: matt tags: trunk, v1.24 | |
2011-09-11
| ||
12:51 | Starting point for server implemntation check-in: 598ddd3327 user: matt tags: trunk | |
Changes
Modified db.scm from [b9e48d30f0] to [dacaca63c6].
︙ | ︙ | |||
60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | final_logf TEXT DEFAULT 'logs/final.log', logdat BLOB, run_duration INTEGER DEFAULT 0, comment TEXT DEFAULT '', event_time TIMESTAMP, fail_count INTEGER DEFAULT 0, pass_count INTEGER DEFAULT 0, CONSTRAINT testsconstraint UNIQUE (run_id, testname, item_path) );") (sqlite3:execute db "CREATE INDEX tests_index ON tests (run_id, testname);") (sqlite3:execute db "CREATE VIEW runs_tests AS SELECT * FROM runs INNER JOIN tests ON runs.id=tests.run_id;") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS test_steps (id INTEGER PRIMARY KEY, test_id INTEGER, stepname TEXT, state TEXT DEFAULT 'NOT_STARTED', status TEXT DEFAULT 'n/a',event_time TIMESTAMP, comment TEXT DEFAULT '', CONSTRAINT test_steps_constraint UNIQUE (test_id,stepname,state));") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS extradat (id INTEGER PRIMARY KEY, run_id INTEGER, key TEXT, val TEXT);") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER PRIMARY KEY, var TEXT, val TEXT, CONSTRAINT metadat_constraint UNIQUE (var));") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS access_log (id INTEGER PRIMARY KEY, user TEXT, accessed TIMESTAMP, args TEXT);") (patch-db db) (patch-db db) ;; yes, need to do it twice BUG FIXME ;; Must do this *after* running patch db (db:set-var db "MEGATEST_VERSION" megatest-version) )) db)) ;;====================================================================== ;; TODO: ;; put deltas into an assoc list with version numbers ;; apply all from last to current ;;====================================================================== (define (patch-db db) (handle-exceptions exn (begin (print "Exception: " exn) (print "ERROR: Possible out of date schema, attempting to add table metadata...") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER, var TEXT, val TEXT, | > > > > > > > > > > > > > > > > > > > > > > > > | > | < > | < | > > > | < < < < | > | > > | | | | | < < < < | | < | | < > | | | | | | | | | 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | final_logf TEXT DEFAULT 'logs/final.log', logdat BLOB, run_duration INTEGER DEFAULT 0, comment TEXT DEFAULT '', event_time TIMESTAMP, fail_count INTEGER DEFAULT 0, pass_count INTEGER DEFAULT 0, first_err TEXT, first_warn TEXT, CONSTRAINT testsconstraint UNIQUE (run_id, testname, item_path) );") (sqlite3:execute db "CREATE INDEX tests_index ON tests (run_id, testname);") (sqlite3:execute db "CREATE VIEW runs_tests AS SELECT * FROM runs INNER JOIN tests ON runs.id=tests.run_id;") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS test_steps (id INTEGER PRIMARY KEY, test_id INTEGER, stepname TEXT, state TEXT DEFAULT 'NOT_STARTED', status TEXT DEFAULT 'n/a',event_time TIMESTAMP, comment TEXT DEFAULT '', CONSTRAINT test_steps_constraint UNIQUE (test_id,stepname,state));") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS extradat (id INTEGER PRIMARY KEY, run_id INTEGER, key TEXT, val TEXT);") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER PRIMARY KEY, var TEXT, val TEXT, CONSTRAINT metadat_constraint UNIQUE (var));") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS access_log (id INTEGER PRIMARY KEY, user TEXT, accessed TIMESTAMP, args TEXT);") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS test_meta (id INTEGER PRIMARY KEY, testname TEXT DEFAULT '', author TEXT DEFAULT '', owner TEXT DEFAULT '', description TEXT DEFAULT '', reviewed TIMESTAMP, iterated TEXT DEFAULT '', avg_runtime REAL, avg_disk REAL, tags TEXT DEFAULT '', CONSTRAINT test_meta_constraint UNIQUE (testname));") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS test_data (id INTEGER PRIMARY KEY, test_id INTEGER, category TEXT DEFAULT '', variable TEXT, value REAL, expected REAL, tol REAL, units TEXT, comment TEXT DEFAULT '', status TEXT DEFAULT 'n/a', CONSTRAINT test_data UNIQUE (test_id,category,variable));") (patch-db db) (patch-db db) ;; yes, need to do it twice BUG FIXME ;; Must do this *after* running patch db (db:set-var db "MEGATEST_VERSION" megatest-version) )) db)) ;;====================================================================== ;; TODO: ;; put deltas into an assoc list with version numbers ;; apply all from last to current ;;====================================================================== (define (patch-db db) (handle-exceptions exn (begin (print "Exception: " exn) (print "ERROR: Possible out of date schema, attempting to add table metadata...") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER, var TEXT, val TEXT, CONSTRAINT metadat_constraint UNIQUE (var));") (if (not (db:get-var db "MEGATEST_VERSION")) (db:set-var db "MEGATEST_VERSION" 1.17))) (let ((mver (db:get-var db "MEGATEST_VERSION")) (test-meta-def "CREATE TABLE IF NOT EXISTS test_meta (id INTEGER PRIMARY KEY, testname TEXT DEFAULT '', author TEXT DEFAULT '', owner TEXT DEFAULT '', description TEXT DEFAULT '', reviewed TIMESTAMP, iterated TEXT DEFAULT '', avg_runtime REAL, avg_disk REAL, tags TEXT DEFAULT '', CONSTRAINT test_meta_constraint UNIQUE (testname));")) (print "Current schema version: " mver " current megatest version: " megatest-version) (cond ((not mver) (print "Adding megatest-version to metadata") ;; Need to recreate the table (sqlite3:execute db "DROP TABLE IF EXISTS metadat;") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER, var TEXT, val TEXT, CONSTRAINT metadat_constraint UNIQUE (var));") (db:set-var db "MEGATEST_VERSION" 1.17)) ((< mver 1.21) (sqlite3:execute db "DROP TABLE IF EXISTS metadat;") (sqlite3:execute db "CREATE TABLE IF NOT EXISTS metadat (id INTEGER, var TEXT, val TEXT, CONSTRAINT metadat_constraint UNIQUE (var));") (db:set-var db "MEGATEST_VERSION" 1.21) ;; set before, just in case the changes are already applied (sqlite3:execute db test-meta-def) (for-each (lambda (stmt) (sqlite3:execute db stmt)) (list "ALTER TABLE tests ADD COLUMN first_err TEXT;" "ALTER TABLE tests ADD COLUMN first_warn TEXT;" ))) ((< mver 1.24) (db:set-var db "MEGATEST_VERSION" 1.24) (sqlite3:execute db "DROP TABLE IF EXISTS test_data;") (sqlite3:execute db "DROP TABLE IF EXISTS test_meta;") (sqlite3:execute db test-meta-def) (sqlite3:execute db "CREATE TABLE IF NOT EXISTS test_data (id INTEGER PRIMARY KEY, test_id INTEGER, category TEXT DEFAULT '', variable TEXT, value REAL, expected REAL, tol REAL, units TEXT, comment TEXT DEFAULT '', status TEXT DEFAULT 'n/a', CONSTRAINT test_data UNIQUE (test_id,category,variable));")) ((< mver megatest-version) (db:set-var db "MEGATEST_VERSION" megatest-version)))))) ;;====================================================================== ;; meta get and set vars ;;====================================================================== ;; returns number if string->number is successful, string otherwise (define (db:get-var db var) |
︙ | ︙ | |||
487 488 489 490 491 492 493 | ;;====================================================================== (define (make-db:test-data)(make-vector 10)) (define-inline (db:test-data-get-id vec) (vector-ref vec 0)) (define-inline (db:test-data-get-test_id vec) (vector-ref vec 1)) (define-inline (db:test-data-get-category vec) (vector-ref vec 2)) (define-inline (db:test-data-get-variable vec) (vector-ref vec 3)) (define-inline (db:test-data-get-value vec) (vector-ref vec 4)) | | > | 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 | ;;====================================================================== (define (make-db:test-data)(make-vector 10)) (define-inline (db:test-data-get-id vec) (vector-ref vec 0)) (define-inline (db:test-data-get-test_id vec) (vector-ref vec 1)) (define-inline (db:test-data-get-category vec) (vector-ref vec 2)) (define-inline (db:test-data-get-variable vec) (vector-ref vec 3)) (define-inline (db:test-data-get-value vec) (vector-ref vec 4)) (define-inline (db:test-data-get-expected vec) (vector-ref vec 5)) (define-inline (db:test-data-get-tol vec) (vector-ref vec 6)) (define-inline (db:test-data-get-units vec) (vector-ref vec 7)) (define-inline (db:test-data-get-comment vec) (vector-ref vec 8)) (define-inline (db:test-data-get-status vec) (vector-ref vec 9)) (define (db:csv->test-data db test-id csvdata) (debug:print 4 "test-id " test-id ", csvdata: " csvdata) (let ((csvlist (csv->list (make-csv-reader (open-input-string csvdata) '((strip-leading-whitespace? #t) (strip-trailing-whitespace? #t)) )))) ;; (csv->list csvdata))) (for-each (lambda (csvrow) (let* ((padded-row (take (append csvrow '(#f #f #f #f #f #f #f #f)) 8)) |
︙ | ︙ | |||
530 531 532 533 534 535 536 | (set! status ;; NB// need to assess each one (i.e. not return operator since need to act if not valid op. (case (string->symbol tol) ;; tol should be >, <, >=, <= ((>) (if (> value expected) "pass" "fail")) ((<) (if (< value expected) "pass" "fail")) ((>=) (if (>= value expected) "pass" "fail")) ((<=) (if (<= value expected) "pass" "fail")) (else (conc "ERROR: bad tol comparator " tol)))))) | | | | | | 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 | (set! status ;; NB// need to assess each one (i.e. not return operator since need to act if not valid op. (case (string->symbol tol) ;; tol should be >, <, >=, <= ((>) (if (> value expected) "pass" "fail")) ((<) (if (< value expected) "pass" "fail")) ((>=) (if (>= value expected) "pass" "fail")) ((<=) (if (<= value expected) "pass" "fail")) (else (conc "ERROR: bad tol comparator " tol)))))) (sqlite3:execute db "INSERT OR REPLACE INTO test_data (test_id,category,variable,value,expected,tol,units,comment,status) VALUES (?,?,?,?,?,?,?,?,?);" test-id category variable value expected tol units (if comment comment "") status))) csvlist))) ;; get a list of test_data records matching categorypatt (define (db:read-test-data db test-id categorypatt) (let ((res '())) (sqlite3:for-each-row (lambda (id test_id category variable value expected tol units comment status) (set! res (cons res (vector id test_id category variable value expected tol units comment status)))) db "SELECT id,test_id,category,variable,value,expected,tol,units,comment,status FROM test_data WHERE test_id=? AND category LIKE ?;" test-id categorypatt) (reverse res))) (define (db:load-test-data db run-id test-name itemdat) (let* ((item-path (item-list->path itemdat)) (testdat (db:get-test-info db run-id test-name item-path)) (test-id (if testdat (db:test-get-id testdat) #f))) ;; (debug:print 1 "Enter records to insert in the test_data table, seven fields, comma separated per line") |
︙ | ︙ | |||
765 766 767 768 769 770 771 | (let ((test-data '()) (curr-test-name #f)) (sqlite3:for-each-row (lambda (testname item_path category variable value comment status) (set! curr-test-name testname) (set! test-data (append test-data (list (list testname item_path category variable value comment status))))) db | | | 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 | (let ((test-data '()) (curr-test-name #f)) (sqlite3:for-each-row (lambda (testname item_path category variable value comment status) (set! curr-test-name testname) (set! test-data (append test-data (list (list testname item_path category variable value comment status))))) db "SELECT testname,item_path,category,variable,td.value AS value,expected,tol,units,td.comment AS comment,td.status AS status FROM test_data AS td INNER JOIN tests ON tests.id=td.test_id WHERE test_id=?;" test-id) (if curr-test-name (set! results (append results (list (cons curr-test-name test-data))))) )) test-ids) (system (conc "mkdir -p " tempdir)) ;; (pp results) |
︙ | ︙ |
Modified megatest.scm from [53a5275b02] to [847bc202b1].
︙ | ︙ | |||
37 38 39 40 41 42 43 | Run data :runname : required, name for this particular test run :state : required if updating step state; e.g. start, end, completed :status : required if updating step status; e.g. pass, fail, n/a Values and record errors and warnings -set-values : update or set values in the megatest db | > > | | | | | 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | Run data :runname : required, name for this particular test run :state : required if updating step state; e.g. start, end, completed :status : required if updating step status; e.g. pass, fail, n/a Values and record errors and warnings -set-values : update or set values in the megatest db :category : set the category field (optional) :variable : set the variable name (optional) :value : value measured (required) :expected : value expected (required) :tol : |value-expect| <= tol (required, can be <, >, >=, <= or number) :units : name of the units for value, expected_value etc. (optional) :first_err : record an error message :first_warn : record a warning message Arbitrary test data loading -load-test-data : read test specific data for storage in the test_data table from standard in. Each line is comma delimited with four fields category,variable,value,comment |
︙ | ︙ | |||
109 110 111 112 113 114 115 116 117 118 | "-logpro" "-m" "-rerun" "-days" "-rename-run" "-to" ;; values and messages ":first_err" ":first_warn" ":value" | > > | | 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | "-logpro" "-m" "-rerun" "-days" "-rename-run" "-to" ;; values and messages ":category" ":variable" ":first_err" ":first_warn" ":value" ":expected" ":tol" ":units" ;; misc "-extract-ods" "-debug" ;; for *verbosity* > 2 ) (list "-h" |
︙ | ︙ | |||
647 648 649 650 651 652 653 | (else status))) ;; transfer relevant keys into a hash to be passed to test-set-status! ;; could use an assoc list I guess. (otherdata (let ((res (make-hash-table))) (for-each (lambda (key) (if (args:get-arg key) (hash-table-set! res key (args:get-arg key)))) | | | 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 | (else status))) ;; transfer relevant keys into a hash to be passed to test-set-status! ;; could use an assoc list I guess. (otherdata (let ((res (make-hash-table))) (for-each (lambda (key) (if (args:get-arg key) (hash-table-set! res key (args:get-arg key)))) (list ":value" ":tol" ":expected" ":first_err" ":first_warn" ":units" ":category" ":variable")) res))) (if (and (args:get-arg "-test-status") (or (not state) (not status))) (begin (debug:print 0 "ERROR: You must specify :state and :status with every call to -test-status\n" help) (sqlite3:finalize! db) |
︙ | ︙ |
Modified runs.scm from [77923710f4] to [3102091d6b].
︙ | ︙ | |||
212 213 214 215 216 217 218 219 220 221 222 223 224 225 | (sqlite3:execute db "UPDATE tests SET first_err=? WHERE run_id=? AND testname=? AND item_path=?;" val run-id test-name item-path))) ;; :first_warn (let ((val (hash-table-ref/default otherdat ":first_warn" #f))) (if val (sqlite3:execute db "UPDATE tests SET first_warn=? WHERE run_id=? AND testname=? AND item_path=?;" val run-id test-name item-path))) ;; need to update the top test record if PASS or FAIL and this is a subtest (if (and (not (equal? item-path "")) (or (equal? status "PASS") (equal? status "WARN") (equal? status "FAIL") (equal? status "WAIVED") (equal? status "RUNNING"))) | > > > > > > > > > > > > > > > > > > > | 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 | (sqlite3:execute db "UPDATE tests SET first_err=? WHERE run_id=? AND testname=? AND item_path=?;" val run-id test-name item-path))) ;; :first_warn (let ((val (hash-table-ref/default otherdat ":first_warn" #f))) (if val (sqlite3:execute db "UPDATE tests SET first_warn=? WHERE run_id=? AND testname=? AND item_path=?;" val run-id test-name item-path))) (let ((category (hash-table-ref/default otherdat ":category" "")) (variable (hash-table-ref/default otherdat ":variable" "")) (value (hash-table-ref/default otherdat ":value" #f)) (expected (hash-table-ref/default otherdat ":expected" #f)) (tol (hash-table-ref/default otherdat ":tol" #f)) (units (hash-table-ref/default otherdat ":units" ""))) (debug:print 4 "category: " category ", variable: " variable ", value: " value ", expected: " expected ", tol: " tol ", units: " units) (if (and value expected tol) ;; all three required (db:csv->test-data db test-id (conc category "," variable "," value "," expected "," tol "," units "," comment ",")))) ;; need to update the top test record if PASS or FAIL and this is a subtest (if (and (not (equal? item-path "")) (or (equal? status "PASS") (equal? status "WARN") (equal? status "FAIL") (equal? status "WAIVED") (equal? status "RUNNING"))) |
︙ | ︙ |
Modified tests/tests/runfirst/main.sh from [e833de03b3] to [e1e013c667].
︙ | ︙ | |||
15 16 17 18 19 20 21 | foo,bal,1.2,1.2,<,,Check for overload foo,alb,1.2,1.2,<=,Amps,This is the high power circuit test foo,abl,1.2,1.3,0.1 foo,bra,1.2,pass,silly stuff faz,bar,10,8mA,,,"this is a comment" EOF | | | 15 16 17 18 19 20 21 22 23 24 | foo,bal,1.2,1.2,<,,Check for overload foo,alb,1.2,1.2,<=,Amps,This is the high power circuit test foo,abl,1.2,1.3,0.1 foo,bra,1.2,pass,silly stuff faz,bar,10,8mA,,,"this is a comment" EOF $MT_MEGATEST -test-status :state COMPLETED :status $? -m "This is a test level comment" :value 10e6 :expected_value 1.1e6 :tol 100e3 :category nada :variable sillyvar :units mFarks :comment "This is the value/expected comment" # $MT_MEGATEST -test-status :state COMPLETED :status FAIL |