Preface

This book is organised as three sub-books; getting started, writing tests and reference.

License
    Copyright 2006-2017, Matthew Welland.

    This document is part of Megatest.

    Megatest is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    Megatest is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with Megatest.  If not, see <http://www.gnu.org/licenses/>.

Why Megatest?

The Megatest project was started for two reasons, the first was an immediate and pressing need for a generalized tool to manage a suite of regression tests and the second was the fact that I had written or maintained several such tools at different companies over the years. I thought a single open source tool, flexible enough to meet the needs of any team doing continuous integrating and or running a complex suite of tests for release qualification would solve some problems for me and for others.

-- Matt Welland, original author of the Megatest tool suite.

Megatest Design Philosophy

Megatest is a distributed system intended to provide the minimum needed resources to make writing a suite of tests and tasks for implementing continuous build for software, design engineering or process control (via owlfs for example) without being specialized for any specific problem space. Megatest in of itself does not know what constitutes a PASS or FAIL of a test or task. In most cases megatest is best used in conjunction with logpro or a similar tool to parse, analyze and decide on the test outcome.

  • Self-checking -Repeatable strive for directed or self-checking test as opposed to delta based tests.

  • Traceable - environment variables, host OS and other possibly influential variables are captured and kept recorded.

  • Immutable - once a test is run it cannot be easily overwritten or modified accidentally.

  • Repeatable - test results can be recreated in the future using all the original variables.

  • Relocatable - the testsuite or automation area can be checked out and the tests run anywhere in the disk hierarchy.

  • Encapsulated - the tests run in self-contained directories and all inputs and outputs to the process can be found in the run areas.

  • Deployable - a testsuite is self-contained and can be bundled with a software project.

Megatest Architecture

Data separation

All data to specify the tests and configure the system is stored in plain text config files. All system state is stored in an sqlite3 database.

Distributed Compute

Tests are launched using the launching system available for the distributed compute platform in use. A template script is provided which can launch jobs on local and remote Linux hosts. Currently megatest uses the network filesystem to call home to your master sqlite3 database. Megatest has been used with the Intel Netbatch system, the lsf batch system and it should be possible to use it with other similar systems.

Installation

Dependencies

Chicken scheme and a number of "eggs" are required for building Megatest. See the script installall.sh in the utils directory of the source distribution for an automated way to install everything needed for building Megatest on Linux.

Getting Started

Getting started with Megatest

Creating a testsuite or flow and your first test or task.

Megatest Internals

server.png

Road Map

Note 1: This road-map continues to evolve and subject to change without notice.

Here is a smattering of ideas for Megatest 2.0

  1. Add variable $MT_RUNPATH = $MT_LINKTREE/$MT_TARGET/$MT_RUNNAME 

[db]
api legacy|new

  1. One big lesson from the 1.63-1.65 generation was that the main.db, 1.db … model was really good at scaling. I’d like to combine that model with the current also-very-good model. Obviously this is a disruptive change. I think making the old model the default and the new model an option for at least one generation would be fair.

  2. Rigorous megatest.config and runconfig.config caching.

    1. Cache the configs in $MT_RUNPATH

    2. Following invocations of –run, -rerun* will calculate the new config but only overwrite the cached file IF changed

  3. If the cached file changes ALL existing tests go from COMPLETED → STALE, I’m not sure what to do about RUNNING tests

  4. !VARS in runconfigs are not exported to the environment. They are accessed via rget as if the ! was not there.

  5. Per test copy commands (crude example below is not correct). 

[testcopy]
%/iind% unison SRC DEST
% cp –r SRC DEST

  1. Test management via pkts (optional?)

    1. Control pkt types: run, kill, rerunclean, clean, archive, status?

    2. Status pkt types: ack, step, status_change

  2. Add nanomsg as a transport option .

RFC M01: Add ability to move runs to other Areas

Purpose: allow shrinking megatest.db data by moving runs to an alternate Megatest area with same keys.

Method: extend db sync to take a different megatest area as a destination.

Design:

  1. add param -destination [area|path]. when specified runs are copied to new area and removed from local db.

  2. the data move would involve these steps

    1. copy the run data to destination area megatest.db

    2. mark the run records as deleted, do not remove the run data on disk

  3. accessing the data would be by running dashboard in the satellite area

  4. future versions of Megatest dashboard should support displaying areas in a merged way.

  5. some new controls would be supported in the config

    1. [setup] ⇒ allow-runs [no|yes] ⇐= used to disallow runs

    2. [setup] ⇒ auto-migrate=[areaname|path] ⇐= used to automatically migrate data to a satellite area.

Branch: This work is taking place on branch v1.65-reduce-records

RFC M02: Move data into completed-runs.db

Purpose: shrink megatest.db data to enable lower load and higher performance.

Method: add a completed-runs.db and automatically move runs data from megatest.db to that db

Design:

  1. completed-runs.db is a full megatest database with complete schema

  2. the data move would involve these steps

    1. copy the run data to completed-runs.db

    2. remove the run data, first from /tmp/…/megatest.db and /tmp/…/megatest_ref.db, followed by megatest.db

  3. accessing the data would be unchanged for most operations.

  4. a mode -full-db will be added which when specified would attach the completed-runs.db to megatest.db before doing the query

  5. mechanisms for moving runs to/from the megatest.db would be added

    1. -reduce-records ⇒ move runs to completed-runs.db

    2. -restore-records ⇒ move runs from completed-runs.db to megatest.db

Branch: This work is taking place on branch v1.65-reduce-records

RFC M03: Automatic homehost migrations

Purpose: Automatically migrate homehost.

Method: Check that there are no tests running, launched or remotehoststart in past ½ hour then if not on homehost migrate the db to current host

Design:

  1. Check that the system is quiescent, i.e. that there are no runs in flight or recently run

  2. Create a lock

  3. Migrate the /tmp cache db to the current host

  4. Update the .homehost file

  5. Remove the lock

Branch: This work not yet started

Architecture Refactor

Goals

  1. Reduce load on the file system. Sqlite3 files on network filesystem can be a burden. [DONE]

  2. Reduce number of servers and frequency of start/stop. This is mostly an issue of clutter but also a reduction in "moving parts". [DONE]

  3. Coalesce activities to a single home host where possible. Give the user feedback that they have started the dashboard on a host other than the home host. [DONE]

  4. Reduce number of processes involved in managing running tests.

Changes Needed

  1. ACID compliant db will be on /tmp and synced to megatest.db with a five second max delay. [DONE]

  2. Read/writes to db for processes on homehost will go direct to /tmp megatest.db file. [DONE]

  3. Read/wites fron non-homehost processes will go through one server. Bulk reads (e.g. for dashboard or list-runs) will be cached on the current host in /tmp and synced from the home megatest.db in the testsuite area. [DONE]

  4. Db syncs rely on the target db file timestame minus some margin. [DONE]

  5. Since bulk reads do not use the server we can switch to simple RPC for the network transport. [DONE]

  6. Test running manager process extended to manage multiple running tests.

Current Items

ww05 - migrate to inmem-db

  1. Switch to inmem db with fast sync to on disk db’s [DONE]

  2. Server polls tasks table for next action

    1. Task table used for tracking runner process [Replaced by mtutil]

    2. Task table used for jobs to run [Replaced by mtutil]

    3. Task table used for queueing runner actions (remove runs, cleanRunExecute, etc) [Replaced by mtutil]

shifting, note that the preceding blank line is needed.

Index