MaxScale/system-test/Documentation/BUILD_TEST_DEBUG.md

Maxscale Continuous Integration and Test Automation

Basics

Jenkins at http://max-tst-01.mariadb.com:8089/ is the main tool for providing build and test services for Maxscale, but note that it cannot be accessed directly (see further down).

A set of virtual machines (VMs) is in use for all building and testing. The VMs are controlled by Vagrant and MDBCI.

3 types of VMs are supported:

• Libvirt/qemu
• AWS
• Docker containers

All Jenkins jobs are in Jenkins job Builder YAML format and are available here.

VMs for every build or test run are created from clean Vagrant box.

To speed up regular builds and tests there are a set of constantly running VMs:

• Build machine centos_7_libvirt_build
• Test VMs set centos_7_libvirt-mariadb-10.0-permanent

Build and test Jenkins jobs generate e-mails with results report. In addition, test run logs are stored here.

Accessing max-tst-01

Jenkins running on max-tst-01 can only be accessed from localhost. In order to be able to access it from your computer, an ssh tunnel needs to be setup.

• Ensure that your public key is available on max-tst-01.
• Then setup the ssh tunnel:

$ ssh -f -N -L 8089:127.0.0.1:8089 vagrant@max-tst-01.mariadb.com

After that, Jenkins on max-tst-01 can be accessed by browsing to http://localhost:8089.

In the following it is assumed that the above has been done.

Build

• Build builds Maxscale and creates a binary repository for one Linux distribution.

• Build_all builds Maxscale and creates binary repository for all supported Linux distributions.

The list of supported distributions (Vagrant boxes) is here.

Main Build and Build_all jobs parameters

To start a build, click on Build with Parameters to the left in the Jenkins panel.

scm_source

The place in Maxscale source code.

• for branch - put branch name here
• for commit ID - put Git Commit ID here
• for tag - put refs/tags/<tagName> here

box

The name of Vagrant box to create VM. Box name consists of the name of distribution, distribution version and the name of VM provider. Whole list of supported boxes can be found here

target

Then name of binary repository. Ensure that the name is reasonably unique so that there can be no unintended overwrites. If you are building a personal branch, then using the same value for target as is the value for scm_source should be fine.

Binaries will go to: http://max-tst-01.mariadb.com/ci-repository/<target>/mariadb-maxscale/<distribution_name>/<version>

cmake_flags

Debug build:

-DBUILD_TESTS=Y -DCMAKE_BUILD_TYPE=Debug -DFAKE_CODE=Y -DBUILD_AVRO=Y -DBUILD_CDC=Y

Release build:

-DBUILD_TESTS=N -DFAKE_CODE=N -DBUILD_AVRO=Y -DBUILD_CDC=Y

Build scripts automatically add all other necessary flags (e.g. -DPACKAGE=Y)

do_not_destroy_vm

If yes, then the VM will not be destroyed after the build. Choose that if you, for some specific reason (e.g. debugging), intend to log into the VM later.

repo_path

The location for binary repositories on max-tst-01. Usually no need to change.

try_alrady_running

If yes constantly running VM will be used for build instead of bringing clean machines from scratch. The name of VM is <box_name>_build.

If this VM is not running it will be created from clean box. If new build dependency is introduced and new package have to be installed (and build script is modified to do it) before the build it is necessary to destroy constantly running VM - in this case a new VM will be created and all stuff will be installed.

run_upgrade_test

If yes upgrade test will be executed after build.

Upgrade test is executed on a separate VM.

• First, old version of Maxsale is installed (parameter old_target),
• then upgrade to recently built version is executed,
• then Maxscale is started with minimum configuration (only CLI service is configured, no backend at all), and
• finally, maxadmin is executed to check that Maxscale is running and available.

Regular automatic builds

File ~/jobs_branches/build_all_branches.list on _max-tst-02.mariadb.com contains a list of regexes. If branch name matches one of the regex from this file build_all job will be executed for this branch every night.

Regular builds are executed with run_upgrade_test=yes

Test execution

• run_test creates a set of VMs: 1 VM for Maxscale, 4 VMs for Master/Slave setup and VMs for Galera cluster.

Main Run_test parameters

Before this job can be run, a build job must have been executed first.

target

The name of Maxscale binary repository.

The value used here, should be the same as the value used in target.

box

The name of Vagrant box to create VM.

The value used here, must be the same as the value used in box when the used target was built.

product

MariaDB or MySQL to be installed to all backend machines.

version

Version of MariaDB or MySQL to be installed on all MariaDB machines. The list of versions contains all MariaDB and MySQL major version. Selecting wrong version (e.g. 5.7 for MariaDB or 10.0 for MySQL) causes VM creation error and job failure.

do_not_destroy_vm

If yes all VMs will run until manually destoroy by Destroy job. It can be used for debugging. Do not forget to destroy VMs after debugging is finished.

name

The name of test run. The names of the VMs will be:

• <name>_maxscale,
• <name>_node_000, ..., <name>_node_003 and
• <name>_galera_000, ..., <name>_galera_003-

The name can be used to access VMs:

. ~/build-scripts/test/set_env_vagrant.sh <name>
ssh -i $<vm_name>_keyfile $<vm_name>_whoami@$<vm_name>_network

where <vm_name> can be 'maxscale', 'node_XXX' or 'galera_XXX'.

test_branch

The name of MaxScale repository branch to be used in the test run.

It is usually the same as the one used in scm_source.

slave_name

Host server to for VMs and maxscale-system-test execution

Name	Server
master	localhost
maxtst2	max-tst-02.mariadb.com
maxtst3	max-tst-03.mariadb.com

test_set

Defines tests to be run. See ctest documentation https://cmake.org/cmake/help/v3.7/manual/ctest.1.html for details.

Most common cases:

Arguments	Description
-I 1,5,,45,77	Execute tests from 1 to 5 and tests 45 and 77
-L HEAVY	Execute all tests with 'HEAVY' label
-LE UNSTABLE	Execute all tests except tests with 'UNSTABLE' label

Run_test_snapshot

This job uses already running VMs. The set of VMs have to have snapshot (by default snapshot name is 'clean'). Instead of bringing up all VMs from scratch this job only reverts VMs to the 'clean' state.

The name of running VMs set should be:

<box>-<product>-<version>-permanent

If there is no VMs set with such name it will be created automatically.

If case of broken VMs it can be destroyd with help of Destroy job and Run_test_snapshot should be restarted after it to create a new VMs set.

Only one VMs set on every server can be started for particular box, product and version combination. If two Run_test_snapshot jobs are runnig for the same box, product and version the second one will be waiting until the first job run ios finished. (job sets 'snapshot_lock').

If case if locak is not removed automatically (for example due to Jenkins crash) it can be removed manually:

rm ~/mdbci/centos_7_libvirt-mariadb-10.0-permanent_snapshot_lock

Run_test_labels and Run_test_snapshot_labels

The only difference from Run_test and Run_test_snapshot is the way to define set of tests to execute. *_labels jobs use checkboxes list of test labels.

Labels list have to be maintained manually. It can be created by https://github.com/mariadb-corporation/maxscale-jenkins-jobs/blob/master/create_labels_jobs.sh script.

Regular test runs

Test runs by timer

File

~/jobs_branches/run_test_branches.list

on max-tst-02.mariadb.com contains a list of regexes. If branch name matches one of regexes tests are executed for this branch every day.

The test set is defined also in this file.

Job print_branches_which_matches_regex_in_file http://localhost:8089/view/All/job/print_branches_which_matches_regex_in_file/build can be used to check the list of branches that match regexes.

Job weekly_smoke_run_test_matrix triggers build_regular and run_test_matrix http://localhost:8089/view/test/job/run_test_matrix/ every week for 'develop' branch with test set '-L LIGHT' (all tests with label 'LIGHT')

Test runs by GIT push

File

~/jobs_branches/on_push_maxscale_branches.list

on max-tst-02.mariadb.com contains a list of regexes. If branch name matches one of regexes tests are executed for this branch after every push.

Job print_branches_which_matches_regex_in_file http://localhost:8089/view/All/job/print_branches_which_matches_regex_in_file/build can be used to check the list of branches that match regexes (select on_push_maxscale_branches.list file for branches_list parameter)

Debugging

For regular debugging contsntly running set of VMs is recommended. See documentation here.

Another way is to use do_not_destroy=yes parameter of run_test job. After run_test job executing VMs stay running and can be accessed from the host server. See here for details.