How to run integration tests at warp speed using Docker and tmpfs

(Last Updated On: May 10, 2018)

Introduction

As previously explained, you can run database integration tests 20 times faster! The trick is to map the data directory in memory, and my previous article showed you what changes you need to do when you have a PostgreSQL or MySQL instance on your machine.

In this post, I’m going to expand the original idea, and show you how you can achieve the same goal using Docker and tmpfs.

MariaDB

Getting the Docker image

First of all, you need a Docker image for the database you want to run your integration tests on.

To see what Docker images you have on your machine, you need to run the docker images command:

> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
oracle/database     12.1.0.2-se2        b5f12a4d9ae0        9 days ago          11.1 GB
bash                latest              c2a000c8aa3c        11 days ago         12.8 MB
oraclelinux         latest              5a42e075a32b        3 weeks ago         225 MB
d4w/nsenter         latest              9e4f13a0901e        4 months ago        83.8 kB

Now, let’s get the latest MariaDB Docker image:

> docker pull mariadb
Status: Downloaded newer image for mariadb:latest

If we rerun docker images, we’ll see the MariaDB Docker image listed as well:

> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
oracle/database     12.1.0.2-se2        b5f12a4d9ae0        9 days ago          11.1 GB
bash                latest              c2a000c8aa3c        11 days ago         12.8 MB
mariadb             latest              7eca0e0b51c9        2 weeks ago         393 MB
oraclelinux         latest              5a42e075a32b        3 weeks ago         225 MB
d4w/nsenter         latest              9e4f13a0901e        4 months ago        83.8 kB

Running the database in a Docker container

To create a new Docker container, we need to use the docker run command:

docker run \
--name mariadb \
-p 3306:3306 \
--tmpfs /var/lib/mysql:rw \
-e MYSQL_ROOT_PASSWORD=admin \
-e MYSQL_USER=hibernate_orm_test \
-e MYSQL_PASSWORD=hibernate_orm_test \
-e MYSQL_DATABASE=hibernate_orm_test  \
-d mariadb

On Windows, you’ll have to use the ^ multi-line separator instead:

docker run ^
--name mariadb ^
-p 3306:3306 ^
--tmpfs /var/lib/mysql:rw ^
-e MYSQL_ROOT_PASSWORD=admin ^
-e MYSQL_USER=hibernate_orm_test ^
-e MYSQL_PASSWORD=hibernate_orm_test ^
-e MYSQL_DATABASE=hibernate_orm_test ^
-d mariadb

The arguments can be explained as follows:

  • --name is used to specify the name of the newly created container. We can then reference the container by this name when we want to stop or start it
  • -p 3306:3306 is used to map the Docker container port to the host machine port so we can access the MariaDB database using the 3306 port from within the host machine
  • --tmpfs /var/lib/mysql:rw is the coolest argument since it allows us to map the MariaDB /var/lib/mysql data directory in-memory using tmpfs
  • -e MYSQL_ROOT_PASSWORD=admin defines the root account password for MariaDB
  • -e MYSQL_USER=hibernate_orm_test creates a new user which we’ll use for testing
  • -e MYSQL_PASSWORD=hibernate_orm_test creates a new password for our testing user
  • -e MYSQL_DATABASE=hibernate_orm_test creates a new MariaDB database

After running the aforementioned docker run command, if we list the current Docker containers using docker ps -a, we can see our newly created mariadb Docker container that’s also up and running:

> docker ps -a
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                    PORTS                    NAMES
2c5f5131566b        mariadb                        "docker-entrypoint..."   12 minutes ago      Up 12 minutes             0.0.0.0:3306->3306/tcp   mariadb
dc280bbfb186        oracle/database:12.1.0.2-se2   "/bin/sh -c $ORACL..."   9 days ago          Exited (137) 7 days ago                            oracle

Using the container name (e.g. mariadb), we can also attach a bash process so that we can inspect the MariaDB Docker container:

> docker exec -i -t mariadb /bin/bash

root@2c5f5131566b:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          59G   23G   34G  41% /
tmpfs           2.2G     0  2.2G   0% /dev
tmpfs           2.2G     0  2.2G   0% /sys/fs/cgroup
/dev/sda1        59G   23G   34G  41% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           2.2G  115M  2.1G   6% /var/lib/mysql
tmpfs           2.2G     0  2.2G   0% /sys/firmware

As you can see from the df -h output, the var/lib/mysql data directory is mapped on tmpfs. Woohoo!

One more thing to do, let’s grant some admin privileges to our test user. We can do it right from the Docker container bash terminal using the mysql -u root -padmin hibernate_orm_test command:

root@2c5f5131566b:/# mysql -u root -padmin hibernate_orm_test
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 6
Server version: 10.1.21-MariaDB-1~jessie mariadb.org binary distribution

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [hibernate_orm_test]>GRANT ALL PRIVILEGES ON *.* TO 'hibernateormtest' WITH GRANT OPTION;

Done!

PostgreSQL

Getting the Docker image

Now, let's do the same for a specific version of PostgreSQL (e.g. 9.5.6)

> docker pull postgres:9.5.6
9.5.6: Pulling from library/postgres
Status: Downloaded newer image for postgres:9.5.6

If we rerun docker images, we’ll see the PostgreSQL Docker image listed as well:

> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
postgres            9.5.6               bd44e8a44ab2        2 weeks ago         265 MB
oracle/database     12.1.0.2-se2        b5f12a4d9ae0        9 days ago          11.1 GB
bash                latest              c2a000c8aa3c        11 days ago         12.8 MB
mariadb             latest              7eca0e0b51c9        2 weeks ago         393 MB
oraclelinux         latest              5a42e075a32b        3 weeks ago         225 MB
d4w/nsenter         latest              9e4f13a0901e        4 months ago        83.8 kB

Running the database in a Docker container

For PostgreSQL, the command is extremely similar:

docker run ^
    --name postgres95 ^
    -p 5432:5432 ^
    --tmpfs /var/lib/postgresql/data:rw ^
    -e POSTGRES_PASSWORD=admin ^
    -d ^
    postgres:9.5.6

The data folder is located under /var/lib/postgresql/data in PostgreSQL, and the other parameters have the same meaning like it was the case with MariaDB.

Testing time

Running all tests (around 400 database integration tests) in the Hibernate documentation folder on MariaDB takes around 30 seconds:

> gradle test -Pdb=mariadb
:documentation:compileTestJava
:documentation:processTestResources
:documentation:testClasses
:documentation:test

BUILD SUCCESSFUL

Total time: 30.339 secs

On PostgreSQL, they take less than 30 seconds:

> gradle test -Pdb=pgsql
:documentation:compileTestJava
:documentation:processTestResources
:documentation:testClasses
:documentation:test

BUILD SUCCESSFUL

Total time: 28.732 secs

Container lifecycle

When you’re doing using the database container, you can stop it as follows:

> docker stop mariadb

or, for PostgreSQL:

> docker stop postgres95

The container is persisted, so you don’t need to rerun all these steps the next time you need it. All you need to do is to start it like this:

> docker start mariadb

or, for PostgreSQL:

> docker start postgres95

If you enjoyed this article, I bet you are going to love my Book and Video Courses as well.

Conclusion

Mapping a RDBMS data directory on tmpfs is even simpler with Docker, and this is especially relevant for MySQL and MariaDB since the DDL operations take significantly more time than on other database systems (e.g. Oracle or SQL Server).

However, even for PostgreSQL, you’ll find a significant improvement for running tests when the data folder is mapped in a RAM drive.

Subscribe to our Newsletter

* indicates required
10 000 readers have found this blog worth following!

If you subscribe to my newsletter, you'll get:
  • A free sample of my Video Course about running Integration tests at warp-speed using Docker and tmpfs
  • 3 chapters from my book, High-Performance Java Persistence, 
  • a 10% discount coupon for my book. 
Get the most out of your persistence layer!

Advertisements

5 thoughts on “How to run integration tests at warp speed using Docker and tmpfs

    1. The Introduction section contains a link which already provides that. Just follow the link for more details about disk vs ram.

  1. Hi!
    Thank you for the great, great post!
    I am trying to replace my H2 database by the SQL Server run on docker (thats the engine, my project uses on production)
    Could you please give a little bit more explanation of the docker tmpfs option?
    In general I understand the concept of mounting a temporary fs in host’s memory.
    However I cannot find any information on what does the “:rw” part in this statement
    ” –tmpfs /var/lib/mysql:rw” actually mean
    Could you help me?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.