MongoDB Incremental Migration Scripts

Imagine having a tool that can automatically detect if you are using JPA and Hibernate properly. Hypersistence Optimizer is that tool!

Introduction

An incremental software development process requires an incremental database migration strategy.

I remember working on an enterprise application where the hibernate.hbm2ddl.auto was the default data migration tool.

Updating the production environment required intensive preparation and the migration scripts were only created on-the-spot. An unforeseen error could have led production data corruption.

Incremental updates to the rescue

The incremental database update is a technical feature that needs to be addressed in the very first application development iterations.

We used to develop our own custom data migration implementations and spending time on writing/supporting frameworks is always working against your current project budget.

A project must be packed with both application code and all associated database schema/data updates scripts. Using incremental migration scripts allows us to automate the deployment process and to take advantage of continuous delivery.

Nowadays you don’t have to implement data migration tools, Flyway does a better job than all our previous custom frameworks. All database schema and data changes have to be recorded in incremental update scripts following a well-defined naming convention.

A RDBMS migration plan addresses both schema and data changes. It’s always good to separate schema and data changes. Integration tests might only use the schema migration scripts in conjunction with test-time related data.

Flyway supports all major relational database systems but for NoSQL (e.g. MongoDB) you need to look somewhere else.

Mongeez

Mongeez is an open-source project aiming to automate MongoDB data migration. MongoDB is schema-less, so migration scripts are only targeting data updates only.

Integrating mongeez

First you have to define a mongeez configuration file:

mongeez.xml

<changeFiles>
    <file path="v1_1__initial_data.js"/>
    <file path="v1_2__update_products.js"/>
</changeFiles>

Then you add the actual migrate scripts:

v1_1__initial_data.js

//mongeez formatted javascript
//changeset system:v1_1
db.product.insert({
    "_id": 1,
    "name" : "TV",
    "price" : 199.99,
    "currency" : 'USD',
    "quantity" : 5,
    "version" : 1
});
db.product.insert({
    "_id": 2,
    "name" : "Radio",
    "price" : 29.99,
    "currency" : 'USD',
    "quantity" : 3,
    "version" : 1
});

v1_2__update_products.js

//mongeez formatted javascript
//changeset system:v1_2
db.product.update(
    {
        name : 'TV'
    },
    {
         $inc : {
             price : -10,
             version : 1
         }
    },
    {
        multi: true
    }
);

And you need to add the MongeezRunner too:

<bean id="mongeez" class="org.mongeez.MongeezRunner" depends-on="mongo">
    <property name="mongo" ref="mongo"/>
    <property name="executeEnabled" value="true"/>
    <property name="dbName" value="${mongo.dbname}"/>
    <property name="file" value="classpath:mongodb/migration/mongeez.xml"/>
</bean>

Running mongeez

When the application first starts, the incremental scripts will be analyzed and only run if necessary:

INFO  [main]: o.m.r.FilesetXMLReader - Num of changefiles 2
INFO  [main]: o.m.ChangeSetExecutor - ChangeSet v1_1 has been executed
INFO  [main]: o.m.ChangeSetExecutor - ChangeSet v1_2 has been executed

Mongeez uses a separate MongoDB collection to record previously run scripts:

db.mongeez.find().pretty();
{
        "_id" : ObjectId("543b69eeaac7e436b2ce142d"),
        "type" : "configuration",
        "supportResourcePath" : true
}
{
        "_id" : ObjectId("543b69efaac7e436b2ce142e"),
        "type" : "changeSetExecution",
        "file" : "v1_1__initial_data.js",
        "changeId" : "v1_1",
        "author" : "system",
        "resourcePath" : "mongodb/migration/v1_1__initial_data.js",
        "date" : "2014-10-13T08:58:07+03:00"
}
{
        "_id" : ObjectId("543b69efaac7e436b2ce142f"),
        "type" : "changeSetExecution",
        "file" : "v1_2__update_products.js",
        "changeId" : "v1_2",
        "author" : "system",
        "resourcePath" : "mongodb/migration/v1_2__update_products.js",
        "date" : "2014-10-13T08:58:07+03:00"
}

Conclusion

To automate the deployment process you need to create self-sufficient packs, containing both bytecode and all associated configuration (XML files, resource bundles, and data migration scripts). Before starting writing your own custom framework, you should always investigate for available open-source alternatives.

Code available on GitHub.

FREE EBOOK

4 Comments on “MongoDB Incremental Migration Scripts

  1. Hi Vlad,

    thanks for the article, still flyway do not support NoSql and companies like Redgate is focused on MS SQL, look like liquibase support JSON format, mongeez last commit belong to 3 years ago, did MongoDB have any tool or framework for manage changes?

    Regards,
    Ali

    • Better use a SQL database which can scale just fine and has much better tooling.

  2. Hi Vlad,

    Thank you for this article, there are definitely to few articles about MongoDB database versioning. However using Mongeez uses db.eval() which is deprecated may not be a good choice in the long term. Writing migrations in plain JavaScript is not recommended but the only alternative is to use MongDB driver and write migrations in Java. From this point of view Mongobee (https://github.com/mongobee/mongobee ) may be interesting.

    On the other hand maybe a better idea is to use migrate-mongo (https://www.npmjs.com/package/migrate-mongo), and keep migrations separate from the Java code?

    I wish that tools like Flyway or Liquidbase support MongoDb soon. Because now it is really hard to find something reliable.

    Regards,
    Michal

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.