{ martowen.com }

Mongo Deployment Scripts with TypeScript

Jul 20, 2018
8 minutes

In 10 years of working with MongoDB I’ve found that developing against it is great, but putting changes live is not so great, due to the fact it’s impossible to say for sure how the live database compares to the database you’ve been developing against.

Add to this duplication data due de-normalised data, and you end up with risky deployments which can cause stress, and hesitation to make changes that affect your Mongo data.

As part of the deployment process in my current company, we use SQL Server Database Tools for generating a deployment script against the database in each environment. At the time of writing I don’t know of a comparable tool for MongoDB, and the schemaless nature of Mongo databases makes it unlikely that one will ever be developed.

Our deployment process also involves a DBA running a deployment script that we hand over to them, and we can’t make any assumptions about the client that they will run the script from.

With all this in mind, I worked on a way to use TypeScript and Webpack to build up a Mongo script for each release that would let us develop strongly typed scripts which came bundled with any dependencies that they required.

Why not just use Mongoose?

The NodeJS Mongo driver Mongoose allows you to define a schema for your Mongo database, and connect to it via that. This is preferable to the solution I describe in this post, and will likely be what I use in the future.

I found it acceptable to roll my own MongoDB types for the following reasons:

  1. The limitation of not being able to guarantee NodeJS will be available on the machine of the DBA running the script. My requirement was that we would have a standalone JavaScript file that could run in the MongoDB shell with a command such as mongo 127.0.0.1/db_name <script.js>
  2. I was moving from a situation in which we had many small snippets of Mongo maintenance JavaScript being copied and pasted and passed around. Any solution which allowed us to organise these in a way that allowed us to import pre-defined library code and bundle them into a release script would be a massive improvement.
  3. The database had a small number of Collections (less than 10) which needed to be scripted against, so it was possible to write entity classes against them.

Developing a TypeScript Mongo Shell script

TypeScript type definition

To start with I needed a TypeScript type definition file for the Mongo DB shell. One doesn’t exist currently, as I’m pretty sure the assumption is you won’t do such complex scripting against the shell. I gradually built up a definition file as I went along, for each object and method I required.

Here’s an excerpt of what I ended up with:

declare namespace MongoShell {
    export class ISODate {
        constructor(str: string);
    }

    export class Database {
        User: Collection; // The 'User' Entity will be defined below
        hostInfo(): HostInfo;
    }

    export class Collection {
        count(query?: any, options?: any): Number;
        find(query?: any, options?: any): Cursor;
        findOne(query: any, options?: any): any;
        update(query: any, update: any, options?: any): WriteResult;
        insert(
            document: any,
            writeConcern?: any,
            ordered?: boolean
        ): WriteResult;
        remove(query: any, justOne?: boolean): WriteResult;
        aggregate(query: any): Cursor;
        findAndModify(query: any): WriteResult;
    }

    export class Cursor {
        count(query: any, options?: any): Number;
        forEach(func: Function): void;
        toArray(): Array<any>;
        next(): any;
    }

    export class WriteResult {
        nMatched: number;
        nUpserted: number;
        nModified: number;
        writeConcernError?: WriteConcernError;
    }

    export class WriteConcernError {
        code: number;
        errmsg: string;
    }
}

The User Collection within Database is the name of an actual MongoDB Collection. You will need one of these definitions for each Collection in your database.

Once they are in a .t.ds file in your types folder they can be made available in a script by declaring MongoShell.Database as a variable:

declare var db: MongoShell.Database;

Defining entities

The Entity base class

A type definition for the Mongo shell is only the start, we still need types for the Documents in our Collections in order to develop scripts around them.

I started with an Entity abstract class:

import * as _ from "lodash";
import * as guidHelpers from "../../lib/guidHelpers";

declare var db: MongoShell.Database;

export abstract class Entity {
    protected _id: guidHelpers.BinData;
    protected _t: string = Object.getPrototypeOf(this).constructor.name;
    get id(): guidHelpers.BinData {
        return this._id;
    }

    IsDeleted: boolean;
    CreatedOn: MongoShell.ISODate;
    UpdatedOn: MongoShell.ISODate;

    // If either a string or Guid Id is passed it will be used, otherwise a new
    // one will be generated by guidHelpers.newGuid() (this Guid generation is not recommended)
    constructor(strId?: string, binData?: guidHelpers.BinData) {
        if (!_.isUndefined(strId)) {
            this._id = guidHelpers.CSUUID(strId);
            this.find();
        } else if (!!_.isUndefined(strId)) {
            this._id = binData;
        } else {
            this._id = guidHelpers.CSUUID(guidHelpers.newGuid());
        }
    }

    protected abstract find(): void;
}

A helper library

The guidHelpers library is the uuidhelpers.js file that has been part of the MongoDB C# Driver source code for years and is useful for converting Guid strings into BinData.

I defined a simple guidHelpers.d.ts file in order to be able to use it:

export class BinData {
    toUUID(): string;
    toJUUID(): string;
    toCSUUID(): string;
    toRawCSUUID(): string;
    toPYUUID(): string;
    toHexUUID(): string;
    hex(): string;
}

export function HexToBase64(hex: string): string;
export function Base64ToHex(base64: string): string;
export function UUID(uuid: string): BinData;
export function JUUID(uuid: string): BinData;
export function CSUUID(uuid: string): BinData;
export function PYUUID(uuid: string): BinData;
export function newGuid(): string;

An Entity implementation

With that Entity class in place I can now define entities:

import * as _ from "lodash";
import * as guidHelpers from "../../lib/guidHelpers";
import { Entity } from "./Entity";
import { UpdateResult } from "../../myMongo/UpdateResult";

export class User extends Entity {

  FirstName: string;
  Surname: string;
  DivisionId: guidHelpers.BinData;

  static fromDocument(doc: any): User {
    var user = new User();
    if (!_.isUndefined(doc)) {
        user._id = doc["_id"];

        user.FirstName = doc.FirstName;
        user.Surname = doc.Surname;
        user.DivisionId = doc.DivisionId;

        user.IsDeleted = doc.IsDeleted;
        user.CreatedOn = doc.CreatedOn;
        user.UpdatedOn = doc.UpdatedOn;
    }
    return user;
  }

  constructor(strId?: string) {
    super(strId);
  }

  protected find(): void {
    const userDoc = db.User.findOne({ '_id': this.id });
    if (userDoc != null) {
        const userObj = User.fromDocument(userDoc);
        _.assign(this, userObj);
    }
  }

  static findByDivision(divisionId: guidHelpers.BinData): Array<User> {
    return db.User.find({ 'DivisionId': divisionId, 'IsDeleted': false  }).forEach(User.fromDocument));
  }

  softDelete(): UpdateResult {
    var writeResult = db.User.update(
        { "_id": this._id },
        { "$set": { "IsDeleted": true } },
        { multi: false  }
    );

    return new UpdateResult(writeResult);
  }
}

Using Entities

I’ve implemented a softDelete method and a static findByDivision method in the User Entity above to illustrate how I could now call this from a deployment script without worrying about any of the underlying shell commands:

import * as _ from "lodash";
import * as guidHelpers from "../../lib/guidHelpers";
import { User } from "../../myDb/entities/User";

// The MongoDB Shell 'print' function
declare function print(input: any): void;

const divisionToSoftDeleteUsersFrom: guidHelpers.BinData = guidHelpers.CSUUID(
    "e062111b-7841-4f17-956d-580d433c8c44"
);

export function up(): void {
    _.each(
        User.findByDivision(divisionToSoftDeleteUsersFrom),
        (user: User): void => {
            const updateResult = user.softDelete();
            print(
                `Soft deleted User Id ${user.id} named '${user.FirstName} ${
                    user.Surname
                }' ${updateResult.toString()}`
            );
        }
    );
}

At this point I start to get concerned that the User Entity will get out of sync with the schema in the database, as the actual Entities in the application code (written in C# in my case) are different to these TypeScript ones.

Context-aware scripts running against different environments

A nice additional feature that our team was able to add to this scripting framework was environment-specific configurations for each of the scripts, using TypeScript generics.

import * as _ from "lodash";

declare var db: MongoShell.Database;

export class Environment {
    protected db: MongoShell.Database;
    hostInfo: MongoShell.HostInfo;

    Name: string;
    EnvironmentType: EnvironmentType;
    CustomerBase: CustomerBase;
    DataCentre: DataCentre;

    constructor() {
        this.db = db;
        this.hostInfo = db.hostInfo();
        this.identityEnvironment();
    }

    protected identityEnvironment() {
        switch (this.hostInfo.system.hostname) {
            case "mdb1.uk.prod":
            case "mdb2.uk.prod":
                this.EnvironmentType = EnvironmentType.Production;
                this.DataCentre = DataCentre.UK;
                break;

            case "mdb1.us.prod":
            case "mdb2.us.prod":
                this.EnvironmentType = EnvironmentType.Production;
                this.DataCentre = DataCentre.US;
                break;

            case "mdb1.uk.staging":
            case "mdb2.uk.staging":
                this.EnvironmentType = EnvironmentType.Staging;
                this.DataCentre = DataCentre.UK;
                break;

            case "localhost":
                this.EnvironmentType = EnvironmentType.LocalDev;
                this.DataCentre = DataCentre.None;
                break;
        }
    }

    selectEnvironmentConfig<T>(
        environmentConfig: Array<EnvironmentConfig<T>>
    ): EnvironmentConfig<T> {
        var environment = new Environment();

        const configToUse = _.find(
            environmentConfig,
            (config: EnvironmentConfig<T>): boolean => {
                return (
                    config.EnvironmentType === environment.EnvironmentType &&
                    config.DataCentre === environment.DataCentre
                );
            }
        );

        return configToUse;
    }
}

export class EnvironmentConfig<T> {
    Name: string;
    EnvironmentType: EnvironmentType;
    DataCentre: DataCentre;
    Config: T;
}

export enum EnvironmentType {
    LocalDev = "LocalDev",
    Staging = "Staging",
    Production = "Production"
}

export enum DataCentre {
    None = "None",
    UK = "UK",
    US = "US"
}

This means that the person running the script doesn’t need to specify which config to use. The script will determine it based on the value of hostInfo.system.hostname. If the script defines a config object, it can then be specified for each environment:

import * as _ from "lodash";
import { Division } from "../../myDb/entities/Division";
import {
    Environment,
    EnvironmentType,
    CustomerBase,
    DataCentre
} from "../helpers/Environment";

// The MongoDB Shell 'print' function
declare function print(input: any): void;

class DivisionConfig {
    divisionIdsToDelete: Array<number>;
}

const environmentConfig: Array<EnvironmentConfig<DivisionConfig>> = [
    {
        Name: "Local Development",
        EnvironmentType: EnvironmentType.Development,
        CustomerBase: CustomerBase.Mixed,
        DataCentre: DataCentre.None,
        Config: {
            divisionIdsToDelete: [4, 5, 6]
        }
    },
    {
        Name: "UK Staging",
        EnvironmentType: EnvironmentType.Staging,
        CustomerBase: CustomerBase.Mixed,
        DataCentre: DataCentre.UK,
        Config: {
            divisionIdsToDelete: [10, 11, 12]
        }
    },
    {
        Name: "UK Production",
        EnvironmentType: EnvironmentType.Production,
        CustomerBase: CustomerBase.Mixed,
        DataCentre: DataCentre.UK,
        Config: {
            divisionIdsToDelete: [10, 11, 12]
        }
    }
];

const environment = new Environment();
const config = environment.selectEnvironmentConfig(environmentConfig);

function deleteDivisions(divisionIds: Array<number>) {
    _.each(divisionIds, (divisionId: number): void => {
        const updateResult = Division.remove(divisionId);
        print(`Deleted DivisionId: ${divisionId}, Result: ${updateResult}`);
    });
}

deleteDivisions(config.Config.divisionIdsToDelete);

So in this example the Config is set correctly according to the environment in which the script is run. Ideally the Ids would be the same in between environments to help with validating the script, but it doesn’t need to be.

Once the deleteDivisions.js script is built it will be as simple as handing it to a DBA and running:

mongo mdb.uk.prod/db deleteDivisions.js
Older Git Submodules vs Git Subtrees
Newer Leader Election in Kubernetes with the...