Flask Mongo Engine with Azure Cosmos DB

Hello World!

This article we will do a light touch on Cosmos DB; specifically the Mongo API from Cosmos DB and using that API from Mongo Engine.  I think one of the great things about Cosmos DB’s Mongo API is that I simply swap out my connection strings and guess what; it works!  This means not only can I use Mongo Engine, but I can use PyMongo or any other framework for any language that connects to Mongo.

Getting Started with Cosmos DB

I’m not going to rewrite everything; but here are some key docs you may be interested in:

I think the key items to take away are the following:

  1. Easy to Setup, Configure and Use from any language
  2. Globally Scale Able with several options for data consistency
  3. Elastically scale-able for dev purposes as well as production.

Hooking Up Mongo Engine

So I’m using python 3.5 for Tensor Flow.  The first thing to do is pip install.  Its pretty easy.

pip install flask-mongoengine

After this; I’m going to assume you have a larger, real project.  I am going to assume you have the following files

  • app.py : Application Entry Point and Configuration
  • api.py : restful api defined here
  • datamodel.py : some mongo data model

The key difference to Cosmos DB and other Mongo setups is that Cosmos DB forces secure communications.  Luckily this is just some simple configuration.

App.py

Below is the key code configurations you need in your app.py.  Notice ‘ssl’ and ‘ssl_cer_reqs’.  This is not documented anywhere on their sites; but turns out you can just pass this along and it makes it into the underlying engine configuration.  Awesome!

from optparse import OptionParser
import os
from flask import Flask, url_for, request, jsonify
from flask_restful import Resource, Api
from flask_mongoengine import MongoEngine
from flask_debugtoolbar import DebugToolbarExtension
import ssl

basedir = os.path.abspath(os.path.dirname(__file__))
app = Flask(__name__)
db = MongoEngine()

app.config['MONGODB_SETTINGS'] = {
    'db' : 'YOURDB',
    'host' : 'YOURS.documents.azure.com',
    'port' : 10255,
    'username' : 'USERNAME',
    'password' : 'PASSWORD',
    'ssl' : True,
    'ssl_cert_reqs' : ssl.CERT_NONE
}
app.config['DEBUG_TB_PANELS'] = [
    'flask_debugtoolbar.panels.versions.VersionDebugPanel',
    'flask_debugtoolbar.panels.timer.TimerDebugPanel',
    'flask_debugtoolbar.panels.headers.HeaderDebugPanel',
    'flask_debugtoolbar.panels.request_vars.RequestVarsDebugPanel',
    'flask_debugtoolbar.panels.config_vars.ConfigVarsDebugPanel',
    'flask_debugtoolbar.panels.template.TemplateDebugPanel',
    'flask_mongoengine.panels.MongoDebugPanel',
    'flask_debugtoolbar.panels.logger.LoggingPanel',
    'flask_debugtoolbar.panels.route_list.RouteListDebugPanel',
    'flask_debugtoolbar.panels.profiler.ProfilerDebugPanel'
]

db.init_app(app)
toolbar = DebugToolbarExtension(app)

from api.py import api
app.register_blueprint(api)

def main(options, args):
    host_addr = '0.0.0.0'
    host_port = 80
    if(options.ip):
        host_addr = options.ip
    if(options.local):
        host_addr = '127.0.0.1'
        local = True
    if(options.port):
        print('binding to port: ' + str(options.port))
        host_port = options.port
    app.run(host = host_addr, port=host_port)

if __name__ == '__main__':
    parser = OptionParser()
    parser.add_option('-d', '--debug', dest='debug', help='Starts Server in debug mode')
    parser.add_option('-p', '--port', dest='port', help='Specifies which port to start server on')
    parser.add_option('-l', '--local', dest='local', help='Starts server up on local host')
    parser.add_option('-i', '--ip', dest='ip', help='Starts server up on specified ip')
    (options, args) = parser.parse_args()
    main(options, args)

So we have quite a lot of code here; but most of this is just some common flask stuff.  The really big ticket items are the debug panels and the mongo configuration.  Great news all this works inside a docker container; just make sure you open ports 4000 and whatever your flask app runs on.

datamodel.py

So this code is just a simple data model; this is well documented.

from flask_mongoengine import MongoEngine

db = MongoEngine()

class datamodel(db.Document):
    field = db.StringField()

So mongo engine and python work together to make some magic stuff that is completely invisible to us.  We don’t need to pass a context around etc etc; it just does magic and this hooks right in and it works against Cosmos DB; Freaking awesome.  Another note; MongoEngine has a variety of out of the box field types including image, reference and gosh a ton of stuff and it just handles that for you and CosmosDB just plugs away as if it were just regular ol’ Mongo.

api.py

So this is partially documented and partially not.  Mongo Engine appears to have been built for more MVC style applications; but I don’t really want to serve out my data in that fashion; I have too many app targets and my api model is a bit more sophisticated.  I tend to build apis around relevent work streams as opposed to pages; though basic crud like pages often exist; I will still expose the data in a restful manner enabling me to have a single back end for all front ends.  Due to this; I had to change a few things.  Here is a sample api which returns paginated results as json as well as saves new data models.

from flask import Blueprint, request, jsonify
import json
from datamodel import datamodel

api= Blueprint('api', __name__, url_prefix='/api/datamodel')

@api.route('/page', methods=['POST'])
def page():
    '''
    Does pagination
    '''
    form = request.form
    dpage = form['page']
    perpage = form['perpage']
    pag_audits = datamodel.objects.paginate(page=int(dpage), per_page=int(perpage))
    pag_res = []
    for item in pag_audits.items:
        pag_res.append(item.to_json())
    return jsonify({'audits' : pag_res})

@api.route('/new', methods=['POST'])
def new():
    '''
    creates a new datamodel
    '''
    form = request.form
    data = datamodel()
    data.field = form['field']
    data.save()
    return jsonify({'status' : 'success'})

The unique piece here is around the fact that you can call item_tojson() on individual items and you need to arrange them into a list and then return the individual jsonified items back as a jsonified list.  Careful not to jsonify too much or you might get extra escape characters.

SQL vs Cosmos DB vs Mongo

Alright so all of this is great; but why did I pick Cosmos DB instead of SQL or raw Mongo?  It boiled down primarily to pricing.  The use case for me specifically involves significant amounts of data and individual pieces of data can be quite large.  Cosmos vs SQL on pricing; Cosmos wins every time.  So what about performance SQL vs Cosmos?  Cosmos claims super speed and though I may grow very large data as long as I structure it properly I will be able to quickly query.  I don’t think the large pricing difference between the two is going to be worth the slight extra performance I might receive.  Most of my hard work is done in number crunching on gpus and cpus, not data searching.  So why Cosmos DB over Mongo DB?  Well the thought here is I will use both.  I will offer a SaaS application with Cosmos DB as my backend in the cloud and on premise; I can simply deploy a Mongo DB.  Having Cosmos DB as my backend means I can quickly and easily elastically scale across the globe.  Perfect; just deploy more stateless containers all over the place and my data is ready to roll and I don’t have to think about a thing.  Makes life easy.  I can pay for that.

Summary

So we covered some overview of Cosmos DB, how to set up the Mongo API w/ MongoEngine and integrate it into a restful flask application and also some compare and contrast with a few other database systems.

Leave a Reply

Your email address will not be published. Required fields are marked *