Using SocketIO to publish Mongo database change streams to clients

Mongo Change Streams and SocketIO Web Sockets: Abaganon

Create a NodeJS app that streams Mongo data changes over web sockets.

This is a full tutorial on creating a NodeJS app that listens for changes in a Mongo database and streams those changes to clients who connect to the app through a SocketIO web socket. The tutorial will also include the setup on the client side and the Mongo configuraiton.

What This Tutorial Covers

  1. Installing & Configuring Mongo For Change Streams
  2. Creating A NodeJS App With SocketIO & Mongo Driver
  3. Configuring A Back End For Basic Mongo Updates
  4. Configuring Client Side SocketIO

What You Need For This Tutorial

Docker

Mongo Set Up

Mongo requires either a replica set or a sharded cluster in order to enable change streams. We’ll be using a replica set. The following file is the Docker Compose configuration for a mongo replica set along with the builds and deployments for the back and front ends.

  • The extra_hosts field is needed to connect the Mongo instances into a replica set.
  • The rs_config.js file (discussed below) is the script that will configure the replica set.
  • A redis instance is added, which is used to store stateful data on socket connections, allowing the actual socket service to scale as it needed.

SocketIO Web Sockets

We’ll set up our sockets server here. First we need to connect to the Mongo replica set and set up the SocketIO server. This is done in the database() and sockets() functions. Once they resolve, we set up our change streams in watchChangeStreams(). In that function, we connect to whatever databases we want to listen to and then listen to changes in that database with the watch() function.

Mongo Back End

To test all this out, we need a back end for updating data in Mongo. Below is a very simple ExpressJS app that has one route for updating documents in a collection called “test”. A key thing to notice is that we’re adding Mongo Binary UUIDs to all documents. You could really use anything, but binary UUIDs are very efficient. Although not used in this tutorial, it’s highly recommended that you create an index on the UUID.

The Front End

On the front end, we have two simple functions. The first, connect(), connects to the SocketIO server and joins a room based on the UUID provided. The second, update(), updates or inserts a document with the provided UUID and arbitrary data. If you do not provide a UUID, a random one will be generated on the back end.

Dunners!

Overall I think it’s a fairly easy architecture to grasp though there are a lot of moving pieces and gotchas when figuring this out. One final important note is that if you want to use this kind of architecture in production, you’ll need to add security to determine who can listen to which UUIDs. That’s too much extra stuff to go over in this tutorial, but when I implemented it at my company, the general gist was to store an httpOnly Oauth token with the client and then use that to ping an Oauth back end service for auth information on the user. Good luck!

Twitter: @ericfossas | Github: efossas