Data synchronization

This document is only useful for deployments where the cloud search server is used in conjunction with a mongodb database. If using the Cloud search server as a standalone database, data synchronization is of almost no importance.

The cloud search and analytics engine can be synchronized with your mongodb database automatically or manually. Automatic synchronization connects to the mongodb database, copies a snapshot of the data and then continue to sync the data by tailing the oplog of the mongodb database. For manual synchronization, you are responsible for inserting, deleting and updating data in the search engine using the CRUD api. You typically will perform manual synchronization in your app code after the object is saved in the database.

Automatic synchronization

Open the mongo shell and connect to the search server using the connection string retrieved from the task manager. You will then be able to run any of the given below commands.

Connecting via the mongo shell steps.

Download the mongo shell if not already installed on your machine.
Retrieve the connection string from the task manager using CSAS actions > Connect
Open the command prompt or terminal and CD to the directory in which you extracted the mongo shell files from step 1.
Enter the below command to connect to the search server.
mongo -u "provide-username" -p "provide-password"
If you have enabled SSL, use the below command.
mongo -u "provide-username" -p "provide-password" --ssl --sslAllowInvalidCertificates

Automatic data syncing accepts only the below data tyoes from the mongo wire protocol. If an attribute is not of the type below, the attribute is ignored as if it does not exist in the source document. For almost all use cases, the below data types are more then enough. Note you can however query the data with regex types and so on. The below data type restriction is only for DML operations on automatic data synchronization.

Null
Boolean
Int32, Int64
Double
String, Binary
ObjectId
Datetime, Timestamp
Array
Document

1

Synchronize your entire mongodb database as is.

db.runCommand({ syncData : "flowers-db", connection : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/flowers-db", oplog : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/local?authSource=flowers-db" })

Remarks

This is the simplest form of synchronization to perform. The search engine becomes a replica of your mongodb database. All inserts, deletes and updates are replicated to the search engine. Collections from the remote database flowers-db, will be named flowers-db.collectionName in the search engine.
2

Synchronize one or more classes/collections from a mongodb database

db.runCommand({ syncData : "flowers-db", connection : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/flowers-db", oplog : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/local?authSource=flowers-db", collections : ["products", "reviews"] })

Remarks

The above command will copy and synchronize only the two collections products and reviews from your mongodb database.
3

Synchronize specific fields only from one or more classes/collections from a mongodb database

db.runCommand({ syncData : "flowers-db", connection : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/flowers-db", oplog : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/local?authSource=flowers-db", collections : [ { name : "products", project : { title : 1, description : 1, _created_at : 1, price : 1, _rperm : 1 }}, "reviews" ] })

Remarks

The above command will only copy the collections products and reviews. For the products class, only the title, description, price, _created_at and _rperm fields will be copied. All other fields in the documents will be discarded by the search engine. For parse server users, the _rperm field will allow you to keep your ACLS as is, the _created_at and _updated_at fields can be helpful to keep in the search engine. Copying specific fields avoids unneccesary duplication and can significantly save storage space in some use cases.
4

Synchronize documents that evaluates to true based on a given criteria from a mongodb database

db.runCommand({ syncData : "flowers-db", connection : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/flowers-db", oplog : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/local?authSource=flowers-db", collections : [ { name : "products", project : { title : 1, description : 1, _created_at : 1, price : 1, _rperm : 1 }, query : { in_stock : true }} ] })

Remarks

The above command will only copy documents from the products collection only when the product is actually in stock. Only the title, description, price, _created_at and _rperm fields will be copied after the document matches the criteria given.
5

Synchronize all collections but exclude one or more collections from a mongodb database

db.runCommand({ syncData : "flowers-db", connection : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/flowers-db", oplog : "mongodb://flowers-db:flowers-pass@db-flowers-1.nodechef.com/local?authSource=flowers-db", exclude_collections : ["logs","settings"] })

Remarks

The above command will copy all collections from the remote database but the logs and settings collections.
6

Drop collections from an active syncData

db.runCommand({ resetSyncData : "flowers-db", exclude_collections : ["logs","settings","payments"] })

Remarks

You can reset the exclude_collections array at any time, this will automatically stop syncing data for the specified collections. Note previously synced data will not be removed, you will be responsible for doing this using a drop collection command.
7

Reset collections, query criteria or projection of an active syncData

db.runCommand({ resetSyncData : "flowers-db", collections : [ { name : "products", project : { title : 1, description : 1, _created_at : 1, price : 1, _rperm : 1, likes_count : 1, reviews_count : 1 }, query : { in_stock : true }} ] })

Remarks

You can reset the collections array at any time, this will only affect the incomming data from the oplog. Data that had previously been synced remains unchanged. If you are adding new fields and the new fields already existed in the mongodb database, you could trigger a forced update such as setting an attribute to its original value. This will cause the updates to be logged in the oplog which will then trigger these updates on the search engine as well. Fields that are to be removed will not be removed until the document is updated again after the projection was reset.

Manual synchronization

Manaul synchronizations gives you complete control on the names of the classes, as well as the fields populated in the search engine but requires a bit more code to accomplish. The example given below is for parse server deployments. For meteorjs and node js, you will have to figure out when it is best to update, delete or insert into the search engine in your app code. It is generally sound practice to update the search engine once the operation has completed on the database.

1

Insert or update document using afterSave hooks.

var mongodb = require( '../node_modules/parse-server/node_modules/mongodb' ) , MongoClient = mongodb.MongoClient; var search_db; // connect once and avoid pool polution. MongoClient.connect( process.env.NC_CSA_SERVER, function ( err, db ) { if ( !err ) { search_db = db; } } ); Parse.Cloud.afterSave( 'Products', function ( request ) { if ( search_db ) { var o = request.object.toJSON(); // update the object if it exists else insert new. // in this example only the title and description fields are of interest. search_db.collection( "Products" ).update( { _id: o._id }, { title: o.title, description: o.description }, { upsert: true, w: 1 }, function ( err, r ) { if ( err ) console.log( "Failed writing document with error: " + err ); } ); } } );
2

Delete documents using afterDelete hooks.

var mongodb = require( '../node_modules/parse-server/node_modules/mongodb' ) , MongoClient = mongodb.MongoClient; var search_db; // connect once and avoid pool polution. MongoClient.connect( process.env.NC_CSA_SERVER, function ( err, db ) { if ( !err ) { search_db = db; } } ); Parse.Cloud.afterDelete( 'Products', function ( request ) { // Get Algolia objectID if ( search_db ) { var objectID = request.object.id; search_db.collection( "Products" ).remove( { _id: objectID }, { w: 1 }, function ( err, r ) { if ( err ) console.log( "Failed deleting document with error: " + err ); } ); } } );
3

Import an existing collection.
Applies to both nodejs and parse server.

var mongodb = require( '../node_modules/parse-server/node_modules/mongodb' ) , MongoClient = mongodb.MongoClient; MongoClient.connect( process.env.DATABASE_URI, function ( err, db ) { // connect to your mongodb database to read documents. if ( !err ) { // connect to the search engine.. MongoClient.connect( process.env.NC_CSA_SERVER, function ( err, search_db ) { if ( !err ) { var batch_array = new Array( 1000 ); // insert the documents in batches of 1000 // use each to stream the documents instead of toArray as // the entire collection is been read.. db.collection('products').find( ).each( function ( err, doc ) { if ( batch_array.length == 1000 ) { var insert_array = batch_array; search_db.collection( "products" ).insert( insert_array, { w: 1 }, function ( err, result ) { if ( err ) console.log( err ); } ); batch_array = new Array( 1000 ); } // insert into the search engine only the required fields // or the entire documents as is. // Note _rperm is the ACL in case you need it later when reading. batch_array.push( { _id: doc['_id'], title: doc['title'], description: doc['description'], _rperm: doc['_rperm'], _created_at : doc['_created_at'] } ); } ); if ( batch_array.length > 0 ) { //insert the final batch.. search_db.collection( "products" ).insert( batch_array, { w: 1 }, function ( err, result ) { if ( err ) console.log( err ); } ); } } } ); } else { console.log( err ); response.error( err ); } } );