Live Query - (Query tailing)

Live query allows you to create a tailable cursor on any collection in the document database. There is no requirement for the collection to be capped. Tailable queries support all operators including sorts.

A tailable query with the sort operator must have a corresponding index or compound index on the sort field(s)

Query tailing is intended to replace legacy approaches to building real time apps such as polling or tailing the replication log of the database.


This feature is only available from your Nodechef Cloud Search instance. None of the examples provided below will work on a mongodb database. To deploy a cloud search instance, login and navigate to the dashboard. From deployments > deploy cloud search, you can then spin up a cloud search server

Usage

The example provided below uses the standard mongodb node.js driver. If you are using a different driver, refer to the driver reference for tailing a capped collection.

var MongoClient = require('mongodb').MongoClient; MongoClient.connect("mongodb://localhost/test", function(err,db) { var filter = { title : "tekken", region : "us-east" }; // set MongoDB cursor options var cursorOptions = { tailable: true, awaitdata: true, numberOfRetries: -1 }; // create stream and listen // game_scores is not a capped collection. var stream = db.collection('system.subscriptions.game_scores').find(filter, cursorOptions) .stream(); stream.on('data', function(document) { console.log(document); // relay this document to the client via a web socket and so on. }); })

Remarks

System.subscriptions is a special namespace that allows you to create a scalable pub/sub system on any collection that is not a capped a collection. Note when the above query executes, it will only return results for documents that are inserted, deleted or updated after the cursor is created. If you need to for example, show the last ten documents in your app and continue to subscribe for new documents, you must issue a regular query for the last 10 documents and also create a seperate query using the system.subcriptions namespace prefix to receive new documents.

On Inserts

When a document is inserted which matches the query document of any of the pending cursors, the output document shown below should be expected by the listening client

# Note: If the cursor was created with a valid projection, the newly inserted document # will be transformed using the projection document. { state : 2, new_val : <projection_of_document_inserted> }

On Updates

When a document is updated which matches the query document of any of the pending cursors, the output document shown below should be expected by the listening client

{ state : 4, old_val : <projection_of_document_before_update>, new_val : <projection_of_document_after_update> }

On Deletes

When a document is deleted which matches the query document of any of the pending cursors, the output document shown below should be expected by the listening client

{ state : 1, old_val : <projection_of_deleted_document>, }

Streaming sort - On insert, update, delete

var cursorOptions = { tailable: true, awaitdata: true, numberOfRetries: -1 }; var stream = db.collection('system.subscriptions.game_scores').find(filter, cursorOptions) .addQueryModifier( "$orderby", { scrore : -1 } ) .comment( "liveQuerySortSetOnly=1&liveQuerySortLimit=15" ) .stream();

Remarks

Using the sort operator together with a tailable cursor on the 10gen sponsored node.js driver (2.2.5) throws an exception. To work around this you can use the addQueryModifier on the cursor to specify the sort criteria for the query as seen above. If you are using a legacy driver which allows you to use the sort method directly on a tailable cursor, you should defer to using that instead.

The driver will kill the cursor when it reads the total number of documents equal to the limit set on the query. This however is undesirable when the intent is to implement a live query system which needs to alert the client of new changes. As such you can specify the limit criteria using the propery "liveQuerySortlimit" in the comment string of the query. Note the comment parameters are encoded as a regular URL query string.


Cursor response when the sort operator is used

{ add : [ { index : <index_of_document_in_top_x_list> obj : <projection_of_document> } ], remove : [ <projection_of_document_no_longer_in_top_x_list>, ], set : [ <projection_of_document> ] }

Remarks

The document returned will either contain the set array attribute only or a combination of the remove and add array attribute

The set array attribute always returns in sorted order all the documents contained in the query. That is if the query was to sort on a field and limit by 15, if there are 15 documents in the collection, set is guaranteed to always contain all the 15 documents in the requested sort order. If you set the parameter "liveQuerySortSetOnly=1" in the comment of the query, the document returned by the cursor will always contain the "set" array. This is much easier to work with as for example you can simply rewrite the HTML of your web app whenever the cursor returns a document.

The add array returns documents and their positions in the list. For example if the query is to sort and limit by 15, the add array returns documents and the position of the document in the entire set. Note you will only receive the add array if the you did not set the "liveQuerySortSetOnly=1". You will only go this route if you are returning very large documents and transmitting the entire query set over the wire is inefficient. However, this is typically rare for most cases.

The remove array contains one or more documents to be removed from the query set. Note you will only receive the remove array attribute if the you did not set the "liveQuerySortSetOnly=1".

Performance of streaming sort operations

Streaming sorts require an index on the sort attribute as well as the limit clause on the query. Changes are computed incrementally but can be cpu intensive in cases when the cardinality of the sort index is high and a filter conditon was specified with the query.