Lazy loading Backbone collections with Promises
Written by , November 12, 2013 Until recently, when our customers created a campaign, they could choose to target the whole world, the U.S. or create a list of specific countries and U.S. metros (e.g. the San Francisco Bay Area, the Atlanta metro, etc.).
Last week, we added the ability to target or exclude countries, metros, regions (e.g. states in the U.S., provinces in Canada,…), cities and postal codes. That meant doing a significant refactor of our geolocation code, which lead me to (finally) use Deferred objects.
We have under 200 countries and 250 metros in our database, so it was not a problem to bootstrap the whole list in our Backbone app on page load. However, regions are in the thousands and cities in the hundreds of thousands. We couldn’t just simply load everything anymore: we turned to lazy loading.
With perfect timing, Jeremy Ashkenas’ keynote presentation for BackboneConf was put online. He talks about a few Backbone patterns that he came across, including one to lazily load models in a collection. Here is the segment: (duration: ~1’30”)
The idea is that the collection may or may not have loaded the model you’re interested in at this moment. If it’s loaded, the collection can give it to you right away. But if it’s not, it needs to get it asynchronously from the server. One way would be to have your view manage this:
var user = users.get(id);
if (user) {
this.renderUser(user);
} else {
user = new User({id: id});
user.fetch({
success: function(user, response, options) {
this.renderUser(user);
}
});
}This works, but it’s hard to read, it’s messy and it doesn’t separate concerns well: the view shouldn’t need to check if a model is loaded or not. On top of that, you would need the same thing in any views that want to get a model.
The solution of course is to move this logic into the collection itself, and Jeremy discusses using Deferred objects instead of passing a callback to the lookup method. Here is the code from his slides:
// View code
users.lookup(id).then(function() {
...
});
// Collection code
lookup: function(id) {
var model;
if (model = this.get(id)) {
return $.Deferred().resolveWith(this, model);
} else {
model = new Model({id: id});
return model.fetch();
}
}The interesting part is in this line: return $.Deferred().resolveWith(this, model);. I.e. even if we have the model loaded locally, the collection returns a Deferred object, but it resolves right away with the local model. This way, the interface is the same in both cases and the view doesn’t need to worry about this at all. Very neat solution.
However, the code above doesn’t work as-is, so I wanted to write down how I got it to work. (admittedly, I might have missed something or Jeremy just put the slide as an illustration, not to be taken literally)
The issue is that in one case the method returns a Deferred object resolved with a Backbone model, while in the second case we return what model.fetch() returns, which is a jqXHR. For our purposes, we just need to know that a jqXHR is an XMLHttpRequest that implements the Promise interface. When that jqXHR finally resolves (when the AJAX request finishes), the function passed to then() gets data, textStatus, jqXHR as arguments.
What we want to return however is a Backbone model to be consistent with the first case. After some slight refactoring, here is what I have:
lookup: function(id) {
var model;
var deferred = $.Deferred();
if (model = this.get('id')) {
deferred.resolveWith(this, model);
} else {
model = new Model({id: id});
model.fetch({
success: function(model, response, options) {
deferred.resolveWith(this, model);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}One of the things to note is that I don’t return a Deferred object anymore but a Promise. The difference is that the Promise lets you add callbacks to the Deferred but doesn’t let you resolve it. It’s best practice in most cases to let whatever code created the object resolve it as well.
I like this approach a lot. It’s elegant and easy to read and adapts well to different cases.
For example, in my geolocation case, I don’t want only one model at a time, but instead I want to do auto-completion and get a list of geolocations that match a specific string. In some cases, the geolocation type will be completely pre-loaded (for countries and metros); in others, it will be fetched on-demand.
The difference with the code above is that the success callback option for Backbone.Collection.fetch doesn’t return the list of models that were fetched. Instead the callback arguments are collection, response, options. So I need a bit more code.
Here is a modified version for when you want to call collection.fetch:
// `type`: type of geolocation
// `query`: query string you want to match your models against
// `this.prefetched`: array of what types are prefetched (set in initialize())
search: function(type, query) {
var matchingModels;
var deferred = $.Deferred();
// Here we check if the requested type is fully prefetched
if (this.prefetched[type]) {
// match() would be implemented however you need it to
matchingModels = this.match(type, query);
deferred.resolveWith(this, matchingModels);
} else {
var queryData = {
type: type,
query: query
};
this.fetch({
data: queryData,
success: function(collection, response, options) {
// At this point, the new models will have been added to the
// collection
var matchingModels = collection.match(type, query);
deferred.resolveWith(this, matchingModels);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}I can now create my collection with some fully bootstrapped geolocation types (countries and metros) and some that need to be fetched on-demand and the views don’t need to know which is which.
One drawback of having a match method in the collection is that I’m duplicating some server logic on the front end. The two could potentially clash, e.g. if my server takes the query string and sends back all cities whose name starts with the query string, while the collection matches a model if the query string is in the name, the displayed results won’t be consistent and may vary based on the order in which I make various queries.
I haven’t come up with a good solution to that. I could cache the list of IDs sent back by the server for each query string but a different problem appears for the prefetched types: I would need to also create list of IDs for any substring for the prefetched models.
For now, since I’ll be keeping a simple matching logic, duplicating the logic will do.
Finally, I need to add a cache to my collection to keep track of the models I have already fetched from the server, because with the code above you could be making the same request multiple times.
When search is called, I just need to check if the requested type is fully prefetched. If it isn’t, I need to check if I have already queried that string for that type:
// `type`: type of geolocation
// `query`: query string you want to match your models against
// `this.prefetched`: array of what types are prefetched (set in initialize())
// `this.requestCache`: object that keeps a cache of the strings that have already
// been queried from the server for each geo type
search: function(type, query, limit) {
var matchingModels;
var cache = this.requestCache[type];
var deferred = $.Deferred();
if (this.prefetched[type] ||
cache && cache[query] && cache[query] >= limit) {
// match() would be implemented however you need it to
matchingModels = this.match(type, query, limit);
deferred.resolveWith(this, matchingModels);
} else {
var queryData = {
type: type,
query: query,
limit: limit
};
this.requestCache[type] = this.requestCache[type] || {};
this.requestCache[type][query] = limit;
this.fetch({
data: queryData,
success: function(collection, response, options) {
// At this point, the new models will have been added to the
// collection
var matchingModels = collection.match(type, query, limit);
deferred.resolveWith(this, matchingModels);
}
});
}
// Returning a Promise so that only this function can modify
// the Deferred object
return deferred.promise();
}I also added a limit attribute to be able to request a certain number of objects. I save that value in a dictionary keyed on the query string, so that I know how many of the already-cached objects will match this query for that geolocation type. For example, I could have some parts of the application that need only 20 items and some where I want to show 100 items. If I already searched for 100 items for a specific query string, I don’t need to make a new query for the part of the app where I only need 20 items. This saves some additional roundtrips to the server and make the application feel snappier.
I could still improve this further by delaying the call to fetch so that I can gather up queries on multiple types at once. So if I were to call:
collection.search('city', 'san').then(printGeos);
collection.search('region', 'san').then(printGeos);only one server call could be made to get results for both types. But I’ll leave that as an exercise for now.
Another area of improvement would be to add ways to manage the cache and the size of the collection. It’s probably a good idea to be able to clear out the cache over time and to make sure the collection doesn’t grow too big. But, this too will have to wait.
To conclude, I now have a lazy-loaded collection that I can partially prefetch as needed, and I limit the number of server queries by caching the query terms. Deferred and Promise objects let me present a single, elegant interface to the views.