Caching SharePoint Data Locally with SPServices and HTML5’s Web Storage

The SharePoint SOAP Web Services are fast. In fact, I think they are as fast as the newer REST Web Services in many cases. The old, crufty SOAP Web Services even provide batching, something that the REST services don’t yet do. (Andrew Connell (@andrewconnell) has been beating the drum about this with Microsoft for months now, and we all hope they get this OData capability into SharePoint’s REST services sooner rather than later.)

Even though the SOAP services are fast, sometimes they just aren’t fast enough. In some of those cases, it may make sense to store some of your data in the browser’s Web storage so that it’s there on the client during a session or across sessions. Web storage is an HTML5 capability that is available in virtually all browsers these days, even Internet Explorer 8.

The best candidates for this type of storage (IMO) are list contents that are used as references and that don’t have a high number of changes. As an example, you might decide to store a list of countries in Web storage rather than loading them from the Countries list every time a page loads. Even though the read from the list is fast, it has to take *some* time. There’s the retrieval time, and then there is also any processing time on the client side. For instance, if you have dozens of fields per country and you need to load them into a complex JavaScript structure, that takes time, too. If those data chores are making your page loads seem laggy, then consider using local storage.

There are three main ways you can store data locally to improve performance. I’m not going to go into all of their intricacies, but I will give you some rules of thumb. There are a lot of nuances to this, so before you dive in, do some studying about how it all works.

Cookies

For small pieces of data, you should consider using cookies. Contrary to just about every article out there in the press, cookies are not bad. They can store up to 4k of data each for you, which you can read back when the user returns to the page. There’s a excellent little jQuery plugin I use to facilitate this called, aptly, jquery-cookie. You can download it (for free!) from GitHub here. Cookies persist across sessions.

Session Storage

Session storage is the flavor of Web storage that allows you to store data just for the duration of the session. Think of a session as a browser lifespan. Once you close the browser, the session storage is gone. Both session storage and local storage sizes are limited by the browser you are using. If you want to know if Web storage is available in your browser of choice, take a look at “Can I use“. The amount of storage each browser gives you is a moving target, but it’s per domain.

Local Storage

Local storage takes Web storage one step further. The data stored in local storage persists across browser sessions. In fact, it usually won’t go away until you explicitly delete it. (Consider this fact when you are gobbling up local storage in your development process.)

So What?

The trick with using these storage mechanisms is managing the data you’ve put in local storage as a cache. That data can go past its expiration date, either because some changes were made to the underlying data source or the cache itself has become corrupted. The latter is more difficult to deal with, so here I’ll focus on the former.

JavaScript – like most other programming languages – lends itself to building wrapper functions that add additional layers of abstraction on top of underlying functionality. Too many levels of abstraction can make things confusing, but with careful thought and smart code writing, you can build abstractions that serve you well.

In a recent client project, I found that as list data volumes were increasing, the pages in my SPServices- and KnockoutJS-driven application were loading more and more slowly. I’m building on top of SharePoint 2007 in this case, so even if I wanted to use REST, I couldn’t, nor do I believe that it would automatically make anything faster. If we had better servers running things, that might make a huge difference, but we have no control over that in the environment.

What I wanted was a reusable wrapper around SPGetListItemsJson (which itself is a wrapper around the SOAP List Web Service’s GetListItemChangesSinceToken and SPService’s SPXmlToJson) that would let me check local storage for a cached data source (list data), read either the entire data source or just the deltas from the SharePoint list, load the data into my application, and then update the cache appropriately.

The getDataSource function below is what I’ve come up with so far. There’s some setup to use it, so let me explain the parameters it takes:

  • ns – This is the namespace into which you want to load the data. In my applications these days, following the lead from the patterns Andrew and Scot Hillier (@scothillier) have published, I usually have a namespace defined that looks something like ProjectName.SubProjectName.DataSources. The “namespace” is simply a complex JavaScript object that contains most of my data and functions.
  • dataSourceName – The name that I want to give the specific data source within ns. In my example above with the Countries list I would use “Countries”.
  • params – This is the big magilla of the parameters. It contains all of the values that will make my call to SPGetListItemsJson work.
  • cacheItemName – This is the name of the item I want to store in Web storage. In the Countries example, I would use “ProjectName.SubProjectName.DataSources.Countries”.
  • storageType – Either “localStorage” or “sessionStorage”. If I expect the data to change regularly, I’d probably use sessionStorage (this gives me a clean data load for each session). If the data is highly static, I’d likely use localStorage.

And here’s the code:

/* Example:
getDataSource(ProjectName.SubProjectName.DataSources, "Countries", params: {
  webURL: "/",
  listName: "Countries",
  CAMLViewFields: "<ViewFields>" +
      "<FieldRef Name='ID'/>" +
      "<FieldRef Name='Title'/>" +
      "<FieldRef Name='Population'/>" +
      "<FieldRef Name='CapitalCity'/>" +
      "<FieldRef Name='Continent'/>" +
    "</ViewFields>",
  CAMLQuery: "<Query>" +
      "<OrderBy><FieldRef Name='ID'/></OrderBy>" +
    "</Query>",
  CAMLRowLimit: 0,
  changeToken: oldToken,
  mapping: {
      ows_ID:{"mappedName":"ID","objectType":"Counter"},
      ows_Title:{"mappedName":"Title","objectType":"Text"},
      ows_Population:{"mappedName":"Population","objectType":"Integer"},
      ows_CapitalCity:{"mappedName":"CapitalCity","objectType":"Text"},
      ows_Continent:{"mappedName":"Continent","objectType":"Lookup"},
    }
  }, "ProjectName.SubProjectName.DataSources.Countries"
)
*/

function getDataSource(ns, dataSourceName, params, cacheItemName, storageType) {

  var dataReady = $.Deferred();

  // Get the data from the cache if it's there
  ns[dataSourceName] = JSON.parse(window[storageType].getItem(cacheItemName)) || new DataSource();
  var oldToken = ns[dataSourceName].changeToken;
  params.changeToken = oldToken;

  // Read whatever we need from the dataSource
  var p = $().SPServices.SPGetListItemsJson(params);

  // Process the response
  p.done(function() {
    var updates = this.data;
    var deletedIds = this.deletedIds;
    var changeToken = this.changeToken;

    // Handle updates/new items
    if (oldToken !== "" && updates.length > 0) {
      for (var i = 0; i < updates.length; i++) {
        var thisIndex = ns[dataSourceName].data.binaryIndexOf(updates[i], "ID");
        // If the item is in the cache, replace it with the new data
        if (thisIndex > -1) {
          ns[dataSourceName].data[thisIndex] = updates[i];
          // Otherwise, add the new item to the cache
        } else {
          ns[dataSourceName].data.splice(-thisIndex, 0, updates[i]);
        }
      }
    } else if (oldToken === "") {
      ns[dataSourceName] = this;
    }
    // Handle deletes
    for (var i = 0; i < deletedIds.length; i++) {
      var thisIndex = ns[dataSourceName].data.binaryIndexOf({
        ID: deletedIds[i]
      }, "ID");
      ns[dataSourceName].data.splice(thisIndex, 1);
    }
    // Save the updated data back to the cache
    if (oldToken === "" || updates.length > 0 || deletedIds.length > 0) {
      // Save the new changeToken
      ns[dataSourceName].changeToken = changeToken;
      window[storageType].setItem(cacheItemName, JSON.stringify(ns[dataSourceName]));
    }
    dataReady.resolve();
  });
  return dataReady.promise();
}

Some of the nice things about this function:

  • It’s generic. I can call it for any list-based data source in SharePoint. (I started out building it for one data source and then generalized it.)
  • I call call it during a page life cycle to refresh the application data anytime I want or on a schedule, perhaps with setInterval.
  • I can set a lot of parameters to cover a lot of different use cases.
  • Each time I call it, it updates the cache (if it needs to) so that the next time I call it I get a “fresh” copy of the data.
  • It only loads the data that it needs to, by using the GetListItemChangesSinceToken capabilities.

And some downsides:

  • Since I know what data I’m working with in my application and that it will fit into the Web storage easily, I’m not worrying about failed saves.
  • If the cache does become corrupt (not something I expect, but there’s always Murphy), I’m not handling it at all.

If you decide to try this out, you’ll need a few auxiliary functions as well:

/* DataSource constructor */
function DataSource() {
  this.changeToken = "";
  this.mapping = {};
  this.data = [];
  this.deletedIds = [];
}

/** Adapted from http://oli.me.uk/2013/06/08/searching-javascript-arrays-with-a-binary-search/
 *
 * Performs a binary search on the host array.
 * @param {*} searchObj The object to search for within the array.
 * @param {*} searchElement The element in the object to compare. The objects in the array must be sorted by this element.
 * @return {Number} The index of the element. If the item is not found, the function returns a negative index where it should be inserted (if desired).
 */
Array.prototype.binaryIndexOf = function(searchObj, searchElement) {

  var minIndex = 0;
  var maxIndex = this.length - 1;
  var currentIndex;
  var currentElement;

  var searchValue = searchObj[searchElement];

  while (minIndex <= maxIndex) {
    currentIndex = (minIndex + maxIndex) / 2 | 0;
    currentElement = this[currentIndex];

    if (currentElement[searchElement] < searchValue) {
      minIndex = currentIndex + 1;
    } else if (currentElement[searchElement] > searchValue) {
      maxIndex = currentIndex - 1;
    } else {
      return currentIndex;
    }
  }

  return ~maxIndex;
}

Similar Posts

14 Comments

  1. As usuall, another excellent article Marc! Quick question, do you and/or your colleagues use SPServices in SharePoint 2013? I used to love applying SPServices to SP 2010 solutions, however I have been writing my own REST API calls in SP 2013? I think you can still use SPServices in 2013, just have not tried. Your thoughts? Keep up the great work!

    1. Scott:

      SPServices still works just great in SharePoint 2013. The SOAP Web Services have been deprecated, but I don’t see any signs that they will disappear anytime soon. SharePoint Designer still uses them to talk to the server, for instance, and there’s a lot of legacy code that would break if they are shut off.

      If you’re starting development of something new in 2013, then REST is probably safer from a longevity standpoint, but you may not be able to do some of the things you can do with SOAP, still.

      M.

  2. Nice article Marc (as usual)! Thought i’d add that a nice way to handle local storage caching if you’ve got multiple lists to get data from (lots of REST-driven webparts on a homepage, anyone?) is to hit the Lists endpoint for the entire web and $select the last modified timestamp against the stamp on your cached data (so you only fetch data from a list if it’s been updated and your cache is stale). Cuts down on page loads dramatically! Looking forward to a follow up post about dealing with cache corruption ;-)

    1. Nice suggestion, Bruce. Here I’m working in SharePoint 2007, so REST isn’t available. I suppose I could use GetListCollection to do something similar with the SOAP Web Services, though. In my specific case, the data is in several webs, so I think checking each list for the deltas is probably just as efficient.

      M.

  3. Interesting stuff Marc. After reading I’m trying to get it working in my environment. I’m having an issue with the line
    ns[dataSourceName] = JSON.parse(window[storageType].getItem(cacheItemName)) || new DataSource();

    I’m getting an error ‘DataSource’ is undefined. Is there another script I need to include?

    Thanks
    Russ

  4. Hello Marc,

    Thanks and question.
    Have you considered also to option a local storage Db like PouchDb which can store a whole lot more data?

    Kind regards,
    Mario

    1. Mario:

      There are quite a few options here (more all the time, really), but I wanted to focus on what you could do cross-browser easily. It’s also an example based on my actual work, and this is how I solved it.

      M.

  5. Marc, thanks for this great article and sharing your experience! You sure helped me big time in learning to tame the Sharepoint beast over the years!

    As this is five years old already, are you aware of any changes to the functionality of the changeToken in recent Sharepoint revisions (I’m working on 2013)?

    Because for the life of me I’m unable to get a proper response when using it (SPServices.SPGetListItemsJson with changeToken provided). Neither with my own code nor with your exact code from this site (adjusted for variable names and paths, of course).

    I’m always getting an array back, with its length being (correctly) equal to the amount of touched list entries. Only that the array entries are simply empty proto objects. No reference to changed field, value or ID – nothing!

    I tried to provide the “All Items” view ID as “viewName” option or a dedicated “viewFields” string or even none of those (which should result to a fallback to default view), all to no avail.

    I also searched the web back and forth, but never found a reference to known problems (except for Microsoft stating that “PermMask” needs to be defined as “viewField” on lists with unique item permissions, though providing that viewField does not change the response for me).

    So in a nutshell: I do get a response as an array, whose length equals the number of changed list entries – but there is no content inside this array, so I can’t update my local copy, but instead have to re-load the whole list if any change occured. I do have full access rights to the list.

    Any idea what might be the culprit here or what settings I may have to check/change?

    1. @Ralph:

      It’s hard to say without seeing your actual code. The fact that you are getting the right number of items in the array would lead me to believe you aren’t requesting the columns correctly somehow.

      M.

Leave a Reply to Bruce Albany (@BruceAlbany) Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.