Caching SharePoint Data Locally with SPServices and HTML5’s Web Storage

The SharePoint SOAP Web Services are fast. In fact, I think they are as fast as the newer REST Web Services in many cases. The old, crufty SOAP Web Services even provide batching, something that the REST services don’t yet do. (Andrew Connell (@andrewconnell) has been beating the drum about this with Microsoft for months now, and we all hope they get this OData capability into SharePoint’s REST services sooner rather than later.)

Even though the SOAP services are fast, sometimes they just aren’t fast enough. In some of those cases, it may make sense to store some of your data in the browser’s Web storage so that it’s there on the client during a session or across sessions. Web storage is an HTML5 capability that is available in virtually all browsers these days, even Internet Explorer 8.

The best candidates for this type of storage (IMO) are list contents that are used as references and that don’t have a high number of changes. As an example, you might decide to store a list of countries in Web storage rather than loading them from the Countries list every time a page loads. Even though the read from the list is fast, it has to take *some* time. There’s the retrieval time, and then there is also any processing time on the client side. For instance, if you have dozens of fields per country and you need to load them into a complex JavaScript structure, that takes time, too. If those data chores are making your page loads seem laggy, then consider using local storage.

There are three main ways you can store data locally to improve performance. I’m not going to go into all of their intricacies, but I will give you some rules of thumb. There are a lot of nuances to this, so before you dive in, do some studying about how it all works.

Cookies

For small pieces of data, you should consider using cookies. Contrary to just about every article out there in the press, cookies are not bad. They can store up to 4k of data each for you, which you can read back when the user returns to the page. There’s a excellent little jQuery plugin I use to facilitate this called, aptly, jquery-cookie. You can download it (for free!) from GitHub here. Cookies persist across sessions.

Session Storage

Session storage is the flavor of Web storage that allows you to store data just for the duration of the session. Think of a session as a browser lifespan. Once you close the browser, the session storage is gone. Both session storage and local storage sizes are limited by the browser you are using. If you want to know if Web storage is available in your browser of choice, take a look at “Can I use“. The amount of storage each browser gives you is a moving target, but it’s per domain.

Local Storage

Local storage takes Web storage one step further. The data stored in local storage persists across browser sessions. In fact, it usually won’t go away until you explicitly delete it. (Consider this fact when you are gobbling up local storage in your development process.)

So What?

The trick with using these storage mechanisms is managing the data you’ve put in local storage as a cache. That data can go past its expiration date, either because some changes were made to the underlying data source or the cache itself has become corrupted. The latter is more difficult to deal with, so here I’ll focus on the former.

JavaScript – like most other programming languages – lends itself to building wrapper functions that add additional layers of abstraction on top of underlying functionality. Too many levels of abstraction can make things confusing, but with careful thought and smart code writing, you can build abstractions that serve you well.

In a recent client project, I found that as list data volumes were increasing, the pages in my SPServices- and KnockoutJS-driven application were loading more and more slowly. I’m building on top of SharePoint 2007 in this case, so even if I wanted to use REST, I couldn’t, nor do I believe that it would automatically make anything faster. If we had better servers running things, that might make a huge difference, but we have no control over that in the environment.

What I wanted was a reusable wrapper around SPGetListItemsJson (which itself is a wrapper around the SOAP List Web Service’s GetListItemChangesSinceToken and SPService’s SPXmlToJson) that would let me check local storage for a cached data source (list data), read either the entire data source or just the deltas from the SharePoint list, load the data into my application, and then update the cache appropriately.

The getDataSource function below is what I’ve come up with so far. There’s some setup to use it, so let me explain the parameters it takes:

  • ns – This is the namespace into which you want to load the data. In my applications these days, following the lead from the patterns Andrew and Scot Hillier (@scothillier) have published, I usually have a namespace defined that looks something like ProjectName.SubProjectName.DataSources. The “namespace” is simply a complex JavaScript object that contains most of my data and functions.
  • dataSourceName – The name that I want to give the specific data source within ns. In my example above with the Countries list I would use “Countries”.
  • params – This is the big magilla of the parameters. It contains all of the values that will make my call to SPGetListItemsJson work.
  • cacheItemName – This is the name of the item I want to store in Web storage. In the Countries example, I would use “ProjectName.SubProjectName.DataSources.Countries”.
  • storageType – Either “localStorage” or “sessionStorage”. If I expect the data to change regularly, I’d probably use sessionStorage (this gives me a clean data load for each session). If the data is highly static, I’d likely use localStorage.

And here’s the code:

/* Example:
getDataSource(ProjectName.SubProjectName.DataSources, "Countries", params: {
  webURL: "/",
  listName: "Countries",
  CAMLViewFields: "<ViewFields>" +
      "<FieldRef Name='ID'/>" +
      "<FieldRef Name='Title'/>" +
      "<FieldRef Name='Population'/>" +
      "<FieldRef Name='CapitalCity'/>" +
      "<FieldRef Name='Continent'/>" +
    "</ViewFields>",
  CAMLQuery: "<Query>" +
      "<OrderBy><FieldRef Name='ID'/></OrderBy>" +
    "</Query>",
  CAMLRowLimit: 0,
  changeToken: oldToken,
  mapping: {
      ows_ID:{"mappedName":"ID","objectType":"Counter"},
      ows_Title:{"mappedName":"Title","objectType":"Text"},
      ows_Population:{"mappedName":"Population","objectType":"Integer"},
      ows_CapitalCity:{"mappedName":"CapitalCity","objectType":"Text"},
      ows_Continent:{"mappedName":"Continent","objectType":"Lookup"},
    }
  }, "ProjectName.SubProjectName.DataSources.Countries"
)
*/

function getDataSource(ns, dataSourceName, params, cacheItemName, storageType) {

  var dataReady = $.Deferred();

  // Get the data from the cache if it's there
  ns[dataSourceName] = JSON.parse(window[storageType].getItem(cacheItemName)) || new DataSource();
  var oldToken = ns[dataSourceName].changeToken;
  params.changeToken = oldToken;

  // Read whatever we need from the dataSource
  var p = $().SPServices.SPGetListItemsJson(params);

  // Process the response
  p.done(function() {
    var updates = this.data;
    var deletedIds = this.deletedIds;
    var changeToken = this.changeToken;

    // Handle updates/new items
    if (oldToken !== "" && updates.length > 0) {
      for (var i = 0; i < updates.length; i++) {
        var thisIndex = ns[dataSourceName].data.binaryIndexOf(updates[i], "ID");
        // If the item is in the cache, replace it with the new data
        if (thisIndex > -1) {
          ns[dataSourceName].data[thisIndex] = updates[i];
          // Otherwise, add the new item to the cache
        } else {
          ns[dataSourceName].data.splice(-thisIndex, 0, updates[i]);
        }
      }
    } else if (oldToken === "") {
      ns[dataSourceName] = this;
    }
    // Handle deletes
    for (var i = 0; i < deletedIds.length; i++) {
      var thisIndex = ns[dataSourceName].data.binaryIndexOf({
        ID: deletedIds[i]
      }, "ID");
      ns[dataSourceName].data.splice(thisIndex, 1);
    }
    // Save the updated data back to the cache
    if (oldToken === "" || updates.length > 0 || deletedIds.length > 0) {
      // Save the new changeToken
      ns[dataSourceName].changeToken = changeToken;
      window[storageType].setItem(cacheItemName, JSON.stringify(ns[dataSourceName]));
    }
    dataReady.resolve();
  });
  return dataReady.promise();
}

Some of the nice things about this function:

  • It’s generic. I can call it for any list-based data source in SharePoint. (I started out building it for one data source and then generalized it.)
  • I call call it during a page life cycle to refresh the application data anytime I want or on a schedule, perhaps with setInterval.
  • I can set a lot of parameters to cover a lot of different use cases.
  • Each time I call it, it updates the cache (if it needs to) so that the next time I call it I get a “fresh” copy of the data.
  • It only loads the data that it needs to, by using the GetListItemChangesSinceToken capabilities.

And some downsides:

  • Since I know what data I’m working with in my application and that it will fit into the Web storage easily, I’m not worrying about failed saves.
  • If the cache does become corrupt (not something I expect, but there’s always Murphy), I’m not handling it at all.

If you decide to try this out, you’ll need a few auxiliary functions as well:

/* DataSource constructor */
function DataSource() {
  this.changeToken = "";
  this.mapping = {};
  this.data = [];
  this.deletedIds = [];
}

/** Adapted from http://oli.me.uk/2013/06/08/searching-javascript-arrays-with-a-binary-search/
 *
 * Performs a binary search on the host array.
 * @param {*} searchObj The object to search for within the array.
 * @param {*} searchElement The element in the object to compare. The objects in the array must be sorted by this element.
 * @return {Number} The index of the element. If the item is not found, the function returns a negative index where it should be inserted (if desired).
 */
Array.prototype.binaryIndexOf = function(searchObj, searchElement) {

  var minIndex = 0;
  var maxIndex = this.length - 1;
  var currentIndex;
  var currentElement;

  var searchValue = searchObj[searchElement];

  while (minIndex <= maxIndex) {
    currentIndex = (minIndex + maxIndex) / 2 | 0;
    currentElement = this[currentIndex];

    if (currentElement[searchElement] < searchValue) {
      minIndex = currentIndex + 1;
    } else if (currentElement[searchElement] > searchValue) {
      maxIndex = currentIndex - 1;
    } else {
      return currentIndex;
    }
  }

  return ~maxIndex;
}

Talko – Talk. Share. Do. – Predicting a Winner

Image from http://www.talko.com/

Image from http://www.talko.com/

Yesterday, a lot of us read about Ray Ozzie’s latest venture, Talko,  in the media. The guy does know how to get the word out. That and the fact that he basically invented Lotus Notes, built and sold Groove to Microsoft, became Microsoft’s chief technical officer or chief software architect (or whatever, depending on where you read it), and left Microsoft to, well, “Beyond that, Ray has no plans at this time“. Oh, and Groove is under the hood in OneDrive for Business. Comments on that last one withheld.

All kidding aside, in my playing around with Talko yesterday with Kris Wagner (@SharePointKris) and a few of my other nerd friends, I think Ray and the boys may have Another Big Thing here.

The big benefit of Talko in my mind is to bring all of the disjointed “conversations” we have across different tools on our phones together in one interface. How often do you start a “conversation” in texts (maybe surreptitiously from an another meeting), then move to email (at your desk after the meeting), then maybe call (once you realize that it’s easier than typing)?

With Talko, that can all happen in one annotated stream. You can flip from mode to mode in the stream painlessly. That gives you a mixed media record of the *real* conversation. No more hunting for “Where did Kris say that thing about…”. I can picture people basically living in the Talko app rather than flipping from one app to another. Once they have our captive attention, they can do with us what they choose.

This is what Skype or Lync should or could be, IMO. Ray’s going to give them a run for their money. I can hear the network admin conversations starting already about how to block Talko in the “enterprise”. Like Dropbox, it’s going to fill a need that people didn’t know they had and organizational secrets will be flying through the airwaves. I predict it’s a winner.

KnockoutJS – Creating a Comma-Delimited List of Values

KnockoutJS LogoI’ve been building a lot of great stuff with KnockoutJS lately. It seems that it can enable many – if not all – of the development requirements I have these days, whether I’m building a single-page application (SPA) or just adding a snippet of content to an existing page. I can even build KnockoutJS-enabled “Web Parts” by dropping  a Content Editor Web Part [CEWP] into a page. It doesn’t matter what version of SharePoint it is.

The KnockoutJS documentation is generally very good, but sometimes I find the examples lacking a bit. It’s always tempting to make examples show off too much, which often leads to showing off too little. (I have this same problem with my SPServices documentation and examples.)

A common use case is wanting to display a set of values in a comma-delimited list. Let’s take this example. Say I have an observable array of Ticker objects, like so:

self.Tickers = ko.observableArray([
  {lookupId: 1, lookupValue:"SPLS"},
  {lookupId: 2, lookupValue:"APPL"},
  {lookupId: 3, lookupValue:"GOOG"}
]);

Because I’m pulling data from SharePoint, I want to hang onto the lookupId value along with the text value, which is what I want to display. Because of this a simple Tickers.join(“, “) doesn’t cut it.

I’d like to display the list of tickers like this:

SPLS, APPL, GOOG

Pretty simple, right? But after a little Binglage, I couldn’t find a concise example, thus this post.

If you check the KnockoutJS documentation for foreach, you’ll see that there is a variable available called $index. The $index variable gives you the zero-based index of the current array item.

So if I use foreach on Tickers:

<div data-bind="template: { name: 'Ticker', foreach: Tickers }"></div>

I can use the $index to determine if I should emit a comma. We don’t want to see a comma after the last value, so it requires this small bit of finesse.

<script type="text/html" id="Ticker"><span data-bind="visible: $index() > 0">, </span><span data-bind="text: lookupValue"></span></script>

It’s a little bit bass-ackward, but if the $index value is greater than zero – which it is for all values except the first one, where the index is zero – then we *prepend* a comma to the value.

Yes, I’m using a separate template to emit the tickers. That’s mainly because in my case I’m doing a little bit more than what I’m showing here. However, by creating a separate template, I have a reusable piece of markup. that I can use in many places.

I hope someone out there finds this useful!

References

WordPress and Blackbird Pie: “There was a problem connecting to Twitter”

I have been using WordPress longer than a lot of people. I think I first moved from Windows Live’s blogging platform to WordPress around 2007. Because I’ve been here a long time, I’ve got some plugins that have been around for a long time. One of those is the Twitter Blackbird Pie WordPress Plugin.

The Blackbird Pie plugin was great when it came out. It let you use an embed code to add a live tweet directly into a post. It looked something like this:

[blackbirdpie id="507693704581500928"]

The id was the unique id for one individual tweet. Twitter was just getting popular and this seemed really cool.

Later, Blackbird Pie got smarter and you could just give it the URL for the tweet on Twitter’s site.

[blackbirdpie url="http://twitter.com/sympmarc/status/507693704581500928"]

The WordPress crew noticed the Blackbird Pie plugin and they even added it to WordPress in 2010. Instead of using the [blackbirdpie] embed code, you could just drop the url directly into your post. But dang it, it still wasn’t working for me.

I assumed the problem was that Twitter kept changing their interface and APIs and Blackbird Pie stopped working, regardless how you used it. For probably a couple of years now, all my Blackbird Pie -enabled tweet links have been broken. Each one has been showing the message “There was a problem connecting to Twitter”.

I’ve tried quite a few times to fix this, but I’ve gotten nowhere. All of the forum posts out there say things like “It was such a great plugin. Why doesn’t it work anymore?” The owner of the plugin seemed to give up on it, too.

WordPress 4.0 “Benny” came out today and I immediately upgraded, as I am want to do. One of the things I noticed in the release notes was that Twitter embeds are supposed to just work natively. “Well, why haven’t they been working on my site?”, I asked myself.

I don’t know why it occurred to me to do it, but I decided to uninstall the Blackbird Pie plugin. Presto, change-o, all was right with the world. Well, with tweets embedded in my test post. The problem was the Blackbird Pie plugin itself!

You can fix this on your blog by searching for all of the Blackbird Pie embed codes and switching to plain old WordPress embeds.

  • Go to Plugins and Deactivate and/or Deletel the Blackbird Pie plugin
  • Search for all of your posts with the [blackbirdpie] embed code by going to http://YourWordPressBlog/wp-admin/edit.php?s=blackbird
  • For each of those posts, edit it and copy the url to the tweet
  • Delete the embed code line
  • Paste the tweet url into your post

That should do it!

Adding Geolocation Columns to SharePoint Lists

One of the cool things that came along in SharePoint 2013 was Geolocation fields in lists. Using Geolocation fields, we can display Map Views in SharePoint lists. This capability is an awesome way to add visualization to your UI and can really add value in many business processes.

Map View

Image Source: http://msdn.microsoft.com/en-us/library/office/jj656773(v=office.15).aspx

Think about it. You could show:

  • A map of your customers in a region
  • A map of your office’s location on its Intranet page
  • Deliveries you’ve made in the last 30 days
  • etc.

Unfortunately, at the present time in SharePoint 2013 on premises and SharePoint Online (Office365), “The Geolocation column is not available by default in SharePoint lists. To add the column to a SharePoint list, you have to write code.” (See How to: Add a Geolocation column to a list programmatically in SharePoint 2013)

For whatever reason, there’s no way to add a new Geolocation field via the UI. Instead you have to go through some hoops with PowerShell or script. See the following articles for great tutelage on how to do this:

As I understand it, there’s an MSI package (SQLSysClrTypes.msi)that must be installed manually on the Web Front Ends (WFEs) in an on premises installation to enable Geolocation fields, but this is already in place in SharePoint Online. Given this, we should be able to add Geolocation columns to lists via the UI without PowerShell or admin intervention.

This doesn’t sit well with me. At some of my clients, getting an admin to add something to the WFEs or run PowerShell requires an act of Congress. These Geolocation capabilities are too powerful a SharePoint feature to keep them under wraps.

To wit, I’ve created a suggestion called Adding Geolocation Fields to SharePoint Lists on the Office Development UserVoice site. If you’d like to be able to use Geolocation fields in your SharePoint solutions – at least on Office365 – head on over and cast your vote(s). Remember, Microsoft is listening!