Caching SharePoint Data Locally with SPServices and HTML5’s Web Storage

The SharePoint SOAP Web Services are fast. In fact, I think they are as fast as the newer REST Web Services in many cases. The old, crufty SOAP Web Services even provide batching, something that the REST services don’t yet do. (Andrew Connell (@andrewconnell) has been beating the drum about this with Microsoft for months now, and we all hope they get this OData capability into SharePoint’s REST services sooner rather than later.)

Even though the SOAP services are fast, sometimes they just aren’t fast enough. In some of those cases, it may make sense to store some of your data in the browser’s Web storage so that it’s there on the client during a session or across sessions. Web storage is an HTML5 capability that is available in virtually all browsers these days, even Internet Explorer 8.

The best candidates for this type of storage (IMO) are list contents that are used as references and that don’t have a high number of changes. As an example, you might decide to store a list of countries in Web storage rather than loading them from the Countries list every time a page loads. Even though the read from the list is fast, it has to take *some* time. There’s the retrieval time, and then there is also any processing time on the client side. For instance, if you have dozens of fields per country and you need to load them into a complex JavaScript structure, that takes time, too. If those data chores are making your page loads seem laggy, then consider using local storage.

There are three main ways you can store data locally to improve performance. I’m not going to go into all of their intricacies, but I will give you some rules of thumb. There are a lot of nuances to this, so before you dive in, do some studying about how it all works.

Cookies

For small pieces of data, you should consider using cookies. Contrary to just about every article out there in the press, cookies are not bad. They can store up to 4k of data each for you, which you can read back when the user returns to the page. There’s a excellent little jQuery plugin I use to facilitate this called, aptly, jquery-cookie. You can download it (for free!) from GitHub here. Cookies persist across sessions.

Session Storage

Session storage is the flavor of Web storage that allows you to store data just for the duration of the session. Think of a session as a browser lifespan. Once you close the browser, the session storage is gone. Both session storage and local storage sizes are limited by the browser you are using. If you want to know if Web storage is available in your browser of choice, take a look at “Can I use“. The amount of storage each browser gives you is a moving target, but it’s per domain.

Local Storage

Local storage takes Web storage one step further. The data stored in local storage persists across browser sessions. In fact, it usually won’t go away until you explicitly delete it. (Consider this fact when you are gobbling up local storage in your development process.)

So What?

The trick with using these storage mechanisms is managing the data you’ve put in local storage as a cache. That data can go past its expiration date, either because some changes were made to the underlying data source or the cache itself has become corrupted. The latter is more difficult to deal with, so here I’ll focus on the former.

JavaScript – like most other programming languages – lends itself to building wrapper functions that add additional layers of abstraction on top of underlying functionality. Too many levels of abstraction can make things confusing, but with careful thought and smart code writing, you can build abstractions that serve you well.

In a recent client project, I found that as list data volumes were increasing, the pages in my SPServices- and KnockoutJS-driven application were loading more and more slowly. I’m building on top of SharePoint 2007 in this case, so even if I wanted to use REST, I couldn’t, nor do I believe that it would automatically make anything faster. If we had better servers running things, that might make a huge difference, but we have no control over that in the environment.

What I wanted was a reusable wrapper around SPGetListItemsJson (which itself is a wrapper around the SOAP List Web Service’s GetListItemChangesSinceToken and SPService’s SPXmlToJson) that would let me check local storage for a cached data source (list data), read either the entire data source or just the deltas from the SharePoint list, load the data into my application, and then update the cache appropriately.

The getDataSource function below is what I’ve come up with so far. There’s some setup to use it, so let me explain the parameters it takes:

  • ns – This is the namespace into which you want to load the data. In my applications these days, following the lead from the patterns Andrew and Scot Hillier (@scothillier) have published, I usually have a namespace defined that looks something like ProjectName.SubProjectName.DataSources. The “namespace” is simply a complex JavaScript object that contains most of my data and functions.
  • dataSourceName – The name that I want to give the specific data source within ns. In my example above with the Countries list I would use “Countries”.
  • params – This is the big magilla of the parameters. It contains all of the values that will make my call to SPGetListItemsJson work.
  • cacheItemName – This is the name of the item I want to store in Web storage. In the Countries example, I would use “ProjectName.SubProjectName.DataSources.Countries”.
  • storageType – Either “localStorage” or “sessionStorage”. If I expect the data to change regularly, I’d probably use sessionStorage (this gives me a clean data load for each session). If the data is highly static, I’d likely use localStorage.

And here’s the code:

function getDataSource(ns, dataSourceName, params, cacheItemName, storageType) {
  /* e.g., getDataSource(ProjectName.SubProjectName.DataSources, "Countries", params: {
      webURL: "/",
      listName: "Countries",
      CAMLViewFields: "<ViewFields>" +
          "<FieldRef Name='ID'/>" +
          "<FieldRef Name='Title'/>" +
          "<FieldRef Name='Population'/>" +
          "<FieldRef Name='CapitalCity'/>" +
          "<FieldRef Name='Continent'/>" +
        "</ViewFields>",
      CAMLQuery: "<Query>" +
        "<OrderBy><FieldRef Name='ID'/></OrderBy>" +
      "</Query>",
      CAMLRowLimit: 0,
      changeToken: oldToken,
      mapping: {
          ows_ID:{"mappedName":"ID","objectType":"Counter"},
          ows_Title:{"mappedName":"Title","objectType":"Text"},
          ows_Population:{"mappedName":"Population","objectType":"Integer"},
          ows_CapitalCity:{"mappedName":"CapitalCity","objectType":"Text"},
          ows_Continent:{"mappedName":"Continent","objectType":"Lookup"},
        }
      }, "ProjectName.SubProjectName.DataSources.Countries"
    )
*/

  var dataReady = $.Deferred();

  // Get the data from the cache if it's there
  ns[dataSourceName] = JSON.parse(window[storageType].getItem(cacheItemName)) || new DataSource();
  var oldToken = ns[dataSourceName].changeToken;
  params.changeToken = oldToken;

  // Read whatever we need from the dataSource
  var p = $().SPServices.SPGetListItemsJson(params);

  // Process the response
  p.done(function() {
    var updates = this.data;
    var deletedIds = this.deletedIds;
    var changeToken = this.changeToken;

    // Handle updates/new items
    if (oldToken !== "" && updates.length > 0) {
      for (var i = 0; i < updates.length; i++) {
        var thisIndex = ns[dataSourceName].data.binaryIndexOf(updates[i], "ID");
        // If the item is in the cache, replace it with the new data
        if (thisIndex > -1) {
          ns[dataSourceName].data[thisIndex] = updates[i];
          // Otherwise, add the new item to the cache
        } else {
          ns[dataSourceName].data.splice(-thisIndex, 0, updates[i]);
        }
      }
    } else if (oldToken === "") {
      ns[dataSourceName] = this;
    }
    // Handle deletes
    for (var i = 0; i < deletedIds.length; i++) {
      var thisIndex = ns[dataSourceName].data.binaryIndexOf({
        ID: deletedIds[i]
      }, "ID");
      ns[dataSourceName].data.splice(thisIndex, 1);
    }
    // Save the updated data back to the cache
    if (oldToken === "" || updates.length > 0 || deletedIds.length > 0) {
      // Save the new changeToken
      ns[dataSourceName].changeToken = changeToken;
      window[storageType].setItem(cacheItemName, JSON.stringify(ns[dataSourceName]));
    }
    dataReady.resolve();
  });
  return dataReady.promise();
}

Some of the nice things about this function:

  • It’s generic. I can call it for any list-based data source in SharePoint. (I started out building it for one data source and then generalized it.)
  • I call call it during a page life cycle to refresh the application data anytime I want or on a schedule, perhaps with setInterval.
  • I can set a lot of parameters to cover a lot of different use cases.
  • Each time I call it, it updates the cache (if it needs to) so that the next time I call it I get a “fresh” copy of the data.
  • It only loads the data that it needs to, by using the GetListItemChangesSinceToken capabilities.

And some downsides:

  • Since I know what data I’m working with in my application and that it will fit into the Web storage easily, I’m not worrying about failed saves.
  • If the cache does become corrupt (not something I expect, but there’s always Murphy), I’m not handling it at all.

KnockoutJS – Creating a Comma-Delimited List of Values

KnockoutJS LogoI’ve been building a lot of great stuff with KnockoutJS lately. It seems that it can enable many – if not all – of the development requirements I have these days, whether I’m building a single-page application (SPA) or just adding a snippet of content to an existing page. I can even build KnockoutJS-enabled “Web Parts” by dropping  a Content Editor Web Part [CEWP] into a page. It doesn’t matter what version of SharePoint it is.

The KnockoutJS documentation is generally very good, but sometimes I find the examples lacking a bit. It’s always tempting to make examples show off too much, which often leads to showing off too little. (I have this same problem with my SPServices documentation and examples.)

A common use case is wanting to display a set of values in a comma-delimited list. Let’s take this example. Say I have an observable array of Ticker objects, like so:

self.Tickers = ko.observableArray([
  {lookupId: 1, lookupValue:"SPLS"},
  {lookupId: 2, lookupValue:"APPL"},
  {lookupId: 3, lookupValue:"GOOG"}
]);

Because I’m pulling data from SharePoint, I want to hang onto the lookupId value along with the text value, which is what I want to display. Because of this a simple Tickers.join(“, “) doesn’t cut it.

I’d like to display the list of tickers like this:

SPLS, APPL, GOOG

Pretty simple, right? But after a little Binglage, I couldn’t find a concise example, thus this post.

If you check the KnockoutJS documentation for foreach, you’ll see that there is a variable available called $index. The $index variable gives you the zero-based index of the current array item.

So if I use foreach on Tickers:

<div data-bind="template: { name: 'Ticker', foreach: Tickers }"></div>

I can use the $index to determine if I should emit a comma. We don’t want to see a comma after the last value, so it requires this small bit of finesse.

<script type="text/html" id="Ticker"><span data-bind="visible: $index() > 0">, </span><span data-bind="text: lookupValue"></span></script>

It’s a little bit bass-ackward, but if the $index value is greater than zero – which it is for all values except the first one, where the index is zero – then we *prepend* a comma to the value.

Yes, I’m using a separate template to emit the tickers. That’s mainly because in my case I’m doing a little bit more than what I’m showing here. However, by creating a separate template, I have a reusable piece of markup. that I can use in many places.

I hope someone out there finds this useful!

References

Detecting the Current User’s Regional Settings with jQuery

I noticed a nice little trick going by in my Twitter feeds today.
2014-08-20_19-09-43I’ve always wanted to know how to do this. Displaying the right formats for dates and times as well as the right offset from UTC is a real challenge from the client side. There’s a fantastic library called Moment.js that I use on just about every project these days, but it only does the full job if you know the user’s locale and timezone.

The post on StackExchange gave a little snippet of script which came from an old MSDN Forums post from 2009. Kudos to Peter Holpar for the original idea.

The script basically screen-scrapes the values from the Language and Region settings page. I don’t mean that disparagingly at all; screen-scraping is how the SPGetCurrentUser function in my SPServices works. If it gets the job done and is reliable, the old art of screen-scraping – or should we call it plumbing the DOM with AJAX? – is just great.

Note that you’ll need to have jQuery loaded to use its AJAX function.

Here’s the script as-is from the StackExchange post. It was originally intended for SharePoint 2010.

var url = ctx.HttpRoot+"/_layouts/regionalsetng.aspx?Type=User";
$.get(url,function(data){
    $(data).find("select[name='ctl00$PlaceHolderMain$ctl00$ctl00$DdlwebLCID']").find(":selected").each(function(){
        var lcid  = $(this).attr("value");
        var cultureInfo  = $(this).text();
    });
});

I just tested it out in SharePoint Online on Office365 and this is the version that works for me. Note that I simplified the jQuery selectors to look just for the important ending text in the name attribute. I’ve also re-scoped the variables so that they are available outside the $.get call. If you want to use this snippet, you’ll probably need to adapt it for your environment. The alert at the end is jut there to show you that it works.

var url = _spPageContextInfo.webAbsoluteUrl+"/_layouts/regionalsetng.aspx?Type=User";
var lcid, cultureInfo, timeZone;

$.get(url,function(data){
  $(data).find("select[name$='LCID'] option:selected").each(function(){
      lcid  = $(this).attr("value");
      cultureInfo  = $(this).text();
  });
  $(data).find("select[name$='TimeZone'] option:selected").each(function(){
    timeZone  = $(this).text();
  });

  alert(lcid + "::" + cultureInfo + "::" + timeZone);
});

Renegade Idea: Expose REST Endpoints in the SharePoint UI

Here’s an idea I had today based on a conversation I had with some of the other SharePoint MVPs: expose the REST endpoints and documentation right in the UI for those who have the appropriate permissions. It could be a little icon at the bottom of any page where there are relevant end points. All content is security trimmed anyway, so no one is going to see an endpoint they can’t use (though if they can fathom the docs they can know what endpoints there are available).

Think about it:

  • You’re on a List Settings page, and you can see the endpoint for that
  • You’re on a workflow status page…
  • You’re in Central Admin on the Create Site Collection page…
  • You’re on a Term Set Administration page…
  • You’re on a List View page…

All of the APIs are more obtuse than they need to be. In many cases, the docs are either very long-winded without explaining basic operations or just content stubs. By adding these direct connections from the matching UI pages, it would:

  • Satisfy the developer needs for easy to understand documentation
  • Educate appropriate [power] users about how to talk to the devs about what they want
  • Potentially get telemetry to Microsoft about what people want in the APIs and their docs

There’s far too much of the “I can’t look it up in the dictionary until I know how to spell it already” going on with the APIs in SharePoint in general – going back to the SOAP Web Services and beyond. The platform needs to be more open, not less. Adding these direct links to the REST endpoints and their docs would be an incredibly powerful step in that direction.

What do you think of this idea? It’s mine alone and even if it’s the best idea I’ve ever had (probably not) there no guarantee that it would make any sense to the Product Group. Leave a comment, though, and add your own thoughts about how to make the SharePoint APIs more usable.

If you think this is a good idea, vote for it on the Expose REST Endpoints in the SharePoint UI item on the Office Development UserVoice site.

20140801-225509-82509986.jpg
This is a piece of art from a gallery in Hoi An, Vietnam showing a pig. As you can see, it’s even possible to make a pig look nice with a little creativity and work.

Using the Signature Pad jQuery Plugin with SharePoint & InfoPath

Have you ever needed to capture signatures in SharePoint forms? What about InfoPath forms? I’m betting many people have seen a need to do this but have balked at the cost or complexity of the available solutions.

A client of mine named Cody Sellers (@codyjsellers), who works for Mercom Corporation had just such a need. Mercom is a smallish company that didn’t want to spend tens of thousands of dollars on a “real” eSig solution. Cody did some research and found a great jQuery plugin and turned to me for help implementing it. In this case we’re working with SharePoint 2013 on premises, but this should work with any version of SharePoint with some modification.

2014-06-18_13-38-20

The Signature Pad plugin comes from a clearly smart guy named Thomas J Bradley. Based on the work I’ve done with it so far, it’s well written with good documentation. Even better: it just plain works.

Signature Pad: A jQuery plugin for assisting in the creation of an HTML5 canvas based signature pad. Records the drawn signature in JSON for later regeneration.

The plugin allows you to capture signatures based on typed or drawn input and can be saved as JSON or a PNG image.

In our case, we wanted to be able to add a signature block at the bottom of multiple InfoPath forms. To make this work well, we decided that we would add the script into the master page. Yes, that may seen foolhardy, but a large majority of the work that is going to be happening in this Site Collection is filling out these forms.

Because of this, we needed a clear convention to follow in every form so that we could locate all of the signature fields (some forms have more than one) and add the signature pad capability reliably.

One of the hardest things about working with InfoPath forms client-side is that there are almost no sensible hooks in the emitted DOM. Everything is a .NET-like element id, like

ctl00_ctl33_g_3782ce51_9259_4854_80a0_e6355e54b690_FormControl0_V1_I1_T6

Since those ids can change anytime you change the form, you really want to give yourself something more solid to hook into. (At least we know that InfoPath won’t change anymore and the basic form elements are constant!)

To ensure that we could easily add the signature pad to any existing signature fields, we wrapped each signature field in an InfoPath section with its ScreenTip set to “==Signature==”.

InfoPath section with its ScreenTip set to ==Signature==

InfoPath section with its ScreenTip set to ==Signature==

The logic works like this:

  • Check to see if there’s a div with its id ending in ‘XmlFormView’ – this is the container for an InfoPath form. If there isn’t one, do nothing.
  • Find sections in the form that have their ScreenTip set to “==Signature==”. Sections have a fieldset container, and the title is set to the ScreenTip value. In other words, any section with its ScreenTip set to “==Signature==” will be treated as a signature area by the code.
  • Find the input element inside that section. This is the field where we want to store the JSON representation of the signature.
  • If the input element has no value, render the signature pad for signing (new forms)
  • If the input element already has a value, show the signature it contains (edit and display forms)

We decided to store the JSON representation of the signature in a field in each InfoPath form. Since the JSON represents the vectors from an HTML5 canvas, it’s a nice, standard way to store it. Images would require some separate repository or further encoding to be stored in a text field. Since the plugin can both emit and reconstitute the JSON into a signature, it’s a good way to go.

We ended up with the code below for the simplest case. (It’ll get more complicated as we get into the business rules, but this post is about the technology to capture and display the signatures.) When we create a new item in the newifs.aspx form, the Signature column will be empty and thus we will show the signature pad. If there’s already a value in the field, then we’re on the edit or display form, and signing has already happened so we just display the signature we have.

I’ve added comments that hopefully make the code simple enough to follow. As with all code you read on the Web, this isn’t just a drop-it-in-and-it’ll-work thing – you’ll end up tailoring it for sure. (One would think I wouldn’t need these caveats, but…)

$(document).ready(function() {
    // Find the guts of the InfoPath form in the page
    var infoPathContainer = $("div[id$='XmlFormView']");
    // if there's no InfoPath form, then we have nothing to do here (escape early)
    if (infoPathContainer.length !== 0) {
        setupSignatures(infoPathContainer);
    }
});
// There's an InfoPath form in the page, so look for signature fields and set them up
function setupSignatures(infoPathContainer) {
    // The Signature "signature" in the form. This makes it easy to select the proper form elements
    var signatureSignature = "==Signature==";
    // The signature field should be inside a section with its tooltip=signatureSignature
    var signatureContainer = infoPathContainer.find("fieldset[title='" + signatureSignature + "']");
    // The Signature field is the lone input element in the section
    var signatureBox = signatureContainer.find("input");
    // We may have multiple signature fields in the form
    signatureBox.each(function() {
        var thisSignatureBox = $(this);
        // We need a reference to the fieldset which represents the section several times
        var thisSignatureFieldset = thisSignatureBox.closest("fieldset");
        // Clean up the display by hiding the field we're using to store the JSON
        thisSignatureBox.hide();
        // The signature data is the value of the signatureBox
        var signatureData = thisSignatureBox.val();
        // If there's no signature yet...
        if (signatureData === "") {
            // Add the appropriate markup to the page
            thisSignatureBox.before(buildSignatureBox());
            // Bind to the click event for the 'Ready to Sign' button
            thisSignatureFieldset.find(".signature-ready").click(function() {
                $(this).toggle();
                $(this).next("div").toggle();
            });
            // Activate the signature pad in drawOnly mode
            var signatureArea = thisSignatureFieldset.find(".signature-box").signaturePad({
                drawOnly: true,
                lineTop: 125,
                output: thisSignatureBox,
                onBeforeValidate: function(context, settings) {
                    thisSignatureBox.focus(); // Needed to fire the change events bound to the field
                    thisSignatureBox.blur(); // Needed to fire the change events bound to the field
                    thisSignatureBox.hide(); // In case it becomes visible again
                }
            });
            // When the user clicks the button below the signature pad, validate
            thisSignatureFieldset.find(".signature-done").click(function() {
                thisSignatureFieldset.find("p.error").remove();
                signatureArea.validateForm();
            });
            // If we already have signature data, just show the existing signature
        } else {
            // Add the appropriate markup to the page
            thisSignatureBox.before(buildSignatureDisplay());
            // Activate the signature pad in displayOnly mode
            thisSignatureFieldset.find(".sigPad").signaturePad({
                displayOnly: true
            }).regenerate(signatureData);
        }
    });
}
// Function to emit the markup for the signature pad in signing mode
function buildSignatureBox() {
    var signatureBox = "<div class='signature-box'>" +
        "<input class='signature-ready' type='button' value='Ready to sign'/>" +
        "<div style='display:none;'>" +
        "<ul class='sigNav'>" +
        "<li class='drawIt'><a href='#draw-it'>Sign Here</a></li>" +
        "<li class='clearButton'><a href='#clear'>Clear</a></li>" +
        "</ul>" +
        "<div class='sig sigWrapper'>" +
        "<canvas class='pad' width='700' height='150'></canvas>" +
        "<input type='hidden' name='output' class='output'>" +
        "</div>" +
        "<input class='signature-done' type='button' value='Capture signature'/>" +
        "</div>" +
        "</div>";
    return signatureBox;
}
// Function to emit the markup for the signature pad in display mode
function buildSignatureDisplay() {
    var signatureDisplay = "<div class='sigPad signed'>" +
        "<div class='sigWrapper'>" +
        "<canvas width='700' height='150' class='pad'></canvas>" +
        "</div>" +
        "</div>";
    return signatureDisplay;
}

Here are some screenshots from our proof of concept:

The form when it loads

The form when it loads

Ready to sign

Ready to sign

Signed

Signed

Signature displayed on the display form

Signature displayed on the display form

Snippet of the data that is stored for a signature

Snippet of the data that is stored for a signature