SPServices 2013.01ALPHA4 Returns a Deferred Object (Promise)

I’ve just posted a new alpha (ALPHA4) for SPServices 2013.01. This alpha implements the return of a deferred object, aka a promise, as well as executing a completefunc (if provided) for backward compatibility. Note that the deferred object capability was introduced in jQuery 1.5, so this version of SPServices requires that version or greater. I’ve done my initial testing with jQuery 1.8.2.

So, what does this mean to you, the SPServices user? Well, in the short term for most of you it may not mean anything unless you understand the first paragraph above. If you do, I could really use some help in testing the deferred object capability.

Here’s how I’ve implemented it. There are two big differences in the core SPServices function. First, I’ve set it up so that it runs a completefunc if one is provided, just has it always has, but also returns a promise. This means the code has gone from this:

var cachedXML;
var status = null;

if(opt.cacheXML) {
  cachedXML = $("body").data(msg);
}

if(cachedXML === undefined) {
  // Make the Ajax call
  $.ajax({
    url: ajaxURL,                      // The relative URL for the AJAX call
    async: opt.async,                    // By default, the AJAX calls are asynchronous.  You can specify false to require a synchronous call.
    beforeSend: function (xhr) {              // Before sending the msg, need to send the request header
      // If we need to pass the SOAPAction, do so
      if(WSops[opt.operation][1]) {
        xhr.setRequestHeader("SOAPAction", SOAPAction);
      }
    },
    type: "POST",                      // This is a POST
    data: msg,                        // Here is the SOAP request we've built above
    dataType: "xml",                    // We're getting XML; tell jQuery so that it doesn't need to do a best guess
    contentType: "text/xml;charset='utf-8'",        // and this is its content type
    complete: function(xData, Status) {
      if(opt.cacheXML) {
        $("body").data(msg, xData);        // Cache the results
      }
      cachedXML = xData;
      status = Status;
      opt.completefunc(cachedXML, status);        // When the call is complete, do this
    }
  });

} else {
  opt.completefunc(cachedXML, status);            // Call the completefunc
}

to this:

// Check to see if we've already cached the results
var cachedXML;
if(opt.cacheXML) {
  cachedXML = promisesCache[msg];
}

if(typeof cachedXML === "undefined") {

  // Finally, make the Ajax call
  var spservicesPromise = $.ajax({
    // The relative URL for the AJAX call
    url: ajaxURL,
    // By default, the AJAX calls are asynchronous.  You can specify false to require a synchronous call.
    async: opt.async,
    // Before sending the msg, need to send the request header
    beforeSend: function (xhr) {
      // If we need to pass the SOAPAction, do so
      if(WSops[opt.operation][1]) {
        xhr.setRequestHeader("SOAPAction", SOAPAction);
      }
    },
    // Always a POST
    type: "POST",
    // Here is the SOAP request we've built above
    data: msg,
    // We're getting XML; tell jQuery so that it doesn't need to do a best guess
    dataType: "xml",
    // and this is its content type
    contentType: "text/xml;charset='utf-8'",
    complete: function(xData, Status) {
      // When the call is complete, call the completefunc if there is one
      if($.isFunction(opt.completefunc)) {
        opt.completefunc(xData, Status);

      }
    }
  });

  spservicesPromise.then(
    function() {
      // Cache the promise if requested
      if(opt.cacheXML) {
        promisesCache[msg] = spservicesPromise;
      }
    },
    function() {
                                // TODO: Allow for fail function
    }
  );

  // Return the promise
  return spservicesPromise;

} else {
  // Call the completefunc if there is one
  if($.isFunction(opt.completefunc)) {
    opt.completefunc(cachedXML, null);
  }
  // Return the cached promise
  return cachedXML;
}

I don’t usually post the differences in the code in such detail, but I want to get some eyeballs on it so that I can gather yeas or nays on the implementation. I wanted to keep it as clean as possible yet maintain 100% backward capability. I seem to have accomplished both in all of my testing, but I’d love feedback. I’m especially curious what folks think I should do in the failure case of the .then() function. In the past I’ve not done anything, leaving it to the SPServices user to decide what they would like to do. The only two options I can think of are:

  1. Add a retry loop to give it a few more shots before giving up
  2. Pop up an error message of some sort

Neither of these two options feel right to me. If the .ajax() call fails, in my experience it’s because the server isn’t responding 99.9999% of the time. Trying again won’t usually solve that unless it’s simply a lag after an IISRESET. Popping up an error doesn’t really help, either, as it would have to be some sort of generic “Contact your administrator” nonsense, which I avoid assiduously. Thoughts?

The second change is that I’ve moved from using the .data() jQuery function for caching to caching the promises in an array, as you can see in the code above. This difference should be invisible to all end users and to most developers who use SPServices. However, since it means a difference in how the caching works since I introduced it in v0.7.2, I want to be sure that I get people testing that as well.

Finally, as a further test of the deferred object capability, in this alpha the SPArrangeChoices function also uses the returned promise for GetList as a test to improve the performance and user experience. This function was a good candidate for the first internal test of the new architecture, as it is relatively straightforward and only makes one Web Services call to GetList. I’ll be implementing more uses of promises inside the library as it makes sense before finalizing this release.

So if you’re a hard-core SPServices user, please give this one a test for me. Obviously I’m interested in any regression, but I’d also like to know how the returned promises work for you if you can write new calls against this version. As an illustration of how you can use this new capability, here’s what I’ve done in SPArrangeChoices:

// Rearrange radio buttons or checkboxes in a form from vertical to horizontal display to save page real estate
$.fn.SPServices.SPArrangeChoices = function (options) {

  var opt = $.extend({}, {
    listName: $().SPServices.SPListNameFromUrl(),          // The list name for the current form
    columnName: "",          // The display name of the column in the form
    perRow: 99,            // Maximum number of choices desired per row.
    randomize: false        // If true, randomize the order of the options
  }, options);

  var columnFillInChoice = false;
  var columnOptions = [];
  var out;

  // Get information about columnName from the list to determine if we're allowing fill-in choices
  var thisGetList = $().SPServices({
    operation: "GetList",
    async: false,
    cacheXML: true,
    listName: opt.listName
  });

  // when the promise is available...
  thisGetList.done(function() {
    $(thisGetList.responseXML).find("Field[DisplayName='" + opt.columnName + "']").each(function() {
      // Determine whether columnName allows a fill-in choice
      columnFillInChoice = ($(this).attr("FillInChoice") === "TRUE") ? true : false;
      // Stop looking;we're done
      return false;
    });
[...]
  });
}; // End $.fn.SPServices.SPArrangeChoices

Here’s another chunk of test code that I’ve been using to see that the caching is working as I’d like. The Census Data list on my Demos site has over 3000 items in it, so it’s a good list to test timings with, as it takes a comparatively long time to send the data down the wire. I’ve got the async option and the completefunc commented out below; you can play around with the combinations here as I have if you’d like.

function getIt() {
  var outUl = $("#WSOutput ul");

  logTime(outUl, "Start: ");
  var getListItemsPromise = $().SPServices({
    cacheXML: true,
//    async: false,
    operation: "GetListItems",
    webURL: "/Demos/",
    listName: "Census Data"
//    completefunc: function (xData, Status) {
//      logTime(outUl, "completefunc: " + $(xData.responseXML).SPFilterNode("rs:data").attr("ItemCount"));
//    }
  });
  logTime(outUl, "End: ");

    getListItemsPromise.done(function() {
    logTime(outUl, "promiseComplete: " + $(getListItemsPromise.responseXML).SPFilterNode("rs:data").attr("ItemCount"));
    });
}

function logTime(o, t) {
  o.append("<li>" + new Date() + " :: " + t + "</li>");
}

with this simple markup:

<input type="button" onclick="getIt();" value="Get It"/>
<div class="ms-vb" style="width:100%;" id="WSOutput"><ul></ul></div>

Let me know what you think, and stay tuned for more changes in this release, which you can keep track of in the Issue Tracker.

Thanks go to Scot Hillier (@scothillier) for opening my eyes to the value of deferred objects by showing examples in several of his recent sessions.

SPServices Futures: Moving to jQuery’s Deferred Objects and More

As I gear up to work on the next release of SPServices, I want to make some pretty fundamental changes/improvements to the internal plumbing. What I’m hoping to do with this post is to gather any ideas and feedback that the user community has about the implementation before I go too far with things.

SPServices

Deferred Object and Promises

In all prior versions, we’ve relied on the completefunc (which I made up) to process the results we get back from a Web Services operation call. In the meantime, jQuery has moved forward significantly.

In jQuery 1.5, we got the first deferred object capabilities. I was loathe to change the internal workings of SPServices at the time because we had a substantial user base, even then.

Now the user base is significantly larger (as of this writing, there have been almost 75,000 [!] total downloads), but I think it’s time to add the deferred capabilities into the core SPServices function. More and more people have been exposed to deferred processing capabilities in script at this point, so it will be less of a shock to existing SPServices users (I hope!).

Oddly, I’ve only had one request (at least that I can remember) to add deferred processing into SPServices. It came from a Codeplex user named MgSam in a thread s/he titled JQuery Deferred compatibility, and I’ll admit that I blew it off a bit when he raised it as a possibility.

My strongest impetus for considering this fundamental architecture shift is having seen Scott Hillier talk several times about Utilizing Promises in SharePoint 2013 Apps. In actual fact, his talk applies to JavaScript (and thus jQuery) development more than it applies to SharePoint 2013 development. We can use the deferred methodology even if we’re running our scripts on SharePoint 2007. It’s a way to manage requests in the script and has little to do with the version of SharePoint.

In doing this, my goal will be to keep full backward compatibility so that all existing code continues to function with no changes, but I may deprecate the current completefunc focused methodology, either in this next release or shortly thereafter.

in the simplest form, it will look something like this (borrowing from MgSam’s example in the thread above):

function myCode(someOtherListId) {
    /* Ideally, I should be able to do this call SPListNameFromUrl asynchronously
     * as well, perhaps add an optional "async" parameter
     * which, if used, returns the jqXHR object and invokes
     * a callback upon completion rather than blocking until
     * the data returns. This is not the main point of this code though.
     */
    var currentListId = $().SPServices.SPListNameFromUrl();

    //Call 1
    var a = $().SPServices({
        operation: "GetListItems",
        async: true,
        listName: currentListId,
        completefunc: function(xData) {
            //Do something useful
        }
    });

    //Call 2
    var b = $().SPServices({
        operation: "GetList",
        async: true,
        listName: currentListId,
        completefunc: function(xData) {
            //Do something useful
        }
     });

     //Call 3
     var c = $().SPServices({
        operation: "GetListItems",
        async: true,
        listName: someOtherListId,
        completefunc: function(xData) {
            //Do something useful
        }
     });

     /* Now I want to do something once all 3
      * asynchronous calls have completed. Under the current
      * model, there's no easy way to do this.
      * I either have to hand-roll some ugly code,
      * or contort my completefuncs so that they work with
      * jQuery's Deferred.
     */
}

Moving toward the deferred object approach will also mean that I’ll replace the simplistic caching mechanism I put into place using the .data() function in v0.7.2. There will be no visible difference in the way it works, but the underlying mechanism will be deferred promises instead.

SharePoint 2007

I’m also considering finally ending development specifically for SharePoint 2007. I’ve continued to use WSS 3.0 as my development environment since the first release of SPServices so that I can “test up” rather than down *and* up.

I still have a lot of people contacting me with questions who are using SharePoint 2007. In fact, some of my best clients are still on SharePoint 2007. It’s still a great platform that works well for many people who are still using it.

That said, there are some capabilities in SharePoint 2010 and now SharePoint 2013 that I haven’t tried to take advantage of because they aren’t there in SharePoint 2007. It isn’t that I want to abandon the SharePoint 2007 folks, but I’m considering making the main focus SharePoint 2010.

I’m very interested in hearing people’s thoughts on that. There’s no easy way for me to know what versions everyone out there has, so chime in.

Version Numbering

These changes will definitely merit a more significant version bump than just going from 0.7.2 to 0.7.3, of course. Is it finally time to ship a 1.0.0? Or maybe I should just decide to go to 5.3.8 or something. The numbers mean nothing, of course, as long as they go in a consistent direction. I’m even toying with 2013.01. Any thoughts on this?

Thanks to everyone out there who uses SPServices, especially those of you who have contacted me to let me know how it has helped you accomplish your business goals. If you have a story to tell, please consider writing it up for my SPServices Stories series. I’m planning to keep that series going as long as there are good stories to tell.