Moving from SPServices to Rest, Part 5: Using Promises with SOAP and REST

Wouldn’t it be great if we could drive flying cars like the DeLorean in Back to the Future? Unfortunately, even though October 21, 2015 has passed us by, that’s not a reality yet. The science fiction writer Arthur C. Clarke famously said,Any sufficiently advanced technology is indistinguishable from magic.” Sometimes JavaScript promises can feel a little bit like magic – at least until you get good at working with them.

Paul Nigh's '' Back to the Future DeLorean Time Machine - Image source:

Paul Nigh’s ‘’ Back to the Future DeLorean Time Machine
Image source:

What are promises?

Both SOAP calls using SPServices and REST calls return promises. This is the case primarily because we are usually using a similar transport mechanism: maybe jQuery’s .ajax() function, which sends HTTP requests. (We don’t have to use jQuery, but it’s one of the more commonly used libraries to do this sort of thing. If you’re using one of the alternatives, you’ll need to do a little translation here.)

Promises are very cool mechanisms that work something like this. I think of it as a bit of magic, maybe like an act on America’s Got Talent.

When you make a call to SPServices or REST, you can assign the result of that call to a variable. That variable is a promise, which is sort of like an empty envelope. There’s no data in it when you first get it, but the call you just made promises to fill up the envelope for you. When the data is available and any conversion is finished, the original call sort of taps you on the shoulder and says, “You’ve got data!” (Say that in your best AOL voice.) At that point, you can start to do whatever work you want to do with the data.

Pretty neat, right? But, why does it matter? Well, when you are making AJAX calls, it can take a while for the server to respond. Servers are usually very powerful but they can get busy, or you may have asked for something complicated, or you may have asked for a lot of data, or the server may simply be tuned very badly. In any of those cases there will be a delay between the time you make the request and the time the response is available to you. The delay may be a few milliseconds or even a minute or two (though that probably indicates a poorly thought out request or a badly tuned server). During the time that delay occurs, you don’t want your JavaScript code to sit there waiting. Instead, you want to handle the call asynchronously meaning you don’t wait for it to finish; you elect to be notified when it finishes. That’s where promises come in.

Each browser reacts slightly differently to synchronous calls. Internet Explorer is probably the worst of the bunch, as it really hangs up until the request is complete. This makes for an unpleasant user experience, which ought never to be our goal. Because of this, using a promises pattern is important to ensure that your application feels responsive and peppy to your users.

What does converting to the promises pattern get us?

If you’ve been using SPServices for a long time, you may be setting async: false in many or all of your calls. This means you are making the calls synchronously, hanging up the browser. By moving to the promises pattern while still using SPServices, you’ll be setting yourself up for making REST calls efficiently down the road.

Let’s look at a simple example where we are just retrieving some items from a Custom List.

  operation: "GetListItems",
  async: false,
  listName: "MyCustomList",
  completefunc: function(xData,status) {

    $(xData.responseXML).SPFilterNode("z:row").each(function() {
      // Process the data


In this example, a SOAP payload will be sent to the server, and our code will sit and wait until the response comes back because we have specified async: false. If we’re only making this one call on page load and there isn’t a lot of data coming back, that might be fine, as the response will come back quickly, so the user may not even notice.

But if the server is busy or we’ve asked for a lot of data, our user will be wondering why the browser is locked up. To avoid this, let’s switch to using a promise.

var p = $().SPServices({
  operation: "GetListItems",
  listName: "MyCustomList"

p.done(function() {

    $(p.responseXML).SPFilterNode("z:row").each(function() {
      // Process the data


Here we’ve switched things around. We make the call to get the data, obtain the promise from the call and put it in variable p and then move on. Below the call, we have p.done(…), which means “When the response is here and ready to process…” Our JavaScript keeps executing from there and the browser is available to the user in the meantime. They might leave the page, click on some other actions, whatever.

When the promise is fulfilled, our code in the p.done(…) block executes. Now this will feel pretty weird to you until you get the hang of it. In the interim since you made the call, the DOM may have changed and the user may have done other things. You need to take those possibilities into account as you process the data you’ve received in the response.

Why convert now?

Converting to the promises pattern while you still are using SPServices to make SOAP calls has some advantages.

First, you already know how your code works. It may have taken you a while to get it up and running. As you’re thinking about how to move from SOAP to REST, you may not want to change too many things at once.

Find a simple call using SPServices and convert it to the promises model so that you understand how it works. Your user experience will improve even if you don’t move to REST yet. Even better, you can make this change in SharePoint 2007 or 2010 without having to switch to REST at all.

By the way, the call above in REST would look something like:

  url: _spPageContextInfo.webAbsoluteUrl +
  method: "GET",
  headers: {
    "Accept": "application/json; odata=verbose"
  success: function(data) {
    success(data); // If the call is successful, process the response
  error: function(data) {
    failure(data); // Do something with the error

The success and failure functions will only be called once the promise from the $.ajax function is resolved, so it’s somewhat equivalent to the promise-based call using SPServices above.


Since one of my goals in this series is to make moving from SOAP to REST less painful, I want to give you smaller steps you can take to get there without hurting yourself. Moving to the promises pattern now – whether you decide to go fully to REST or not – will put you in a better place when you do decide to take the plunge. Even better, it will prepare you for more “modern” Web development down the road, where you may be consuming data from multiple services. Those services may all be within SharePoint Online or Office 365 or they may be from separate sources altogether. Many of those services will be exposed via REST and the promises pattern will enable you to manage all of that data flying around more effectively.

This article was also published on IT Unity on 11/6/2015. Visit the post there to read additional comments.

Using ETags in SharePoint REST Calls to Manage Concurrency Control

Have you ever needed to generate a unique identifying number, sort of like the ID column in a list? Well, until I started learning the REST APIs, I had no idea how we could ever do this in SharePoint. With the adherence to the OData standards, we can do this and more using eTags to manage concurrency.

Concurrency control is a thorny problem in computer science. It refers to the situation where more than one person requests to change (either via update or delete) a piece of data stored somewhere. The only way you can tell if someone else has changed the data before you try is to read it from the source again. In the meantime, someone may have changed the data – lather, rinse, repeat.

People come up with all sorts of fragile concurrency control in their applications, and you see the results of that in systems from time to time. Have you ever scratched your head and been *sure* that you updated something, only to see the old value(s)? It may have been poor concurrency control that caused it.

In the prior SOAP-based calls supported by SPServices, we had no reliable way to manage concurrency. In most cases, we would build things so that the last person in wins – whomever requests a change last overwrites any prior changes. This is fine in many, many cases, but if more than one person is likely to make changes to data – and you want to manage those changes cleanly – it falls down fast.

With the move to using REST more fully in SharePoint 2013 and SharePoint Online in Office365, we have very strong concurrency control capabilities. For this to work well, the standard implements a concept called “eTags” which lets your code know if there is a concurrency problem. This works well because there is a collaboration between the server and the client about what constitutes a change and how to manage it.

Let’s look at my unique identifier problem as an example to explain how this all can work for you. In an application I’m building, I need to generate unique Document Set names based one a set of rules. In other words, it’s not sufficient to use something like the Document ID Service – I need to be able to build the names using some of my own logic which manifests the business rules.

As part of these names, I need to generate a unique, sequential number. In other words, the first “thing” will be named something like “AAAA-0001”, a subsequent “thing” may be named “BBBB-0001”, the next might be “BBBB-0002”, then “AAAA-0002”, etc., where each of the numbering schemes is independent. The Document Set ID in the list won’t do it.

As you can see, we have a unique numbering scheme based on a step in a process. (There are more varieties than just AAAA and BBBB.) We need to keep track of each unique scheme separately.

We’ve got a Configuration list in a SharePoint site. I’ll often build one of these lists for a project, and it simply contains parameter name/value pairs, along with a description of how it is used.

Configuration list

As you can see, for each step in the process, we’re maintaining a unique “last sequence number” in the Configuration list. (Yes, our real step names are more descriptive than AAAA, BBBB, CCCC!)

To make this work, I’ve got a function that does the following:

  1. Read all of the parameters from the Configuration list. It’s no more “expensive” to read all the parameters than just the one we need, so we read all of them.
  2. Find the value we need
  3. Attempt to write the value + 1 back into the list, passing back the eTag in the “IF-MATCH” parameter in the header of the request
  4. If we have a concurrency issue, then go back to step 2 above, and repeat until we are successful
  5. If success, pass back the value for the sequence number

In this particular application, the likelihood of two people clicking the “Get Name” button at the exact same time is very low, but we still want to manage this eventuality so that we don’t end up with two Document Sets with the same name.

The call to the function looks something like this:

$.when(ProjectName.Functions.GetNextSeq(ProjectName.DataSources.Steps.Abbreviation)).then(function() {
    nameInput.val(ProjectName.DataSources.Steps.Abbreviation + "-" + this); });

I call GetNextSeq with the abbreviation for the step, e.g., “AAAA” or “BBBB”. Because the GetNextSeq function passes back a jQuery promise, I can say basically:

Wait until $.when GetNextSeq is done, then build up the name for the Document Set

The GetNextSeq function looks something like this:

ProjectName.Functions.GetNextSeq = function(abbrev, p) {

  // If we've passed in a deferred object, then use it, else create a new one (this enables recursion)
  var deferred = p || $.Deferred();

  // Get the information from the Configuration list
  ProjectName.DataSources.Configuration = {};
  ProjectName.Promises.Configuration = $.ajax({
    url: _spPageContextInfo.siteServerRelativeUrl +
      "/_api/web/lists/getbytitle('Configuration')/items?" +
    method: "GET",
    headers: {
      "Accept": "application/json; odata=verbose"
    success: function(data) {

      /* Looping through the list items creates a JavaScript object like: 
          { "LastSeqAAAA" : { paramValue: "3", ID: "1", etag: "3" },
          { "LastSeqBBBB" : { paramValue: "103", ID: "1", etag: "110" }
      for (var i = 0; i < data.d.results.length; i++) {
        var thisParam = data.d.results[i];
        ProjectName.DataSources.Configuration[thisParam.Title] = {
          paramValue: thisParam.ParamValue,
          ID: thisParam.ID,
          etag: thisParam["__metadata"].etag


      // Next, we try to save LastSeqXXXX back into the list, using the eTag we got above
        url: _spPageContextInfo.siteServerRelativeUrl +
          "/_api/web/lists/getbytitle('Configuration')/items(" + ProjectName.DataSources.Configuration["LastSeq" + abbrev].ID + ")",
        type: "POST",
        contentType: "application/json;odata=verbose",
        data: JSON.stringify({
          "__metadata": {
            "type": "SP.Data.ConfigurationListItem"
          "ParamValue": (parseInt(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue) + 1).toString()
        headers: {
          "Accept": "application/json;odata=verbose",
          "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
          "X-HTTP-Method": "MERGE",
          "IF-MATCH": ProjectName.DataSources.Configuration["LastSeq" + abbrev].etag
        success: function(data) {
          // If the write is successful (response 204 (No Content)), resolve the promise with the value we should use for the sequence number (padded with leading zeroes)
          deferred.resolveWith(pad(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue, 4));
        error: function(data, a, b) {
          // If the server sends back a 412 response (Precondition Failed), then we have a concurrency issue, so call the function again with the existing deferred object
          if (data.status === 412) {
            ProjectName.Functions.GetNextSeq(abbrev, deferred);
    error: function(data) {
      alert('API error: ' + data);

  // Return the deferred object to the calling code
  return deferred;


There’s one other function I call above, and it’s just a little utility function used to pad the sequence numbers with leading zeroes.

function pad(num, size) {
	var s = num + "";
	while (s.length < size) s = "0" + s;
	return s;

While my requirements here may not match yours, the basic method probably will sooner or later. We want to read a value (item) from a list and then make an update to it based on some user action. Using eTags, we can do this reliably.

You don’t need to care what the eTag values are, but SharePoint seems to use a number much like a version number to manage this. If the item has been edited once, you’ll probably get back “1”, twice, probably will give you “2”, etc. The key here is, though, that you don’t care what the eTag value is: just whether it has changed. Even that doesn’t matter, as the 412 response from the server lets you know this is the case. By sending in the prior eTag value with your request, the server can tell you if there’s a problem. It’s up to you to handle that problem.

What if we want to do things the old way, like we did with SOAP? We simply omit the “IF-MATCH” parameter or pass “IF-MATCH”: “*”, meaning “I’ll accept any old value of the eTag – write away!”


This article was also published on IT Unity on 11/3/2015. Visit the post there to read additional comments.

Reasons to [Still] Love SharePoint’s SOAP Web Services

There are still reasons to love the SOAP Web Services (hopefully using SPServices!) even though they have been deprecated by Microsoft in SharePoint 2013 (and one can assume to upcoming SharePoint 2016) and SharePoint Online.

There are times when the REST API just doesn’t provide functionality we need, and the SOAP Web Services are a useful fallback when that’s the case.

I’ve run into two very specific instances where this was the case lately. Since Microsoft is actively trying to get REST up to snuff, I’ve added requests to the SharePoint User Voice for both of these. If you would like to see these fixed, please go to the User Voice links and vote them up. While we’re waiting, though, it’s SOAP to save the day.

Recurring Calendar Events

It’s extremely common to have recurring events in a SharePoint Calendar list. Usually this type of event is used for a recurring meeting, or it could just be a placeholder to remind a team to check something, etc.

If you’re building client-side functionality, you may want to do something like retrieving all of the occurrences of events to show in a widget like fullcalendar. Unfortunately, the REST API doesn’t know from recurrence.

There’s a sample at the OfficeDev/PnP repo, but if you run through it, you’ll see it’s pretty ugly. Ingenious, but ugly.

You can pass a CAML fragment into your REST call to augment the OData request, and here’s a different example of how that can work:

// Example from
  url: _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/getbytitle('Tasks')/GetItems(query=@v1)?@v1={\"ViewXml\":\"<View><Query><Where><And><Neq><FieldRef Name='Status' /><Value Type='Choice'>Completed</Value></Neq><Membership Type='SPWeb.Groups'><FieldRef Name='AssignedTo' /></Membership></And></Where></Query></View>\"}", 
  type: "POST", 
  headers: { 
      "X-RequestDigest": $("#__REQUESTDIGEST").val(), 
      "Accept": "application/json;odata=verbose", 
      "Content-Type": "application/json; odata=verbose" 
  success: function (data) { 
    if (data.d.results) { 
      // TODO: handle the data  
      alert('handle the data'); 
  error: function (xhr) { 
     alert(xhr.status + ': ' + xhr.statusText); 

But if you need to do that, I don’t see why you wouldn’t just use SOAP.

User Voice: Add support for recurring events in the REST API

Lookup Site Columns in Other Webs

A good information architecture usually includes some list-based lookup columns. A simple example might be a list stored in the root site of a Site Collection to contain information about all of your organization’s offices. You might store things like:

  • Name of Office (using the Title column)
  • Address
  • City
  • State
  • Postal Code
  • Office Manager
  • Phone Number
  • Industry Focus
  • etc.

Then you might have a Site Column called Office Name that is a Lookup column into the Title column of that list. This enables you to offer a consistent list of options for Office Name to choose from throughout the Site Collection along with the other information (as needed), either for display purposes or for inclusion as data in other lists.

If you try to make a REST call which requests data in a column like this – when you aren’t in same site as the list – you’ll get an error like this:

{"error":{"code":"-1, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The field 'MyLookupColumn' is not supported in query. The lookup list is in another web."}}}

Uh-uh, it’s back to SOAP again.

User Voice: Enable support for lookup columns in other webs in the REST API


In case you were going to ask, I avoid CSOM/JSOM. I see it as a dead end – in this context, at least. More of Microsoft’s effort is going into REST, which is based on Web standards rather than a Microsoft-only approach.

What about you? Do you still see uses for SOAP where REST won’t do it? Respond in the comments if you know of other holes in the REST Swiss cheese.

Retrieving High Priority Tasks with SharePoint’s Query REST API

fullcalendar month view javascriptThis will be a quick post. Recently, when I wanted to make a REST call to retrieve all of the high priority tasks from a set of task lists, I figured it would be no big deal. I’d just add something into the querytext and be done with it. Unfortunately, syntax is always a cruel bedfellow.

I tried all sorts of things, like:

PriorityOWSCHCS:(1) High
PriorityOWSCHCS=(1) High
PriorityOWSCHCS='(1) High'
Priority:'(1) High'

Someone like Mikael Svenson (@mikaelsvenson) – search guru that he is – would just look at this and tell me immediately what to do, but I try not to bug him unless it’s something complicated. Oddly, I couldn’t find a single example out there of someone doing this. It would seem to be a common request: show me all of the high priority tasks in the Site Collection so that I can focus on them.

In this case, we’re showing those tasks on a fullcalendar on the home page of a parent site which has many subsites, each for a project. Fullcalendar is an excellent, full-featured JavaScript calendar widget that you can pretty easily add into a SharePoint page. You just have to feed it data in a format it can understand. So we want to retrieve all of those project tasks using the search API and massage them into the fullcalendar format.

The filtering solution turned out to be too easy:


Here’s the full code that I came up with for this part of my KnockoutJS view model:

// Only get tasks with Priority='(1) High'
var projectTasks = [];
  url: _spPageContextInfo.webAbsoluteUrl + "/_api/search/query?" +
    "querytext=%27ContentTypeId:0x01080023AEF73C31A00C4C87FD5DB6FD82F6EE* Priority:1%27" +
  type: "GET",
  headers: {
    "accept": "application/json;odata=verbose",
  success: function(data) {

    $.each(data.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results, function(index, item) {

      var searchResult = [];

      // Normalize the fields
      $.each(this.Cells.results, function(i, prop) {
        searchResult[prop.Key] = prop.Value; // { Value: prop.Value, ValueType: prop.ValueType };

        calendar: "Project Tasks",
        color: "rgb(100,150,148)",
        title: searchResult.Title,
        path: searchResult.Path,
        eventDate: $.fullCalendar.moment(searchResult.StartDateOWSDATE),
        endDate: $.fullCalendar.moment(searchResult.DueDateOWSDATE)
  error: function(error) {

If nothing else, the next time I need to do this, I’ll hit something when I search!