Using ETags in SharePoint REST Calls to Manage Concurrency Control

Have you ever needed to generate a unique identifying number, sort of like the ID column in a list? Well, until I started learning the REST APIs, I had no idea how we could ever do this in SharePoint. With the adherence to the OData standards, we can do this and more using eTags to manage concurrency.

Concurrency control is a thorny problem in computer science. It refers to the situation where more than one person requests to change (either via update or delete) a piece of data stored somewhere. The only way you can tell if someone else has changed the data before you try is to read it from the source again. In the meantime, someone may have changed the data – lather, rinse, repeat.

People come up with all sorts of fragile concurrency control in their applications, and you see the results of that in systems from time to time. Have you ever scratched your head and been *sure* that you updated something, only to see the old value(s)? It may have been poor concurrency control that caused it.

In the prior SOAP-based calls supported by SPServices, we had no reliable way to manage concurrency. In most cases, we would build things so that the last person in wins – whomever requests a change last overwrites any prior changes. This is fine in many, many cases, but if more than one person is likely to make changes to data – and you want to manage those changes cleanly – it falls down fast.

With the move to using REST more fully in SharePoint 2013 and SharePoint Online in Office365, we have very strong concurrency control capabilities. For this to work well, the standard implements a concept called “eTags” which lets your code know if there is a concurrency problem. This works well because there is a collaboration between the server and the client about what constitutes a change and how to manage it.

Let’s look at my unique identifier problem as an example to explain how this all can work for you. In an application I’m building, I need to generate unique Document Set names based one a set of rules. In other words, it’s not sufficient to use something like the Document ID Service – I need to be able to build the names using some of my own logic which manifests the business rules.

As part of these names, I need to generate a unique, sequential number. In other words, the first “thing” will be named something like “AAAA-0001”, a subsequent “thing” may be named “BBBB-0001”, the next might be “BBBB-0002”, then “AAAA-0002”, etc., where each of the numbering schemes is independent. The Document Set ID in the list won’t do it.

As you can see, we have a unique numbering scheme based on a step in a process. (There are more varieties than just AAAA and BBBB.) We need to keep track of each unique scheme separately.

We’ve got a Configuration list in a SharePoint site. I’ll often build one of these lists for a project, and it simply contains parameter name/value pairs, along with a description of how it is used.

Configuration list

As you can see, for each step in the process, we’re maintaining a unique “last sequence number” in the Configuration list. (Yes, our real step names are more descriptive than AAAA, BBBB, CCCC!)

To make this work, I’ve got a function that does the following:

  1. Read all of the parameters from the Configuration list. It’s no more “expensive” to read all the parameters than just the one we need, so we read all of them.
  2. Find the value we need
  3. Attempt to write the value + 1 back into the list, passing back the eTag in the “IF-MATCH” parameter in the header of the request
  4. If we have a concurrency issue, then go back to step 2 above, and repeat until we are successful
  5. If success, pass back the value for the sequence number

In this particular application, the likelihood of two people clicking the “Get Name” button at the exact same time is very low, but we still want to manage this eventuality so that we don’t end up with two Document Sets with the same name.

The call to the function looks something like this:

$.when(ProjectName.Functions.GetNextSeq(ProjectName.DataSources.Steps.Abbreviation)).then(function() {
    nameInput.val(ProjectName.DataSources.Steps.Abbreviation + "-" + this); });
});

I call GetNextSeq with the abbreviation for the step, e.g., “AAAA” or “BBBB”. Because the GetNextSeq function passes back a jQuery promise, I can say basically:

Wait until $.when GetNextSeq is done, then build up the name for the Document Set

The GetNextSeq function looks something like this:

ProjectName.Functions.GetNextSeq = function(abbrev, p) {

  // If we've passed in a deferred object, then use it, else create a new one (this enables recursion)
  var deferred = p || $.Deferred();

  // Get the information from the Configuration list
  ProjectName.DataSources.Configuration = {};
  ProjectName.Promises.Configuration = $.ajax({
    url: _spPageContextInfo.siteServerRelativeUrl +
      "/_api/web/lists/getbytitle('Configuration')/items?" +
      "$select=ID,Title,ParamValue",
    method: "GET",
    headers: {
      "Accept": "application/json; odata=verbose"
    },
    success: function(data) {

      /* Looping through the list items creates a JavaScript object like: 
          { "LastSeqAAAA" : { paramValue: "3", ID: "1", etag: "3" },
          { "LastSeqBBBB" : { paramValue: "103", ID: "1", etag: "110" }
          etc.
      */
      for (var i = 0; i < data.d.results.length; i++) {
        var thisParam = data.d.results[i];
        ProjectName.DataSources.Configuration[thisParam.Title] = {
          paramValue: thisParam.ParamValue,
          ID: thisParam.ID,
          etag: thisParam["__metadata"].etag
        }

      }

      // Next, we try to save LastSeqXXXX back into the list, using the eTag we got above
      $.ajax({
        url: _spPageContextInfo.siteServerRelativeUrl +
          "/_api/web/lists/getbytitle('Configuration')/items(" + ProjectName.DataSources.Configuration["LastSeq" + abbrev].ID + ")",
        type: "POST",
        contentType: "application/json;odata=verbose",
        data: JSON.stringify({
          "__metadata": {
            "type": "SP.Data.ConfigurationListItem"
          },
          "ParamValue": (parseInt(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue) + 1).toString()
        }),
        headers: {
          "Accept": "application/json;odata=verbose",
          "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
          "X-HTTP-Method": "MERGE",
          "IF-MATCH": ProjectName.DataSources.Configuration["LastSeq" + abbrev].etag
        },
        success: function(data) {
          // If the write is successful (response 204 (No Content)), resolve the promise with the value we should use for the sequence number (padded with leading zeroes)
          deferred.resolveWith(pad(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue, 4));
        },
        error: function(data, a, b) {
          // If the server sends back a 412 response (Precondition Failed), then we have a concurrency issue, so call the function again with the existing deferred object
          if (data.status === 412) {
            ProjectName.Functions.GetNextSeq(abbrev, deferred);
          }
        }
      });
    },
    error: function(data) {
      alert('API error: ' + data);
    }
  });

  // Return the deferred object to the calling code
  return deferred;

}

There’s one other function I call above, and it’s just a little utility function used to pad the sequence numbers with leading zeroes.

function pad(num, size) {
	var s = num + "";
	while (s.length < size) s = "0" + s;
	return s;
}

While my requirements here may not match yours, the basic method probably will sooner or later. We want to read a value (item) from a list and then make an update to it based on some user action. Using eTags, we can do this reliably.

You don’t need to care what the eTag values are, but SharePoint seems to use a number much like a version number to manage this. If the item has been edited once, you’ll probably get back “1”, twice, probably will give you “2”, etc. The key here is, though, that you don’t care what the eTag value is: just whether it has changed. Even that doesn’t matter, as the 412 response from the server lets you know this is the case. By sending in the prior eTag value with your request, the server can tell you if there’s a problem. It’s up to you to handle that problem.

What if we want to do things the old way, like we did with SOAP? We simply omit the “IF-MATCH” parameter or pass “IF-MATCH”: “*”, meaning “I’ll accept any old value of the eTag – write away!”

Resources:

This article was also published on IT Unity on 11/3/2015. Visit the post there to read additional comments.

Reasons to [Still] Love SharePoint’s SOAP Web Services

There are still reasons to love the SOAP Web Services (hopefully using SPServices!) even though they have been deprecated by Microsoft in SharePoint 2013 (and one can assume to upcoming SharePoint 2016) and SharePoint Online.

There are times when the REST API just doesn’t provide functionality we need, and the SOAP Web Services are a useful fallback when that’s the case.

I’ve run into two very specific instances where this was the case lately. Since Microsoft is actively trying to get REST up to snuff, I’ve added requests to the SharePoint User Voice for both of these. If you would like to see these fixed, please go to the User Voice links and vote them up. While we’re waiting, though, it’s SOAP to save the day.

Recurring Calendar Events

It’s extremely common to have recurring events in a SharePoint Calendar list. Usually this type of event is used for a recurring meeting, or it could just be a placeholder to remind a team to check something, etc.

If you’re building client-side functionality, you may want to do something like retrieving all of the occurrences of events to show in a widget like fullcalendar. Unfortunately, the REST API doesn’t know from recurrence.

There’s a sample at the OfficeDev/PnP repo, but if you run through it, you’ll see it’s pretty ugly. Ingenious, but ugly.

You can pass a CAML fragment into your REST call to augment the OData request, and here’s a different example of how that can work:

// Example from https://karinebosch.wordpress.com/my-articles/caml-designer-for-sharepoint-2013/
$.ajax({ 
  url: _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/getbytitle('Tasks')/GetItems(query=@v1)?@v1={\"ViewXml\":\"<View><Query><Where><And><Neq><FieldRef Name='Status' /><Value Type='Choice'>Completed</Value></Neq><Membership Type='SPWeb.Groups'><FieldRef Name='AssignedTo' /></Membership></And></Where></Query></View>\"}", 
  type: "POST", 
  headers: { 
      "X-RequestDigest": $("#__REQUESTDIGEST").val(), 
      "Accept": "application/json;odata=verbose", 
      "Content-Type": "application/json; odata=verbose" 
  }, 
  success: function (data) { 
    if (data.d.results) { 
      // TODO: handle the data  
      alert('handle the data'); 
    } 
  }, 
  error: function (xhr) { 
     alert(xhr.status + ': ' + xhr.statusText); 
  } 
});

But if you need to do that, I don’t see why you wouldn’t just use SOAP.

User Voice: Add support for recurring events in the REST API

Lookup Site Columns in Other Webs

A good information architecture usually includes some list-based lookup columns. A simple example might be a list stored in the root site of a Site Collection to contain information about all of your organization’s offices. You might store things like:

  • Name of Office (using the Title column)
  • Address
  • City
  • State
  • Postal Code
  • Office Manager
  • Phone Number
  • Industry Focus
  • etc.

Then you might have a Site Column called Office Name that is a Lookup column into the Title column of that list. This enables you to offer a consistent list of options for Office Name to choose from throughout the Site Collection along with the other information (as needed), either for display purposes or for inclusion as data in other lists.

If you try to make a REST call which requests data in a column like this – when you aren’t in same site as the list – you’ll get an error like this:

{"error":{"code":"-1, Microsoft.SharePoint.SPException","message":{"lang":"en-US","value":"The field 'MyLookupColumn' is not supported in query. The lookup list is in another web."}}}

Uh-uh, it’s back to SOAP again.

User Voice: Enable support for lookup columns in other webs in the REST API

 

In case you were going to ask, I avoid CSOM/JSOM. I see it as a dead end – in this context, at least. More of Microsoft’s effort is going into REST, which is based on Web standards rather than a Microsoft-only approach.

What about you? Do you still see uses for SOAP where REST won’t do it? Respond in the comments if you know of other holes in the REST Swiss cheese.

Retrieving High Priority Tasks with SharePoint’s Query REST API

fullcalendar month view javascriptThis will be a quick post. Recently, when I wanted to make a REST call to retrieve all of the high priority tasks from a set of task lists, I figured it would be no big deal. I’d just add something into the querytext and be done with it. Unfortunately, syntax is always a cruel bedfellow.

I tried all sorts of things, like:

PriorityOWSCHCS:(1) High
PriorityOWSCHCS=(1) High
PriorityOWSCHCS='(1) High'
Priority:'(1) High'

etc.
Someone like Mikael Svenson (@mikaelsvenson) – search guru that he is – would just look at this and tell me immediately what to do, but I try not to bug him unless it’s something complicated. Oddly, I couldn’t find a single example out there of someone doing this. It would seem to be a common request: show me all of the high priority tasks in the Site Collection so that I can focus on them.

In this case, we’re showing those tasks on a fullcalendar on the home page of a parent site which has many subsites, each for a project. Fullcalendar is an excellent, full-featured JavaScript calendar widget that you can pretty easily add into a SharePoint page. You just have to feed it data in a format it can understand. So we want to retrieve all of those project tasks using the search API and massage them into the fullcalendar format.

The filtering solution turned out to be too easy:

Priority:1

Here’s the full code that I came up with for this part of my KnockoutJS view model:

// Only get tasks with Priority='(1) High'
var projectTasks = [];
$.ajax({
  url: _spPageContextInfo.webAbsoluteUrl + "/_api/search/query?" +
    "querytext=%27ContentTypeId:0x01080023AEF73C31A00C4C87FD5DB6FD82F6EE* Priority:1%27" +
    "&amp;selectproperties=%27Title,StartDateOWSDATE,DueDateOWSDATE,Path,Body,PriorityOWSCHCS%27",
  type: "GET",
  headers: {
    "accept": "application/json;odata=verbose",
  },
  success: function(data) {

    $.each(data.d.query.PrimaryQueryResult.RelevantResults.Table.Rows.results, function(index, item) {

      var searchResult = [];

      // Normalize the fields
      $.each(this.Cells.results, function(i, prop) {
        searchResult[prop.Key] = prop.Value; // { Value: prop.Value, ValueType: prop.ValueType };
      });

      projectTasks.push({
        calendar: "Project Tasks",
        color: "rgb(100,150,148)",
        title: searchResult.Title,
        path: searchResult.Path,
        eventDate: $.fullCalendar.moment(searchResult.StartDateOWSDATE),
        endDate: $.fullCalendar.moment(searchResult.DueDateOWSDATE)
      });
    });
  },
  error: function(error) {
    alert(JSON.stringify(error));
  }
});

If nothing else, the next time I need to do this, I’ll hit something when I search!

Software Adoption: A People Problem, or A Technology Problem?

I was a guest recently on the TechnologyAdvice Expert Interview Series to share my thoughts on the intersection of sales, marketing, and technology. The series, which is hosted by TechnologyAdvice’s Josh Bland, explores a variety of business and technology landscapes through conversations with industry leaders.

In this episode we discussed corporate cultures, global collaboration, and the upcoming SharePoint Technology Conference (SPTechCon) in Boston August 24-27.

Below are a few highlights from our conversation:

TechnologyAdvice: How does SharePoint help businesses and employees collaborate in a world where everything is always connected?

Marc Anderson: SharePoint has had a rather long lifespan for enterprise software — lots of things don’t last as long as SharePoint has. In the earlier days, it was very focused on teamwork– small groups of people trying to accomplish specific things. The goals might have been bigger, but that was really the focus.

Where Microsoft is going now, is they’ve moved their focus to offering services rather than software. They’re actually at the forefront of where businesses are going over the next five to ten years, whether they know they will or not. It’s not about the technology so much. The fact that it’s in the cloud or that it’s got a blue logo or something really doesn’t make any difference.

Over the last 10 to 15 years, we’ve really seen a change in the way corporate cultures and structures work compared to the way they used to. The Industrial Revolution and 1950’s military industrial complex were very hierarchical, very siloed. Now inefficient organizations are getting flatter. They’re putting teams together to solve specific problems and they’re having those teams break apart again, then go off and form new teams, so each person in that team might go do different things.

We’re seeing a lot more agile, small organizations able to accomplish things in different ways than they used to. That’s where SharePoint’s really going with some of the stuff they’re rolling out on Office 365, with more machine learning. It’s not artificial intelligence but it’s trying to get you the information you need right before you think you needed it, or before you even realize you needed it. And that’s cool stuff.

It’s shifting from team members putting things into a document repository and just knowing it’s in there, to the point where content gets put in front of you because you’re going to need it in your work soon. And with the Office Graph and Delve, we’re starting to see some real evidence that’s going to be the way people are working in the future.

TA: If people are not in an office but around the world working on different projects, something like SharePoint can continue to grow and become more powerful. Do you agree with that?

Anderson: Absolutely. In almost all cases, even a small company like mine with two people – we’re a global company. I know that sounds ridiculous because we both live in Boston, but we work with customers all over the place. There’s no geographical barrier anymore, which is part of what you’re alluding to. People are out on the road more, and they need to stay tethered somehow to their home-base. But they also need to be able to work together with people who could be anywhere.

You might have a conference call with someone in Shanghai and then work on the document with somebody in Amsterdam. That’s a normal day in this global economy. Twenty or thirty years ago those were huge exceptions. SharePoint, which gives us a large surface area to enable remote and geographically dispersed work, is a win.

It lets us work asynchronously. We don’t have to be in a room having a meeting. Meetings are god-awful things anyway, but we don’t have to be in a room talking about something. We can do it asynchronously using technology and work when we need to work or when we’re able to work. I think it changes our lifestyle, it changes our incentives, it changes the culture of organization, and SharePoint’s been a big part of that.

TA: What common collaborative software challenges do you see?

Anderson: In many ways, I don’t see a lot of the challenges as technology challenges. They’re more business and cultural challenges. One of the things I’ve seen working with clients — I’ve been in consulting for decades now — is the aspiration is rarely reached. And very specifically with SharePoint, people bring it in and they want to change the way their company works, and they end up storing documents in SharePoint and that’s about it.

Getting from the execution to the promise of what they may have thought they wanted to do — or even not realized they could do — that’s one of the biggest challenges. The challenge is in enabling their organization to work differently. Some people talk about it as an adoption problem. SharePoint is in a lot of organizations, but a lot of people don’t use it. I don’t see it as an adoption problem, I see it as we’re not offering good enough capabilities to make people want to use it.

It’s really not about the technology per se. It’s about the implementation, it’s about the cultural stuff around it, it’s about incentives. It’s about helping people understand how they can be better at what they do and enjoy that. If they don’t enjoy it, they’re just going to dig their heels in and avoid it. So those are some of the biggest challenges, whether it’s cloud or on premises, or whether you’ve got to get a fatter pipe to get the data across. Those are all solvable problems. Those are technical problems.

The harder challenge is the soft stuff around the technology. People are softer than computers. They’re quirky and they don’t like change. Working on the things outside the technology can be even more important challenges to overcome.

This podcast was created and published by TechnologyAdvice. Interview conducted by Josh Bland.