Making Your REST Calls Simpler by Changing the Metadata Setting

When you talk to SharePoint using REST, you have some options about what the data you get or send looks like. Early on, when REST first arrived on the scene for SharePoint, we HAD to specify odata=verbose. This meant that we had to be very specific about what our data was going to look like, especially when it came to updates.

If you don’t use odata=verbose, then you’re telling the server to accept something different. You’re basically telling the server what “dialect” of odata you are choosing to speak for the current transaction.

If you look at this post, you’ll see the options:

JSON Light support in REST SharePoint API released

Just to recap, the options are:

accept: application/json; odata=verbose

This is probably what you’re used to, and requires the heaviest payload. Using this option, you’ll get the maximum amount of information about the data coming your way, and you’ll also have to send more as well.

accept: application/json; odata=minimalmetadata

This is the middle ground. You’ll get some metdata, and you’ll also need to send some, but it’s less.

accept: application/json; odata=nometadata

This option means we won’t get any metadata and we also don’t have to send any.

accept: application/json

If you don’t specify the odata setting at all, it will default to odata=minimalmetadata.

You can use these settings both on the inbound and outbound part of your REST calls.

Content-Type: "accept: application/json; odata=minimalmetadata"

means that you are sending data that contains minimal metadata.

Likewise,

Accept: "accept: application/json; odata=verbose"

means that you want to receive data with full metadata information.

In many cases, you’ll want to use odata=verbose as you’re debugging and switch to odata=nometadata once you move into production. The metadata tells you a lot about the data coming and going and can be helpful as you build up your calls. If you don’t make that switch, there will be more data going down the wire, though in many cases that doesn’t matter too much. Since I learned to code back when nibbles were expensive storage, I tend to want to reduce data size as much as I can, though.

Unfortunately, I don’t think many SharePoint developers realize what all this can do for you. I hear people say all the time that REST calls are too “chatty”. In many cases, that’s simply not true; you developers are making things too chatty!

This allows you to switch from something like:

var data = {
  "__metadata": {
    "type": "SP.Data.FC_x0020_RunsListItem"
  },
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json; odata=verbose",
    "content-type": "application/json;odata=verbose",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

to:

var data = {
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json",
    "content-type": "application/json;odata=nometadata",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

This is a real example from some of my code in a client project. It may not look like a huge savings, but if you are passing a lot of data either way, it can make a difference.


UPDATE on 3 May: Tip from Mikael Svenson (@mikaelsvenson) – If you’re running on premises SharePoint 2013, you may need to enable the multiple metadata formats for JSON. See:

Using ETags in SharePoint REST Calls to Manage Concurrency Control

Have you ever needed to generate a unique identifying number, sort of like the ID column in a list? Well, until I started learning the REST APIs, I had no idea how we could ever do this in SharePoint. With the adherence to the OData standards, we can do this and more using eTags to manage concurrency.

Concurrency control is a thorny problem in computer science. It refers to the situation where more than one person requests to change (either via update or delete) a piece of data stored somewhere. The only way you can tell if someone else has changed the data before you try is to read it from the source again. In the meantime, someone may have changed the data – lather, rinse, repeat.

People come up with all sorts of fragile concurrency control in their applications, and you see the results of that in systems from time to time. Have you ever scratched your head and been *sure* that you updated something, only to see the old value(s)? It may have been poor concurrency control that caused it.

In the prior SOAP-based calls supported by SPServices, we had no reliable way to manage concurrency. In most cases, we would build things so that the last person in wins – whomever requests a change last overwrites any prior changes. This is fine in many, many cases, but if more than one person is likely to make changes to data – and you want to manage those changes cleanly – it falls down fast.

With the move to using REST more fully in SharePoint 2013 and SharePoint Online in Office365, we have very strong concurrency control capabilities. For this to work well, the standard implements a concept called “eTags” which lets your code know if there is a concurrency problem. This works well because there is a collaboration between the server and the client about what constitutes a change and how to manage it.

Let’s look at my unique identifier problem as an example to explain how this all can work for you. In an application I’m building, I need to generate unique Document Set names based one a set of rules. In other words, it’s not sufficient to use something like the Document ID Service – I need to be able to build the names using some of my own logic which manifests the business rules.

As part of these names, I need to generate a unique, sequential number. In other words, the first “thing” will be named something like “AAAA-0001”, a subsequent “thing” may be named “BBBB-0001”, the next might be “BBBB-0002”, then “AAAA-0002”, etc., where each of the numbering schemes is independent. The Document Set ID in the list won’t do it.

As you can see, we have a unique numbering scheme based on a step in a process. (There are more varieties than just AAAA and BBBB.) We need to keep track of each unique scheme separately.

We’ve got a Configuration list in a SharePoint site. I’ll often build one of these lists for a project, and it simply contains parameter name/value pairs, along with a description of how it is used.

Configuration list

As you can see, for each step in the process, we’re maintaining a unique “last sequence number” in the Configuration list. (Yes, our real step names are more descriptive than AAAA, BBBB, CCCC!)

To make this work, I’ve got a function that does the following:

  1. Read all of the parameters from the Configuration list. It’s no more “expensive” to read all the parameters than just the one we need, so we read all of them.
  2. Find the value we need
  3. Attempt to write the value + 1 back into the list, passing back the eTag in the “IF-MATCH” parameter in the header of the request
  4. If we have a concurrency issue, then go back to step 2 above, and repeat until we are successful
  5. If success, pass back the value for the sequence number

In this particular application, the likelihood of two people clicking the “Get Name” button at the exact same time is very low, but we still want to manage this eventuality so that we don’t end up with two Document Sets with the same name.

The call to the function looks something like this:

$.when(ProjectName.Functions.GetNextSeq(ProjectName.DataSources.Steps.Abbreviation)).then(function() {
    nameInput.val(ProjectName.DataSources.Steps.Abbreviation + "-" + this); });
});

I call GetNextSeq with the abbreviation for the step, e.g., “AAAA” or “BBBB”. Because the GetNextSeq function passes back a jQuery promise, I can say basically:

Wait until $.when GetNextSeq is done, then build up the name for the Document Set

The GetNextSeq function looks something like this:

ProjectName.Functions.GetNextSeq = function(abbrev, p) {

  // If we've passed in a deferred object, then use it, else create a new one (this enables recursion)
  var deferred = p || $.Deferred();

  // Get the information from the Configuration list
  ProjectName.DataSources.Configuration = {};
  ProjectName.Promises.Configuration = $.ajax({
    url: _spPageContextInfo.siteServerRelativeUrl +
      "/_api/web/lists/getbytitle('Configuration')/items?" +
      "$select=ID,Title,ParamValue",
    method: "GET",
    headers: {
      "Accept": "application/json; odata=verbose"
    },
    success: function(data) {

      /* Looping through the list items creates a JavaScript object like: 
          { "LastSeqAAAA" : { paramValue: "3", ID: "1", etag: "3" },
          { "LastSeqBBBB" : { paramValue: "103", ID: "1", etag: "110" }
          etc.
      */
      for (var i = 0; i < data.d.results.length; i++) {
        var thisParam = data.d.results[i];
        ProjectName.DataSources.Configuration[thisParam.Title] = {
          paramValue: thisParam.ParamValue,
          ID: thisParam.ID,
          etag: thisParam["__metadata"].etag
        }

      }

      // Next, we try to save LastSeqXXXX back into the list, using the eTag we got above
      $.ajax({
        url: _spPageContextInfo.siteServerRelativeUrl +
          "/_api/web/lists/getbytitle('Configuration')/items(" + ProjectName.DataSources.Configuration["LastSeq" + abbrev].ID + ")",
        type: "POST",
        contentType: "application/json;odata=verbose",
        data: JSON.stringify({
          "__metadata": {
            "type": "SP.Data.ConfigurationListItem"
          },
          "ParamValue": (parseInt(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue) + 1).toString()
        }),
        headers: {
          "Accept": "application/json;odata=verbose",
          "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
          "X-HTTP-Method": "MERGE",
          "IF-MATCH": ProjectName.DataSources.Configuration["LastSeq" + abbrev].etag
        },
        success: function(data) {
          // If the write is successful (response 204 (No Content)), resolve the promise with the value we should use for the sequence number (padded with leading zeroes)
          deferred.resolveWith(pad(ProjectName.DataSources.Configuration["LastSeq" + abbrev].paramValue, 4));
        },
        error: function(data, a, b) {
          // If the server sends back a 412 response (Precondition Failed), then we have a concurrency issue, so call the function again with the existing deferred object
          if (data.status === 412) {
            ProjectName.Functions.GetNextSeq(abbrev, deferred);
          }
        }
      });
    },
    error: function(data) {
      alert('API error: ' + data);
    }
  });

  // Return the deferred object to the calling code
  return deferred;

}

There’s one other function I call above, and it’s just a little utility function used to pad the sequence numbers with leading zeroes.

function pad(num, size) {
	var s = num + "";
	while (s.length < size) s = "0" + s;
	return s;
}

While my requirements here may not match yours, the basic method probably will sooner or later. We want to read a value (item) from a list and then make an update to it based on some user action. Using eTags, we can do this reliably.

You don’t need to care what the eTag values are, but SharePoint seems to use a number much like a version number to manage this. If the item has been edited once, you’ll probably get back “1”, twice, probably will give you “2”, etc. The key here is, though, that you don’t care what the eTag value is: just whether it has changed. Even that doesn’t matter, as the 412 response from the server lets you know this is the case. By sending in the prior eTag value with your request, the server can tell you if there’s a problem. It’s up to you to handle that problem.

What if we want to do things the old way, like we did with SOAP? We simply omit the “IF-MATCH” parameter or pass “IF-MATCH”: “*”, meaning “I’ll accept any old value of the eTag – write away!”

Resources:

This article was also published on IT Unity on 11/3/2015. Visit the post there to read additional comments.

Getting User Information with the SharePoint 2013 REST API

Sometimes the tiniest little throw-away comment on an article out there somewhere can prove useful to someone. Sometimes, it’s even a comment I’ve made.

2014-01-21_16-43-35That’s Andrew “AC” Clark (@bitterac) who tweeted. I may get a club soda out of it, but at Andrew’s suggestion, I figured I’d put up a post as well.

There’s far less documentation about SharePoint 2013’s REST capabilities than most of us would like. As has happened so many times in the past, the blogosphere fills in the gaps.

The post where Andrew Clark found my comment was a great one by Andrew Connell (@andrewconnell) about Applying Filters to Lookup Fields with the SharePoint 2013 REST API.

Andrew Connell (this post is a bit of an Andrew festival – an AC festival, at that) gave extremely useful (and hard to find) info about how to filter based on the values of lookup columns in SharePoint lists using REST. He covered regular lookup columns as well as Managed Metadata columns.

What it comes down to is using the projection to the source of the lookup or Managed Metadata column and then filtering base on the original, underlying value. We do this in REST for SharePoint using the $expand operator.

My comment was about needing to figure out the projection for an Author (Created By) column. The columns Created By and Modified By are populated by SharePoint automagically and the data used for that population is stored in the User Information List. This is a semi-hidden list which exists in the root of each Site Collection. I say semi-hidden because it’s actually the list you’re looking at when you go into the People and Groups view in Site Settings.

If you need to retrieve the name of the Author in a REST call, you can make it work by adding the projection for the Author column.

/_api/web/lists/getbytitle(listname)/items?$select=Title,Author/ID,Author/Title&$expand=Author/ID,Author/Title

Going a little further, you can request any of the data you’re used to seeing on the userdisp.aspx page (_layouts/15/userdisp.aspx?Force=true).

/_api/web/lists/getbytitle(listname)/items?$select=Title,Author/ID,Author/FirstName,Author/LastName,Author/Title,Author/Department,Author/SipAddress&$expand=Author/ID

It doesn’t really make a lot of sense when you look at it (at least not to me), but by adding the $expand clause, you can retrieve the corresponding info the the User Information List. It feel like you’d need to specify where that list is or something, but under the covers that happens for you.

It seems that by providing each of the values you want in both the $select and the $expand operators, it works. I couldn’t find any documentation on this anywhere. AC2 (the Connell one) has suggested to me several times that it makes sense to simply look at the OData standard. As he said in a reply to me on the same post:

For me, there’s simply nothing better than the raw SDKs & specs on the www.odata.org site. Aside from that, I’ll query www.StackOverflow.com. Then, as a SharePoint guy, you then need to look at WCF Data Services and see what it does not support in the OData v3 spec & the same is true for SharePoint 2013.

I think this stuff is about as clear as mud, but the main reason is that I haven’t spent a lot of time with it yet. REST calls are just a different flavor of how we ask for data in other languages. It’s simply a matter of getting the accent right.