Moving from SPServices to REST, Part 6: Converting UpdateListItems to REST

This entry is part 6 of 6 in the series Moving from SPServices to REST

While the majority of calls using SOAP with SPServices use GetListItems to get items from lists, changing data with UpdateListItems is probably a close second. In this case, we’re altering data based on some user action. In a way, it’s like repainting your car (which I’ve never done). The shape and structure of the car stays the same, but you make it look better with new paint. New data can be like that – after all, the list or library itself doesn’t change.

Image source: http://www.internetbillboards.net/2015/03/27/a-step-by-step-diy-guide-to-give-your-car-a-new-paint-job/

Image source: http://www.internetbillboards.net/2015/03/27/a-step-by-step-diy-guide-to-give-your-car-a-new-paint-job/

Let’s look at a simple example of an update: deleting a list item. This is probably the simplest update because all we have to pass in is the ID of the list item and ask for a delete.

var itemID = 1;
var p = $().SPServices({
  operation: "UpdateListItems",
  listName: "Tasks",
  updates: "<Batch OnError='Continue'> " +
    "<Method ID='1' Cmd='Delete'>" +
    "<Field Name='ID'>" + itemID + "</Field>" +
    "</Method>" +
    "</Batch>"
});
p.done({
  // Manage success or failure here
});

To delete a document in a library, it’s just a little bit harder. We need to pass in the FileRef (where the file lives) in addition to the ID.

var itemID = 1;
var fileRef = "http://mydomain/libName/fileName";
var p = $().SPServices({
  operation: "UpdateListItems",
  listName: "Tasks",
  updates: "<Batch OnError='Continue'> " +
    "<Method ID='1' Cmd='Delete'>" +
    "<Field Name='ID'>" + itemID + "</Field>" +
    "<Field Name='FileRef'>" + fileRef + "</Field>" +
    "</Method>" +
    "</Batch>"
});
p.done({
  // Manage success or failure here
});

In both cases, we pass in what’s called an update, which contains a batch. We can request multiple actions in the batch, but in this case, I’m keeping it very simple and just requesting one action: deleting the item with ID = itemID (and FileRef = fileRef for the library). As you can see, I also need to pass in the Cmd = “Delete” to tell the server what I want it to do. My other options are “New” and “Update”.

If our delete is successful, we simply get a result that tells us the error code was 0x00000000. If there was an issue, we’ll get something else (usually 0x81020016) which may not be all that descriptive of the error.

The call to delete a list item looks like this in REST:

var itemID = 1;
var p = $.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/GetByTitle('Tasks')/items(" + itemID + ")",
  method: "POST",
  headers: {
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": "DELETE",
    "IF-MATCH": "*",
    success: successFunction,
    error: errorFunction
  }
});

We can check what’s happened one of two ways. You can see above that I’ve set a variable called p to the results of the REST call. I’m also doing something with success and failure within the call. This is redundant, but I’m showing here to demonstrate the two methods (which I probably should have covered in my previous article Part 5: Using Promises with SOAP and REST, as several people have pointed out).

Examining p and acting on it is working with a promise; acting based on the success or failure inside the call is called a callback. You’ll find no end of arguing about which of the two is more “correct”, but it’s really a stylistic choice. Usually if you have lots going on and need to manage it all together, people will agree that working with promises can be easier to follow. There’s nothing wrong with using callbacks as long as you can follow your program logic.

If there’s an issue in the call (regardless which approach you take), you may an error back that looks something like this:

{"error":{"code":"-2130575338, System.ArgumentException","message":{"lang":"en-US","value":"Item does not exist. It may have been deleted by another user."}}}

In this case, I’ve tried to delete an item which doesn’t exist. Note that with REST, unlike with SOAP, no response on an update or delete is good news – you only get a response if something’s gone wrong.

Because I’m changing some data on the server, the request has to be a POST rather than a GET. POSTs are a little harder to set up than GETs because we have to demonstrate to the server that we aren’t bad guys by passing a little more information about who we are and where we are coming from.

Keep in mind that these examples don’t reflect what you would need to do in Apps Add-Ins; that authorization flow has been written about in many other places. These examples show what you would do if you are loading your JavaScript directly in the page, perhaps by including it in a Content Editor Web Part (CEWP) using the Content Link or in a Script Editor Web Part (SEWP).

We pass this extra authorization information in the request header:

"X-RequestDigest": document.getElementById("__REQUESTDIGEST").value

In every SharePoint page in SharePoint 2010 and above, there is a request digest value stored in the page in a hidden element with the id of “__REQUESTDIGEST”. Note that’s two underscores up front, not just one. In the line above I’m using simple JavaScript (no real need for jQuery here) to grab that value.

In both REST examples above I’m doing what I think of as a “selfish delete”. By specifying

"IF-MATCH": "*"

I’m telling the server that I don’t care if anyone else has changed the item or document – I just want it gone. Check out my post on using Etags if you’re interested in how to avoid collisions in requests.

To delete a document in a library, I do basically the same thing:

var itemID = 1;
var p = $.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/GetByTitle('Documents')/items(" + itemID + ")",
  method: "POST",
  headers: {
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": "DELETE"
    "IF-MATCH": "*",
  }
});
p.then({});
p.fail({});

Now let’s look at something a little harder. We’ll update a field (column) in a list item instead. I’ll keep it simple, but we’ll change the item’s Title.

Here’s how we’d do in in SOAP:

var itemID = 1;
var p = $().SPServices({ 
  operation: "UpdateListItems", 
  listName: "Tasks", 
  updates: "<Batch OnError='Continue'> " +
      "<Method ID='1' Cmd='Update'>" +
      "<Field Name='ID'>" + itemID + "</Field>" + 
      "<Field Name='Title'>This is the new title text</Field>" + 
      "</Method>" + 
      "</Batch>"
});
p.then({});
p.fail({});

As before, I need to pass in the batch, but this time the Cmd is “Update”, meaning that I want to change something about the list item. As you can see, I’m just changing the Title to some arbitrary text.

This SOAP request return us the item or document as it looks after the write. We can inspect that result if we want to update an array or something else we’re maintaining client side.

In REST we need to pass in the data we want the server to use in the update a little differently – no surprise, probably. We still pass the itemID on the url, but the data we want to write goes into a data element. We also need to “stringify” the data because it’s usually represented in our code as a JavaScript object. When we look at JSON data in the developer tools of our favorite browser, it looks like strings, but it’s actually stored as binary content. Since we can only pass text to the server, we need to do this stringify conversion.

var itemID = 1;
var payload = {
    "Title": "This is the new title text"
  };

var p = $.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
      "/_api/web/lists/GetByTitle('Tasks')/items(" +
        itemID +
      ")",
  method: "POST",
  data: JSON.stringify(payload),
  contentType: "application/json;odata=nometadata",
  headers: {
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method":"MERGE",
    "IF-MATCH": "*"
  }
});

In this example, I’m choosing to pass the data using the contentType with odata=nometadata set. You may see other examples with odata=verbose, and in that case, we must pass a metadata value in the payload, like this:

var payload = {
    "__metadata": {
      "type": "SP.Data.SurveysListItem"
    },
    "Title": "This is the new title text"
  };

Passing the metadata basically adds additional checks to the veracity of our data, and can be a good idea.

I’m still doing a selfish write here (“IF-MATCH”: “*”), not caring whether anyone else has made changes. As with SOAP this means that the last person in wins.

Here’s a table that gives an overview of how REST and SOAP compare when you want to do updates to your SharePoint content.

  SOAP REST
Authorization None needed; our request is made using the current user’s permissions X-RequestDigest

document.getElementById(“__REQUESTDIGEST”).value

Actions Passed in the Cmd element of the update CAML

[New, Update, Delete]

X-HTTP-Method

[PUT, MERGE, PATCH, DELETE]

Data Contained in the updates as strings using CAML notation Contained in the data element and the content must must be “stringified”
Collision Control None – last person wins Using ETags, we can learn of collisions, but we must manage those collisions in our code.

 

References:

This article was also published on IT Unity on 12/7/2015. Visit the post there to read additional comments.

Moving from SPServices to Rest, Part 5: Using Promises with SOAP and REST

This entry is part 5 of 6 in the series Moving from SPServices to REST

Wouldn’t it be great if we could drive flying cars like the DeLorean in Back to the Future? Unfortunately, even though October 21, 2015 has passed us by, that’s not a reality yet. The science fiction writer Arthur C. Clarke famously said,Any sufficiently advanced technology is indistinguishable from magic.” Sometimes JavaScript promises can feel a little bit like magic – at least until you get good at working with them.

Paul Nigh's 'TeamTimeCar.com' Back to the Future DeLorean Time Machine - Image source: https://en.wikipedia.org/wiki/DeLorean_time_machine

Paul Nigh’s ‘TeamTimeCar.com’ Back to the Future DeLorean Time Machine
Image source: https://en.wikipedia.org/wiki/DeLorean_time_machine

What are promises?

Both SOAP calls using SPServices and REST calls return promises. This is the case primarily because we are usually using a similar transport mechanism: maybe jQuery’s .ajax() function, which sends HTTP requests. (We don’t have to use jQuery, but it’s one of the more commonly used libraries to do this sort of thing. If you’re using one of the alternatives, you’ll need to do a little translation here.)

Promises are very cool mechanisms that work something like this. I think of it as a bit of magic, maybe like an act on America’s Got Talent.

When you make a call to SPServices or REST, you can assign the result of that call to a variable. That variable is a promise, which is sort of like an empty envelope. There’s no data in it when you first get it, but the call you just made promises to fill up the envelope for you. When the data is available and any conversion is finished, the original call sort of taps you on the shoulder and says, “You’ve got data!” (Say that in your best AOL voice.) At that point, you can start to do whatever work you want to do with the data.

Pretty neat, right? But, why does it matter? Well, when you are making AJAX calls, it can take a while for the server to respond. Servers are usually very powerful but they can get busy, or you may have asked for something complicated, or you may have asked for a lot of data, or the server may simply be tuned very badly. In any of those cases there will be a delay between the time you make the request and the time the response is available to you. The delay may be a few milliseconds or even a minute or two (though that probably indicates a poorly thought out request or a badly tuned server). During the time that delay occurs, you don’t want your JavaScript code to sit there waiting. Instead, you want to handle the call asynchronously meaning you don’t wait for it to finish; you elect to be notified when it finishes. That’s where promises come in.

Each browser reacts slightly differently to synchronous calls. Internet Explorer is probably the worst of the bunch, as it really hangs up until the request is complete. This makes for an unpleasant user experience, which ought never to be our goal. Because of this, using a promises pattern is important to ensure that your application feels responsive and peppy to your users.

What does converting to the promises pattern get us?

If you’ve been using SPServices for a long time, you may be setting async: false in many or all of your calls. This means you are making the calls synchronously, hanging up the browser. By moving to the promises pattern while still using SPServices, you’ll be setting yourself up for making REST calls efficiently down the road.

Let’s look at a simple example where we are just retrieving some items from a Custom List.

$().SPServices({
  operation: "GetListItems",
  async: false,
  listName: "MyCustomList",
  completefunc: function(xData,status) {

    $(xData.responseXML).SPFilterNode("z:row").each(function() {
      // Process the data
    });

  }
});

In this example, a SOAP payload will be sent to the server, and our code will sit and wait until the response comes back because we have specified async: false. If we’re only making this one call on page load and there isn’t a lot of data coming back, that might be fine, as the response will come back quickly, so the user may not even notice.

But if the server is busy or we’ve asked for a lot of data, our user will be wondering why the browser is locked up. To avoid this, let’s switch to using a promise.

var p = $().SPServices({
  operation: "GetListItems",
  listName: "MyCustomList"
});

p.done(function() {

    $(p.responseXML).SPFilterNode("z:row").each(function() {
      // Process the data
    });

});

Here we’ve switched things around. We make the call to get the data, obtain the promise from the call and put it in variable p and then move on. Below the call, we have p.done(…), which means “When the response is here and ready to process…” Our JavaScript keeps executing from there and the browser is available to the user in the meantime. They might leave the page, click on some other actions, whatever.

When the promise is fulfilled, our code in the p.done(…) block executes. Now this will feel pretty weird to you until you get the hang of it. In the interim since you made the call, the DOM may have changed and the user may have done other things. You need to take those possibilities into account as you process the data you’ve received in the response.

Why convert now?

Converting to the promises pattern while you still are using SPServices to make SOAP calls has some advantages.

First, you already know how your code works. It may have taken you a while to get it up and running. As you’re thinking about how to move from SOAP to REST, you may not want to change too many things at once.

Find a simple call using SPServices and convert it to the promises model so that you understand how it works. Your user experience will improve even if you don’t move to REST yet. Even better, you can make this change in SharePoint 2007 or 2010 without having to switch to REST at all.

By the way, the call above in REST would look something like:

$.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
      "/_api/web/lists/getbytitle('MyCustomList')/items",
  method: "GET",
  headers: {
    "Accept": "application/json; odata=verbose"
  },
  success: function(data) {
    success(data); // If the call is successful, process the response
  },
  error: function(data) {
    failure(data); // Do something with the error
  }
});

The success and failure functions will only be called once the promise from the $.ajax function is resolved, so it’s somewhat equivalent to the promise-based call using SPServices above.

Conclusion

Since one of my goals in this series is to make moving from SOAP to REST less painful, I want to give you smaller steps you can take to get there without hurting yourself. Moving to the promises pattern now – whether you decide to go fully to REST or not – will put you in a better place when you do decide to take the plunge. Even better, it will prepare you for more “modern” Web development down the road, where you may be consuming data from multiple services. Those services may all be within SharePoint Online or Office 365 or they may be from separate sources altogether. Many of those services will be exposed via REST and the promises pattern will enable you to manage all of that data flying around more effectively.

This article was also published on IT Unity on 11/6/2015. Visit the post there to read additional comments.

Moving from SPServices to REST, Part 4: Converting GetListItems to REST

This entry is part 4 of 6 in the series Moving from SPServices to REST

Summary: Because of the significant inconsistencies in the SOAP Web Services, converting your calls to REST will be a different process for each of them. In this article, we’ll look at the real workhorse of SPServices: GetListItems.

When my mechanic works on my car, sometimes he tells me about some different tires or parts I might consider instead of what was on the car since I got it. It might be to improve performance or to give me a bit more life out of the car than I might get otherwise. (I have a good mechanic.)

Officina Meccanica by efilpera https://flic.kr/p/5vvSy9

Photo credit: Officina Meccanica by efilpera https://flic.kr/p/5vvSy9

As I’ve suggested in prior articles in the series, you can start moving individual calls from SOAP to REST anytime that you decide. If you’re still running an older version of SharePoint (no shame there), then it probably won’t be until you upgrade to SharePoint 2013. If you’re already on SharePoint 2010 or 2013, then you can start anytime.

When it comes time to converting some SOAP calls to REST calls, what will that process look like? In this article, we’re going to take a look at converting the workhorse of the SOAP Web Services: GetListItems.

GetListItems lets us read items from a list or library (virtually all content in SharePoint is stored in a list or library – a library is just a fancy list that is document-centric). Even basic information about users is stored in each Site Collection in the User Information List. GetListItems does a lot for us in SPServices-based development; it’s the most-commonly used function by just about every developer who uses SPServices and it sits underneath most of the value-added functions like SPCascadeDropdowns and SPDisplayRelatedItems.

The basic structure of the simplest GetListItems call is this:

$().SPServices({
  operation: "GetListItems",
  listName: "Tasks"
});

We make a call to SPServices, specify the GetListItems operation, and specify the list from which we want to get data. That’s about it. The results of this call will contain the items in the Tasks list (in the current site) as they are displayed in the Tasks list’s default view, which by default would be AllTasks.aspx.

Usually we’d have a bit more code wrapped around this call, maybe something like this:

var p = $().SPServices({
  operation: "GetListItems",
  listName: "Tasks"
});

p.done(function() {

  // Process the response

});

Note that I’m using a promise above so that I know when the call has completed rather than making a synchronous call. More on that in a later article.

So what does the analogous REST call look like? It would be something like this:

$.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/getbytitle('Tasks')/items",
  method: "GET",
  headers: {
    "Accept": "application/json; odata=verbose"
  },
  success: function(data) {
    success(data); // If the call is successful, process the response
  },
  error: function(data) {
    failure(data); // Do something with the error
  }
});

Not so different when you get past the sides and trimming, right? The specifics of what we’re asking for in the REST call are all in the url object. The way REST works is that we pass information about what we want the server to do for us as parameters on the url. In this case, we have:

url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/getbytitle('Tasks')/items",

This means:

  • Go to the context of _spPageContextInfo.webAbsoluteUrl, which is the current site. In SPServices, I figure this out for you, or you can choose to specify a different site with the webURL parameter.
  • Call the lists service: /_api/web/lists
  • Select the Tasks list using its name: /getbytitle(‘Tasks’)
  • And finally, get the items: /items

The rest of the REST call is just packaging and what we want to do with the result.

Where the two methods start to diverge more is when we start to request things in more specific ways, for example all of the outstanding tasks for a particular user. But there’s an imperfect isomorphism here. For virtually everything we can do with the SOAP Web Services, we can do the same thing in REST just by adding the analogous parameters to the REST call.

Here’s how some of the most common parameters in the request payload map from SOAP to REST:

SOAP Option(SPServices synonym) REST Comments
ViewFields(CAMLViewFields) $select Choose the columns you would like to retrieve. With both SOAP and REST we get some data we don’t explicitly request, but by specifying only the columns we need we can reduce the payload sizes.
Query(CAMLQuery) $filter, $orderby Specify which items in the list we would like to return and how we would like them sorted.
RowLimit(CAMLRowLimit) $limit Say how many items matching the Query we would like to receive. In SOAP we can specify 0 to get all matching items; in REST we can omit the parameter to get all the matching items. Otherwise, we can specify any integer as the limit.
ViewName(CAMLViewName) NA ViewName lets you choose the view you would like to get in the response. There’s no REST equivalent here. I’ve always discouraged using this option in SOAP because it’s too easy to change the view settings and cause unintended consequences.
QueryOptions (CAMLQueryOptions) NA In SOAP, this lets us specify some options that change how the data is returned to us. For example, we can ask for all of the attachment URLs rather than just a Boolean which tells us that there are attachments.
NA $expand This option in REST has no direct equivalent in SOAP. $expand allows us to indicate that we would like the values for a relationship – rather than just the indices – using a projection. This is important with Lookup columns and Person or Group columns.

References:

As you can see, it’s really not that scary. It’s the equivalent of translating Spanish into French, but even easier since all of the words are in English (sorry rest of the world – I’m going to play American on this one).

Here’s a slightly more complicated example where I’m asking for items from a Workflow Tasks list.

First, the SOAP request:

SOAPRequest

And then the equivalent REST request:

REST request for items from a Workflow Tasks list

Here’s how the two calls break down based on the table I showed above:

SOAP REST
CAMLViewFields <Viewfields><FieldRef Name=’ID’/><FieldRef Name=’Title’/><FieldRef Name=’AssignedTo’/><FieldRef Name=’RelatedItems’/>

<FieldRef Name=’Priority’/>

<FieldRef Name=’Status’/>

<FieldRef Name=’DueDate’/>

<FieldRef Name=’Author’/>

</ViewFields>

$select $select=ID, Title, AssignedTo/Id, AssignedTo/Title, RelatedItems, Priority, Status, DueDate, Author/Id, Author/Title
CAMLQuery <OrderBy><FieldRef Name=’Created’ Ascending=’False’/></OrderBy> $filter, $orderby $orderby=Created Desc
NA $expand $expand=AssignedTo,Author

Note that in the SOAP call, I get the ID and Title (name) of each person or group column automatically when I request the column value. If I needed any more information about those people, I’d have to read it from another source, like the User Information List or User Profile. I also can specify a CAMLQueryOption that returns more of the basic information about users, like email address, SIP, etc.:

CAMLQueryOptions: "<QueryOptions><ExpandUserField>True</ExpandUserField></QueryOptions>"

In SOAP, most options that we pass go in the request payload. With REST, some of the options – for instance, what format we’d like the response to take – go in the request header.

If we don’t specify an accept parameter in the header, then we get XML in the response. If you’re used to parsing XML, then that might make you happy, but we usually want JSON these days.

To get JSON, we specify:

accept: application/json; odata=verbose

On premises in SharePoint 2013, this is the only option at the moment. In SharePoint Online (Office 365), if we don’t want all the metadata that comes along with the verbose option, we have a couple other options.

accept: application/json; odata=minimalmetadata
accept: application/json; odata=nometadata

The best way to understand the differences between these options is to try them out. When payload size matters (and it usually does), you should think about receiving less metadata to reduce the size of the responses.

References:

It’s the nuances that will trip us up. In SOAP, we can’t join lists together; we have to pull the data from each list and do the “join” on the client side. In REST we have what are called projections – using $expand – that let us do some lookups into other lists, for example to get information from a lookup list.

But back to GetListItems. Here are some examples of REST calls to get data back from a Tasks list showing some common parameter settings.

Get all tasks assigned to me

/_api/web/lists/getbytitle('Development Tasks')/items?$expand=AssignedTo&amp;$select=Title,AssignedTo/Title&amp;$filter=AssignedTo/Title eq 'Anderson, Marc'

Note that the format of your username will vary based on how you have things set up. Using an account may be a better choice, but I wanted to make the example clear.

Get all tasks that have not been completed, showing the status and the % complete

/_api/web/lists/getbytitle('Development Tasks')/items?$expand=AssignedTo&amp;$select=Title,AssignedTo/Title,Status,PercentComplete&amp;$filter=Status ne 'Completed'

Getting the data in these various ways is of course just the first step. Once you have the data, you’ll have to process it in some way in order to display it to the user, do calculations on it or whatever you’re trying to accomplish. That part of the exercise isn’t in the scope of this article, so you’ll have to look elsewhere for tricks for those pieces.

In the next article, we’ll look at how JavaScript promises come into play with SOAP and REST. I hope that you’ve been using promises with SPServices calls since I added support for them in version 2013.01, but if you haven’t, moving to REST is an excellent time to start doing so.

This article was also published on IT Unity on July 23, 2015. Visit the post there to read additional comments.

Moving from SPServices to REST, Part 3: JSON vs. XML

This entry is part 3 of 6 in the series Moving from SPServices to REST

Summary: When you move from SOAP to REST, you’ll want to understand the differences between XML and JSON as well. Modern Web developers are most likely to use JSON, and this is a great time to switch.

Most auto mechanics understand that there are several different measurement systems – and therefore toolsets – that they have to use to fix up cars, whether under the hood or on the outside body. Where I sit here in the USA, it’s our unique system versus almost everyone else’s (metric). That’s sort of where we are if we’re still using SOAP and XML instead of REST and JSON.

2014 Fiat 500

Under the hood of a 1956 T-Bird by Mark Turnauckas https://flic.kr/p/f14uV8

The SOAP Web Services speak XML. That is, we send XML in our requests and we get XML in the responses. These days many developers feel those services are “chatty” and that “XML is bloated.” I’m not so sure that’s really the case, but I can go along with it.

When we receive XML in the response from a SOAP call to the server, we have to parse it apart so that we can use the data. The most common operation that people use in SPServices is GetListItems, which allows us to pull list data from the server to the client. With a call to GetListItems we get a packet back that looks something like this from a Tasks list:

call-to-getlistitems-returns-task-list-2

With the SOAP Web Services, XML is our only option. There’s no way to request anything else. REST, on the other hand, is usually used to request JSON.

To bring SPServices more into the modern way of doing things, I added a function in 0.7.1 called SPXmlToJson and then another in 2014.01 called SPGetListItemsJson. These two functions make it easy – really easy – to convert XML to JSON. You don’t have to know much about either method of storing data to get JSON back.

With REST we can still choose to get XML in the response, but we also have the option to request that the server send us JSON. Because JSON is a text representation of JavaScript objects, the conversion is much more straightforward, with just a call to $.parseJSON (assuming you’re using jQuery).

JSON is preferable because it doesn’t require much – if any – translation when we want to use it. You may already be converting the XML you get back with SPServices calls into JSON, either manually or with a call to SPXmlToJson or SPGetListItemsJson.

JSON stands for JavaScript Object Notation. It’s a way to store data in a complex, object-based JavaScript structure rather than what many of you may be used to doing with individual variables.

For example, when I get a task back from GetListItems, I could create a set of variables, like this:

var thisId = $(this).attr("ows_ID");
var thisTitle = $(this).attr("ows_Title");
var thisDueDate = $(this).attr("ows_DueDate");

When I answer questions in the SPServices Discussions, a lot of the code people post does something like this. It’s the way I used to do things, too.

With a call to REST, I get an object back that is all ready to go.

call-to-rest-gives-object-back-3

Taking it one step further, I can build a constructor function that takes the task as I get it back from a call to the server and converts it into a JSON structure of my own choosing, perhaps adding a few new properties I may need down the road:

function Task(data) {
  this.ID = data.ID;
  this.Title = data.Title;
  this.DueDate = moment(data.DueDate);
  this.Status = data.Status;
  // etc.

  this.Complete = this.Status === "Completed" ? true : false;

};

Note the conversion of the DueDate in the function above. In most of my applications these days, I use the MomentJSJavaScript library. It gives us a huge number of date- and time-oriented functions. Dealing with date/time conversions and arithmetic can be extremely messy, and MomentJS makes it easy. So rather than simply storing the text-based value of DueDate that I get back in the XML or JSON from a call, I store the DueDate converted into a “moment,” which is the lingua franca to use MomentJS’s helpful functions.

In the code examples above, I’ve gone from the rather brute force method I used to use where I used a lot of individual variables, to using a JSON representation, to using a constructor function to “build” each task from the results of a call. The first two of these methods require basically the same amount of code, but the second is far easier to work with when we pass the task back out to whatever code we want to work with it next. The third approach, while it introduces yet another level of abstraction, gives us even more flexibility, as we can decide to add a new element to any tasks inside the constructor function and it will be available in tasks everywhere.

Using SPXmlToJson and SPGetListItemsJson is another way to move your code forward without actually converting to the REST services right away, say if you are still on SharePoint 2007 or 2010. In fact, one of the main reasons I wrote these two functions was so that I could start working with JSON in a project on SharePoint 2007 using KnockoutJS. Yes, you can use the tools and frameworks all the cool kids are using even on ancient platforms.

In Part 2, I outlined a way for you to think about abstracting your CRUD calls using SPServices so that you could slide REST underneath at a later point. By using the two XML to JSON capabilities in SPServices, you can also start returning JSON to your applications from those abstractions, taking yourself one step closer to REST.

There are a few caveats here. I didn’t try to match the JSON formats that the SharePoint 2013 REST services return when I built my functions, though the way I hand you the data are similar. Similar doesn’t mean the same, of course.

Here’s an example. When you request list items using SPGetListItemsJson and don’t provide any custom mapping of your own, SPUser fields, which are also known as Person or Group columns, are returned in a JSON object that looks like this:

request-list-items-using-spgetlistitemsjson-returns-json-object-4

If you make a REST call to SharePoint 2013, SPUser fields are returned in a JSON object that looks like this, assuming you have requested the Id and Title using a projection (more on this in a future article in the series):

spuser-fields-returned-in-json-object-5

As you can see, there are some naming and formatting differences. I return “userId” and the REST call returns “Id”, for instance. In my opinion those differences are manageable, but it will take some work when you switch to REST because your application code will expect things in the format that SPGetListItemsJson provides. This is another reason why constructor functions can be helpful: they give us a place to do the isomorphic mapping from one data representation to another.

The upshot here is that you can:

  1. Abstract your CRUD operations as in Part 2, and
  2. Return JSON to your application rather than XML

As before, your application shouldn’t care where the data lives, how it’s stored or how we get it; it should only care about processing data that has a consistent format.

If you follow these suggestions, your code might go from something like this inline request:

var p = $().SPServices({
  operation: "GetListItems",
  ListName: "Tasks",
  CAMLViewFields: "<ViewFields>" +
                "<FieldRef Name='Title'/>" +
                "<FieldRef Name='AssignedTo'/>" +
                "<FieldRef Name='RelatedItems'/>" +
                "<FieldRef Name='Priority'/>" +
                "<FieldRef Name='Status'/>" +
                "<FieldRef Name='DueDate'/>" +
                "<FieldRef Name='Author'/>" +
                "</ViewFields>"
});

To this abstracted request:

// Get all the Tasks data
ProjectName.Promises.Tasks = $().ProjectName.Tasks.readAll();

// Read items from the Tasks list
$.fn.ProjectName.Tasks.readAll = function(options) {

  var opt = $.extend({}, {}, $.fn. ProjectName.defaults, options);
  var p = $().SPServices({
    operation: "GetListItems",
    ListName: "Tasks",
    CAMLViewFields: "<ViewFields>" +
        "<FieldRef Name='Title'/>" +
        "<FieldRef Name='AssignedTo'/>" +
        "<FieldRef Name='RelatedItems'/>" +
        "<FieldRef Name='Priority'/>" +
        "<FieldRef Name='Status'/>" +
        "<FieldRef Name='DueDate'/>" +
        "<FieldRef Name='Author'/>" +
      "</ViewFields>"
  });

  return p;

}; // End $.fn. ProjectName.Tasks.readAll

To this abstracted JSON-based request:

// Get all the Tasks data
ProjectName.Promises.Tasks = $().ProjectName.Tasks.readAll();

// Read items from the Tasks list
$.fn.ProjectName.Tasks.readAll = function(options) {

  var opt = $.extend({}, {}, $.fn. ProjectName.defaults, options);
  var p = $().SPServices.SPGetListItemsJson({
        listName: opt.TasksList,
        CAMLViewFields: opt.TasksListViewFields,
        CAMLQuery: camlQuery,
        CAMLQueryOptions: opt.TasksListQueryOptions,
        mappingOverrides: opt.TasksListMappingOverrides,
        changeToken: opt.changeToken
  });

  return p;

}; // End $.fn. ProjectName.Tasks.readAll

“Hold on here,” you might say. “That’s more code than I was writing before!” Absolutely true. And if you have a little snippet in a Content Editor Web Part (CEWP) that just reads a few items from a list and that’s it, then this effort might not be worth it. However, if you are building Single Page Applications (SPAs) or even little “portlets” for use throughout your Site Collection(s), it may make total sense.

For instance, if you have a customized Tasks or Announcements list in every Team Site that you are reading from and writing to, then abstracting those calls and storing them centrally may be a great idea. I do this sort of thing all the time. I load my data access operations into the master page in the Site Collection and then use them reliably from subsite to subsite. If you’re working on a larger team – which I tend not to do – it will require coordination, but it will also give you the ability to parse out atomic programming tasks across your team. Maybe one person is really good at data access and another is really good at HTML and CSS. This approach lets you parse out the work based on everyone’s skills and interests.

But I digress. The goal of this series is to help you move from SOAP to REST over time, not reorganize the roles on your team. Hopefully you can see that taking a few of these steps now or when you begin your next project, you’ll be putting yourself in a better position for the long run. These approaches may not help with everything you are using SOAP to accomplish – the SOAP services still cover more surface area than CSOM or REST, though that is changing – but it will put your most frequently-used sections of code in a good spot when it is time to move to REST.

Those auto mechanics know that there isn’t a huge difference between using a metric spanner and an Imperial wrench (other than getting everyone to agree on the terminology). XML and JSON are just dialects or different systems that accomplish basically the same thing: wrapping data up in packets that can be deciphered easily on the other end.

This article was also published on IT Unity on April 17, 2015. Visit the post there to read additional comments.

Moving from SPServices to REST, Part 2: New Patterns for SPServices Development

This entry is part 2 of 6 in the series Moving from SPServices to REST

Summary: You can improve your SPServices-based code today to prepare for converting to REST calls later. In this article, Marc introduces some new patterns for your JavaScript code to make SOAP calls easier to maintain and easier to move to REST when you decide it’s time.

Remember that older car I drive? Well, over the years, I’ve had work done on it a few times to buff out scratches or pull out dents. In this article, I’ll talk about how you can get your code in better shape to prepare for REST down the road.

Patterns to improve SPServices-based code; moving SPServices to REST
Photo Credit: “Erie LaSalle Body Shop & Car Care Center” by Thomas Hawk on Flickr https://flic.kr/p/8bA2yH

Over the last year or so, I’ve started to follow some new patterns in writing my JavaScript code. The ideas came primarily from a smart client of mine named Doug Coutts. He came to me for some help setting up some data access methods using SPServices in a SharePoint 2010 environment.

When Doug first got in touch, he sent me an extremely well thought out spreadsheet containing the capabilities he was after. The spreadsheet had about 25 different data access methods in it. Doug had a nice data model built using SharePoint lists as the repositories. Each list would have a set of specific data access methods based on the expected functionality in the application. At first I was simply going to build out a decent set of those methods to get him started, but I ended up building most of them in the original spreadsheet plus quite a few more we figured out we’d need as we went along in the development process.

So why is this all important to you, gentle reader? Well, Doug’s patterns, along with others I’ve learned from great articles from Scot Hillier and others, are almost exactly what I’m recommending for you to move to in order to start preparing for switching from the SOAP Web Services to REST over time.

As I’ll admit to anyone who listens, I’m sort of a hack. I write code that works, and it usually works very well, but I’m not much of a patterns and practices kind of guy. Because of the types of projects I work on with SharePoint, I’m not usually part of some big development team. Instead, I tend to end up building smaller, self-contained “application-lets” if you will, which don’t require more hands than mine.

But over the last few years as the SharePoint team has been moving more and more toward client-side code and I’ve picked apart how they are doing it (Let’s continue to improve that documentation for the client-side code, guys!), I’ve learned more and better ways to do what I’ve already been doing. Even inside SPServices, I make continuous improvements. On some levels, that’s a bad idea, as it can mean that I end up breaking things that worked just fine before – we call those regressions in the software biz – or I simply do work for more academic purposes. However, each time I do a new release of SPServices, I’m less embarrassed about what’s inside and it simply works better.

The new patterns I’m suggesting here will be sort of like that, but with two clear purposes:

  • Improving your code base to make your code as it works today easier to maintain
  • Getting you more ready to move components of your code to REST when you decide it is time

Some of you will already be doing these things, but having seen a lot of the bits and pieces of how you all use SPServices in the wild and in the Codeplex Discussions, I know that many of you don’t.

Here’s the basic idea. While SPServices is already an abstraction on the SOAP Web Services, I suggest that you add yet another abstraction layer in your SPServices-based code.

Say we have a list called Tasks, basically the same as the out-of-the-box Tasks list that shows up in SharePoint Team Sites and elsewhere. Tasks themselves aren’t very complicated: we have a Title, a few descriptive fields that tell us some things about each task, and some dates that tell us when we need to do the task. However, there may be 5 or 6 different ways we want to interact with the list. None of them will shock you, but they are things like:

  • Create a new Task
  • Edit an existing Task
  • Delete a Task
  • Get all of the Tasks created by a specific person
  • Get all the Tasks assigned to a specific person
  • etc.

Nothing earth-shattering, right? Just the basic CRUD (Create, Read, Update, Delete) operations, and a few other methods that are only a bit more complicated. But we may need to call some of the functions in multiple places in the application. You may see where I’m heading here: when there is a strong possibility that we will use a given method in more than one place in our code, it pays to abstract that method out and not repeat it.

Not only do we have some unique script (may I finally call it code?) that drives each individual page in the application (in this case each of the dozen or so pages has its own unique set of business logic), we also have a separate script file that contains all of the data access methods, making them available for reuse across the page logic. This idea is known as abstraction, and with abstraction and modularity we can make our development chores a lot easier. We’re usingKnockoutJS in our specific project, but that doesn’t really matter much when it comes to this part of the pattern.

Overall, the data access script file looks a lot like the way I’ve structured SPServices itself. The structure looks something like this:

  • Surrounding function
  • Defaults
  • A Namespace per data source
  • Individual data access functions for the data source

Inside the data access module, I’ve created each data access method inside a single namespace for the application – let’s call it TasksApp. When we talk about namespaces in JavaScript, we really mean complex objects that limit our impact on other people’s code. Each list – or data source – has its own namespace inside TasksApp, and then we have the series of functions. They look something like this:

  • TasksApp.Tasks.create
  • TasksApp.Tasks.save
  • TasksApp.Tasks.delete
  • etc.

Here’s what the TasksApp.Tasks.delete function’s code looks like. It’s a nice one to show because it’s so simple.

$.fn.TasksApp.Tasks.delete = function(options) {

  var opt = $.extend({}, {
    task: null
  }, $.fn.TasksApp.defaults, options);

  var result = $.Deferred();
  var p = $.Deferred();

  var p = $().SPServices({
    operation: "UpdateListItems",
    listName: opt.tasksList,
    batchCmd: "Delete",
    ID: opt.task.ID
  });

  p.done(function() {

    var errorCode = $(p.responseXML).find("Result &amp;amp;amp;gt; ErrorCode").text();

    if (errorCode === "0x00000000") {
      // Success

      result.resolveWith(opt.task.ID);
    } else {
      // Something went wrong
      result.resolveWith(-1);
    }

  });

  return result.promise();

}; // End $.fn.TasksApp.Tasks.delete

What this does for me is give me one function I can call from anywhere in the application when I want to delete a task. I simply pass a task object (this is defined elsewhere, but consists of the task in an object like { ID: 12, Title: ‘This is the title’, AssignedTo: ‘Anderson, Marc’, … }) into the function, and I know that it will be deleted. I’m using promises internally in the function, and I’m also returning a promise that will eventually tell me whether the call to the function succeeds or not. I have nicely chunked out functions with mnemonic names that we can call from anywhere in the application without any duplication of the actual calls to the SOAP Web Services.Notice what else this does for me: I have encapsulated the Web Services call. All I need to know in my application is that the function will delete a task, wherever it lives. In this case, it’s in a SharePoint list, but we could decide to store our tasks in SQL or any other storage location later.

Even more importantly, I can decide at any point that I want to start deleting tasks via REST. Since I know what I need to pass into the function and what it does, I don’t need to care how it does its thing. Because I have a nice encapsulation, I can decide I want to convert one particular function, all the delete functions, or all of the functions in the entire application, but it’s up to me what pace I decide to take.

Let’s say I decide to switch this one TasksApp.Tasks.delete function to REST. It simply becomes this:

$.fn.TasksApp.Tasks.delete = function(options) {

  var opt = $.extend({}, {
    task: null
  }, $.fn.TasksApp.defaults, options);

  var result = $.Deferred();
  var p = $.Deferred();

var p = $.ajax({
       url: "http://siteurl/_api/web/lists/GetByTitle('Tasks')/items(" + opt.task.ID + ")",
       method: "POST",
       headers: {
         Authorization: "Bearer " + accessToken
     X-RequestDigest: form digest value
         "IF-MATCH": etag or "*"
         "X-HTTP-Method":"DELETE"}
     });

  p.done(function() {

    var errorCode = $(p.responseXML).find("Result &amp;amp;amp;gt; ErrorCode").text();

    if (errorCode === "0x00000000") {
      // Success

      result.resolveWith(opt.ID);
    } else {
      // Something went wrong
      result.resolveWith(-1);
    }

  });

  return result.promise();

}; // End $.fn.TasksApp.tasks.delete

Any code that calls the function TasksApp.Tasks.delete function simply expects to receive a promise back that will be resolved either with the ID of the task that was deleted or -1 if there was a problem. The calling code doesn’t care how it happens, just that it will be informed about the outcome.

About two-thirds of the way through the development cycle in SharePoint 2010, Doug asked me what it would take to upgrade everything to SharePoint 2013. I thought a minute and said, “Copying the files into the SharePoint 2013 environment.” And you know what? It was just about that easy. Take that, managed code developers!

We took a backup of the development Site Collection and restored it into the 2013 environment. Then I simply copied and pasted the script, CSS and HTML files we had built into the 2013 environment. The “conversion” was a big copy of the files from a Document Library using SharePoint Designer 2010 and a paste into a Document Library using SharePoint Designer 2013.

Caveat: there was *one* thing I needed to fix. While “protocol-less” script references are considered good practice, SharePoint Designer doesn’t deal well with them and tries to fix them up; breaking them in the process. So I did have to repair those script reference hrefs, but I didn’t need to rewrite a single line of code!

At that point we were in SharePoint 2013 and I was calling the SOAP Web Services with SPServices. Shouldn’t I have switched to REST calls right away? We decided not to tackle that then because we wanted to finish the application and get it released. I’ve pushed to do some replacement in subsequent versions, and we’ve done it where it makes sense. It’s not a very big deal because we have the discrete data access methods in place. In fact, we have replaced only the methods we were using during updates with REST calls and can tackle others later. We may continue to decide which calls fall into each category simply by what the REST services support or we may decide based on level of effort, but the level of effort won’t be high, I guarantee it.

This article was also published on IT Unity on April 6, 2015. Visit the post there to read additional comments.