Moving from SPServices to REST, Part 2: New Patterns for SPServices Development

This entry is part 2 of 2 in the series Moving from SPServices to REST

Summary: You can improve your SPServices-based code today to prepare for converting to REST calls later. In this article, Marc introduces some new patterns for your JavaScript code to make SOAP calls easier to maintain and easier to move to REST when you decide it’s time.

Remember that older car I drive? Well, over the years, I’ve had work done on it a few times to buff out scratches or pull out dents. In this article, I’ll talk about how you can get your code in better shape to prepare for REST down the road.

Patterns to improve SPServices-based code; moving SPServices to REST
Photo Credit: “Erie LaSalle Body Shop & Car Care Center” by Thomas Hawk on Flickr https://flic.kr/p/8bA2yH

Over the last year or so, I’ve started to follow some new patterns in writing my JavaScript code. The ideas came primarily from a smart client of mine named Doug Coutts. He came to me for some help setting up some data access methods using SPServices in a SharePoint 2010 environment.

When Doug first got in touch, he sent me an extremely well thought out spreadsheet containing the capabilities he was after. The spreadsheet had about 25 different data access methods in it. Doug had a nice data model built using SharePoint lists as the repositories. Each list would have a set of specific data access methods based on the expected functionality in the application. At first I was simply going to build out a decent set of those methods to get him started, but I ended up building most of them in the original spreadsheet plus quite a few more we figured out we’d need as we went along in the development process.

So why is this all important to you, gentle reader? Well, Doug’s patterns, along with others I’ve learned from great articles from Scot Hillier and others, are almost exactly what I’m recommending for you to move to in order to start preparing for switching from the SOAP Web Services to REST over time.

As I’ll admit to anyone who listens, I’m sort of a hack. I write code that works, and it usually works very well, but I’m not much of a patterns and practices kind of guy. Because of the types of projects I work on with SharePoint, I’m not usually part of some big development team. Instead, I tend to end up building smaller, self-contained “application-lets” if you will, which don’t require more hands than mine.

But over the last few years as the SharePoint team has been moving more and more toward client-side code and I’ve picked apart how they are doing it (Let’s continue to improve that documentation for the client-side code, guys!), I’ve learned more and better ways to do what I’ve already been doing. Even inside SPServices, I make continuous improvements. On some levels, that’s a bad idea, as it can mean that I end up breaking things that worked just fine before – we call those regressions in the software biz – or I simply do work for more academic purposes. However, each time I do a new release of SPServices, I’m less embarrassed about what’s inside and it simply works better.

The new patterns I’m suggesting here will be sort of like that, but with two clear purposes:

  • Improving your code base to make your code as it works today easier to maintain
  • Getting you more ready to move components of your code to REST when you decide it is time

Some of you will already be doing these things, but having seen a lot of the bits and pieces of how you all use SPServices in the wild and in the Codeplex Discussions, I know that many of you don’t.

Here’s the basic idea. While SPServices is already an abstraction on the SOAP Web Services, I suggest that you add yet another abstraction layer in your SPServices-based code.

Say we have a list called Tasks, basically the same as the out-of-the-box Tasks list that shows up in SharePoint Team Sites and elsewhere. Tasks themselves aren’t very complicated: we have a Title, a few descriptive fields that tell us some things about each task, and some dates that tell us when we need to do the task. However, there may be 5 or 6 different ways we want to interact with the list. None of them will shock you, but they are things like:

  • Create a new Task
  • Edit an existing Task
  • Delete a Task
  • Get all of the Tasks created by a specific person
  • Get all the Tasks assigned to a specific person
  • etc.

Nothing earth-shattering, right? Just the basic CRUD (Create, Read, Update, Delete) operations, and a few other methods that are only a bit more complicated. But we may need to call some of the functions in multiple places in the application. You may see where I’m heading here: when there is a strong possibility that we will use a given method in more than one place in our code, it pays to abstract that method out and not repeat it.

Not only do we have some unique script (may I finally call it code?) that drives each individual page in the application (in this case each of the dozen or so pages has its own unique set of business logic), we also have a separate script file that contains all of the data access methods, making them available for reuse across the page logic. This idea is known as abstraction, and with abstraction and modularity we can make our development chores a lot easier. We’re usingKnockoutJS in our specific project, but that doesn’t really matter much when it comes to this part of the pattern.

Overall, the data access script file looks a lot like the way I’ve structured SPServices itself. The structure looks something like this:

  • Surrounding function
  • Defaults
  • A Namespace per data source
  • Individual data access functions for the data source

Inside the data access module, I’ve created each data access method inside a single namespace for the application – let’s call it TasksApp. When we talk about namespaces in JavaScript, we really mean complex objects that limit our impact on other people’s code. Each list – or data source – has its own namespace inside TasksApp, and then we have the series of functions. They look something like this:

  • TasksApp.Tasks.create
  • TasksApp.Tasks.save
  • TasksApp.Tasks.delete
  • etc.

Here’s what the TasksApp.Tasks.delete function’s code looks like. It’s a nice one to show because it’s so simple.

$.fn.TasksApp.Tasks.delete = function(options) {

  var opt = $.extend({}, {
    task: null
  }, $.fn.TasksApp.defaults, options);

  var result = $.Deferred();
  var p = $.Deferred();

  var p = $().SPServices({
    operation: "UpdateListItems",
    listName: opt.tasksList,
    batchCmd: "Delete",
    ID: opt.task.ID
  });

  p.done(function() {

    var errorCode = $(p.responseXML).find("Result > ErrorCode").text();

    if (errorCode === "0x00000000") {
      // Success

      result.resolveWith(opt.task.ID);
    } else {
      // Something went wrong
      result.resolveWith(-1);
    }

  });

  return result.promise();

}; // End $.fn.TasksApp.Tasks.delete

What this does for me is give me one function I can call from anywhere in the application when I want to delete a task. I simply pass a task object (this is defined elsewhere, but consists of the task in an object like { ID: 12, Title: ‘This is the title’, AssignedTo: ‘Anderson, Marc’, … }) into the function, and I know that it will be deleted. I’m using promises internally in the function, and I’m also returning a promise that will eventually tell me whether the call to the function succeeds or not. I have nicely chunked out functions with mnemonic names that we can call from anywhere in the application without any duplication of the actual calls to the SOAP Web Services.Notice what else this does for me: I have encapsulated the Web Services call. All I need to know in my application is that the function will delete a task, wherever it lives. In this case, it’s in a SharePoint list, but we could decide to store our tasks in SQL or any other storage location later.

Even more importantly, I can decide at any point that I want to start deleting tasks via REST. Since I know what I need to pass into the function and what it does, I don’t need to care how it does its thing. Because I have a nice encapsulation, I can decide I want to convert one particular function, all the delete functions, or all of the functions in the entire application, but it’s up to me what pace I decide to take.

Let’s say I decide to switch this one TasksApp.Tasks.delete function to REST. It simply becomes this:

$.fn.TasksApp.Tasks.delete = function(options) {

  var opt = $.extend({}, {
    task: null
  }, $.fn.TasksApp.defaults, options);

  var result = $.Deferred();
  var p = $.Deferred();

var p = $.ajax({
       url: "http://siteurl/_api/web/lists/GetByTitle('Tasks')/items(" + opt.task.ID + ")",
       method: "POST",
       headers: {
         Authorization: "Bearer " + accessToken
     X-RequestDigest: form digest value
         "IF-MATCH": etag or "*"
         "X-HTTP-Method":"DELETE"}
     });

  p.done(function() {

    var errorCode = $(p.responseXML).find("Result > ErrorCode").text();

    if (errorCode === "0x00000000") {
      // Success

      result.resolveWith(opt.ID);
    } else {
      // Something went wrong
      result.resolveWith(-1);
    }

  });

  return result.promise();

}; // End $.fn.TasksApp.tasks.delete

Any code that calls the function TasksApp.Tasks.delete function simply expects to receive a promise back that will be resolved either with the ID of the task that was deleted or -1 if there was a problem. The calling code doesn’t care how it happens, just that it will be informed about the outcome.

About two-thirds of the way through the development cycle in SharePoint 2010, Doug asked me what it would take to upgrade everything to SharePoint 2013. I thought a minute and said, “Copying the files into the SharePoint 2013 environment.” And you know what? It was just about that easy. Take that, managed code developers!

We took a backup of the development Site Collection and restored it into the 2013 environment. Then I simply copied and pasted the script, CSS and HTML files we had built into the 2013 environment. The “conversion” was a big copy of the files from a Document Library using SharePoint Designer 2010 and a paste into a Document Library using SharePoint Designer 2013.

Caveat: there was *one* thing I needed to fix. While “protocol-less” script references are considered good practice, SharePoint Designer doesn’t deal well with them and tries to fix them up; breaking them in the process. So I did have to repair those script reference hrefs, but I didn’t need to rewrite a single line of code!

At that point we were in SharePoint 2013 and I was calling the SOAP Web Services with SPServices. Shouldn’t I have switched to REST calls right away? We decided not to tackle that then because we wanted to finish the application and get it released. I’ve pushed to do some replacement in subsequent versions, and we’ve done it where it makes sense. It’s not a very big deal because we have the discrete data access methods in place. In fact, we have replaced only the methods we were using during updates with REST calls and can tackle others later. We may continue to decide which calls fall into each category simply by what the REST services support or we may decide based on level of effort, but the level of effort won’t be high, I guarantee it.

This article was first published on IT Unity on April 6, 2015. Visit the post there to read additional comments.

Unexpected Error from GetListItemsChangesSinceToken Using SPServices

An unexpected result from GetListItemChangesSinceToken using SPServicesThis is another post in the vein of “it happened to me and it took me a long time to figure it out, so why should you go through the same pain”?

I’ve got an application running at a client that sits on top of SharePoint 2007. It uses KnockoutJS and lots of other modern Web development goodies, even though we’re on a very old SharePoint version. In fact, most of the time, I’m debugging in Internet Explorer 8, because that’s still the corporate standard.

Fun stuff, right? Well, the application serves them great, and I still get to have fun because it’s modern development.

Over the last few weeks, I’ve been trying to figure out why some of my calls to the SPServices function SPGetListItemsJson haven’t been returning what I expected. The calls were happening and I was getting valid responses in Firefox and Chrome, but in IE8, the data wasn’t all there. I was using localStorage to store a local cache of the data so I had data from weeks back there, but more recent data wasn’t coming back from my calls. (See Caching SharePoint Data Locally with SPServices and HTML5’s Web Storage for information on storing data locally in the browser cache.)

You can say what you want about my skills, but debugging Web Services calls in IE8 is well-nigh impossible. There’s simply no straightforward way to see the packets going out and back. Adding to the difficulty, Fiddler won’t run properly in this environment and the machine I remote into is locked down so that I can’t update it. Yup, three hands tied behind my back.

I finally got into the bowels of one of the calls using the IE8 Developer Tools (ick) and grabbed the innerHtml from the call. Here’s what I saw:

<xml:namespace prefix = soap />
<xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <?XML:NAMESPACE PREFIX = [default] http://schemas.microsoft.com/sharepoint/soap/ NS = "http://schemas.microsoft.com/sharepoint/soap/" />
    <GetListItemChangesSinceTokenResponse xmlns="http://schemas.microsoft.com/sharepoint/soap/">
      <GetListItemChangesSinceTokenResult>
        <listitems xmlns:rs="urn:schemas-microsoft-com:rowset">
          <Changes>
            <Id ChangeType="InvalidToken">
            </Id>
          </Changes>
          <?xml:namespace prefix = rs />
          <rs:data ItemCount="0">
          </rs:data>
        </listitems>
      </GetListItemChangesSinceTokenResult>
    </GetListItemChangesSinceTokenResponse>
  </soap:Body>
</soap:Envelope>

Note the highlighted lines. I’d never run into that response before, so off the to MSDN documentation for GetListItemsChangesSinceToken, which underlies the SPGetListItemsJson function.

The basic documentation page for GetListItemsChangesSinceToken doesn’t say anything about InvalidToken and I’d never run into it before, so I had no idea what it meant. But in an article marked as

This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

(Hey, we’re working with SharePoint 2007 here) there’s an explanation of several other possible responses to calls to the function:

Table 4: Change Events

Change Events Description
<Id ChangeType=”InvalidToken” /> The token is either invalid or old. You must do a full synchronization.
<Id ChangeType=”Restore” /> The list has been restored from the recycle bin or from a backup. You should perform a full synchronization.
<List></List> If the list schema has changed, or if a change token was not provided, the complete list is returned. The format is the same as returned by GetList.

In the first two cases above, the client should ignore other changes and do a full reconciliation of the list.

(See: https://msdn.microsoft.com/en-us/library/cc264031(v=office.12).aspxThe )

Like I said, I’ve never seen this response, so I had never built anything into SPGetListItemsJson to handle it.

The issue really came down to the fact that I was using localStorage rather than sessionStorage and I’d not built in a capability to occasionally throw away the localStorage cache and fully refresh it. Yes, my bad.

The quick fix was to switch to storing the cache in sessionStorage instead. On a lot of levels, that’s a good idea, anyway. Using localStorage means that I have a persistent cache, but if something goes wrong with that cache – corruption, or something exactly like this – there’s no quick fix. By switching to sessionStorage, I’ve made the cache volatile in that it will be purged at the end of each browser session. Now when someone has a problem with their cache (something they won’t ever understand from a user perspective), I can simply say “Please shut down your browser and restart. Let me know if that fixes the problem.”

I hope this helps someone else along the way.

Moving from SPServices to REST, Part 1: Introduction

This entry is part 1 of 2 in the series Moving from SPServices to REST

In this series, you’ll learn how to transition from the SOAP Web Services – which SPServices supports – to REST. If your organization is making a transition to SharePoint 2013 or Office 365 & SharePoint Online (and anything hybrid in between), this series will help you develop new patterns, migrate your codebase, use OData effectively and more.

It’s been a nice, long ride for SPServices. If you aren’t familiar with SPServices, it’s freely available on Codeplex, Microsoft’s waning open source repository.

SPServices is a jQuery library that abstracts SharePoint’s Web Services and makes them easier to use. It also includes functions that use the various Web Service operations to provide more useful (and cool) capabilities. It works entirely client side and requires no server install.

The Web Services that SPServices abstracts for you are the older SOAP Web Services, sometimes known as XML services, or ASMX services. As I’ve said repeatedly at conferences and in posts (like this one) before, I think we still have many miles to go with the SOAP Web Services. Microsoft officially “deprecated” the SOAP Web Services as of the launch of SharePoint 2013, but I can’t see them fully disappearing for a good long time given the number of different ways they are used.

Two API sets are still supported in the SharePoint 2013 framework for backward compatibility, but we recommend that you not use them for new projects: the ASP.NET (asmx) web services, and direct Remote Procedure Calls (RPC) calls to the owssvr.dll file.

See: https://msdn.microsoft.com/en-us/library/office/jj164060.aspx#DeprecatedAPIs

I started writing this series just after SPTechCon San Francisco in 2014, having talked to Andrew Connell and Scot Hillier about how I might be able to contribute to IT Unity for the site’s launch. Obviously the series fell by the wayside for quite a while, but these topics are even more relevant now than they were when they first came to me.

Like many of you, I drive an older model car. It is fun to drive, is comfortable and seems to be in great shape. The car may last me a while, yet from time to time I think about what I might want to drive next.

Moving from SPServices to REST is like deciding to trade in your reliable car for a model with more safety features. You should do it, eventually.

Photo Credit: “FIAT 600” by Andrea~S on Flickr http://ow.ly/wqxGN

At some point, we’ll all want to head into the dealer and get ourselves something shiny and new. To me, that something new is REpresentational State Transfer, also known as REST.

While SharePoint’s client-side object model (CSOM) can do a lot, it’s a SharePoint-specific technology. REST, however, is a Web standard. Many Web developers use REST to get their work done every day outside the SharePoint world. It’s time for those of us who develop on top of SharePoint to catch up with the rest of the Web development community. Most SharePoint developers have caught onto this reality in the last year or so.

When REST was first offered as an option in SharePoint 2010, it let us get some work done with lists, but that was about it. With SharePoint 2013 and even more so on Office365 with SharePoint Online and the Office365 APIs, the SharePoint team at Microsoft continues to give us more and more RESTful – or REST-driven – capabilities. New REST services have been introduced regularly and the existing services have been extended many times.

If you’ve loved working with SPServices, or at least you’ve gotten some good work done with it, this series of articles will help you to make the transition from the SOAP Web Services that SPServices wraps to the more open – and undoubtedly more robust over time – REST services offered in SharePoint 2013 and Office365. If you’re still running SharePoint 2007 or 2010 – as I know many of you are – this should still be a valuable series to you. By making some adjustments in your development style and the coding patterns that you follow, you can put yourself in an excellent position to move to REST when it becomes appropriate for you to do so.

Here are some of the things I plan to cover in this series:

  • New patterns I suggest you follow with your SPServices code so that you’ll be better prepared to move to REST when the time comes. A side benefit of these new patterns will be better modularity and reusability in your SPServices-based code.
  • You will learn ways to move parts of your code from SPServices to use REST even now with less effort than you might think. We’ll talk about where the most likely places in your codebase probably are and how to get started.
  • We’ll look at how the functionality in the SOAP Web Services maps to the current REST services and where there are still gaps.
  • We’ll talk about where OData comes in and how best to understand the OData componentry that sits under the SharePoint-specific implementation of REST. There are some differences from the standard (these gaps are narrowing all the time), and it’s important to know what those differences are and why they matter.
  • As Microsoft adds more functionality to REST in SharePoint 2013 and Office 365, I’ll write new articles in the series to augment the base set of content. It’s not your father’s Microsoft anymore – the SharePoint team is moving fast and improvements are going to be coming Fast and Furious (see what I did there?)

It’s hard to give up the old car that you’ve driven for years, but sooner or later, it’s time. Maybe the engine starts knocking too much or the springs in the seat get uncomfortable. Maybe you just want to try out something new. It is sad letting the old girl go, but at some point, it’s simply time. I once held onto a car until the engine simply stopped working. You don’t want to be in a similar situation with your SharePoint functionality built on top of SOAP. Let’s start to plan for the future, and let’s do it together.

Cooler with better design and safety features. Your new code using REST, using practices favored by the rest of the Web development community.

Photo Credit: “2014 Fiat 500” by Abdullah AlBargan on Flickr http://ow.ly/wqynJ

This article was first published on IT Unity on March 25, 2015. Visit the post there to read additional comments.

Great Resource: Responsive SharePoint from Eric Overfield

2015-03-20_10-55-33This week I was working on moving a design into SharePoint 2010. The design was made up of the standard sorts of things: HTML mockups with graphics, CSS, etc. The mockups are using Twitter Bootstrap as the styling framework.

I’ve done this sort of branding implementation many times, and it’s always interesting because what I get to start from is different every single time, even if it comes from the same designer. Over time we all change our approaches and styles because we learn better ways of doing things. That’s a good thing, but it means that every design implementation is like a new experience, albeit with many common components.

When I’m implementing a design in SharePoint, I find that there are use one of three basic approaches, regardless which version of SharePoint it is:

  • From the top down – In this case, I’m changing SharePoint’s master pages very little. We might want to change some colors, fonts, and basic functionality, but it still will look like SharePoint from a structural standpoint: banner across the top, Quick Launch down the left, etc. Here I just make a copy of default.master, v4.master, or seattle.master – depending on the version – and add some CSS and script references.
  • From the bottom up – This is the most extreme branding, where I start with a minimal master page and build things from the ground up. This is most common with Internet-facing sites, but can also be the case for an Intranet. Everything is customized in the master page: the HTML markup, the content placeholders and their placement, etc.
  • Something in between – This was the case with the current project. Even though the end goal is a site that look little like SharePoint, it doesn’t require a total disemboweling of the master page.

For the latter two bullets, we have some excellent choices available to us which have been created by the SharePoint community. To me, the shining lights here are Randy Drisgill (@drisgill), Eric Overfield (@EricOverfield), and Kyle Schaffer (@kyleschaeffer). (That’s an alphabetical list by last name; they are all equally awesome.)

All three of these brilliant guys have free starter solutions on Codeplex:

In this particular project, Eric’s master pages seemed to be the best match. As I was struggling to get the ribbon to behave with Bootstrap, I realized that I didn’t need to fight the battle again myself and that Eric’s master page would probably be just the tickets. Indeed it was! After grabbing his project, I was able to get the design up and running in just a matter of hours.

If you’re even considering coming up with a responsive and/or Bootstrap-driven design, save yourself a lot of trouble and look at Kyle or Eric’s work It probably won’t be exactly what you want out of the box, but they have solved a huge majority of the problems you’ll run into already. This is what makes the SharePoint community so awesome. This is really deep capability and work that is out there for you to use – for free.