Making Your REST Calls Simpler by Changing the Metadata Setting

When you talk to SharePoint using REST, you have some options about what the data you get or send looks like. Early on, when REST first arrived on the scene for SharePoint, we HAD to specify odata=verbose. This meant that we had to be very specific about what our data was going to look like, especially when it came to updates.

If you don’t use odata=verbose, then you’re telling the server to accept something different. You’re basically telling the server what “dialect” of odata you are choosing to speak for the current transaction.

If you look at this post, you’ll see the options:

JSON Light support in REST SharePoint API released

Just to recap, the options are:

accept: application/json; odata=verbose

This is probably what you’re used to, and requires the heaviest payload. Using this option, you’ll get the maximum amount of information about the data coming your way, and you’ll also have to send more as well.

accept: application/json; odata=minimalmetadata

This is the middle ground. You’ll get some metdata, and you’ll also need to send some, but it’s less.

accept: application/json; odata=nometadata

This option means we won’t get any metadata and we also don’t have to send any.

accept: application/json

If you don’t specify the odata setting at all, it will default to odata=minimalmetadata.

You can use these settings both on the inbound and outbound part of your REST calls.

Content-Type: "accept: application/json; odata=minimalmetadata"

means that you are sending data that contains minimal metadata.

Likewise,

Accept: "accept: application/json; odata=verbose"

means that you want to receive data with full metadata information.

In many cases, you’ll want to use odata=verbose as you’re debugging and switch to odata=nometadata once you move into production. The metadata tells you a lot about the data coming and going and can be helpful as you build up your calls. If you don’t make that switch, there will be more data going down the wire, though in many cases that doesn’t matter too much. Since I learned to code back when nibbles were expensive storage, I tend to want to reduce data size as much as I can, though.

Unfortunately, I don’t think many SharePoint developers realize what all this can do for you. I hear people say all the time that REST calls are too “chatty”. In many cases, that’s simply not true; you developers are making things too chatty!

This allows you to switch from something like:

var data = {
  "__metadata": {
    "type": "SP.Data.FC_x0020_RunsListItem"
  },
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json; odata=verbose",
    "content-type": "application/json;odata=verbose",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

to:

var data = {
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json",
    "content-type": "application/json;odata=nometadata",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

This is a real example from some of my code in a client project. It may not look like a huge savings, but if you are passing a lot of data either way, it can make a difference.


UPDATE on 3 May: Tip from Mikael Svenson (@mikaelsvenson) – If you’re running on premises SharePoint 2013, you may need to enable the multiple metadata formats for JSON. See:

Getting the Reply Count for a SharePoint Discussion Using the REST API

The other day I was creating a custom UI for a SharePoint 2013 Discussion list. To me, the out-of-the-box UI in SharePoint 2013 is definitely a step forward from previous versions. It’s still pretty rudimentary, though, and my client wanted something “more like other forums”.

SharePoint’s Discussion lists are sort of like Document Sets, in that the original post is a Discussion Content Type which inherits from Folder and the replies are the Message Content Type, which inherits from Item. So there aren’t any Documents involved, but Discussions are once again glorified Folders.

Since I’m using KnockoutJS on this project, I can make the UI look like pretty much anything. One thing we wanted to display was a reply count per thread. This seemed easy enough, but it could get expensive to retrieve all of the replies just to count them for the UI. Unfortunately, there was no obvious column (like “Replies” maybe?) to give me the answer.

Trolling around SharePoint StackExchange, I ran across a post by my friend Rothrock entitled How to get ItemChildCount of DocumentSet (folder) using REST Api asking a pretty similar question, but in his case it was about Document Sets. There ought to be a field like ItemChildCount or something, but he couldn’t find it, either. We’re used to a field like that using SPServices and SOAP, but there didn’t seem to be anything analogous in REST.

Luckily, someone who goes by ECM4D answered Rothrock with this example. (There were typos, but this was the idea.)

/_api/web/lists/getbytitle('your_list')/items?
  $select=ID,Title,Folder/ItemCount
  &$expand=Folder/ItemCount
  &$filter=FSObjType eq 1

By expanding the Folder, we can get at an ItemCount easily, just as we’d like to expect. This is yet another example where the documentation for the REST services simple doesn’t go deep enough to help us. Because much of the documentation is example-based – and this isn’t in any of the examples – we’re out of luck.

I ended up with something pretty similar, and it works great. Note that the ItemCount includes *all* replies, which means replies to the original post as well as replies to replies. If you wanted just replies to the original post, you’d need some other method.

MyProject.Promises.Discussions = $.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/getbytitle('Discussion')/items?" +
    "$select=ID,Title,FileRef,IsFeatured,Created,Author/Title,Folder/ItemCount" +
    "&$expand=Author,Folder",
  method: "GET",
  headers: {
    "Accept": "application/json; odata=verbose"
  }
});

This call – using jQuery’s $.ajax function – gets me the basic info about the Discussion items I need:

  • ID and FileRef let me provide links for the user to use to go deeper into the threads
  • Title is the Subject of the original post
  • IsFeatured tells me if it’s a featured post so that I can highlight it in some way
  • Created and Author tell me who started the thread and when
  • Folder/ItemCount is that mysterious count of replies I was looking for

All pretty easy, really, once I found that thread on SharePoint StackExchange. One of the best ways to learn the ins and out of the REST endpoints is to troll those public forums. Be forewarned, though: you’re just as likely to find something that doesn’t work as something that does.

Get all SharePoint Document Library Files and Folders at a ServerRelativeUrl in One REST Call

Recently, I was building a directory tree view of a Document Library for a client. Yes, you can say this shouldn’t be necessary. We can just tag the documents with metadata and we won’t need folders at all. Unfortunately, that’s not always the way people want to work.

To build this, I started by calling the _api/Web/Lists/getbytitle('Document Library Name')/items endpoint. I figured I’d just get all the documents in the library and sort out the display from the array. I got this working pretty well and in my test environment with a few hundred documents, it worked great.

Then – boom. There was a little requirement I didn’t know about: quite a few of the Document Libraries where we wanted to use this had more than 5000 documents. I stupidly hadn’t thought of that possibility. In my experience, it’s pretty unusual to see Document Libraries with that many documents, though it definitely happens.

Why does 5000 documents matter? Well, as of SharePoint 2010, we went from being able to request as many items as we wanted to an item limit of 5000. I’ll admit it’s not usually a great idea to request that many items from the client, but sometimes we need to.

I had two choices:

  • Add some paging logic to the call. This would mean that if there were more than 5000 items, I’d simply make Math.ceil(documentCount / 5000) calls to get them. In smaller Document Libraries, it would still be one call; as the number of documents went up, it would take more calls.
  • Be smart about it. Just request the objects (files and folders) in the root of the Document Library, and then on-demand, only request what I needed as the user expanded each folder.

The former would have been a little easier, but with larger libraries those calls could get pretty slow. The latter idea was really the “enterprise” way to do it. The problem was that the _api/Web/Lists/getbytitle('Document Library Name')/items endpoint didn’t really give me the right info to do it well.

So I turned to a different endpoint: _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl'). This is a newer endpoint and is designed for doing stuff like this. We can pass in the relative URL – maybe something like “/sites/SiteA/SubSiteA/LibraryName/TopFolder/SubFolderA/SubFolderB” – and get back just the files and folders for that relative path.

It takes two calls for this, though:

  • _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')/Folders
  • _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')/Files

That would work, but it seemed a bit inefficient. Wouldn’t it be better to get the files and folders at the same time?

Off I went to Bingle. Luckily, I found a post on SharePoint StackExchange pretty quickly from jkr asking the same thing: Get all Files and Folders in one call. Vadim Gremyachev replied with the trick.

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?$expand=Folders,Files

With this one call, we can get the info about the file and the folders together in one complex object.

Figure 1: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/$expand=Files,Folders

Figure 1: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/$expand=Files,Folders

As I said, this endpoint is perfect for building something like a directory tree.

There’s not a lot of good documentation for this endpoint (surprise!). You can find some examples of calls on the MSDN page Files and folders REST API reference, but no examples of the results. If you download the SharePoint 2013 REST Syntax (wall posters) you get some more clues.

The Files result provides results like those shown in Figure 2. As far as I can tell, there’s no way to control what fields you get back, as using $select has no effect.

Figure 2: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Files

Figure 2: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Files

The Folders result provides results like those shown in Figure 3. As far as I can tell, there’s no way to control what fields you get back here either, as using $select has no effect.

Figure 3: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Folders

Figure 3: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Folders

Note that in the Folders results, there are also Files and Folders objects, so the idea of recursion is there, though the objects are deferred. Because each folder has a ServerRelativeUrl value, you can dig as deep as you need to.

If you know you only need to go a few layers deep, you can also do things like:

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?

$expand=Folders,Folders/Folders,Folders/Folders/Folders

or

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?

$expand=Files,Folders/Files,Folders/Folders/Files

Both of these calls will get you three folders deep, which may be enough for some things you might want to do. I could also see using a call like these latter ones to get a bit ahead of your user to reduce the “chatter” on the line. That would make your array processing on the client side a little more complex, but could be worth it.

With some spiffy recursion in your framework of choice, you can build some very nice user interfaces with data like this. But that’s for another post…

Using the Signature Pad jQuery Plugin with SharePoint & InfoPath – Redux

I’m using Thomas Bradley’s Signature Pad plugin for a project, which I’ve used successfully before. The twist this time is that I want to save the signature as an image rather than just as JSON.

There’s a method called getSignatureImage()  that works just fine to grab the signature as a base64 string, like so (this is the result for an empty canvas):

data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAACWCAYAAAA8Els6AAAIjElEQ…QIECBg8LoBAgQIECBAgACBtIDBm65XOAIECBAgQIAAgQtmc1xdE+aNPAAAAABJRU5ErkJggg==

(Note the “…”; it’s a much longer string.)

Snazzy signatureI was having a problem saving the signature to a library successfully. Uploading the file was easy, but the image file was always ending up corrupted. It didn’t matter if I uploaded to a Document library or Picture Library; no joy.

I knew I was missing something obvious. I tried removing the leading “data:image/png;base64,”. I tried different values for Content-Type, etc. It had to be something about the way I was creating the file.

In the end, I got some great advice from a colleague. The base64 content has to be *decoded* so that we can save it. This is what worked:

// Get the base64-encoded image from the plugin
var img = signatureArea.getSignatureImage();
var outfile = fakefilename(); // I'm creating a unique filename to use for saving in my testing

$.ajax({
    url: _spPageContextInfo.webAbsoluteUrl +
        "/_api/web/getfolderbyserverrelativeurl('/path/to/my/picture/library/')/files/add(overwrite=true, url='" + outfile + "')",
    type: "POST",
    data: convertDataURIToBinary(img),
    processData: false,
    headers: {
        "accept": "application/json;odata=verbose",
        "X-RequestDigest": $("#__REQUESTDIGEST").val()
    },
    success: function() {
        $("#sig-file").attr("src", outfile); // Show the image file in the form
    }
});

// Useful function "borrowed" from http://sharepoint.stackexchange.com/questions/60417/cant-upload-a-non-text-file-to-sharepoint-app-via-rest-api
// Decodes the base64 text data back into the binary data representation of the image file
var convertDataURIToBinary = function(dataURI) {
    var base64Marker = ";base64,";
    var base64Index = dataURI.indexOf(base64Marker) + base64Marker.length;
    var base64 = dataURI.substring(base64Index);
    var raw = window.atob(base64);
    var rawLength = raw.length;
    var array = new window.Uint8Array(new window.ArrayBuffer(rawLength));

    for (i = 0; i < rawLength; i++) {
        array[i] = raw.charCodeAt(i);
    }
    return array;
};

Are We on the Cusp of a Citizen Developer Revolution?

I ran across an article on ZDNet today entitled The advent of the citizen developer. In the article,  posits that we may finally be at the cusp of a citizen development revolution.

Yes!

I’ve been arguing for years – in one way or another (See, for example, The Middle Tier Manifesto: An Alternative Approach to Development with Microsoft SharePoint) – that the citizen developers in any organization are the lynch pins to IT success. This gray area, where the work isn’t just configuration and isn’t “real” development has been critical to the success of SharePoint in many organizations for years. What we call it is less important than what it is: the lifeblood of technology use and success in the modern organization.

Unfortunately, the response from most IT departments is that these very capable citizen developers will simply screw everything up and – oh, no! – require support the IT department can’t provide. This attitude is truly at IT’s peril. Instead of being seen as having a customer service attitude, they become known as the “NO! people”. “NO!” should almost never be the answer; this simply sends end users off to use external tools, which are usually far better that what IT provides to the organization. Thus, the very security that IT claims to be protecting falls apart. The end users are happy, but only in violation of corporate governance, whether it is explicit or not.

When I speak at conferences I often talk about IT offering Functions as a Service (FaaS) to the organization. (For example, see slide 12 in my recent presentation at SharePoint konference 2016 in Munich Alternative Approaches to Solution Development in SharePoint and Office 365.) By providing a set of reliable building blocks, whether vetted external JavaScript libraries or internally developed functionality that citizen developers can incorporate in their solutions, IT simply must get on board with this work. Offering consultative services in support of citizen developers must be cheaper and more productive than not doing so in the long run, though I can’t point to a study to prove it.

Even Microsoft is starting to understand how important these citizen developers are. At the recent SPTechCon in Austin, Seth Patton, the Global Senior Director for SharePoint and OneDrive product marketing even had a slide in his keynote acknowledging that citizen developers *exist*, which is a HUGE step forward. If Microsoft is thinking about empowering citizen developers, shouldn’t you be?

Continued developer empowerment

‘Continued developer empowerment’ from Seth Patton’s SPTechCon Austin 2016 keynote

Here’s hoping that now and into the 2020s become the decade(s) of citizen developers, even though they have been around as long as there has been computing. If it weren’t for the dabblers and hobbyists inside organizations, we wouldn’t have evolved to the places we’re in now. These people are the ones who push the envelope in directions that makes sense to the business, not just to some power grabbers in IT.