Microsoft’s May 4 SharePoint Announcements and Client Side Development Futures

Bill Gates 1981

Image source: http://www.biography.com/news/bill-gates-biography-facts

In the beginning, there was Bill, and Bill was good. Bill revolutionized the concept of computing (along with a handful of others). But somewhere along the way, something happened. It’s not clear exactly what. Maybe it was the disaster of Vista. Maybe it was the reign of Ballmer. What it was exactly doesn’t really matter anymore.

Microsoft had become a monolithic, unresponsive thing, with few ways to penetrate its outer ranks. We felt like the chimps at the monolith; we had little control. We didn’t know what Microsoft was anymore.

Image source: http://www.salon.com/writer/david_mcraney/

Image source: http://www.salon.com/writer/david_mcraney/

 

Instead of listening to its customers, Microsoft responded with reasons why they knew best, why they would take care of us, even if it didn’t feel like they were taking care of us at all. When we asked for things, we didn’t get them.

I'm sorry, Dave. I'm afraid can't do that.

I’m sorry, Dave. I’m afraid can’t do that. Image source: https://i.ytimg.com/vi/qDrDUmuUBTo/maxresdefault.jpg

We no longer felt a connection with Microsoft. They weren’t watching our back; they weren’t listening to us; they didn’t seem to know what we wanted.

Then something changed. It changed very quickly. None of us really understood what was changing, but we knew it was good.

2001 Stargate

Image source: http://www.sci-fi-o-rama.com/2008/12/11/2001-a-space-odyssey-2/

Fast forward to today: May 4, 2016. Today’s announcements about SharePoint feel like the culmination of several years of recovery. It feels like things have come around again. Even an old cynic like me can be a Microsoft fan now without regretting it half the time.

Back to the Future

Image source: http://www.pcadvisor.co.uk/how-to/photo-video/how-watch-back-future-ii-on-21-october-2015-at-cinema-trilogy-anniversary-30th-today-3627401/

Today Jeff Teper and his crew have both turned back time and looked forward to show us a wide vision of the new Microsoft and the #FutureOfSharePoint. It feels fresh; it feels right; it feels like we’re taking a wild ride together.

What were the big stories coming out of today’s event (which fell on a date with a science fiction theme from some other movie)?

Well, to me, the bullet points are:

  • SharePoint is alive and kicking – working out for a long marathon run. More than 200,000 organizations use SharePoint today, and an extraordinary community of more than 50,000 partners and 1 million developers make up a $10 billion solutions ecosystem around SharePoint.
  • The heart of SharePoint is the documents that make our work work. But rather than looking for the files you need, those files will find you. Almost any device – any time.
  • The modern intranet has evolved, and Microsoft is out ahead of it. The new mobile SharePoint app, the new SharePoint home page, and new modern team sites may well be ahead of the market. Many organizations will take years to catch up to what Microsoft has on offer.
  • SharePoint 2016 went to General Availability today, meaning it’s available in most sales channels.  That in itself is news but not unexpected. We heard more today about what SharePoint 2016 really is: a springboard to the future.
  • Security, security, security – This stuff fails to excite me, though I know it is exceedingly important to many.

The SharePoint Framework

It probably won’t surprise you to hear the single most exciting thing to me about today’s announcements is the new SharePoint Framework. One of the best things about this new development option is that it’s additive: it doesn’t replace anything or make anything obsolete. If you’re a server side developer, you’re not losing anything. If you’ve invested in the Add-In model, you can still make hay with that. But if you’re like me and have seen how your clients’ or customers’ faces light up when they see well-built client side functionality, then you can walk with pride now. What the new SharePoint Framework does is elevate JavaScript development on top of SharePoint to a first class citizen.

I was ecstatic to see an early iteration of the SharePoint Framework early this year at a Developer Kitchen (DevKit to insiders) in Redmond. What we put our hands on and built things with was the culmination of a journey I’ve had a lot of fun making. The ability to use JavaScript on top of SharePoint has always been there. What’s happened is that the general view of that approach has gone from ridicule to cautious interest to outright excitement. The SharePoint Framework is the right next step.

At the DevKit, senior members of the SharePoint Product Group – like Vesa Juvonen, Chakkaradeep “Chaks” Chandran, Luca Bandinelli, Dan Kogan, and many more –  worked side by side with us to understand the warts of the new approach, taking away notes on ways to fix it. The DevKit was no holds barred – when they didn’t have an answer, they said so; when they wanted to see if we had better ideas, they asked. They didn’t have all the answers then, and they probably don’t now. I actually think that’s a great thing for a few reasons:

  1. They realize that the Framework will evolve, and they want it to evolve based on our input and feature requests.
  2. Monolithic thinking is gone. The Product Group will be adding new bits and bobs fast.
  3. The Product Group will be using the SharePoint Framework to build out new “experiences” themselves. No more “secret” tools like in the past – they will feel the pain of any issues right along with us.

    In fact, we have built the new experiences for our new mobile app, SharePoint Online and OneDrive for Business, including the new document library and list experiences, using the SharePoint Framework.

  4. The Framework is framework-agnostic. It’s designed to support today’s hot development frameworks like KnockoutJS, AngularJS, and ReactJS. Because it is agnostic, the next shiny new framework that comes along should work just fine, too.

This is going to be the future of SharePoint, without abandoning the past. What it does is legitimize client side development to a level that hasn’t been possible before. We’ll have enterprise-grade tooling to build client side Web Parts and full scale solutions we’ve never had before. Even if you’ve been doing client side development already and have a reliable set of methods (as I do), you’ll appreciate the new focus and will be able to take advantage of it gradually.

The-SharePoint-framework-an-open-and-connected-platform-2

We’ll have a page and part model that allows us to “take over” as much as we need. I expect many of you will simply build new client side Web Parts. Others will build full Single Page Applications using the SharePoint Framework. We’ll have several app “shapes” we can use to build even more robust solutions.

The-SharePoint-framework-an-open-and-connected-platform-3

As I said, we get new tooling, including a new client side SharePoint Workbench to test our work. The SharePoint Framework will come out of the gate with many of the features we need, with more coming all the time. Remember, the SharePoint Product Group has committed to use the SharePoint Framework to build new features into SharePoint. If it doesn’t work for them – and thus us – they will fix it. We’ve never had that level of development parity before.

We get a canvas that is an evolution of the existing Web Part Zone. Within that canvas, we can take over the entire thing, or just parts of it. (You’ll recognize the canvas from the Delve Blog experience.) Microsoft takes care of the chrome and the services we need to make the components on the canvas sing, including the Data Broker, Caching, Authentication, and Telemetry. That’s right, we can even tap into the same telemetry engine that Microsoft uses so that we can understand how our solutions are being used. We even get responsive design right out of the box from the canvas. We need to be sure to build our stuff correctly to take care of that responsive nature, but we don’t need to set up extra stuff for that responsiveness.

We get a new property panel that’s the evolution of the Tool Pane in existing Web Parts. We can add our own properties and functionality to the property panel based on what we are building – all with commonly known client side tools.

Speaking of tools, there are some pieces of this that many current SharePoint client side devs probably aren’t using. There WILL be a learning curve, but it will be worth it. People who already know how to do Web Development – separate from SharePoint – will probably know at least some of these tools; others are new:

  • Yeoman for initial application setup
  • Visual Studio Code (you can probably insert your favorite IDE here at some point) and later Visual Studio
  • NodeJS to give us a local JavaScript development environment
  • NPM to grab existing open source packages
  • TypeScript to bring real typing to the JavaScript experience. We can also use plain old JavaScript if we’re more comfortable there. (I find that C# developers like TypeScript because it gives them the comfort of “true” classes.)
  • Gulp to build our applications, package them, and deploy them to our own CDNs
  • SharePoint Workbench for testing

SharePoint Framework Tool Chain

From my experience at the DevKit, some of this is still a little rough around the edges, but I expect that it’s becoming more polished on a daily basis. Remember, this isn’t that unfriendly Microsoft – they are shipping improvements weekly based on our input and requests. We’re going to be on this ride together, and it’s going to be really fun.

There’s plenty more to read about the SharePoint Framework and more in the official posts from today’s event on the Office Blog:

As Jeff said in one of the posts coming out of today’s event: The future starts today.

Roads? Where we're going, we don't need roads

Image source: http://pixel.nymag.com/imgs/daily/vulture/2014/12/22/back-to-the-future/22-back-to-the-future-001.nocrop.w529.h316.jpg

Making Your REST Calls Simpler by Changing the Metadata Setting

When you talk to SharePoint using REST, you have some options about what the data you get or send looks like. Early on, when REST first arrived on the scene for SharePoint, we HAD to specify odata=verbose. This meant that we had to be very specific about what our data was going to look like, especially when it came to updates.

If you don’t use odata=verbose, then you’re telling the server to accept something different. You’re basically telling the server what “dialect” of odata you are choosing to speak for the current transaction.

If you look at this post, you’ll see the options:

JSON Light support in REST SharePoint API released

Just to recap, the options are:

accept: application/json; odata=verbose

This is probably what you’re used to, and requires the heaviest payload. Using this option, you’ll get the maximum amount of information about the data coming your way, and you’ll also have to send more as well.

accept: application/json; odata=minimalmetadata

This is the middle ground. You’ll get some metdata, and you’ll also need to send some, but it’s less.

accept: application/json; odata=nometadata

This option means we won’t get any metadata and we also don’t have to send any.

accept: application/json

If you don’t specify the odata setting at all, it will default to odata=minimalmetadata.

You can use these settings both on the inbound and outbound part of your REST calls.

Content-Type: "accept: application/json; odata=minimalmetadata"

means that you are sending data that contains minimal metadata.

Likewise,

Accept: "accept: application/json; odata=verbose"

means that you want to receive data with full metadata information.

In many cases, you’ll want to use odata=verbose as you’re debugging and switch to odata=nometadata once you move into production. The metadata tells you a lot about the data coming and going and can be helpful as you build up your calls. If you don’t make that switch, there will be more data going down the wire, though in many cases that doesn’t matter too much. Since I learned to code back when nibbles were expensive storage, I tend to want to reduce data size as much as I can, though.

Unfortunately, I don’t think many SharePoint developers realize what all this can do for you. I hear people say all the time that REST calls are too “chatty”. In many cases, that’s simply not true; you developers are making things too chatty!

This allows you to switch from something like:

var data = {
  "__metadata": {
    "type": "SP.Data.FC_x0020_RunsListItem"
  },
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json; odata=verbose",
    "content-type": "application/json;odata=verbose",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

to:

var data = {
  "Title": run.RunID,
  "RunDate": run.RunDate,
  "OperatorId": run.Operator.Id,
  "DataIDs": dataIds.join(","),
  "FCData": angular.toJson(run.samples)
};

var request = {
  method: 'POST',
  url: url,
  data: JSON.stringify(data),
  headers: {
    "Accept": "application/json",
    "content-type": "application/json;odata=nometadata",
    "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
    "X-HTTP-Method": method,
    "IF-MATCH": "*"
  }
};

This is a real example from some of my code in a client project. It may not look like a huge savings, but if you are passing a lot of data either way, it can make a difference.


UPDATE on 3 May: Tip from Mikael Svenson (@mikaelsvenson) – If you’re running on premises SharePoint 2013, you may need to enable the multiple metadata formats for JSON. See:

Getting the Reply Count for a SharePoint Discussion Using the REST API

The other day I was creating a custom UI for a SharePoint 2013 Discussion list. To me, the out-of-the-box UI in SharePoint 2013 is definitely a step forward from previous versions. It’s still pretty rudimentary, though, and my client wanted something “more like other forums”.

SharePoint’s Discussion lists are sort of like Document Sets, in that the original post is a Discussion Content Type which inherits from Folder and the replies are the Message Content Type, which inherits from Item. So there aren’t any Documents involved, but Discussions are once again glorified Folders.

Since I’m using KnockoutJS on this project, I can make the UI look like pretty much anything. One thing we wanted to display was a reply count per thread. This seemed easy enough, but it could get expensive to retrieve all of the replies just to count them for the UI. Unfortunately, there was no obvious column (like “Replies” maybe?) to give me the answer.

Trolling around SharePoint StackExchange, I ran across a post by my friend Rothrock entitled How to get ItemChildCount of DocumentSet (folder) using REST Api asking a pretty similar question, but in his case it was about Document Sets. There ought to be a field like ItemChildCount or something, but he couldn’t find it, either. We’re used to a field like that using SPServices and SOAP, but there didn’t seem to be anything analogous in REST.

Luckily, someone who goes by ECM4D answered Rothrock with this example. (There were typos, but this was the idea.)

/_api/web/lists/getbytitle('your_list')/items?
  $select=ID,Title,Folder/ItemCount
  &$expand=Folder/ItemCount
  &$filter=FSObjType eq 1

By expanding the Folder, we can get at an ItemCount easily, just as we’d like to expect. This is yet another example where the documentation for the REST services simple doesn’t go deep enough to help us. Because much of the documentation is example-based – and this isn’t in any of the examples – we’re out of luck.

I ended up with something pretty similar, and it works great. Note that the ItemCount includes *all* replies, which means replies to the original post as well as replies to replies. If you wanted just replies to the original post, you’d need some other method.

MyProject.Promises.Discussions = $.ajax({
  url: _spPageContextInfo.webAbsoluteUrl +
    "/_api/web/lists/getbytitle('Discussion')/items?" +
    "$select=ID,Title,FileRef,IsFeatured,Created,Author/Title,Folder/ItemCount" +
    "&$expand=Author,Folder",
  method: "GET",
  headers: {
    "Accept": "application/json; odata=verbose"
  }
});

This call – using jQuery’s $.ajax function – gets me the basic info about the Discussion items I need:

  • ID and FileRef let me provide links for the user to use to go deeper into the threads
  • Title is the Subject of the original post
  • IsFeatured tells me if it’s a featured post so that I can highlight it in some way
  • Created and Author tell me who started the thread and when
  • Folder/ItemCount is that mysterious count of replies I was looking for

All pretty easy, really, once I found that thread on SharePoint StackExchange. One of the best ways to learn the ins and out of the REST endpoints is to troll those public forums. Be forewarned, though: you’re just as likely to find something that doesn’t work as something that does.

Get all SharePoint Document Library Files and Folders at a ServerRelativeUrl in One REST Call

Recently, I was building a directory tree view of a Document Library for a client. Yes, you can say this shouldn’t be necessary. We can just tag the documents with metadata and we won’t need folders at all. Unfortunately, that’s not always the way people want to work.

To build this, I started by calling the _api/Web/Lists/getbytitle('Document Library Name')/items endpoint. I figured I’d just get all the documents in the library and sort out the display from the array. I got this working pretty well and in my test environment with a few hundred documents, it worked great.

Then – boom. There was a little requirement I didn’t know about: quite a few of the Document Libraries where we wanted to use this had more than 5000 documents. I stupidly hadn’t thought of that possibility. In my experience, it’s pretty unusual to see Document Libraries with that many documents, though it definitely happens.

Why does 5000 documents matter? Well, as of SharePoint 2010, we went from being able to request as many items as we wanted to an item limit of 5000. I’ll admit it’s not usually a great idea to request that many items from the client, but sometimes we need to.

I had two choices:

  • Add some paging logic to the call. This would mean that if there were more than 5000 items, I’d simply make Math.ceil(documentCount / 5000) calls to get them. In smaller Document Libraries, it would still be one call; as the number of documents went up, it would take more calls.
  • Be smart about it. Just request the objects (files and folders) in the root of the Document Library, and then on-demand, only request what I needed as the user expanded each folder.

The former would have been a little easier, but with larger libraries those calls could get pretty slow. The latter idea was really the “enterprise” way to do it. The problem was that the _api/Web/Lists/getbytitle('Document Library Name')/items endpoint didn’t really give me the right info to do it well.

So I turned to a different endpoint: _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl'). This is a newer endpoint and is designed for doing stuff like this. We can pass in the relative URL – maybe something like “/sites/SiteA/SubSiteA/LibraryName/TopFolder/SubFolderA/SubFolderB” – and get back just the files and folders for that relative path.

It takes two calls for this, though:

  • _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')/Folders
  • _api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')/Files

That would work, but it seemed a bit inefficient. Wouldn’t it be better to get the files and folders at the same time?

Off I went to Bingle. Luckily, I found a post on SharePoint StackExchange pretty quickly from jkr asking the same thing: Get all Files and Folders in one call. Vadim Gremyachev replied with the trick.

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?$expand=Folders,Files

With this one call, we can get the info about the file and the folders together in one complex object.

Figure 1: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/$expand=Files,Folders

Figure 1: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/$expand=Files,Folders

As I said, this endpoint is perfect for building something like a directory tree.

There’s not a lot of good documentation for this endpoint (surprise!). You can find some examples of calls on the MSDN page Files and folders REST API reference, but no examples of the results. If you download the SharePoint 2013 REST Syntax (wall posters) you get some more clues.

The Files result provides results like those shown in Figure 2. As far as I can tell, there’s no way to control what fields you get back, as using $select has no effect.

Figure 2: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Files

Figure 2: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Files

The Folders result provides results like those shown in Figure 3. As far as I can tell, there’s no way to control what fields you get back here either, as using $select has no effect.

Figure 3: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Folders

Figure 3: Complex object returned from _api/Web/GetFileByServerRelativeUrl()/Folders

Note that in the Folders results, there are also Files and Folders objects, so the idea of recursion is there, though the objects are deferred. Because each folder has a ServerRelativeUrl value, you can dig as deep as you need to.

If you know you only need to go a few layers deep, you can also do things like:

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?

$expand=Folders,Folders/Folders,Folders/Folders/Folders

or

_api/Web/GetFolderByServerRelativeUrl('folderRelativeUrl')?

$expand=Files,Folders/Files,Folders/Folders/Files

Both of these calls will get you three folders deep, which may be enough for some things you might want to do. I could also see using a call like these latter ones to get a bit ahead of your user to reduce the “chatter” on the line. That would make your array processing on the client side a little more complex, but could be worth it.

With some spiffy recursion in your framework of choice, you can build some very nice user interfaces with data like this. But that’s for another post…

Uploading Attachments to SharePoint Lists Using REST

In a previous post, I walked through Uploading Attachments to SharePoint Lists Using SPServices. Well, in the “modern” world, I want to use REST whenever I can, especially with SharePoint Online.

Add an attachmentI ran into a challenge figuring out how to make an attachment upload work in an AngularJS project. There are dozens of blog posts and articles out there about uploading files to SharePoint, and some of them even mention attachments. But I found that not a single one of the worked for me. It was driving me nuts. I could upload a file, but it was corrupt when it got there. Images didn’t look like images, Excel couldn’t open XLSX files, etc.

I reached out to Julie Turner (@jfj1997) and asked for help, but as is often the case, when you’re not as immersed in something it’s tough to get the context right. She gave me some excellent pointers, but I ended up traversing some of the same unfruitful ground with her.

Finally I decided to break things down into the simplest example possible, much like I did in my earlier post with SOAP. I created a test page called UploadTEST with a Content Editor Web Part pointing to UploadText.html, below:

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.3/jquery.js"></script>

<input id="my-attachments" type="file" fileread="run.AttachmentData" fileinfo="run.AttachmentInfo" />

<script type="text/javascript" src="/sites/MySiteCollection/_catalogs/masterpage/_MyProject/js/UploadTEST.js"></script>

Like I said, I wanted to keep it simple. Here’s what I’m doing:

  • In line 1, I load a recent version of jQuery from cdnjs. On the theory that Angular just adds another layer of complexity, I wanted to try just jQuery, which is useful for its $.ajax function, among many other things.
  • In line 3, I’ve got an input field with type=”file”. In “modern” browsers, this gives us a familiar file picker.
  • In line 5, I’m loading my script file called UploadTest.js, below:
$(document).ready(function() {

 var ID = 1;
 var listname = "UploadTEST";

 $("#my-attachments").change(function() {

  var file = $(this)[0].files[0];

  var getFileBuffer = function(file) {

   var deferred = $.Deferred();
   var reader = new FileReader();

   reader.onload = function(e) {
    deferred.resolve(e.target.result);
   }

   reader.onerror = function(e) {
    deferred.reject(e.target.error);
   }

   reader.readAsArrayBuffer(file);

   return deferred.promise();
  };

  getFileBuffer(file).then(function(buffer) {

   $.ajax({
    url: _spPageContextInfo.webAbsoluteUrl +
     "/_api/web/lists/getbytitle('" + listname + "')/items(" + ID + ")/AttachmentFiles/add(FileName='" + file.name + "')",
    method: 'POST',
    data: buffer,
    processData: false,
    headers: {
     "Accept": "application/json; odata=verbose",
     "content-type": "application/json; odata=verbose",
     "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
     "content-length": buffer.byteLength
    }
   });

  });

 });
});

Here’s how this works – and yes, it does work!

  • I’ve got everything wrapped in a $(document).ready() to ensure the page is fully loaded before my script runs.
  • Lines 3-4 just set up some variables I could use for testing to keep things simple.
  • In line 6, I bind to the change event for the input field. Whenever the user chooses a file in the dialog, this event will fire.
  • In line 8, I’m getting the information about the file selected from the input element.
  • Lines 10-26 is the same getFileBuffer function I used with SOAP; it’s where we use a FileReader to get the contents of the selected file into a buffer.
  • The function in lines 28-42 runs when the file contents have been fully read. The getFileBuffer function returns a promise, and that promise is resolved when the function has gotten all of the file contents. With a large file, this could take a little while, and by using a promise we avoid getting ahead of ourselves. Here we make the REST call that uploads the attachment.
    • The URL ends up looking something like: “/my/site/path/_api/web/lists/getbytitle(‘UploadTEST’)/items(1)/AttachmentFiles/add(FileName=’boo.txt’)”
    • The method is a POST because we’re writing data to SharePoint
    • The data parameter contains the “payload”, which is the buffer returned from the getFileBuffer function.
    • The headers basically tell the server what we’re up to:
      • Accept doesn’t really come into play here unless we get a result back, but it says “send me back json and be verbose”
      • content-type is similar, but tells the server what it is getting from us
      • X-RequestDigest is the magic string from the page that tells the server we are who it thinks we are
      • content-length tells the server how many bytes it should expect

This worked for me on a basic Custom List (UploadTEST). So now I knew it was possible to upload an attachment using REST.

I took my logic and shoved it back into my AngularJS page, and it still worked! However, when I switched from $.ajax to AngularJS’s $http, I was back to corrupt files again. I’m still not sure why that is the case, but since I’m loading jQuery for some other things, anyway, I’ve decided to stick with $.ajax for now instead. If anyone reading this has an idea why $http would cause a problem, I’d love to hear about it.

Once again, I’m hoping my beating my head against this saves some folks some time. Ideally there would be actual documentation for the REST endpoints that would explain how to do things. Unfortunately, if there is for this, I can’t find it. I’d also love to know if /add allows any other parameters, particularly whether I can overwrote the attachment if one of the same name is already there. Maybe another day.