When Using Angular, Be Sure to Convert to JSON Using `angular.toJson`

This is a quick one, but if it has the potential to bite you, you’ll thank me for it later.

Image from https://memegenerator.net/Slap-My-Head

I use AngularJS generally these days when I want to write solutions in SharePoint, whether on premises or in SharePoint Online. Yes, that’s the older version of Angular, meaning versions in the 1.x range.

When you’re using any JavaScript to work with SharePoint data, though, you’ll probably end up converting the data from string-based JSON to JavaScript objects and if you want to save them back to SharePoint, you’ll then convert the data from JavaScript objects back to string-based JSON.

There are two primary ways for do the second conversion:

  • JSON.stringify – All “modern” browsers support this function and it works great. If you’ve been doing JavaScript development for a while, you’ll know this function very well – so well that you may rarely think about it when you use it.
  • angular.toJson – This is AngularJS’s version of the idea, but it’s smart specifically for objects which are under Angular’s control. In many cases, Angular augments objects with its own properties to keep track of things going on in the page. For example, $$hashkey helps Angular keep track of what objects have changed since a digest cycle.

Depending on how you manage the objects which contain your SharePoint data, you may end up with some of these additional properties on things. If you try to save a list or library item with one of these unrecognizable properties (to SharePoint, at least), you may get an error.

Because of this, it’s important to either:

  1. Delete the properties which will be seen by SharePoint as invalid before you use JSON.stringify in advance of the writes, or
  2. Use angular.toJson instead

I would recommend the latter, after getting bitten by my own code earlier this week. I had taken a great function Julie (@jfj1997) had adapted from Andrew Connell’s (@andrewconnell) earlier work on REST batching (See the article Make batch requests with the REST APIs for a link to AC’s examples.) and embedded it into a solution. Everything worked just fine for quite a while, and then we started noticing a few items missing when we expected them to have been loaded into a list – and it was our favorite type of bug: sporadic. It turned out that several of the items had Angular properties and were throwing errors in the batch loads. This was my fault – when I decided to use the function, I didn’t change JSON.stringify to angular.toJson. Once I did, all was right as rain.

Lesson learned for me (yet again!): be sure you understand exactly what the code you borrow really does.

Advertisements

Auth, Auth, Auth – The Bane of Our Development Existence

I was reading through one of Bob German’s (@Bob1German) recent (and great!) posts about Porting REST calls to SharePoint Framework and thinking about a conversation Bob, Julie (@jfj1997) and I had at our “salon”* when I was reminded of a very specific truth about our working days.

Image from New York City Department of Transportation on Flickr

I would venture to say that some ridiculously high percentage of all of our time developing with SharePoint and Office 365 comes down to figuring out why we can’t get something to work due to authorization or authentication issues. Some say this is a necessary evil to create enterprise software, but I say it’s an unnecessary problem. Unfortunately, documentation about auth stuff tends to be tremendously opaque and difficult to understand. In the same way that I expect the server to be there and I want nothing to do with it directly, I want auth stuff to just work – easily.

Sadly, this has never been the case with so-called “enterprise-class” software. On some level, I think the obtuseness of auth is there on purpose to keep the bar high enough to exclude lower end hackers. Unfortunately, we all get caught up in the kudzu of auth as a corollary effect.

Image from NatalieMaynor on Flickr

Ok, so that’s my editorial on auth. I wish it were easier, but it isn’t.

Recently in one of my Single Page Applications (SPAs) for a client, we kept getting weird failures in posting data to a list. Weird mainly in that I never saw the error on my end, my client is far away, and screen sharing is tricky due to the technical proficiency on that end. I’m not dissing anyone; they are great at what they do, but debugging JavaScript with me is not in their wheelhouse.

If you write software, you know that the worst bugs to squash are those that happen sporadically, and only on someone else’s machine – especially if you don’t have direct access to that machine. As usually, though, it was simply me doing something dumb with auth – which is not easy. Have I mentioned that?

Basically, the problem was that while I was fetching the request digest from the page (this is a “classic” page in SharePoint Online), I wasn’t ever refreshing the token. In this application, people move around from page to page enough and use the application for short enough time periods that we simply hadn’t seen the problem in testing.

Smart people will think “Marc’s an idiot” on this one, but I play the fool so you don’t have to.

The code comes from a service I use everywhere in my applications for this particular client. It’s basically a set of utility functions that are useful when you’re using Angular with SharePoint. I’ve built the SPA using AngularJS (1.x), so my code below represents that. However, similar logic can work with jQuery or whatever instead of AngularJS and $q. I adapted it from some Typescript code Julie was using, so credit there, with a bit of add on from me. I’ve added a bunch of comments and have also left some of the console logging I used in debugging – but commented out.

The data we get back looks something like this:

The thing we care about most is the FormDigestValue , as that’s what we use to make our POSTs to SharePoint while it’s valid. But also note that there is an attribute in the returned JSON for FormDigestTimeoutSeconds. That’s the number of seconds that this particular token will be valid. In every tenant where I’m using the code, that works out to be 30 minutes. However, there may well be a setting which can change that, or Microsoft may change the time span on us. Because of this, I use the value to calculate how often to request a new token: T-2 minutes.

I’m pretty sure that there are pages in SharePoint which don’t do this correctly, so I’m not feeling like too much of an idiot. For example, when we click on an “app” – usually a list or library to you and me – in Site Contents, it opens in a new browser tab. Very often when I go back to that Site Contents page – when it has been sitting there for a while – I’ll get an unauthorized error. This may be fixed by now, though it has happened to me a lot.

I hope this is helpful. Using functions like this, we can make the whole auth thing easier on ourselves – and there zero reason to need to write this code from scratch every time. Store int in one place and if anything changes, you’ll have one place to fix it later.


  • We have occasional “software salons” – usually in one of our homes – where we simply get together to kick around what’s going on in our work lives, interesting things we’ve solved, etc. It’s a tremendously educational and useful exercise. I would encourage you to do something similarly informal if you don’t have enough water cooler time. At Sympraxis, we have no water cooler.

Yes, Virginia, You Can Get More than 5000 SharePoint Items with REST

If you haven’t been paying any attention, you might not know that I loathe the 5000 item limit in SharePoint. I may have mentioned it here and here and here and a bunch of other places, but I’m not sure. But given it’s there, we often need to work around it.

No 5000 item limit

I’ve written this little function before, but alas today I needed it again but couldn’t find any previous incarnations. That’s what this blog is for, though: to remind me of what I’ve already done!

In this case, I’m working with AngularJS 1.x. We have several lists that are nearing the 5000 item level, and I don’t want my code to start failing when we get there. We can get as many items as we want if we make repeated calls to the same REST endpoint, watching for the __next link in the results. That __next link tells us that we haven’t gotten all of the items yet, and provides us with the link to get the next batch, based on how we’ve set top in the request.

Here’s an example. Suppose I want to get all the items from a list which contains all of the ZIP codes in the USA. I just checked, and that’s 27782 items. That’s definitely enough to make SharePoint angry at us, what with that 5000 item limit and all. Let’s not get into an argument about whether I need them all or not. Let’s just agree that I do. It’s an example, after all.

Well, if we set up our requests right, and use my handy-dandy recursive function below, we can get all of the items. First, let’s look at the setup. It should look pretty similar to anything you’ve done in an AngularJS service. I set up the request, and the success and error handlers just like I always do. Note I’m asking for the top 5000 items, using "&$top=5000" in my REST call.

If there are fewer than 5000 items, then we don’t have a problem; the base request would return them all. Line 32 is what would do that “normal” call. Instead, I call my recursive function below, passing in the request only, even though the function can take two more arguments: results and deferred.

The recursive function simply keeps calling itself whenever it sees that the __next attribute of the response is present, signifying there is more data to fetch. It concatenates the data into a single array as it goes. In my example, there would be 6 calls to the REST endpoint because there are 27782 / 5000 = 5.5564 “chunks” of items.

Image from https://s-media-cache-ak0.pinimg.com/564x/16/bb/03/16bb034cb3b6b0bdc66d81f47a95a59f.jpg

Image from https://www.pinterest.com/pin/323062973239383839/

NOW, before a bunch of people get all angry at me about this, the Stan Lee rule applies here. If you have tens of thousands of items and you decide to load them all, don’t blame me if it takes a while. All those request can take a lot of time. This also isn’t just a get of of jail free card. I’m posilutely certain that if we misuse this all over the place, the data police at Microsoft will shut us down.

In actual fact, the multiple calls will be separated by short periods of time to us, which are almost eternities to today’s high-powered servers. In some cases, you might even find that batches of fewer than 5000 items may be *faster* for you.

In any case, don’t just do this willy-nilly. Also understand that my approach here isn’t as great at handling errors as the base case. Feel free to improve on it and post your improvements in the comments!

How Can I Customize List Forms in SharePoint Online?

The other day, a client asked me a pretty classic set of questions about customizing SharePoint Online list forms. As with some other arenas of endeavor in Office 365, there is more than one way of doing things and it can be confusing.

I am looking at options to customize List forms on sites in SPO and I am trying not to have to deal with code.

My first choice has been InfoPath Designer but I know this is being deprecated and it seems like some of my sites are not allowing the use of InfoPath to customize my forms. [This was an internal issue, not an InfoPath issue.]

I know I could add web parts to the form pages and use JavaScript / jQuery or I could try and edit in [SharePoint] Designer but without the design view I am hesitant to mess around there too much.

Do you have any other tools you recommend for customizing List Forms?

Here’s an expanded version of the answer I gave my client, which incorporates some additional ideas and feedback I gleaned from Dan Holme (@DanHolme) and Chris McNulty (@cmcnulty2000) at Microsoft. (It’s nice being in Redmond for MVP Summit this week, where I can easily catch these guys in the hallway!)


The answer is the classic “it depends”. The main thing it depends on is what type of customization you actually want to do. There are a number of approaches and each has its pros and cons.

Adding JavaScript to the out-of-the-box forms

This is still possible, but I would discourage it in many cases, even though this has been my bread and butter approach for years. The out-of-the-box forms are changing, and “script injection” to do DOM manipulation is less safe. Remember you’re working on a service now, and it can change anytime.

SPServices

Unfortunately, this means that getting things like cascading dropdowns into forms is becoming harder than it used to be with SPServices. It’s not that you shouldn’t use that approach, but it does mean that the clock is ticking on how long it will continue to work. At lthis point. I’m recommending building entirely bespoke custom forms rather than adding JavaScript to the existing forms, though I still do plenty of the latter.

InfoPath

InfoPath

Yes, it’s deprecated or retired or whatever, but Microsoft will support InfoPath through 2026 at this point. InfoPath is often overkill – you can do a lot with list settings – but at the same time, it can still be useful. Keep in mind that using InfoPath means you’ll need to stick with the classic view for the list or library.

PowerApps + Flow

PowerAppsFlow

These new kids on the block are the successors to InfoPath, as announced at Microsoft Ignite. They can do a lot, but they may or may not meet your needs at this point. They did reach GA recently.

PowerApps would be how you would build the forms themselves and with Flow you can add automation – what we’ve called workflow in the past.

PowerApps embedding is “coming soon”. This will allow us to embed PowerApps into the list form context, as shown in the screenshot below. This will be a GREAT thing for a lot of list scenarios. At that point, I think the need for InfoPath will be greatly diminished.

PowerApps Embed

SharePoint Framework (SPFx)

SharePoint Framework (SPFx)

The SharePoint Framework is the next gen development model for things like custom Web Parts, which will run client side. We can build pretty much anything you want with this, but it’s still in preview. At some point in the not-too-distant future, I think we’ll be building a lot of custom forms using SPFx.

Fully custom forms

AngularJS

KnockoutJS Logo

To create fully custom forms today, you might use development frameworks like AngularJS or KnockoutJS (to name only a few). This can be the right answer, with the goal being to build in a way that can eventually merge into SPFx and the new “modern” pages. Knowing a bit about SPFx is a *very* good idea if you are going to go this route today. You’ll want to build your custom forms in such a way that you aren’t locking yourself out of wrapping them up into the SPFX for improved packaging and deployment down the road.

Third party tools

Because of how I roll, I haven’t used any of the third party tools out there, but there are many. The ones I hear come up most often are Nintex Forms, K2 Appit, and Stratus Forms. Obviously, there’s a economic investment with these choices, as well as a learning curve, so it’s always “your mileage may vary”.

Nintex Forms K2

Stratus Forms


The landscape here continues to change, so keep your eyes and ears open for more news from Microsoft and bloggers like me in the future!

Uploading Attachments to SharePoint Lists Using REST

In a previous post, I walked through Uploading Attachments to SharePoint Lists Using SPServices. Well, in the “modern” world, I want to use REST whenever I can, especially with SharePoint Online.

Add an attachmentI ran into a challenge figuring out how to make an attachment upload work in an AngularJS project. There are dozens of blog posts and articles out there about uploading files to SharePoint, and some of them even mention attachments. But I found that not a single one of the worked for me. It was driving me nuts. I could upload a file, but it was corrupt when it got there. Images didn’t look like images, Excel couldn’t open XLSX files, etc.

I reached out to Julie Turner (@jfj1997) and asked for help, but as is often the case, when you’re not as immersed in something it’s tough to get the context right. She gave me some excellent pointers, but I ended up traversing some of the same unfruitful ground with her.

Finally I decided to break things down into the simplest example possible, much like I did in my earlier post with SOAP. I created a test page called UploadTEST with a Content Editor Web Part pointing to UploadText.html, below:

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.3/jquery.js"></script>

<input id="my-attachments" type="file" fileread="run.AttachmentData" fileinfo="run.AttachmentInfo" />

<script type="text/javascript" src="/sites/MySiteCollection/_catalogs/masterpage/_MyProject/js/UploadTEST.js"></script>

Like I said, I wanted to keep it simple. Here’s what I’m doing:

  • In line 1, I load a recent version of jQuery from cdnjs. On the theory that Angular just adds another layer of complexity, I wanted to try just jQuery, which is useful for its $.ajax function, among many other things.
  • In line 3, I’ve got an input field with type=”file”. In “modern” browsers, this gives us a familiar file picker.
  • In line 5, I’m loading my script file called UploadTest.js, below:
$(document).ready(function() {

 var ID = 1;
 var listname = "UploadTEST";

 $("#my-attachments").change(function() {

  var file = $(this)[0].files[0];

  var getFileBuffer = function(file) {

   var deferred = $.Deferred();
   var reader = new FileReader();

   reader.onload = function(e) {
    deferred.resolve(e.target.result);
   }

   reader.onerror = function(e) {
    deferred.reject(e.target.error);
   }

   reader.readAsArrayBuffer(file);

   return deferred.promise();
  };

  getFileBuffer(file).then(function(buffer) {

   $.ajax({
    url: _spPageContextInfo.webAbsoluteUrl +
     "/_api/web/lists/getbytitle('" + listname + "')/items(" + ID + ")/AttachmentFiles/add(FileName='" + file.name + "')",
    method: 'POST',
    data: buffer,
    processData: false,
    headers: {
     "Accept": "application/json; odata=verbose",
     "content-type": "application/json; odata=verbose",
     "X-RequestDigest": document.getElementById("__REQUESTDIGEST").value,
     "content-length": buffer.byteLength
    }
   });

  });

 });
});

Here’s how this works – and yes, it does work!

  • I’ve got everything wrapped in a $(document).ready() to ensure the page is fully loaded before my script runs.
  • Lines 3-4 just set up some variables I could use for testing to keep things simple.
  • In line 6, I bind to the change event for the input field. Whenever the user chooses a file in the dialog, this event will fire.
  • In line 8, I’m getting the information about the file selected from the input element.
  • Lines 10-26 is the same getFileBuffer function I used with SOAP; it’s where we use a FileReader to get the contents of the selected file into a buffer.
  • The function in lines 28-42 runs when the file contents have been fully read. The getFileBuffer function returns a promise, and that promise is resolved when the function has gotten all of the file contents. With a large file, this could take a little while, and by using a promise we avoid getting ahead of ourselves. Here we make the REST call that uploads the attachment.
    • The URL ends up looking something like: “/my/site/path/_api/web/lists/getbytitle(‘UploadTEST’)/items(1)/AttachmentFiles/add(FileName=’boo.txt’)”
    • The method is a POST because we’re writing data to SharePoint
    • The data parameter contains the “payload”, which is the buffer returned from the getFileBuffer function.
    • The headers basically tell the server what we’re up to:
      • Accept doesn’t really come into play here unless we get a result back, but it says “send me back json and be verbose”
      • content-type is similar, but tells the server what it is getting from us
      • X-RequestDigest is the magic string from the page that tells the server we are who it thinks we are
      • content-length tells the server how many bytes it should expect

This worked for me on a basic Custom List (UploadTEST). So now I knew it was possible to upload an attachment using REST.

I took my logic and shoved it back into my AngularJS page, and it still worked! However, when I switched from $.ajax to AngularJS’s $http, I was back to corrupt files again. I’m still not sure why that is the case, but since I’m loading jQuery for some other things, anyway, I’ve decided to stick with $.ajax for now instead. If anyone reading this has an idea why $http would cause a problem, I’d love to hear about it.

Once again, I’m hoping my beating my head against this saves some folks some time. Ideally there would be actual documentation for the REST endpoints that would explain how to do things. Unfortunately, if there is for this, I can’t find it. I’d also love to know if /add allows any other parameters, particularly whether I can overwrite the attachment if one of the same name is already there. Maybe another day.