Yes, Virginia, You Can Get More than 5000 SharePoint Items with REST

If you haven’t been paying any attention, you might not know that I loathe the 5000 item limit in SharePoint. I may have mentioned it here and here and here and a bunch of other places, but I’m not sure. But given it’s there, we often need to work around it.

No 5000 item limit

I’ve written this little function before, but alas today I needed it again but couldn’t find any previous incarnations. That’s what this blog is for, though: to remind me of what I’ve already done!

In this case, I’m working with AngularJS 1.x. We have several lists that are nearing the 5000 item level, and I don’t want my code to start failing when we get there. We can get as many items as we want if we make repeated calls to the same REST endpoint, watching for the __next link in the results. That __next link tells us that we haven’t gotten all of the items yet, and provides us with the link to get the next batch, based on how we’ve set top in the request.

Here’s an example. Suppose I want to get all the items from a list which contains all of the ZIP codes in the USA. I just checked, and that’s 27782 items. That’s definitely enough to make SharePoint angry at us, what with that 5000 item limit and all. Let’s not get into an argument about whether I need them all or not. Let’s just agree that I do. It’s an example, after all.

Well, if we set up our requests right, and use my handy-dandy recursive function below, we can get all of the items. First, let’s look at the setup. It should look pretty similar to anything you’ve done in an AngularJS service. I set up the request, and the success and error handlers just like I always do. Note I’m asking for the top 5000 items, using "&$top=5000" in my REST call.

If there are fewer than 5000 items, then we don’t have a problem; the base request would return them all. Line 32 is what would do that “normal” call. Instead, I call my recursive function below, passing in the request only, even though the function can take two more arguments: results and deferred.

The recursive function simply keeps calling itself whenever it sees that the __next attribute of the response is present, signifying there is more data to fetch. It concatenates the data into a single array as it goes. In my example, there would be 6 calls to the REST endpoint because there are 27782 / 5000 = 5.5564 “chunks” of items.

Image from https://s-media-cache-ak0.pinimg.com/564x/16/bb/03/16bb034cb3b6b0bdc66d81f47a95a59f.jpg

Image from https://www.pinterest.com/pin/323062973239383839/

NOW, before a bunch of people get all angry at me about this, the Stan Lee rule applies here. If you have tens of thousands of items and you decide to load them all, don’t blame me if it takes a while. All those request can take a lot of time. This also isn’t just a get of of jail free card. I’m posilutely certain that if we misuse this all over the place, the data police at Microsoft will shut us down.

In actual fact, the multiple calls will be separated by short periods of time to us, which are almost eternities to today’s high-powered servers. In some cases, you might even find that batches of fewer than 5000 items may be *faster* for you.

In any case, don’t just do this willy-nilly. Also understand that my approach here isn’t as great at handling errors as the base case. Feel free to improve on it and post your improvements in the comments!

Unity Connect Haarlem 2016 Follow Up

Sorry it’s taken me a little while to get this post up, but I had a great time in Haarlem, The Netherlands the week before last at the Unity Connect conference. The conference is “under new management” – as it were – this year, and though I haven’t attended this particular event in the past, it’s top notch. I look forward to where George Coll (@GeorgeColl) and the other folks from Blue Whale Web (who are running the IT Unity brand now) take things.

The location was pretty amazing, as it was in the Philharmonie Haarlem. It’s a modern concert and performance venue built into a very old building (at least to us Americans!). Check out this photo of Dux Sy (@meetdux) after he presented in the main concert hall.

I delivered two sessions at the conference, and links to the two slide decks are available below.

Several people have asked about the code examples I showed. You can find the small survey in my KO SharePoint repo on Github. The Sliding Quick Launch CSS (courtesy my colleague Julie Turner [@jfj1997]) is in our Sympraxis Conference Demos repo on Github.

I also had the great pleasure of speaking at the Dutch Information Worker User Group (DIWUG) the evening before the conference started, along with Adis Jugo (@adisjugo). The slides from that talk – Creating a Great User Experience in SharePoint – are below.

20161116_214053000_ios

 

Color Codes in SharePoint SPColor Files MUST Be All Caps

With the new world of branding a tenant on Office 365, styling the suite bar and using a theme (aka Composed Look) can take you pretty far toward improving the user experience.

A Composed Look is made up of (potentially):

  • A master page – generally either Seattle or Oslo
  • Theme URL – the spcolor file to use for common classes in a SharePoint page
  • Image URL – an image to use as the background for pages
  • Font Scheme URL – spfont file to use for common classes in a SharePoint page
  • Display order – simply where in the list of Composed Looks yours is displayed

Composed Look Settings

I won’t go into all the details about how this works, but each setting above is somewhat optional: you can use an existing spcolor file, for example, but create your own spfont file. If you want to understand all the mechanics, check out Benjamin Niaulin’s article Step by Step: Create a SharePoint 2013 Composed Look

 

Change the look

Here’s a quick tip about spcolor files. I was tearing my hair out over the last few weeks trying to figure out why sometimes my iterations of an spcolor file were working and other times they weren’t. By not working, I mean that the Composed Look would simply disappear from the Change the look page. I’d revert to my last working version and slowly inch forward again.

It turned out to be something silly and obvious – once you know it!

All of the color codes in the spcolor file MUST be all capitals. So, for instance, this:

would not work, but this:

would. That was *not* easy to spot, but spot it I did.

Don’t let this one bite you; I hope that your Binglage has led you here for a solution and this helps…

SharePoint Saturday Boston 2016 Follow Up

Thanks to everyone who attended my session at SharePoint Saturday Boston on ‘Best Practices for Small-Scale Client-Side Development in SharePoint’. I’ve posted my slides on SlideShare for everyone’s enjoyment.

Be sure to follow my series on Sympraxis Client Side Development Pipeline for more details on some of the approaches and methods I demonstrated in the session.

Thanks to Heather Newman for the photo!

Thanks to Heather Newman for the photo!

Save Your SharePoint Online Tenant: The SharePoint Sandboxed Solutions Inspector

If you’ve been following the “code-based sandbox solutions on Office 365” saga, you know that there is little time left to fix your existing sandbox solutions in Office 365. See: Microsoft Is Removing Code-Based Sandbox Solutions in SharePoint Online – Be Prepared!

Last week, Vesa Juvonen (@vesajuvonen) released a script (New Script Available from Microsoft PnP: Generate list of sandbox solutions from SharePoint Online tenant) that can help you find your sandbox solutions. Surprisingly, what was missing from Vesa’s script was identification of the solutions that contain code. You’d get a list of all your sandbox solutions, but not specifically the ones that were going to cause you problems.

The Rencore Team

Some of the great looking folks at Rencore. Where’s Waldek?

My awesome friends at Rencore – the SFCAF folks – were kind enough to make a free tool available this week to help with even more with your diagnosis and even some of the cures. In Erwin van Hunen’s (@erwinvanhunenpost Introducing the Rencore SharePoint Sandboxed Solutions Inspector, you can learn more about the free tool and how it can help.

On August 31st, 2016 Microsoft is going to shut down support for Sandboxed Solutions with code.

Sandboxed Solutions containing code will be deactivated and this might impact your Office 365 tenant big time!

If you know you need it, just head right on over to the download page.

The SharePoint Sandboxed Solutions Inspector

They have already released several updates to the tool, and are keeping it current based on feedback from the folks using it. Now that’s service – and for a free tool!

But I think the best thing is that the Rencore tool can fix some of the most common issues – most notably the “empty DLL” issue that makes Office 365 think you have code in your sandbox solution when you don’t.

We heard “30 days” when all this started, and now people seem to be taking that as August 31. Don’t leave your users in the lurch – get going on handling this situation.

Oh, and if you’re a vendor or consultant who has written a sandbox solution with code over the last few years: reach out to your client and own it. Get them back on the right road and you’ll be the better for it.

Clean report!

Clean report!