Yes, Virginia, You Can Get More than 5000 SharePoint Items with REST

If you haven’t been paying any attention, you might not know that I loathe the 5000 item limit in SharePoint. I may have mentioned it here and here and here and a bunch of other places, but I’m not sure. But given it’s there, we often need to work around it.

No 5000 item limit

I’ve written this little function before, but alas today I needed it again but couldn’t find any previous incarnations. That’s what this blog is for, though: to remind me of what I’ve already done!

In this case, I’m working with AngularJS 1.x. We have several lists that are nearing the 5000 item level, and I don’t want my code to start failing when we get there. We can get as many items as we want if we make repeated calls to the same REST endpoint, watching for the __next link in the results. That __next link tells us that we haven’t gotten all of the items yet, and provides us with the link to get the next batch, based on how we’ve set top in the request.

Here’s an example. Suppose I want to get all the items from a list which contains all of the ZIP codes in the USA. I just checked, and that’s 27782 items. That’s definitely enough to make SharePoint angry at us, what with that 5000 item limit and all. Let’s not get into an argument about whether I need them all or not. Let’s just agree that I do. It’s an example, after all.

Well, if we set up our requests right, and use my handy-dandy recursive function below, we can get all of the items. First, let’s look at the setup. It should look pretty similar to anything you’ve done in an AngularJS service. I set up the request, and the success and error handlers just like I always do. Note I’m asking for the top 5000 items, using "&$top=5000" in my REST call.

If there are fewer than 5000 items, then we don’t have a problem; the base request would return them all. Line 32 is what would do that “normal” call. Instead, I call my recursive function below, passing in the request only, even though the function can take two more arguments: results and deferred.

The recursive function simply keeps calling itself whenever it sees that the __next attribute of the response is present, signifying there is more data to fetch. It concatenates the data into a single array as it goes. In my example, there would be 6 calls to the REST endpoint because there are 27782 / 5000 = 5.5564 “chunks” of items.

Image from https://s-media-cache-ak0.pinimg.com/564x/16/bb/03/16bb034cb3b6b0bdc66d81f47a95a59f.jpg

Image from https://www.pinterest.com/pin/323062973239383839/

NOW, before a bunch of people get all angry at me about this, the Stan Lee rule applies here. If you have tens of thousands of items and you decide to load them all, don’t blame me if it takes a while. All those request can take a lot of time. This also isn’t just a get of of jail free card. I’m posilutely certain that if we misuse this all over the place, the data police at Microsoft will shut us down.

In actual fact, the multiple calls will be separated by short periods of time to us, which are almost eternities to today’s high-powered servers. In some cases, you might even find that batches of fewer than 5000 items may be *faster* for you.

In any case, don’t just do this willy-nilly. Also understand that my approach here isn’t as great at handling errors as the base case. Feel free to improve on it and post your improvements in the comments!

Unity Connect Haarlem 2016 Follow Up

Sorry it’s taken me a little while to get this post up, but I had a great time in Haarlem, The Netherlands the week before last at the Unity Connect conference. The conference is “under new management” – as it were – this year, and though I haven’t attended this particular event in the past, it’s top notch. I look forward to where George Coll (@GeorgeColl) and the other folks from Blue Whale Web (who are running the IT Unity brand now) take things.

The location was pretty amazing, as it was in the Philharmonie Haarlem. It’s a modern concert and performance venue built into a very old building (at least to us Americans!). Check out this photo of Dux Sy (@meetdux) after he presented in the main concert hall.

I delivered two sessions at the conference, and links to the two slide decks are available below.

Several people have asked about the code examples I showed. You can find the small survey in my KO SharePoint repo on Github. The Sliding Quick Launch CSS (courtesy my colleague Julie Turner [@jfj1997]) is in our Sympraxis Conference Demos repo on Github.

I also had the great pleasure of speaking at the Dutch Information Worker User Group (DIWUG) the evening before the conference started, along with Adis Jugo (@adisjugo). The slides from that talk – Creating a Great User Experience in SharePoint – are below.

20161116_214053000_ios

 

Movement on the SharePoint List 5000 Item Limit!

In October, Eric Alexander (@ejaya2) posted an idea to the SharePoint UserVoice called Prioritize large list management in SharePoint Online. Yesterday I received an alert about it because I had voted for it.

The 5000 item limit has been an albatross around our necks since SharePoint 2010. SharePoint 2007 was the wild west days: we could pile as many items into a list as we wanted and retrieve them without a problem, though perhaps at the expense of performance.

I’ve railed about this limit many times in the past, like here.

No 5000 item limit

If you search the SharePoint UserVoice for “5000“, you’ll find no fewer than 14 suggestions circling this topic. There are probably even more, but without the number 5000 in them.

Luckily, our good friends in Redmond know this is an issue for us. As of yesterday, Eric’s suggestion moved to “Working on it”, at least for SharePoint Online.

Prioritize large list management in SharePoint Online - Working on it

 

 

 

I’m looking forward to what happens here. As Eric notes in his suggestion:

If nothing can be done, then TechNet NEEDS to indicate that the actual limit of SharePoint Online lists and libraries is 5,000 items, not the current architectural limit of 30 million.

The more we want to use SharePoint as a service (SPaaS?), the more important it becomes to get past this limitation.

I’m certain that the work Dan Kogan (@kogandan) and his team are doing on the SharePoint Framework (SPFx) has made it obvious that this limit is a serious issue. (Sometimes you have to take the albatross off your neck and slap someone with it.)

Image from http://elkgrovegovernment.com/with-the-nrc-announcement-has-the-albatross-been-taken-off-elk-groves-neck/

Image from http://elkgrovegovernment.com/with-the-nrc-announcement-has-the-albatross-been-taken-off-elk-groves-neck/

How Can I Customize List Forms in SharePoint Online?

The other day, a client asked me a pretty classic set of questions about customizing SharePoint Online list forms. As with some other arenas of endeavor in Office 365, there is more than one way of doing things and it can be confusing.

I am looking at options to customize List forms on sites in SPO and I am trying not to have to deal with code.

My first choice has been InfoPath Designer but I know this is being deprecated and it seems like some of my sites are not allowing the use of InfoPath to customize my forms. [This was an internal issue, not an InfoPath issue.]

I know I could add web parts to the form pages and use JavaScript / jQuery or I could try and edit in [SharePoint] Designer but without the design view I am hesitant to mess around there too much.

Do you have any other tools you recommend for customizing List Forms?

Here’s an expanded version of the answer I gave my client, which incorporates some additional ideas and feedback I gleaned from Dan Holme (@DanHolme) and Chris McNulty (@cmcnulty2000) at Microsoft. (It’s nice being in Redmond for MVP Summit this week, where I can easily catch these guys in the hallway!)


The answer is the classic “it depends”. The main thing it depends on is what type of customization you actually want to do. There are a number of approaches and each has its pros and cons.

Adding JavaScript to the out-of-the-box forms

This is still possible, but I would discourage it in many cases, even though this has been my bread and butter approach for years. The out-of-the-box forms are changing, and “script injection” to do DOM manipulation is less safe. Remember you’re working on a service now, and it can change anytime.

SPServices

Unfortunately, this means that getting things like cascading dropdowns into forms is becoming harder than it used to be with SPServices. It’s not that you shouldn’t use that approach, but it does mean that the clock is ticking on how long it will continue to work. At lthis point. I’m recommending building entirely bespoke custom forms rather than adding JavaScript to the existing forms, though I still do plenty of the latter.

InfoPath

InfoPath

Yes, it’s deprecated or retired or whatever, but Microsoft will support InfoPath through 2026 at this point. InfoPath is often overkill – you can do a lot with list settings – but at the same time, it can still be useful. Keep in mind that using InfoPath means you’ll need to stick with the classic view for the list or library.

PowerApps + Flow

PowerAppsFlow

These new kids on the block are the successors to InfoPath, as announced at Microsoft Ignite. They can do a lot, but they may or may not meet your needs at this point. They did reach GA recently.

PowerApps would be how you would build the forms themselves and with Flow you can add automation – what we’ve called workflow in the past.

PowerApps embedding is “coming soon”. This will allow us to embed PowerApps into the list form context, as shown in the screenshot below. This will be a GREAT thing for a lot of list scenarios. At that point, I think the need for InfoPath will be greatly diminished.

PowerApps Embed

SharePoint Framework (SPFx)

SharePoint Framework (SPFx)

The SharePoint Framework is the next gen development model for things like custom Web Parts, which will run client side. We can build pretty much anything you want with this, but it’s still in preview. At some point in the not-too-distant future, I think we’ll be building a lot of custom forms using SPFx.

Fully custom forms

AngularJS

KnockoutJS Logo

To create fully custom forms today, you might use development frameworks like AngularJS or KnockoutJS (to name only a few). This can be the right answer, with the goal being to build in a way that can eventually merge into SPFx and the new “modern” pages. Knowing a bit about SPFx is a *very* good idea if you are going to go this route today. You’ll want to build your custom forms in such a way that you aren’t locking yourself out of wrapping them up into the SPFX for improved packaging and deployment down the road.

Third party tools

Because of how I roll, I haven’t used any of the third party tools out there, but there are many. The ones I hear come up most often are Nintex Forms, K2 Appit, and Stratus Forms. Obviously, there’s a economic investment with these choices, as well as a learning curve, so it’s always “your mileage may vary”.

Nintex Forms K2

Stratus Forms


The landscape here continues to change, so keep your eyes and ears open for more news from Microsoft and bloggers like me in the future!

Dear Microsoft: I’m Confused. Can You Help Me Collaborate Well?

Yup, I’ll admit it: I’m confused.

The launch of Microsoft Teams last week is what’s done it. First of all, from everything I’ve seen of the new tool, it’s really cool. I had some trouble getting my head around how to get it up and running in our Sympraxis Office 365 tenant, but now Julie (@jfj1997) and I are taking it for a spin and we like it.

Microsoft Teams

What’s confusing to me is where Microsoft Teams fits into the spectrum of similar offerings, which all look pretty much the same to me on many levels. We have Yammer, Group Conversations, Microsoft Teams, email, Team Site Discussions, and the old SharePoint newsfeed. Now, I’ll allow that the last two are pretty much obsolete, but they are still in the UIs and people see them. This is a lot of choices.

Every time Microsoft comes out with a new “social” tool set, the rhetoric around its positioning tends to be “Hey, we think you like choices, and here’s another choice for you!” It’s absolutely true that everyone works differently, but there are patterns in those different ways of working. While we all may be special flowers (see my comments about millennials below), but we aren’t snowflakes; there is a lot of similarity in the types of things we are trying to solve in our collaborations.

I wish I could find the old slides we used to use when I worked in Renaissance Solutions back in the late 1990s, but I think they have been lost to the electrons of time. When we talked about the different communications mechanisms that we available for people to use for good knowledge management, we listed out 6-8 methods. We talked about them being part of a portfolio of options, each of which had specific set of good use cases. (Some of the terminology has changed, but the ideas haven’t changed all that much.)

For instance, you shouldn’t fire someone over email because it’s an emotional event; that deserves and in-person meeting (generally one-on-one). You also shouldn’t call an all hands company meeting if you’re just reviewing a task checklist if an electronic sharing mechanism works just as well. And so on. These common sense rules are broken all the time, but that doesn’t mean they don’t make simple senss. It’s often simply that no one has provided enough guidance on how to think about things.

One of the big issues I see in our overly connected world these days is misuse of the various options we have at hand. One example is Microsoft support calling me on the phone when I’ve created a support ticket via the Web; generally we should stick to one modality and not shift mid-stream. Like it or not, the initiator gets to choose, and we should only switch by mutual agreement. (“Would you mind if I emailed you some more details?” in a conversation, for instance.) Sometimes my wife and I have this problem, too. We might start a conversation in a text, it jumps over to email, and then becomes a note on the counter. That jumping between methods waters down the interaction and makes it far more confusing. One of the methods was probably the right one, but we’ve subverted that.

What has me confused about Microsoft’s overlapping offerings in the communication spectrum is that they don’t come with guidance about which is good when or for what type of organizations. Instead we see a lot of talk about choice being good. Choice seems good, but when you get right down to it, open choice leads to a certain amount of chaos. Many people I talk to would like some sort of help understanding what Microsoft is thinking, at least, but Microsoft seems unwilling to do this.

When Microsoft is working on the development of a new tool like Microsoft Teams or adding enhancements to an older tool set like Yammer, they must have use cases in mind. (I truly hope this is the case, and I believe it has to be!) But those use cases don’t really make it out to us in the form of helpful portfolio management strategies.

One standard Microsoft answer to things like this is that it’s a “partner opportunity”. That would be an excellent answer if every partner understood how to do this type of positioning, but many don’t. Many partners are technical partners and focus far more on implementing the hardware and software (nay, services). This isn’t a bad thing, it’s just the way it works.

As a consultant who has done a lot of knowledge management work and collaboration and strategy work after that, I suppose I could look at this as an excellent opportunity for me to go out and make a lot of money advising clients about how to manage all of these options. That might be good for me (and hopefully for my clients), but it doesn’t help the ecosystem all that much.

Another chestnut that we hear a lot is “Well, millennial want something different, so…” This is a weird one to me. Sure, younger people may be different, but remember we’re all special flowers, right? People entering the workplace don’t know how to do things in the workplace yet. Simply catering to them – it certainly didn’t happen back when I started working – will just water down effective work styles developed over years. Don’t get me wrong: a lot of work methods are just plain dumb and should be questions, but mindfully and not just because the kids don’t like them.

When I look at the graph below from Avanade, I see some really interesting information. But I also see that many people I know in my generation (I’m a Baby Boomer!)  must be doing things wrong; we don’t act like the stats. I also know millennials who don’t act like millennials. And my son isn’t just like his Gen Z group, either. When I interact with any of these groups, I need to adapt my methods based on what will work at the time, not just what *I* like.

Statistics like this are absolutely useful, but you need to understand your own organization to know what will work. If it’s an organization with lots of closed offices and hierarchy, it’s different from one with an open office plan and a lot of cross-functional work. Age usually has little impact on those constructs. So when you think about what will work for your organization, wouldn’t it be great to have some sort of framework from which to make decisions? A sort of “portfolio management” approach?

I’m reminded of the great work Sadie Van Buren (@sadalit) did a few years ago on her SharePoint Maturity Model. It gave people some very clear ways to think about where they were on the spectrum of success with the platform. The model is a little dusty at this point, but it still makes a lot of sense and can help drive decision-making. (If you check it out and think it’s still useful, please let Sadie or me know. I think it needs a renaissance for Office 365!)

So how can Microsoft un-confuse me, and by extension many of you? Well, I think they need to put their internal politics aside and draw some lines in the sand. For example, regardless how you feel about it, Yammer is good for interactions that require external users because many of the other options don’t provide that capability. A simple statement like that can make some decisions pretty simple: You need external users; you need Yammer. QED. But you rarely see Microsoft making such a clear statement.

Here’s hoping that the smart people in Redmond get on this soon. As the options keep piling up on us, it’s only getting harder to choose. If I were a betting man, I’d pick some winners and loser in this game, too. Knowing what makes sense for specific use cases would reduce risk for organizations who need to make choices and stick with them. Change in most organizations is hard – and expensive. Making good decisions based on good guidance up front makes those changes far more palatable.


nb: I”m publishing this from the plane on the way to Microsoft MVP Summit. It’s best to get this off my chest before I get all jazzed up this week! Maybe I’ll be able to convince some people about this stuff while I’m there.