Wherein I Profess My Love for Document Sets, My Hatred of the 5000 Item Limit, and Some Tips

6 minute read

I love Document Sets. There, I’ve said it. They help us solve so many important business needs, it’s crazy. Unfortunately, because telemetry tells Microsoft that not very many people use Document Sets, they haven’t gotten any love for a long time. On the other hand, I hate the 5000 item limit in lists and libraries because they prevent us from getting good work done for our end users.

With Document Sets, we essentially get folders on steroids. We have a canvas available to us in the form of the Welcome page, which is a Web Part Page we can customize to our heart’s content. That means we an add images, other Web Parts, script (using a CEWP or SEWP), whatever we need in order to make the Document Set sing for our users. We can even push specific metadata from the Document Set level down into the documents within it.

While on the one hand it’s great that Microsoft hasn’t given them any love for a long time (they haven’t broken anything), it will be great when they eventually get the “modern” sheen. (See my comment about not breaking anything – that’s key if the “modern” version is to get any use.)

Today’s episode comes courtesy of one of my clients where we’re using Document Sets to the max in SharePoint Online. It’s a life sciences R&D operation, and we’re tracking most of their significant research data in Document Sets, among other mechanisms. It’s a really cool project, and I often wish I could show more of what we’re doing.

When we first built one of the main libraries using Document Sets as the basis (with 14 different Content Type variants inheriting from Document Set and each other), we talked about how many items would ever be in the library. At the time, 5000 seemed like a huge and distant number. Even so, I added some indices to hedge against it, but clearly not enough indices. It’s been over two years using this system, and we’ve done a bunch of development on top that we couldn’t have predicted originally.

Recently, a couple of things stopped working the way they should. Even though we never expected to, we recently went over the 5000 item limit in the Document Library – 5099 is the current count. Here are summaries of the issues and how we fixed them. The ever wonderful and talented Julie Turner (@jfj1997) came to my rescue on some of it, as you’ll see.

Adjusting the Indices While Over 5000 Items

This has historically been a HUGE problem. Once you cross the 5000 item limit and actually NEED indices on some of your columns, you haven’t been able to create them. When you tried to do so, you’d get an error telling you that you had more than 5000 items, so you couldn’t add an index. Helpful. Off to Sharegate to move some content out, fix the indices,then Sharegate the content back in.

In our Document Set instances, we were getting some errors where we were making REST calls to retrieve items related to the current one. (The Document Sets connect together in pathways of experiments, and we store the ParentID for each item’s parent Document Set.) The REST call would only retrieve one item from the library, since there was a filter:

Unfortunately, ParentID wasn’t indexed, so we were getting 500 errors back from our REST calls. Sigh. I assumed I’d need to shuffle some content out to add the index.

Just on the off chance, I went to add the index anyway, even though we were over the 5000 item. Never hurts to try, right?

Miracle of miracles, I was able to add the index without SharePoint batting an eye. I haven’t had a chance to test this elsewhere, but in this tenant I was able to do what previously was impossible.

If this is indeed the new normal, our lives have indeed gotten a lot easier.

We can add indices to lists and libraries with over 5000 items!

In any case, it solved my immediate problem. Maybe I shouldn’t talk about it so loudly near the tenant in case it changes its mind.

Fixed the broken default view

No one – and I mean no one – likes to see this message on a SharePoint page:

This view cannot be displayed because it exceeds the list view threshold (5000 items) enforced by the administrator.

 

To view items, try selecting another view or creating a new view. If you do not have sufficient permissions to create views for this list, ask your administrator to modify the view so that it conforms to the list view threshold.

We were seeing this horrible messaged in the List View Web Part at the bottom of the Welcome Pages. Since I have code running in the page, I wasn’t 100% sure that it wasn’t my fault.

Since the List View Web Part is only showing documents for this Document Set, it should only want to show a handful for each Document Set; nowhere near 5000. I was starting to think the Document Set was fundamentally broken by design.

Luckily, Julie was online and I asked her to take a look. She had the answer, and probably saved me hours trying to figure out why this was happening.

Her suggestion was to make sure the view doesn’t have “Show all items without folders” set to true. Sure enough, when I checked the view we were using for the Document Sets List View Web Parts, that was the setting. Julie pointed me to the article Manage large lists and libraries in SharePoint, specifically:

If you choose the Show all items without folders option in the Folders section when you create or modify a view in this list or library, you must then use a filter that is based on a simple index to ensure you don’t reach the List View Threshold.

Aha! For whatever reason over the years, we had set that very setting, and that was the problem. By turning that off, everything was right as rain again.

Document Sets Can Have Unique Views

This leads me to a little known thing about Document Sets: we can have a different view at the root of the library than inside the Document Sets. In fact, since you can inherit from Document Sets, you can have a different view per Document Set-based Content Type!

In fact, I was just reminded this yesterday from reading a post from Cameron Dwyer. Sure, it’s a setting on the Document Set settings page, but frankly, I’ve rarely noticed it.

The setting isn’t visible in the Document Set settings page when you create the Content Type, because you aren’t yet in the context of a list. Once you have enabled the Content Type on a list, you’ll see the additional settings.

Here’s the bottom of the Document Set settings in the Content Type definition:


and here’s the bottom of the page in a list context:

Note that the options are slightly different. In the list context we can choose a unique view for the Document Set-based Content Type. That means in my library, I could have 14 different views of the Document Set contents, one per Content Type, should I choose to do so.

Summary

Document sets are awesome. The 5000 item limit is not.

References

Advertisements

9 Comments

  1. I use document sets like crazy, boo to people that dont understand them :), i have at least 6 MAJOR MAJOR implementations of it in our environment, accounting for almost a million documents, and countless little ones. Some of my favorite features are the customizable “home pages” for document sets. Done all kinds of script editor embeds that add some pizzazz to the home page (like custom menus per document set), Also metadata inheritance, item level permissions on document sets (based on people fields), standardize folder creation inside of document sets, and more.

    Reply
  2. It would be useful if they let us change the threshold to a LOWER limit on specific lists, maybe 4500 items. Then, when users reported issues, we could up the limit to 5000 to get them up and running, and then take corrective actions(indexes, views), and then set the threshold down again to test our corrective actions.

    Reply
  3. I’m having the same issue lately with the item limit.
    I have a list where requests are submitted and users can attach files. A SP2013 workflow then creates a document set based on the ID of the list item and moves the attachments to it – the reason for this is so we can maintain version history and the user doesn’t have to mess about with multiple forms/modal dialogs on the new form. The edit form includes an extra button for a modal dialog to view the attachments however, with a nice slimmed down version of the document set welcome page.
    A PowerShell script runs every night telling us which list or library is nearing 4,500 items so we are ready to archive items based on specific criteria. We just have to be mindful that the document set library grows faster, therefore we have to archive both the list and library sooner than expected.

    Interesting post though, indices work well but I’m more inclined to archive for certain projects.

    Reply
  4. I, too, love the document set and enthusiastically evangelize its power whenever I have the opportunity. But. There is one bizarre and annoying—if not super consequential—issue with the document set. Its “new item form” is a server page that’s unchangeable and editable. Also, for some reason, the Title field can -never- be interacted with at new item creation. You can give it a Name (and Microsoft helpfully copies that Name into the Title field on creation), but you can’t give users the Title field on creation.

    Small? Sure. But -really- weird. And its hampered some of my implementations where file name conventions are extremely important. I need my users to give the set a standard file name, but would like to give them a more free-form title field so their projects can have human friendly names.

    Anyway. It’s a small thing, but has always had me scratching my head.

    Reply
    • @Chris:

      In this particular instance, we’ve got some script in the page that creates the Name (thus Title) by gluing together various values from the query string and some other stuff based on the Content Type we are creating. It works well to normalize and enforce the naming conventions.

      M.

      Reply
  5. Thanks Marc. As you know, we announced in May support for an upcoming capability called intelligent indexing. This will use machine learning to discover the need on the fly to add an index intake automatic steps to greatly reduced the frequency of this kind of throttle. in the mean time in may be useful to remind people that indexing always be at it interactively by following the steps in this link https://support.office.com/en-us/article/Add-an-index-to-a-SharePoint-column-f3f00554-b7dc-44d1-a2ed-d477eac463b0

    Reply
    • @Chris:

      Until I see that so-called “intelligent indexing” working in the wild, I don’t have an opinion on it. While it sounds pretty spiffy, there’s no way to know how it will help in these real-life situations over time.
      M.

      Reply

Have a thought or opinion?