Getting Around SharePoint’s Threshold Limits for Large Reference Lists

In SharePoint 2007, we could build lists that contained as many items as we wanted and access them in any way we wanted. We may have done stupid things that were bad for server performance, but if we knew what we were doing there were few limits. We had the mythical “2000 item” rule of thumb (which had little basis in reality in many, many cases) but otherwise it was up to us. (See: Plan for software boundaries (Office SharePoint Server))

In SharePoint 2010, Microsoft introduced threshold limits to protect us from ourselves. This seemed counter intuitive to me, since with the upgrade to SharePoint 2010, one also had to go to 64 bit architecture and put a heck of a lot more iron behind the SharePoint farm. While we could potentially store 30,000,000 items in a list (30 million!), we had a list view threshold of 5000 items. (See: SharePoint Server 2010 capacity management: Software boundaries and limits)

SharePoint 2013 maintains similar limits to 2010, with 5000 items the limit for a list view. (See: Software boundaries and limits for SharePoint 2013)

List View ThrottlingSomehow, as technology has moved forward – significantly – we’ve got more limits.

5000 items is way too many to ever show in a list view, but it may not be if you want to do some client side processing. It doesn’t matter if you’re trying to build a truly bloated, Bad Idea list view or request data using REST, you’re stuck with that 5000 item limit.

But what if you know what you are doing and the 5000 item limit doesn’t work for you? I’ve written in the past about why we shouldn’t change some of the threshold limits (e.g.,  The SharePoint 2010 “List View Lookup Threshold” and Why We Don’t Change It).

If we’re working on Office365, we simply can’t change the limits – Microsoft gets to decide what they are. Sometimes those limits change without notice, too, so it’s truly out of our hands.

Well, what does all this add up to? Let’s take a specific example.

I needed to provide auto complete functionality in an InfoPath form. It was the classic scenario: we had a list of [in this case] 19,000+ cities and towns and we needed a good UX for entering selections into the form. We run into this sort of thing with data like cities and towns all the time. There are too many choices for a dropdown to make any sense, and SharePoint doesn’t offer up any better alternatives. jQueryUI’s autocomplete function is a perfect solution for this sort of thing.

The data was in a spreadsheet, and it’s easy to upload data from a spreadsheet to a SharePoint list. Bingo, bango, done, right? Well, no. It’s impossible to upload data from a spreadsheet *and* index the column. We need to index the column so that we can make type of requests requests that we need for autocomplete – basically, give me all of the items that begin with or contain this string – or else we run into the 5000 item threshold error on our requests.

No problem! We’ll just upload the data and *then* index the column. Not so fast, mister. When you try to index the column you run into – you guessed it – the 5000 item limit. No indexee, no laundry.

So seemingly we’re stuck. But wait – we can just create the list in the UI, add the index, and paste all 19,000 items into the Brave New World of SharePoint 2013’s “data sheet view” aka “Quick Edit”. Sadly, when I tried to paste in more than a few hundred items, the browser hung on me. This was on Office365, and I tried several different browsers. SharePoint Online seems to prefer nibbles to big bites of data. Unless I spent days just doing the pasting, this was never going to happen.

In this case, I decided to simply store the data in a text file containing JSON rather than trying to force it into a list. Thanks to @bpmccullough for the suggestion on Twitter.

The data I originally had received for the sities and towns was in a CSV file. I found a nice little site that would convert the CSV format to JSON (bookmark this one; you’ll need it in this JSON-oriented world).

Once I had the JSON file uploaded to a Document Library, the code below is what I used to set up the autocomplete. We wanted the autocomplete to allow multiple choices, so there’s a little monkeying around to enable that. It wasn’t tremendously peppy (19,000 cities, after all), but it worked. Remember, kids: this is an example. Don’t expect to drop it into your page and have it work.

var citiesAndStates = [];

// Get the City and State data from the file with JSON data
$.getJSON("/Shared%20Documents/CityState.txt", function(data) {
  var results = $(data);
  // Push the data into an array which is appropriate for jQueryUI autocomplete
  citiesAndStates = ($.map(results, function(item) {
    return {
      label: item.State + " - " + item.City, // Show "State - City" [e.g., MA - Boston] for selection
      value: item.City + ", " + item.State // Show "City, State" [e.g., Boston, MA] upon selection
    }
  }));
});

// When the field is available and gets focus, set up the autocomplete behavior
//ctl00_ctl42_g_6069846d_1b7f_4890_b767_2bdc15d2b133_FormControl0_V1_I1_S15_I4_T5
$(document).on("focus", "input[id$='FormControl0_V1_I1_S15_I4_T5']", function() {
  if ($(this).data("autocompleteSet") === undefined) {
    $(this).data("autocompleteSet", true);

    $(this)
    // Don't navigate away from the field on tab when selecting an item
    .bind("keydown", function(event) {
      if (event.keyCode === $.ui.keyCode.TAB && $(this).data("ui-autocomplete").menu.active) {
        event.preventDefault();
      }
    }).autocomplete({
      source: function(request, response) {
        // Delegate back to autocomplete, but extract the last term
        response($.ui.autocomplete.filter(citiesAndStates, extractLast(request.term)));
      },
      minLength: 3,
      focus: function() {
        // Prevent value inserted on focus
        return false;
      },
      select: function(event, ui) {
        var terms = split(this.value);
        // Remove the current input
        terms.pop();
        // Add the selected item
        terms.push(ui.item.value);
        // Add placeholder to get the semicolon-and-space at the end
        terms.push("");
        this.value = terms.join("; ");
        return false;
      }
    });
  }
});

function split(val) {
  return val.split(/;\s*/);
}

function extractLast(term) {
  return split(term).pop();
}

One note here: This approach basically violates my “put all data into a list so that end users can maintain it” rule. However, the approach I came up with was the best one for the situation. It’s still possible for a reasonably savvy (meaning “non-developer”) user to manage the data by downloading the file and using a JSON-aware editor.

The main thing was that we were able to set up something that worked, regardless what the Office365 limits may change to.

SPServices Stories #17: Multiple Form Fields Autocomplete for SharePoint 2010/2013 using JavaScript

This entry is part 16 of 21 in the series SPServices Stories

Introduction

Anyone who follows this series knows that I’m always on the lookout for interesting and useful implementations of SPServices. A few weeks ago, a tweet in my feeds caught my eye:

Anton Khritonenkov Anton Khritonenkov, who is a Technical Lead at Plumsail in Russia, has come up with a nice way to enable autocomplete behavior for multiple list columns based on related lists.

While I’ve got the SPAutocomplete function in SPServices, it’s pretty rudimentary and I usually suggest using the jQueryUI autocomplete function, as Anton does here. The other function in SPServices that is somewhat related to what Anton is doing in SPDisplayRelatedInfo. By using the completefunc there, you could enable some of the behavior that Anton has built, but not as cleanly.

Anton explains things well in his post below, which he was kind enough to allow me to repost here from his original on the CodeProject site.

Multiple Form Fields Autocomplete for SharePoint 2010/2013 using JavaScript

Introduction

Filling forms could be painful sometimes, especially for forms with many fields. In this article I’ll describe approach which can simplify forms filling significantly. In my case I needed to implement order form interface. In fact it contains a lot of fields I can pre-fill based on chosen customer, such as address, ZIP code and account number. All of this data is already stored in separate list called “Customers directory”.

Final requirements could looks like this:

When user starts typing in the customer field, I need to suggest list of customer names from Customers directory. When user chooses customer from suggestions list, I need to read data multiple field values from Customers directory and fill corresponding fields in the order form. User can correct filled values later if needed.

To implement such functionality I used to use JavaScript and this case is not an exception. There are many reasons for this:

  • It well fits Office 365 restrictions.
  • It easily migrates from older SharePoint versions.
  • It is easily to debug without slow SharePoint solution deployment.
  • REST SharePoint 2013 API or SPServices are almost such powerful as server code.
  • Finally I just like it.

In this article I’ll use SPServices jQuery plugin for communication with SharePoint services. It works for SharePoint 2013 as well as for SharePoint 2010. I’ll also use jQuery UI Autocomplete plugin to implement suggestions functionality.

plumFormAutocomplete plugin works well for single field as much as for multiple fields.  Plugin supports text fields only.

Plugin in action looks like this:

How to use plumFormAutocomplete jQuery plugin

You can use this plugin without looking into the code. Firstly, I’ll describe how to use it, then if you still will be interested, you can look inside plugin implementation.

Prerequisites

If you don’t have jQuery, jQuery UI or SPServices, you can download fresh version from official sites. For tutorial purposes I assume that names of downloaded files and folders are following:

  • jquery.min.js 
  • jquery.SPervices.min.js 
  • jquery-ui-1.10.3.custom.min.js
  • jquery-ui-1.10.3.custom.min.css
  • css folder for jQuery UI

I also assume that you have jquery.plumFormAutocomplete.js downloaded from source code of this article (Source code download link).

Upload jquery.min.js, jquery.SPServices.min.js, jquery-ui-1.10.3.custom.min.js and jquery.plumFormAutocomplete.js files to Style Library within your site collection. You also need to upload jQuery UI CSS styles located in the same folder with jquery-ui-1.10.3.custom.min.js. Then open New or Edit form, where autocomplete plugin will be used and add js and CSS links to placeholder PlaceHolderAdditionalPageHead.  You can use following snippet:


Configure and call plugin

Now you are ready to configure and call plugin. For my case plugin call looks like this:

//get control for autocomplete field
var fieldControl = $.getFieldControl('Title');

//call autocomplete plugin for field control
fieldControl.plumFormAutocomplete({
  sourceList: 'Customers directory',
  sourceMatchField: 'Title',
  labelFields: ['Title', 'ZIPCode'],
  labelSeparator: ', ',
  fillConcatenatedLabel: false,
  fieldsMapping: [{sourceField: 'Address', targetField: 'CustAddress'},
    {sourceField: 'AccountNumber', targetField: 'CustAccountNumber'},
    {sourceField: 'ZIPCode', targetField: 'CustZIPCode'}]
});

You can wrap plugin call inside jQuery $(document).ready() function to ensure that code will be executed after page is loaded.

Let us look at this code sample in more detail. Code is divided into two parts:

  1. Get control for autocomplete field
  2. Call autocomplete plugin for field control

For the first step you need to specify internal name of autocomplete field for getFieldControl function. It is ‘Title’ in my case.

In the second step you need to call plugin for received autocomplete field and configure plugin options. Plugin options are structured as a single object as any jQuery plugin options.

Plugin options

  • sourceList – name or GUID of source list, where suggestions will be taken. It is ‘Customers directory’ in my case.
  • sourceMatchField – internal name of the field in the source list. This field will be used to find matching list items for autocomplete keywords.
  • labelFields – an optional parameter, you can specify source list field internal names array. All field values for these  fields will be concatenated with labelSeparator and displayed in autocomplete suggestion as a single string like this: Value1, Value2, …, ValueN.
  • labelSeparator – an optional parameter, it is separator for labelFields concatenation, for example it could be comma with space (‘, ‘).
  • fillConcatenatedLabel – an optional parameter, set true if you need to fill autocomplete textbox with all concatenated labelFields values, set false if you need to fill autocomplete text box only with single field value.
  • fieldsMapping – an optional parameter, it is an array of field mapping objects. Each object declares mapping from source list field to target list field. In my case names of source and target fields are the same. For example Address in Customer directory and Address in Orders list.

Mapping object has following syntax:

{sourceField: 'Internal name of source field', targetField: 'Internal name of target field'}

Note: You can specify only non optional parameters, plugin will work correctly. This plugin works well as single field autocomplete too, just do not fill optional parameters.

Plugin configuration without optional parameters could look like this:

fieldControl.plumFormAutocomplete({
    sourceList: 'Customers directory',
    sourceMatchField: 'Title'
});

Internal implementation of the plugin

Let us look at the full plugin source code. You can download it here.

There are three major part inside the code:

  1. Getting text field input.
  2. Apply jQueryUIi autocomplete plugin and SPServices to get suggestions.
  3. Implementing helper functions.

To get field input I used jQuery selectors and simple regular expression. Unfortunately SharePoint doesn’t provide any method to get field controls from JavaScript, it only stores field internal name inside html comment in the following format:

<!-- FieldName="Title" FieldInternalName="Title" FieldType="SPFieldText" -->

So, I had to parse it to find control I needed. Final function was added to jQuery:

//function gets text field control by internal name
$.getFieldControl = function (fieldInternalName) {
  var regexStr = 'FieldInternalName="' + fieldInternalName + '"'
  var regex = new RegExp(regexStr, 'i');
  var fieldCell = $('td.ms-formbody').filter(function () {
    return (regex).test($(this).html())
  });
  return $(fieldCell.find('input')[0]);
}

In the next step I applied jQuery UI autocomplete plugin and implemented source and select plugin functions. Source function calls source list using SPServices and CAML to get suggestions. When suggestion found, I store all mapped field values inside autcomplete object:

source: function (request, response) {
  var autocompleteVals = [];
  var k = 0;

  $().SPServices({
    operation: "GetListItems",
    async: false,
    listName: options.sourceList,
    CAMLViewFields: getViewFields(options.fieldsMapping),
    CAMLQuery: getCamlQuery(options.sourceMatchField, request.term),
    completefunc: function (xData, Status) {
      $(xData.responseXML).SPFilterNode("z:row").each(function () {
        var queryResult = this;
        var fieldsValues = getFieldsValues(options.fieldsMapping, queryResult);
        var labelText = getLabelText(fieldsValues);
        autocompleteVals[k] = {
          label: labelText,
          value: options.fillConcatenatedLabel ? labelText  :
          extractFieldValue(fieldsValues, options.sourceMatchField),
          fieldsValues: fieldsValues
        };

        k++;

        function getLabelText(fieldValues){
          var result = '';
          if(options.labelFields) {
            for(i = 0; i < options.labelFields.length; i++) {	        		               var fieldName = options.labelFields[i];               var fieldVal = extractFieldValue(fieldValues, fieldName);               if(fieldVal != '') {                 if(i > 0) {
                  result += options.labelSeparator;
                }
                result += fieldVal;
              }
            }
          } else {
            result += extractFieldValue(fieldValues, options.sourceMatchField);
          }
          return result;
        }

      });
      response(autocompleteVals);
    }
  });

}

Select function fills values inside mapped fields according to matched item from source lists. It reads values stored inside ui.item and fills corresponding fields based on suggestion selection.

select: function (event, ui) {
  //Fill all depended fields
  $.each(ui.item.fieldsValues, function () {
    var fieldVal = this;
    var fieldInput = $.getFieldControl(fieldVal.key);

    var outputVal = fieldVal.value;

    if (outputVal) {
      var lookupSeparator = ';#';
      if (outputVal.indexOf(lookupSeparator) != -1) {
        var ind = outputVal.indexOf(lookupSeparator);
        var length = lookupSeparator.length;
        var startInd = ind + length;
        outputVal = outputVal.substring(startInd, outputVal.lenght)
      }
      fieldInput.val(outputVal);
    }

  });
}

Maybe you discovered that there are three helper functions inside plugin: getFieldsValues, getViewFields, and getCamlQuery.

getFieldsValues parses SPServices response and fills autocomplete object according to specified fields mapping.

//get values for all mapped fields
function getFieldsValues(fieldsMapping, queryResult) {
  var result = [];
  $.each(fieldsMapping, function () {
    var fieldMapping = this;
    var val = $(queryResult).attr("ows_" + fieldMapping.sourceField);
    result.push({ key: fieldMapping.targetField, value: val, sourceKey: fieldMapping.sourceField});
  });

  var sourceVal = $(queryResult).attr("ows_" + options.sourceMatchField);
  result.push({ value: sourceVal , sourceKey: options.sourceMatchField});

  return result;
}

getViewFields generates ViewFields xml for CAML query according to fields mapping.

//get view fields for all mapped fields
function getViewFields(fieldsMapping) {
  var result = "";
  var isSourceFieldAdded = false;

  if(fieldsMapping){
    $.each(fieldsMapping, function () {
      var mapping = this;
      var viewField = "";
      result += viewField;
    });

    isSourceFieldAdded = fieldsMapping.filter(function(){
      return this.sourceField == options.sourceMatchField;
    }).length > 0;
  }

  if(!isSourceFieldAdded){
    result += "";
  }

  result += "";
  return result;
}

getCamlQuery generates CAML query according to filter column internal name and keyword from input.

//get CAML query for keyword
function getCamlQuery(colname, keyword) {
  var where = "" + keyword + "";
  var orderBy = "";
  var query = "" + where + orderBy + "";
  return query;
}

Updates

03.10.2013

  • Additional optional parameters added: labelFields, labelSeparator, fillConcatenatedLabel.
  • Reading lookup fields from source list implemented.
  • Minor bug fixes.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Using SPServices with jQueryUI’s Autocomplete Function on InfoPath Forms in SharePoint

Yes, that title says using jQueryUI on InfoPath forms in SharePoint. InfoPath forms are fantastic, except when they just aren’t quite able to do what you want. While InfoPath can help you take your SharePoint list forms to a whole new level, there may be times when you want to add some behavior to those forms which InfoPath simply doesn’t offer.

Guess what? jQuery and jQueryUI can come to the rescue in some of those cases, just as it can with the standard list forms.

I recently worked with a client to do a relatively simple thing, but it was a huge hit. They had a SharePoint list had a Single line of text column, to which the user could add whatever they wanted. However, there was a desire to make suggestions as the user typed, based on the contents of a different list, which had thousands of items.

A dropdown doesn’t make sense in that situation, because there are simply too many items. They also wanted to show matches to what the user had typed, regardless where those characters occurred in the list item.

This is a perfect situation in which to use the idea of autocomplete. We all are familiar with this idea, if not the actual term. You probably see it in action every single day as you use a search engine. As you type your search term(s), the search engine shows suggestions, probably based on some fairly sophisticated algorithms.

Here’s an example from Bing:

image

and from Google:

image

(I can’t help but point out that Google was more on the mark in guessing that I was searching for SPServices, but hey, who’s counting?)

When InfoPath forms are rendered in the browser, they are built of ordinary markup just like any other page, albeit fairly complicated markup, driven by a lot of script.  That doesn’t mean that you can’t add some script of your own as well to add additional capabilities. There are a few peculiarities to this situation, though, which you need to handle.

Here’s a slightly dumbed down version of the script we ended up with.

window.onload = function() {
  window.setTimeout(readyCall, 1000);
}

function readyCall(){

  var externalParties = [];

  $().SPServices({
    operation: "GetListItems",
    listName: "External Parties",
    CAMLViewFields: "",
    async: false,
    completefunc: function (xData, Status) {
      $(xData.responseXML).SPFilterNode("z:row").each(function() {
        externalParties.push($(this).attr("ows_Title"));
      });
    }
  });

  //<input tabIndex="0" title="" class="q_zwfUqJo2fRthHnM4_0 as_zwfUqJo2fRthHnM4_0 b9_zwfUqJo2fRthHnM4_0" id="ctl00_m_g_a226da68_1383_40e3_8410_1ada27d49dcf_FormControl0_V1_I1_T2" aria-invalid="true" style="position: relative;" onfocus="return (TextBox.OnFocus(this, event));" onblur="return (TextBox.OnBlur(this, event));" onpropertychange="return (TextBox.OnPropertyChange(this, event));" type="text" OriginalId="V1_I1_T2" FormId="ctl00_m_g_a226da68_1383_40e3_8410_1ada27d49dcf_FormControl0" ViewDataNode="3" direction="ltr" wrapped="true" ScriptClass="TextBox" VCARD_NAME="91161f891e59461042587839b2504693728ce05a" ?=""/>
  $("input[id$='FormControl0_V1_I1_T2'], input[id$='FormControl0_V1_I1_T3']").autocomplete({
    source: externalParties,
    minLength: 3
  });
}

When InfoPath forms load in the browser, they don’t load with the rest of the page, but instead they are loaded slightly afterward in what amounts to an asynchronous load. Because of that, using $(document).ready(), our trusted jQuery friend, doesn’t work. Instead, as you can see in lines 1-3, we simply wait for 1000 milliseconds (1 second) before we run our script. We found that this was an adequate amount of wait time for our particular form; you might need to adjust this.

In lines 9-19, we use my SPServices library to call the Lists Web Service, using the GetListItems operation. This operation simply reads items from the list based upon the criteria you specify. Once we have the data, we push each of the Title column values into an array called externalParties.

Finally, we call the jQueryUI function autocomplete, using two selectors. In line 21 above, which is commented out, you can see an example of the markup for one of the input elements rendered in the InfoPath form. One of the hardest parts of all of this was to figure out what selector to use. We settled on looking for an input element where the id contained ‘FormControl0_V1_I1_T2’.  (We actually added the autocomplete behavior to two columns in the form, thus the second selector for ‘FormControl0_V1_I1_T3’.)

We added this script into the newifs.aspx and editifs.aspx pages using a trusty Content Editor Web Part (CEWP). Since this was SharePoint 2010, and because it makes for far better code management, we stored the script in a separate file and referenced it using the Content Link.

Bingo-bango, we had a nice, little additional piece of functionality which made the users very happy. This is an example where thinking about all of the tools at your disposal and how you might glue them together into the right solution to get the job done can be the right approach rather than a lot of custom coding.

<UPDATE dateTime=”2011-08-25T23:51″>

My partner in crime for this exercise was Marcel Meth (@marcelmeth), and he’s done a post on it, too, which you can read here. I was able to steal his image of the results, which I’ve added above. Note that the image is not showing the real data but our test data, which was simply a list of the 3000 or so major cities and towns in the US.

</UPDATE>

<UPDATE dateTime=”2011-08-26T09:25″>

I got a question in the comments below about how we added the script to the InfoPath page, and I wanted to add those details here.

  • First, we opened each form in the browser, which launched it in a dialog box.
  • We got the actual URL of the page in the dialog by right-clicking and looking at its properties. The NewForm was newifs.aspx and the EditForm was editifs.aspx (as I mention above).
  • We opened the form page directly in a new browser window and used the toolpaneview=2 Query String parameter trick to put the page into edit mode. This allows you to edit a list form page and add Web Parts to it.
  • We added the CEWP, and put the reference to our script file in the Content Link.
  • Apply and done.

</UPDATE>