Uploading Attachments to SharePoint Lists Using SPServices

Easy!For years people have been asking me how they could upload files using SPServices. I’ve dodged the questions every time because I simply didn’t know how.

On a current client project,  I’m building a slick Single Page Application (SPA) to manage tasks in SharePoint 2010. It’s basically a veneer over the clunky out-of-the-box task list experience. Rather than hopping from page to page, the team using the application can accomplish everything they need to do on a single page.

Every project has specific needs, and for this one I’m using KnockoutJS. But the code snippets I give below are generic enough that you should be able to use them in any context with a little thinking.

It’s been going well, but I had been putting off implementing adding attachments because…well, as I said, I didn’t know how.

One of the benefits of the shift from all server side development to a much more significant focus on client side techniques is that some of the bright minds in the SharePoint development community have turned their eyes toward solving these things, too. Usually it’s on SharePoint 2013 or SharePoint Online at Office 365. However, at least now when I search for “SharePoint JavaScript xxx”, there are likely to be some great hits I can learn from.

In this case, there were two excellent posts, one from James Glading and one from Scot Hillier. (See the Resources section below for links.)

The first step is to enable HTML5 capabilities in IE10. This requires two small changes to the master page. This seems simple, but it can have effects that you don’t expect. In other words, don’t consider this just a “no brainer”. You need to plan for the change across your Site Collection and any impacts it may have.

The first change is to switch from using “XHTML 1.0″ to “HTML” as the DOCTYPE. This is what “turns on” HTML5.

<!DOCTYPE html>

Then, in this project we set the content meta tag to IE=10 because we want to aim for IE10.

<meta http-equiv="X-UA-Compatible" content="IE=10"/>

If your browser base is different, you can, of course, set this differently. In this project, we will require all users of the application to be running IE10 or greater. Firefox and Chrome have supported the bits of HTML5 we need for so long, that there’s little concern about people who choose to use those browsers.

Once we have those two small changes in the master page (we are using a copy of v4.master with no other customizations at the moment), we “have HTML5″. It’s most obvious because some of the branding flourishes I’ve put in like rounded corners show up in IE10 now, rather than just square boxes.

Once we have HTML5 enabled, we can start to use the File API, a.k.a. the FileReader. This is such a simple little thing, that it’s hard to believe it gives us the capability it does.

<input type="file" id="attachment-file-name"/>

That’s it. That simple HTML gives us a file picker on the page. Depending on your browser, it will look something like this:

File Picker

When you make a file selection with this very familiar widget, you can query the element using jQuery.

var file = $("#attachment-file-name").files[0];

Note that we’re after a single file, so we grab the first object in the file list array. We get an object that looks something like this screen shot from Firebug:

File object from the File Picker

Once we have the file object, we can look at what the Lists SOAP Web Service needs to get in order to upload the file. Again, it’s pretty simple. Here is the list of inputs and output from the MSDN documentation page for the Lists.AddAttachment Method.

Parameters

listName
A string that contains either the title or the GUID for the list.
listItemID
A string that contains the ID of the item to which attachments are added. This value does not correspond to the index of the item within the collection of list items.
fileName
A string that contains the name of the file to add as an attachment.
attachment
A byte array that contains the file to attach by using base-64 encoding.

Return Value

A string that contains the URL for the attachment, which can subsequently be used to reference the attachment.

The first three inputs are straightforward. We pass in the name of the SharePoint list, the ID of the list item, and the name of the file we want to attach. It’s that last input parameter called “attachment” that’s a bit tricky.

When we upload a file via an HTTP POST operation – which is what all of the SOAP Web Services use – we have to pass text. But the files we want to upload are as likely as not to be binary files. That’s where the Base64 format comes in. Believe me, you don’t need to know exactly what the format actually is, but you should probably understand that it enables us to send binary files as text bytes. Those bytes are then decoded on the server end so that the file can be stored in all of its original, pristine glory.

Here’s the code I ended up with, skinnied down as much as possible to make it a little clearer to follow. I draw heavily from Scot and James’ posts for this, but nuanced for SPServices. I’ve also stripped out all of the error handling, etc.

  • First, the getFileBuffer function reads the contents of the file into the FileReader buffer. readAsArrayBuffer is an asynchronous method, so we use a jQuery Deferred (promise) to inform the calling function that the processing is done.
  • The contents of the buffer are then converted from an ArrayBuffer – which is what the FileReader gives us – into a Base64EncodedByteArray. This works by passing the buffer through a Uint8Array along the way.
  • Finally, we use the toBase64String method to convert the SP.Base64EncodedByteArray to a Base64String.

Yeah, that’s a lot to swallow, but again, you don’t really need to understand how it works. You just need to know that it does work.

Finally, we call Lists.AddAttachment using SPServices to do the actual upload.

/* From Scot Hillier's post:
http://www.shillier.com/archive/2013/03/26/uploading-files-in-sharepoint-2013-using-csom-and-rest.aspx */
var getFileBuffer = function(file) {

  var deferred = $.Deferred();
  var reader = new FileReader();

  reader.onload = function(e) {
    deferred.resolve(e.target.result);
  }

  reader.onerror = function(e) {
    deferred.reject(e.target.error);
  }

  reader.readAsArrayBuffer(file);

  return deferred.promise();
};

getFileBuffer(file).then(function(buffer) {
  var bytes = new Uint8Array(buffer);
  var content = new SP.Base64EncodedByteArray(); //base64 encoding
  for (var b = 0; b < bytes.length; b++) {
    content.append(bytes[b]);
  }

  $().SPServices({
    operation: "AddAttachment",
    listName: "Tasks",
    listItemID: taskID,
    fileName: file.name,
    attachment: content.toBase64String()
  });

});

Very cool! And as I tweeted yesterday, far easier than I ever would have expected. Yes, it took me a good chunk of a day to figure out, but it definitely works, and pretty fast, too.

If you use this example as a base, you could fairly easily build out some other file uploading functions. Combined with the other attachment-oriented methods in the Lists Web Services, you can also build the other bits of the attachment experience:

  • GetAttachmentCollection – Returns a list of the lit item’s attachments, providing the full path to each that you can use in a list of links.
  • DeleteAttachment – Once you’ve uploaded an attachment and realized it was the wrong one, this method will allow you to delete it.

Moral of the story: Fear not what you do not know. Figure it out.

Resources

What’s the Story for HTML5 with SharePoint 2010? by Joe Li

Uploading Files Using the REST API and Client Side Techniques by James Glading

Uploading Files in SharePoint 2013 using CSOM and REST by Scot Hillier

Lists.AddAttachment Method

This article was also posted at ITUnity on Jun 02, 2014. Visit the post there to read additional comments.

Update 2014-05-28 11:55 GMT-5

Hugh Wood (@HughAJWood) took one look at my code above and gave me some optimizations. Hugh could optimize just about anything; he is *very* good at it. I *think* I can even see the difference with larger files.

getFileBuffer(file).then(function(buffer) {
  var binary = "";
  var bytes = new Uint8Array(buffer);
  var i = bytes.byteLength;
  while (i--) {
    binary = String.fromCharCode(bytes[i]) + binary;
  }
  $().SPServices({
    operation: "AddAttachment",
    listName: "Tasks",
    listItemID: taskID,
    fileName: file.name,
    attachment: btoa(binary)
  });
});

Microsoft Cloud Show Episode 016 – Interview with Marc Anderson on Recent Changes Impacting Customers on Office 365

MSCloudShow

Microsoft Cloud Show

A few weeks back, I sat down (virtually, of course) with Andrew Connell (@AndrewConnell) and Chris Johnson (@LoungeFlyZ) to record an episode of the Microsoft Cloud Show. Andrew was in Florida, I was in Boston, and Chris was way around the world in New Zealand. Ah, the wonders of modern technology.

The only place to stay up to date on everything going on in the Microsoft cloud world including Azure and Office 365.

Whether you are new to the cloud, old hat or just starting to consider what the cloud can do for you this podshow is the place to find all the latest and greatest news and information on what’s going on in the cloud universe.  Join long time Microsoft aficionados and SharePoint experts Andrew Connell and Chris Johnson as they dissect the noise and distill it down, read between the lines and offer expert opinion on what is really going on.  Just the information … no marketing … no BS, just two dudes telling you how they see it.

I was honored to be the very first guest on the show, which already had 15 excellent episodes in the can.

In Episode 016 – Interview with Marc Anderson on Recent Changes Impacting Customers on Office 365, we talked about a number of extremely important things that have been going on with Office365 lately.

I had done a post about one issue that has caused me and users of SPServices the most consternation, Office 365 Update Changes ‘Display Name’ on Required Fields and Andrew had posted about a few others one his blog in Office 365 Needs to Treat the UX as an API if Our Customizations are to Stay Off the Server.

Last week, I released SPServices 2014.01, which addresses the title changes (adding ” Required Field” to the title attribute of some required dropdowns), but there’s a bigger set of issues at play here, as Andrew alludes to in his post.

In the podcast, we talked about the impact of these changes as well as the mindset behind them from the Microsoft side.

If you do any client side development with SharePoint – and that’s where everyone is headed – you owe it to yourself to listen to the podcast. You’ll understand more about what changes to the DOM might mean for you as a developer, or even what might happen to you as a user of customizations that rely on the DOM being stable and predictable.

One things seems certain: we’ll see more changes like the ones we discussed in the podcast and they will have an impact on everyone, not just people replying on Office365. (The same issues started to crop up for people who have applied the December 2013 Cumulative Update (CU) for SharePoint 2010 on premises.)

I want to thank Chris and Andrew for inviting me in for a chat. Assuming I didn’t annoy them too much with my scatological terminology, maybe I’ll be able to visit with them again the next time a round of changes like this pop up and cause ripples in the SharePoint time-space continuum.

Getting Around SharePoint’s Threshold Limits for Large Reference Lists

In SharePoint 2007, we could build lists that contained as many items as we wanted and access them in any way we wanted. We may have done stupid things that were bad for server performance, but if we knew what we were doing there were few limits. We had the mythical “2000 item” rule of thumb (which had little basis in reality in many, many cases) but otherwise it was up to us. (See: Plan for software boundaries (Office SharePoint Server))

In SharePoint 2010, Microsoft introduced threshold limits to protect us from ourselves. This seemed counter intuitive to me, since with the upgrade to SharePoint 2010, one also had to go to 64 bit architecture and put a heck of a lot more iron behind the SharePoint farm. While we could potentially store 30,000,000 items in a list (30 million!), we had a list view threshold of 5000 items. (See: SharePoint Server 2010 capacity management: Software boundaries and limits)

SharePoint 2013 maintains similar limits to 2010, with 5000 items the limit for a list view. (See: Software boundaries and limits for SharePoint 2013)

List View ThrottlingSomehow, as technology has moved forward – significantly – we’ve got more limits.

5000 items is way too many to ever show in a list view, but it may not be if you want to do some client side processing. It doesn’t matter if you’re trying to build a truly bloated, Bad Idea list view or request data using REST, you’re stuck with that 5000 item limit.

But what if you know what you are doing and the 5000 item limit doesn’t work for you? I’ve written in the past about why we shouldn’t change some of the threshold limits (e.g.,  The SharePoint 2010 “List View Lookup Threshold” and Why We Don’t Change It).

If we’re working on Office365, we simply can’t change the limits – Microsoft gets to decide what they are. Sometimes those limits change without notice, too, so it’s truly out of our hands.

Well, what does all this add up to? Let’s take a specific example.

I needed to provide auto complete functionality in an InfoPath form. It was the classic scenario: we had a list of [in this case] 19,000+ cities and towns and we needed a good UX for entering selections into the form. We run into this sort of thing with data like cities and towns all the time. There are too many choices for a dropdown to make any sense, and SharePoint doesn’t offer up any better alternatives. jQueryUI’s autocomplete function is a perfect solution for this sort of thing.

The data was in a spreadsheet, and it’s easy to upload data from a spreadsheet to a SharePoint list. Bingo, bango, done, right? Well, no. It’s impossible to upload data from a spreadsheet *and* index the column. We need to index the column so that we can make type of requests requests that we need for autocomplete – basically, give me all of the items that begin with or contain this string – or else we run into the 5000 item threshold error on our requests.

No problem! We’ll just upload the data and *then* index the column. Not so fast, mister. When you try to index the column you run into – you guessed it – the 5000 item limit. No indexee, no laundry.

So seemingly we’re stuck. But wait – we can just create the list in the UI, add the index, and paste all 19,000 items into the Brave New World of SharePoint 2013′s “data sheet view” aka “Quick Edit”. Sadly, when I tried to paste in more than a few hundred items, the browser hung on me. This was on Office365, and I tried several different browsers. SharePoint Online seems to prefer nibbles to big bites of data. Unless I spent days just doing the pasting, this was never going to happen.

In this case, I decided to simply store the data in a text file containing JSON rather than trying to force it into a list. Thanks to @bpmccullough for the suggestion on Twitter.

The data I originally had received for the sities and towns was in a CSV file. I found a nice little site that would convert the CSV format to JSON (bookmark this one; you’ll need it in this JSON-oriented world).

Once I had the JSON file uploaded to a Document Library, the code below is what I used to set up the autocomplete. We wanted the autocomplete to allow multiple choices, so there’s a little monkeying around to enable that. It wasn’t tremendously peppy (19,000 cities, after all), but it worked. Remember, kids: this is an example. Don’t expect to drop it into your page and have it work.

var citiesAndStates = [];

// Get the City and State data from the file with JSON data
$.getJSON(&quot;/Shared%20Documents/CityState.txt&quot;, function(data) {
  var results = $(data);
  // Push the data into an array which is appropriate for jQueryUI autocomplete
  citiesAndStates = ($.map(results, function(item) {
    return {
      label: item.State + &quot; - &quot; + item.City, // Show &quot;State - City&quot; [e.g., MA - Boston] for selection
      value: item.City + &quot;, &quot; + item.State // Show &quot;City, State&quot; [e.g., Boston, MA] upon selection
    }
  }));
});

// When the field is available and gets focus, set up the autocomplete behavior
//ctl00_ctl42_g_6069846d_1b7f_4890_b767_2bdc15d2b133_FormControl0_V1_I1_S15_I4_T5
$(document).on(&quot;focus&quot;, &quot;input[id$='FormControl0_V1_I1_S15_I4_T5']&quot;, function() {
  if ($(this).data(&quot;autocompleteSet&quot;) === undefined) {
    $(this).data(&quot;autocompleteSet&quot;, true);

    $(this)
    // Don't navigate away from the field on tab when selecting an item
    .bind(&quot;keydown&quot;, function(event) {
      if (event.keyCode === $.ui.keyCode.TAB &amp;&amp; $(this).data(&quot;ui-autocomplete&quot;).menu.active) {
        event.preventDefault();
      }
    }).autocomplete({
      source: function(request, response) {
        // Delegate back to autocomplete, but extract the last term
        response($.ui.autocomplete.filter(citiesAndStates, extractLast(request.term)));
      },
      minLength: 3,
      focus: function() {
        // Prevent value inserted on focus
        return false;
      },
      select: function(event, ui) {
        var terms = split(this.value);
        // Remove the current input
        terms.pop();
        // Add the selected item
        terms.push(ui.item.value);
        // Add placeholder to get the semicolon-and-space at the end
        terms.push(&quot;&quot;);
        this.value = terms.join(&quot;; &quot;);
        return false;
      }
    });
  }
});

function split(val) {
  return val.split(/;\s*/);
}

function extractLast(term) {
  return split(term).pop();
}

One note here: This approach basically violates my “put all data into a list so that end users can maintain it” rule. However, the approach I came up with was the best one for the situation. It’s still possible for a reasonably savvy (meaning “non-developer”) user to manage the data by downloading the file and using a JSON-aware editor.

The main thing was that we were able to set up something that worked, regardless what the Office365 limits may change to.

Finding the SharePoint 2007 / 2010 Thesaurus Files

If you go to TechNet to find out where the thesaurus files for search are so that you can add in your own synonyms, you may be as confused as I was earlier today. It only took me about 20 minutes to figure out, but if three or four people find this post, we’ll have saved enough time for lunch.

The TechNet article you want is Manage thesaurus files (SharePoint Server 2010), though the one for SharePoint 2007 (Edit a thesaurus file (Office SharePoint Server)) is pretty much identical.

In the article, it says

By default, SharePoint Server 2010 installs the thesaurus files for all supported languages at %ProgramFiles%\Microsoft Office Servers\14.0\Data\Office Server\Config. When a search administrator creates a Search service application, the search system automatically copies the thesaurus files from the installation location (including any thesaurus files there that an administrator has edited) to %ProgramFiles%\Microsoft Office Servers\14.0\Data\Office Server\Applications\GUID-query-0\Config, where GUID is the GUID of the new Search service application. The search system performs the same operation on every query server that is running the new Search service application. Thus there is a copy of each thesaurus file on each query server that is running that Search service application.

When I looked in %ProgramFiles%\Microsoft Office Servers\14.0\Data\Office Server\Config, well, there was no %ProgramFiles%\Microsoft Office Servers\14.0\Data\Office Server\Config. Instead, because my client had decided to change the location of the index to another drive, I have to figure out where that actually was. Here’s the trick.

In complex farms, you may have multiple Search Service Application, multiple indices, etc. but these steps should work in most cases.

  • In Central Administration, go to the Search Application -> Central Administration/ Manage service applications / Search Service Application (or whatever you called it)
  • At the bottom of the page, you’ll see a section called ‘Search Application Topology’
  • Click the Modify button and on the next screen look for the ‘Index Partition’ (you may have more than one)
  • Click on the ‘Query Component 0′ link and Edit Properties
  • The field called ‘Location of Index’ contains the root location for the thesaurus files

image

Looking in that folder, you should find folders that look something like this:

image

As noted above, the thesaurus files you want to work with are in the GUID-query-0\Config folder. In my case above, it’s E:\Data\SearchIndex\Office Server\Applications\0f78bae4-05b9-417f-b533-43326409dfcc-query-0\Config

Happy equivalency!

One side note: it boggles my mind that there is no UI to manage synonyms in the thesaurus, but there you go.

Setting a Rich Text Column in a SharePoint Form with jQuery – SharePoint 2010

I have gotten several questions on an older post of mine titled Setting a Rich Text Column in a SharePoint Form with jQuery from a guy named Travis. He’s been struggling to try to use the script in the post to get at the text in a Rich Text Editor (RTE) in SharePoint 2007.

I just took a look at an RTE in SharePoint 2010 in both IE and Firefox. The markup for it is considerably different than it was in SharePoint 2007, and looking at my older post, I can tell from the screenshots that I was in 2007. The good thing is that it looks identical in both browsers in 2010, which wasn’t usually the case in 2007.

Here’s what the column looks like in SharePoint 2010 with the same text typed into it as in the previous example:

9-11-2013 12-56-16 PM

And here’s what the markup looks like in IE10:

9-11-2013 1-04-32 PMThe script in the older post shouldn’t work in any browser, since the markup is different. Rather than a textarea, the typed text goes into a div. It was pretty easy to select the textarea in 2007 because it had its title attribute set to the DisplayName of the column. In 2010, there’s no such obvious “hook” to select on.

Instead, we have to fall back to looking for the column name in the comment which is above each column’s control on the page. To recap, each cell in the left column where the names of the columns show has the CSS class ms-formlabel applied to it. In the right column, where the editing controls are, each cell has the class ms-formbody applied to it.

To find the RTE in the page, we have to loop through all of the ms-formbody cells to find the right one. I do this in SPServices for some things as well, because SharePoint is very inconsistent when it comes to providing a good way to select form elements.

Here’s my findFormField function from SPServices:

// Finds the td which contains a form field in default forms using the comment which contains:
//  <!--  FieldName="Title"
//    FieldInternalName="Title"
//    FieldType="SPFieldText"
//  -->
// as the "anchor" to find it. Necessary because SharePoint doesn't give all field types ids or specific classes.
function findFormField(columnName) {
  var thisFormBody;
  // There's no easy way to find one of these columns; we'll look for the comment with the columnName
  var searchText = RegExp("FieldName=\"" + columnName.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, "\\$&") + "\"", "gi");
  // Loop through all of the ms-formbody table cells
  $("td.ms-formbody, td.ms-formbodysurvey").each(function() {
    // Check for the right comment
    if(searchText.test($(this).html())) {
      thisFormBody = $(this);
      // Found it, so we're done
      return false;
    }
  });
  return thisFormBody;
} // End of function findFormField

So the script to get the HTML value of the RTE – assuming we have the function above available to us – is:

var thisFieldHtml = findFormField("System Description").find(".ms-rtestate-write div").html();
alert(thisFieldHtml);

Unfortunately (fortunately?) SharePoint keeps track of where our cursor is at any given time during the editing process by maintaining two spans inside the div.

<span id="ms-rterangecursor-start"></span><span id="ms-rterangecursor-end"></span>

Here’s what it looks like in an alert when I’ve just typed “big “:

9-11-2013 1-33-09 PMIf we highlight a section of text, then that highlighted text is contained within the two spans. Here I’ve highlighted the word “over”:

<span id="ms-rterangecursor-start"></span>over<span id="ms-rterangecursor-end"></span>

So if we want to get the HTML value of the RTE without the editing spans, we need one more line of code to remove them:

var thisFieldHtml = findFormField("System Description").find(".ms-rtestate-write div");
thisFieldHtml.find("#ms-rterangecursor-start, #ms-rterangecursor-end").remove();
alert(thisFieldHtml.html());

If we just want the text value of the RTE, then we don’t need to bother stripping out the spans:

var thisFieldText = findFormField("System Description").find(".ms-rtestate-write div").text();
alert(thisFieldText);

Note: I’ve also tried this in SharePoint 2013. There, the artificial surrounding div which SharePoint wrapped the Rich Text in with the earlier version is not present, so we don’t need to look inside it:

var thisFieldText = findFormField("Reason for Nomination").find(".ms-rtestate-write").text();
alert(thisFieldText);

Travis, I hope this helps!