Dear Microsoft: Please Fix Retrieving SharePoint Lookup Columns with REST When the Lookup List is in Another Web

I love SharePoint. I really do. I especially love writing client side code to build awesome applications for my clients.

Today’s annoyance, though, comes while I am in the process of rewriting an application I built on SharePoint 2007, porting it to SharePoint Online in Office 365. This ought to feel like a huge leap forward technologically, and in some ways it does. I’m changing all my SOAP calls with SPServices to REST calls. I’m switching from KnockoutJS to AngularJS, which will simply perform better given the profile of the applications. (KnockoutJS was the right choice years ago when I first built the applications, but the data and feature requirements have outgrown it.)

Unfortunately, I’m running into a simple constraint that makes my life a lot harder. When I first started building these applications five years ago, I created what I’ve got to say is a very solid information architecture. It’s withstood shifting needs and requirements in the interim, and I stand by it. One of the aspects of this good information architecture is storing commonly used reference lists in the root site of the Site Collection. By creating a Site Column which is a lookup into each reference list, I can reuse those common reference values throughout my subsites.

This works great in SharePoint 2007 with SOAP calls. When I retrieve items with one of these lookup Site Columns from a list in a subsite, I simply get the ID and Title values, separated by a “;#”. However, when I try to do the same thing with “modern” REST calls, I get an error like this:

I’ve been a good team player, and I’ve suggested they fix this on the SharePoint User Voice in my suggestion Enable support for lookup columns in other webs in the REST API. The votes are up, and it’s been a while.

There’s a workaround, but it’s not very pleasant. (The easiest workaround is to simply stick with SOAP calls and SPServices – I’ve done that in several cases in other projects. But SOAP is officially “deprecated”, so…)

Here’s a specific example. The client I’m working with is in financial services, and they issue recommendations on securities. Those recommendations are very standard, and predictable: Hold, Buy, Sell, etc. In other words, perfect to store in a list in the root site called Recommendations. Why not a Managed Metadata column, you might ask? Well, I also wanted to store several other columns in the Recommendations list, like Description (e.g., “The analyst expects the security to outperform their coverage universe.”), a SortOrder value so I could rearrange the values in dropdowns using SPArrangeChoices, and several other fields which drive configuration of some reports. In other words: great information architecture. The values are all consistent across the various subsites, I store them once, etc. Nice setup.

I created a Site Column back in the beginning called Recommendation, which is a lookup into the title column of the Recommendations list (Hold, Buy, Sell, etc.). I used that Site Column in many Content Types defined on the subsite level. Those Content Types are mainly used in a list I’ll call Notes.

In SOAP with SPServices, I can make this [simplified] call:

This retrieves the items and returns nice JSON for me. Because Recommendation is a lookup column, it comes back as something like “1;#Buy” and that’s easy to turn into a JSON object like:

Easy, peasy.

However, when I try the analogous call in REST:

I get the error:

In other words, there’s no way to $expand the Recommendation column because it comes from an other Web, even though that is ideal information architecture!

The workaround, which André Lage (@aaclage) pointed out in my UserVoice suggestion (but I clearly didn’t get at the time), is to simply ask for the Recommendation column’s ID instead. This isn’t obvious at all:

This doesn’t follow the syntax we’d expect: we need to append “Id” to the end of the lookup column’s InternalName. Of course, this just gets us the ID of the item in the Recommendations list; it doesn’t fetch us the Title value, which is what we really want. Because of this, I need to do a *separate* REST call to get the items from the Recommendations list and merge the values in my client side code.

Now, one could argue that this is more efficient. I don’t ask the server to $expand the values across thousands of notes (yes, there are way more than 5000; I’ve written enough about that lately – I may have mentioned it here and here and here and here), so it gets a break. Retrieving the 5-10 values in the reference list (in this case) is no big deal.

But I have a half dozen or so of these lookup columns to deal with in this application, which means a half dozen extra REST calls, plus the code to merge the values. More work for me, but more importantly a longer wait for the application user when they load the page. I believe that poor UX is what has doomed many a SharePoint roll out, and I loathe creating a poor UX myself. In this case, I’ll make it work, but I’d really like to see this change.

Advertisements

Movement on the SharePoint List 5000 Item Limit!

In October, Eric Alexander (@ejaya2) posted an idea to the SharePoint UserVoice called Prioritize large list management in SharePoint Online. Yesterday I received an alert about it because I had voted for it.

The 5000 item limit has been an albatross around our necks since SharePoint 2010. SharePoint 2007 was the wild west days: we could pile as many items into a list as we wanted and retrieve them without a problem, though perhaps at the expense of performance.

I’ve railed about this limit many times in the past, like here.

No 5000 item limit

If you search the SharePoint UserVoice for “5000“, you’ll find no fewer than 14 suggestions circling this topic. There are probably even more, but without the number 5000 in them.

Luckily, our good friends in Redmond know this is an issue for us. As of yesterday, Eric’s suggestion moved to “Working on it”, at least for SharePoint Online.

Prioritize large list management in SharePoint Online - Working on it

 

 

 

I’m looking forward to what happens here. As Eric notes in his suggestion:

If nothing can be done, then TechNet NEEDS to indicate that the actual limit of SharePoint Online lists and libraries is 5,000 items, not the current architectural limit of 30 million.

The more we want to use SharePoint as a service (SPaaS?), the more important it becomes to get past this limitation.

I’m certain that the work Dan Kogan (@kogandan) and his team are doing on the SharePoint Framework (SPFx) has made it obvious that this limit is a serious issue. (Sometimes you have to take the albatross off your neck and slap someone with it.)

Image from http://elkgrovegovernment.com/with-the-nrc-announcement-has-the-albatross-been-taken-off-elk-groves-neck/

Image from http://elkgrovegovernment.com/with-the-nrc-announcement-has-the-albatross-been-taken-off-elk-groves-neck/

Save Your SharePoint Online Tenant: The SharePoint Sandboxed Solutions Inspector

If you’ve been following the “code-based sandbox solutions on Office 365” saga, you know that there is little time left to fix your existing sandbox solutions in Office 365. See: Microsoft Is Removing Code-Based Sandbox Solutions in SharePoint Online – Be Prepared!

Last week, Vesa Juvonen (@vesajuvonen) released a script (New Script Available from Microsoft PnP: Generate list of sandbox solutions from SharePoint Online tenant) that can help you find your sandbox solutions. Surprisingly, what was missing from Vesa’s script was identification of the solutions that contain code. You’d get a list of all your sandbox solutions, but not specifically the ones that were going to cause you problems.

The Rencore Team

Some of the great looking folks at Rencore. Where’s Waldek?

My awesome friends at Rencore – the SFCAF folks – were kind enough to make a free tool available this week to help with even more with your diagnosis and even some of the cures. In Erwin van Hunen’s (@erwinvanhunenpost Introducing the Rencore SharePoint Sandboxed Solutions Inspector, you can learn more about the free tool and how it can help.

On August 31st, 2016 Microsoft is going to shut down support for Sandboxed Solutions with code.

Sandboxed Solutions containing code will be deactivated and this might impact your Office 365 tenant big time!

If you know you need it, just head right on over to the download page.

The SharePoint Sandboxed Solutions Inspector

They have already released several updates to the tool, and are keeping it current based on feedback from the folks using it. Now that’s service – and for a free tool!

But I think the best thing is that the Rencore tool can fix some of the most common issues – most notably the “empty DLL” issue that makes Office 365 think you have code in your sandbox solution when you don’t.

We heard “30 days” when all this started, and now people seem to be taking that as August 31. Don’t leave your users in the lurch – get going on handling this situation.

Oh, and if you’re a vendor or consultant who has written a sandbox solution with code over the last few years: reach out to your client and own it. Get them back on the right road and you’ll be the better for it.

Clean report!

Clean report!

New Script Available from Microsoft PnP: Generate list of sandbox solutions from SharePoint Online tenant

As I wrote on Monday, the decision to begin removing already deprecated code-based sandbox solutions on SharePoint Online took many people by surprise. Even though the news about the deprecation has been out there since 2014, the abrupt move – especially during a time when many people are on summer vacation – caused consternation for some. Others welcomed the move, in essence saying “good riddance” to a model that never really reached maturity.

But what if you manage a tenant on Office 365 that might have code-based sandbox solutions? Maybe you’ve used outside vendors to build solutions for you and you’re not sure what techniques they have used. Or maybe your own team built some things a few years back, you’ve had some turnover, and you source control isn’t so great. (Not so unusual, frankly.) How do you know what you have and what to do about it? You certainly don’t want functionality your users actually need to stop working unexpectedly. Some of these solutions could be InfoPath forms with code-behind, for example.

Office 365 Dev Patterns & Practices (PnP) Vesa Juvonen (@vesajuvonen) – one of my true Microsoft heroes for what he has done with the PnP set of tools – has come to the rescue, apparently with some help from Karine Bosch (@kboske). They have released a PowerShell script today that promises to “Generate [sic] list of sandbox solutions from SharePoint Online tenant“. (Far be it from me to correct Vesa’s Finglish!)

Generate list of sandbox solutions from SharePoint Online tenant

Generate list of sandbox solutions from SharePoint Online tenant

This script can be used to generate list of sandbox solutions in SharePoint Online tenant. You will need to use tenant administrator account to connect to SharePoint Online and script will generate a list of sandbox solutions to separate txt file, which can be imported to Excel for further analyses.

Note: This script is relatively simple and does not use multi-threading, so execution in larger tenants might take a while. We are looking for further enhancing the script with multi-threading support, if there’s demand for this. Also community contributions on this side are more than welcome.

Output file has following columns

  • URL of the site collection
  • Name of the sandbox solution
  • Author field from the sandbox solution – who uploaded the file
  • Created field from the sandbox solution – when solution was uploaded
  • Status field – 1=Activated, 0=Not activated

What seems to be missing here – at least to me – is the “and this one contains ‘code'” indication, but it’s still going to be very useful. Maybe it’s not simple to tell which solutions contain code? If you have ideas about this, it’s an open source project, so head over there and enhance it!

IMPORTANT: Please note that this script lists ALL sandbox solutions. But only code-based sandbox solutions have been deprecated and are being removed from Office 365. So don’t panic when you see all of your no-code solutions and site templates listed. This is a first step in inventorying your solutions.

The script requires – not surprisingly – that you have the SharePoint Online cmdlets installed. My bet is that there are plenty of Office 365 customers that have never really figured out how to download, install, and use PowerShell against Office 365. Many tenants are run by business users rather than technical types, as befits a powerful SaaS offering. In fact, in many cases, IT doesn’t need to be involved at all. That said, one would hope that those tenant administrators would know whether they have sandbox solution installed. However, see my mention of possible situations above, even if IT was in charge.

Here’s a quick tutorial on how to install those cmdlets in case you need it. In writing this section, I’m stealing the TechNet article Connect to Office 365 PowerShell. (I fear this article may not be available to everyone, as TechNet is part of subscriptions? I’m not really sure.) To do this, you have to be a tenant administrator. If you are, you’re probably the one wondering what you have in any case. It’s a pretty painless process, but if you haven’t used PowerShell – think batch files for servers – then it might be intimidating. I’m just copying the instructions from TechNet here, but I’ll add in some graphics and additional comments over the course of the day, so come back if you have questions. If you want to add any tips, please do so in the comments.

As Vesa always says: Sharing is Caring!

Step 1: Install required software

These steps are required once on your computer, not every time you connect. However, you’ll likely need to install newer versions of the software periodically.

  1. Install the 64-bit version of the Microsoft Online Services Sign-in Assistant: Microsoft Online Services Sign-in Assistant for IT Professionals RTW.

Microsoft Online Services Sign-In Assistant for IT Professionals RTW

2. Install the 64-bit version of the Windows Azure Active Directory Module for Windows PowerShell: Windows Azure Active Directory Module for Windows PowerShell (64-bit version).

Windows Azure Active Directory Module for Windows PowerShell (64-bit version)

Step 2: Open the Windows Azure Active Directory Module

  1. Find and open the Windows Azure Active Directory Module for Windows PowerShell by using one of the following methods based on your version of Windows:
    • Start menu   On the Start menu, enter Azure in the Search programs and files box.
    • No Start menu   Search for Azure using any of these methods:
      • On the Start screen, click an empty area, and type Azure.
      • On the desktop or the Start screen, press the Windows key+Q. In the Search charm, type Azure.
      • On the desktop or the Start screen, move your cursor to the upper-right corner, or swipe left from the right edge of the screen to show the charms. Select the Search charm, and enter Azure.
  2. In the results, select Windows Azure Active Directory Module for Windows PowerShell.
Here's what it looks like on my laptop running Windows 10

Here’s what it looks like on my laptop running Windows 10

Step 3: Connect to your Office 365 subscription

  1. In the Windows Azure Active Directory Module for Windows PowerShell, run the following command.

    In the Windows PowerShell Credential Request dialog box, type your Office 365 work or school account user name and password, and then click OK.

  2. Run the following command.

    Connect-MsolService -Credential $UserCredential - Success!

    Connect-MsolService -Credential $UserCredential – Success!

How do you know this worked?

After Step 3, if you don’t receive any errors, you connected successfully. A quick test is to run an Office 365 cmdlet—for example, Get-MsolUser—and see the results.

If the Get-MsolUser cmdlet runs successfully, you'll see a list of your users

If the Get-MsolUser cmdlet runs successfully, you’ll see a list of your users

If you receive errors, check the following requirements:

  • A common problem is an incorrect password. Run Step 3 again. and pay close attention to the user name and password you enter.
  • The Windows Azure Active Directory Module for Windows PowerShell requires that the Microsoft .NET Framework 3.5.x feature is enabled on your computer. It’s likely that your computer has a newer version installed (for example, 4 or 4.5.x), but backwards compatibility with older versions of the .NET Framework can be enabled or disabled. For more information, see the following topics:
  • Your version of the Windows Azure Active Directory Module for Windows PowerShell might be out of date. To check, run the following command in Office 365 PowerShell or the Windows Azure Active Directory Module for Windows PowerShell:

    If the version number returned is lower than the value 1.0.8070.2, uninstall the Windows Azure Active Directory Module for Windows PowerShell, and install the latest version from the link in Step 1.

  • If you receive a connection error, see this topic: “Connect-MsolService: Exception of type was thrown” error.

An Unpleasant Refiner Bug in SharePoint Online

As always, I don’t know where to report issues with Office 365. So here I am carping in a public channel in the hopes that the right person sees it. This also may happen with SharePoint 2013 on premises, but I can’t recall.

Unpleasant SharePoint search refinerWhen I use a refiner that is based on a Lookup column, I’ll often see something like the attached image. As you can see, there are two clients listed: Anchor Glass and 29. Anchor Glass is the title of the item in the Clients list with the ID of 29, but it still doesn’t make sense (especially to those poor users). It’s as if the indexer isn’t smart enough to handle Lookup columns properly – but only some of the time.

This happens on multiple tenants, with different Lookup columns to different lists, but not consistently, or in any pattern I’ve been able to discern. Generally it’s the Title column we’re looking up into. Using a Lookup column into a list is almost always preferable to using Managed Metadata (IMO) because we can store additional data about the values in the list. For instance, in the Clients list above, we have the client contact info, who works with the client, when we started working with them, what the billings are, etc. Managed Metadata is worthless for that.

But the bug.

Has anyone else seen this? Any ideas on how to work around it?