Dear Microsoft: Please Fix Retrieving SharePoint Lookup Columns with REST When the Lookup List is in Another Web

I love SharePoint. I really do. I especially love writing client side code to build awesome applications for my clients.

Today’s annoyance, though, comes while I am in the process of rewriting an application I built on SharePoint 2007, porting it to SharePoint Online in Office 365. This ought to feel like a huge leap forward technologically, and in some ways it does. I’m changing all my SOAP calls with SPServices to REST calls. I’m switching from KnockoutJS to AngularJS, which will simply perform better given the profile of the applications. (KnockoutJS was the right choice years ago when I first built the applications, but the data and feature requirements have outgrown it.)

Unfortunately, I’m running into a simple constraint that makes my life a lot harder. When I first started building these applications five years ago, I created what I’ve got to say is a very solid information architecture. It’s withstood shifting needs and requirements in the interim, and I stand by it. One of the aspects of this good information architecture is storing commonly used reference lists in the root site of the Site Collection. By creating a Site Column which is a lookup into each reference list, I can reuse those common reference values throughout my subsites.

This works great in SharePoint 2007 with SOAP calls. When I retrieve items with one of these lookup Site Columns from a list in a subsite, I simply get the ID and Title values, separated by a “;#”. However, when I try to do the same thing with “modern” REST calls, I get an error like this:

I’ve been a good team player, and I’ve suggested they fix this on the SharePoint User Voice in my suggestion Enable support for lookup columns in other webs in the REST API. The votes are up, and it’s been a while.

There’s a workaround, but it’s not very pleasant. (The easiest workaround is to simply stick with SOAP calls and SPServices – I’ve done that in several cases in other projects. But SOAP is officially “deprecated”, so…)

Here’s a specific example. The client I’m working with is in financial services, and they issue recommendations on securities. Those recommendations are very standard, and predictable: Hold, Buy, Sell, etc. In other words, perfect to store in a list in the root site called Recommendations. Why not a Managed Metadata column, you might ask? Well, I also wanted to store several other columns in the Recommendations list, like Description (e.g., “The analyst expects the security to outperform their coverage universe.”), a SortOrder value so I could rearrange the values in dropdowns using SPArrangeChoices, and several other fields which drive configuration of some reports. In other words: great information architecture. The values are all consistent across the various subsites, I store them once, etc. Nice setup.

I created a Site Column back in the beginning called Recommendation, which is a lookup into the title column of the Recommendations list (Hold, Buy, Sell, etc.). I used that Site Column in many Content Types defined on the subsite level. Those Content Types are mainly used in a list I’ll call Notes.

In SOAP with SPServices, I can make this [simplified] call:

This retrieves the items and returns nice JSON for me. Because Recommendation is a lookup column, it comes back as something like “1;#Buy” and that’s easy to turn into a JSON object like:

Easy, peasy.

However, when I try the analogous call in REST:

I get the error:

In other words, there’s no way to $expand the Recommendation column because it comes from an other Web, even though that is ideal information architecture!

The workaround, which André Lage (@aaclage) pointed out in my UserVoice suggestion (but I clearly didn’t get at the time), is to simply ask for the Recommendation column’s ID instead. This isn’t obvious at all:

This doesn’t follow the syntax we’d expect: we need to append “Id” to the end of the lookup column’s InternalName. Of course, this just gets us the ID of the item in the Recommendations list; it doesn’t fetch us the Title value, which is what we really want. Because of this, I need to do a *separate* REST call to get the items from the Recommendations list and merge the values in my client side code.

Now, one could argue that this is more efficient. I don’t ask the server to $expand the values across thousands of notes (yes, there are way more than 5000; I’ve written enough about that lately – I may have mentioned it here and here and here and here), so it gets a break. Retrieving the 5-10 values in the reference list (in this case) is no big deal.

But I have a half dozen or so of these lookup columns to deal with in this application, which means a half dozen extra REST calls, plus the code to merge the values. More work for me, but more importantly a longer wait for the application user when they load the page. I believe that poor UX is what has doomed many a SharePoint roll out, and I loathe creating a poor UX myself. In this case, I’ll make it work, but I’d really like to see this change.

Color Codes in SharePoint SPColor Files MUST Be All Caps

With the new world of branding a tenant on Office 365, styling the suite bar and using a theme (aka Composed Look) can take you pretty far toward improving the user experience.

A Composed Look is made up of (potentially):

  • A master page – generally either Seattle or Oslo
  • Theme URL – the spcolor file to use for common classes in a SharePoint page
  • Image URL – an image to use as the background for pages
  • Font Scheme URL – spfont file to use for common classes in a SharePoint page
  • Display order – simply where in the list of Composed Looks yours is displayed

Composed Look Settings

I won’t go into all the details about how this works, but each setting above is somewhat optional: you can use an existing spcolor file, for example, but create your own spfont file. If you want to understand all the mechanics, check out Benjamin Niaulin’s article Step by Step: Create a SharePoint 2013 Composed Look

 

Change the look

Here’s a quick tip about spcolor files. I was tearing my hair out over the last few weeks trying to figure out why sometimes my iterations of an spcolor file were working and other times they weren’t. By not working, I mean that the Composed Look would simply disappear from the Change the look page. I’d revert to my last working version and slowly inch forward again.

It turned out to be something silly and obvious – once you know it!

All of the color codes in the spcolor file MUST be all capitals. So, for instance, this:

would not work, but this:

would. That was *not* easy to spot, but spot it I did.

Don’t let this one bite you; I hope that your Binglage has led you here for a solution and this helps…

Customizing the Suite Bar Theme in your Office 365 Tenant

Part of a good user experience with software is feeling that it is our own. When it comes to SharePoint, which for many people is their Intranet or at least an important work environment, we almost always do some level of branding.

It’s hard to keep up with how we are supposed to brand our Office 365 tenants these days. We have “guidance” from Microsoft that we shouldn’t customize the master pages in our tenants now, and in fact the “modern” “experiences” that are rolling out don’t even use master pages to put together the rendering of the UI.

There are some simple things we can do starting with a vanilla Office 365 tenant to make it feel more like home, though, and they aren’t all that complicated. One of the easiest things to do – and with the widest reach – is to change the theme of the Office 365 suite bar.

By default, the suite bar looks like this:

Suite Bar Default
Making the suite bar your own isn’t that complicated, and this article from Microsoft explains it: Customize your theme in the Office 365 admin center. But I often need to explain these settings to clients, so I figured I’d write up what I tell them.

Here’s what our Sympraxis suite bar looks like:

Sympraxis Suite BarIt’s not fancy or anything, and we don’t mess with any of the components of the suite bar with CSS or JavaScript tricks. We know how to do these things, but it just isn’t worth it. It’s all done by making changes in the Office 365 Admin center.

tip
Note that you need to be a Tenant Administrator to work on these settings. Being a SharePoint Administrator is not enough: you’re changing the suite bar for the entire tenant: SharePoint, Exchange (Mail), OneDrive, even Yammer. I’ve found that the “stickiness” of these changes sometimes varies across these services, but they get there in most cases.

Step by Step Instructions

Navigate to https://portal.office.com/AdminPortal/Home. This is the home page for the admin functions in Office 365. The UI here has been changing frequently, so these screenshots are current in our Sympraxis First Release Tenant as of today. Your “experience” may vary, but hopefully the basic steps will look the same.

Settings / Organization Profile

At the top of the page, you’ll see the information about your organization, such as the name, address, technical contact, etc.

Organization Profile

There is also a section to Manage custom themes for your organization.

Manage themes

Click on the Edit button.

There are a number of things you can change here. I’ll run through them in a little detail.

Select custom logo image

Custom logo imageIf you upload an image here, it will show up in the suite bar in the center.

Custom logo image

You’ll want to make sure the image fits well into the space allotted. The recommended size is “200 x 30 pixels in JPG, PNG, or GIF format, and no larger than 10 KB”. Since this image will load on every page in your Office 365 tenant, you’ll want to make sure the image is the right size and resolution to make it look good and load fast.

If you’d like the image to be a clickable link to something, you can add that in the next field. Since most people use their Office 365 tenant for internal collaboration or as their Intranet, I usually see this link going to the public-facing Internet site for the organization.

Select Background image

Background imageIf you’d like a background image across the entire suite bar, you can upload one here. As above, the image requirements are specific: “1366 x 50 pixels or less in JPG, PNG, or GIF format, and no larger than 15 KB”.

Earlier iterations of this capability came with a set of selectable images. One of the images I’ve seen most often is one with LEGO® tiles. It’s sort of cool, but that sort of image might not be your thing.

Prevent users from overriding custom theming with their own theme

Prevent users

This effectively locks the theme so that no one can override it. Frankly, I’m not sure what this prevents, as we can drop custom CSS into any page which overrides the theme, but here it is…

Set custom colors

Custom colors

Finally, we have a section where we can set custom colors for the suite bar. This is probably the change which will have the biggest impact for your users.

You can change the color from the default Office 365 / Microsoft blue and black to something which is more aligned with your organization’s identity. Even making a little switch like these colors can make your Office 365 tenant feel much more like it belongs to the organization; don’t underestimate the importance of this for the user experience.

You can change three colors here. For Sympraxis, we’ve used our logo’s purple for the Accent color (332F81), white for the Nav bar background color (ffffff), and our logo’s green for the Text and icons (A3A437).

Sympraxis colors

Save your changes

Save changesFinally, save your changes. It will take a few minutes for the changes to take effect across Office 365, but you’ll see them in every application in the suite soon enough.

Note that it’s easy to go back and change these settings or remove them entirely.

Making these changes immediately makes your users feel like the Office 365 belongs to them. They may not even notice the specific changes, but they will feel more at home. Try it out and see!

Save Your SharePoint Online Tenant: The SharePoint Sandboxed Solutions Inspector

If you’ve been following the “code-based sandbox solutions on Office 365” saga, you know that there is little time left to fix your existing sandbox solutions in Office 365. See: Microsoft Is Removing Code-Based Sandbox Solutions in SharePoint Online – Be Prepared!

Last week, Vesa Juvonen (@vesajuvonen) released a script (New Script Available from Microsoft PnP: Generate list of sandbox solutions from SharePoint Online tenant) that can help you find your sandbox solutions. Surprisingly, what was missing from Vesa’s script was identification of the solutions that contain code. You’d get a list of all your sandbox solutions, but not specifically the ones that were going to cause you problems.

The Rencore Team

Some of the great looking folks at Rencore. Where’s Waldek?

My awesome friends at Rencore – the SFCAF folks – were kind enough to make a free tool available this week to help with even more with your diagnosis and even some of the cures. In Erwin van Hunen’s (@erwinvanhunenpost Introducing the Rencore SharePoint Sandboxed Solutions Inspector, you can learn more about the free tool and how it can help.

On August 31st, 2016 Microsoft is going to shut down support for Sandboxed Solutions with code.

Sandboxed Solutions containing code will be deactivated and this might impact your Office 365 tenant big time!

If you know you need it, just head right on over to the download page.

The SharePoint Sandboxed Solutions Inspector

They have already released several updates to the tool, and are keeping it current based on feedback from the folks using it. Now that’s service – and for a free tool!

But I think the best thing is that the Rencore tool can fix some of the most common issues – most notably the “empty DLL” issue that makes Office 365 think you have code in your sandbox solution when you don’t.

We heard “30 days” when all this started, and now people seem to be taking that as August 31. Don’t leave your users in the lurch – get going on handling this situation.

Oh, and if you’re a vendor or consultant who has written a sandbox solution with code over the last few years: reach out to your client and own it. Get them back on the right road and you’ll be the better for it.

Clean report!

Clean report!

Microsoft Is Removing Code-Based Sandbox Solutions in SharePoint Online – Be Prepared!

Though Microsoft announced that sandbox solutions with “code” (this is becoming a more confusing distinction than ever with JavaScript becoming a first class coding citizen!) were deprecated back in 2014, last week’s announcement that sandbox code was being shut off caught many people by surprise.

There was a post that went up last Friday, July 29th, 2016, on the Office Dev Center blog that let us know that they were Removing Code-Based Sandbox Solutions in SharePoint Online. Unfortunately, there’s no date on the post, so quite a few people I shared it with doubted its relevance. But if you go up a level, you can see it was posted on the 29th.

Removing Code-Based Sandbox Solutions in SharePoint Online SharePoint team - Published 07/29/2016

Removing Code-Based Sandbox Solutions in SharePoint Online
SharePoint team – Published 07/29/2016

Here at Sympraxis, we’ve never used Sandbox solutions (client side rulez!), but this quick shutdown seems to be hitting many people hard. I would have thought there would have been  series of reminders, maybe a countdown clock, and some targeted emails to people who are still running this type of solution to help them prepare for the eventuality.

There’s been an active discussion on reddit, confusion in the SharePoint group on Facebook, complaints in the SPYam Yammer network, etc. In other words, the communication either didn’t hit or it hit too late.

Even worse, it seems as though the support people had no idea this was going to happen. As late as last Friday afternoon, this was considered a service issue, with updates coming into the Office 365 Admin Center to explain why it was happening.

Custom Solutions and Workflows - Service degradation

Custom Solutions and Workflows – Service degradation

I really try not to be publicly critical of my friends at Microsoft (I know some of you may find that hard to believe!), but this one could have been handled far better. Microsoft is learning how to be more open, and this is one place where I think they are going to learn some things. There may be good reasons why this shutdown is happening ex post haste, and letting us know what those reasons are would be helpful. With SaaS, whether we like it or not, we’re all riding in the same ship. When it springs a leak, not just the women and children need to know it’s time to head to the lifeboats. And it’s August, when not that many people are running at full steam.

There are clear ways to solve this for your own organization, and good articles explaining how to go about it. But you don’t have a lot of time (we’re hearing as little as 30 days, at least in the rumor mill), so you’d better get cracking!

 

Update 2016-08-01: Check out this post from Dave Feldman (@bostonmusicdave) about getting some non-code solutions to activate: Sandbox Solutions removed from SharePoint Online–Here’s the fix for your Visual Studio developed WSPs to get them to activate. It seems as though the change to SharePoint Online may be blocking some solutions it shouldn’t.

Update 2016-08-02: Apparently, some people are seeing a message like this in the Admin Center. If you aren’t sure if you have sandbox solutions, be sure to check!

MC73347 in the admin center:

We’ve detected that you are using a code-based sandbox solution with your tenant account. Please be advised that we’ve moved forward on our plans to remove code-based sandbox solutions as previously announced in 2014.

As part of the removal process, activation of new code-based sandboxed solutions, as well as updates of existing solutions are no longer available. In approximately 30 days, currently running, code-based sandbox solutions in the SharePoint Online environment will be disabled.

Update 2016-08-03: Be sure to read my follow up post New Script Available from Microsoft PnP: Generate list of sandbox solutions from SharePoint Online tenant