Exclude Specific Content from SharePoint Search Results

This is a quick one, and I’m only posting because it took me a while to find the answer with my Binglage. It’s one of those things I know I can do, but I forget the syntax.

Image from pixabay

Most of the answers I found for this involved going into settings (or running Powershell or writing code – overkill!) to exclude content from results semi-permanently. That’s not what I want in this case.

Say you want to search SharePoint for a specific phrase. In my case today, we want to change the name of the Intranet, so I was searching for the old Intranet name. This would give me the places where documents or pages contained the old Intranet name so I could change it in the content.

When I simply searched for “Old Intranet Name” in Everything, I was getting back a whole lot of documents we put together as we were building the Intranet. The phrase in those documents was irrelevant, so I really wanted to exclude those documents from my search results.

The trick is to exclude the specific path (or any other useful attribute value) so that I reduce the result set down to what I really want. It works basically the same way most search engines do; if you put a value after a dash character, content with that value will be excluded from the results.

The trick is that you need to know what the specific attribute – which in SharePoint is a Managed Property – you need to use for the exclusion to work correctly. In this case, I wanted to exclude an entire path, meaning any document which lived at that URL or below.

While I’m working in SharePoint Online today, this works going all the way back to SharePoint 2007, I believe.

Good search skills are critical in today’s world, so knowing how to exclude content is just as important as including it.


It so happens that Brian Jackett (@BrianTJackett) just tweeted something analogous, so here’s a good reference that shows more possible options.

Reference

Advertisements

SharePoint Online Search Isn’t Displaying What I Expect – Part 1 – Trimmed Duplicates

This entry is part 1 of 1 in the series SharePoint Online Search Isn't Displaying What I Expect

For years, I’ve been skeptical about search indexing in SharePoint, especially in SharePoint Online in Office 365. The fact that we can’t know when a search crawl has run – thus updating the indices – is a huge part of the problem. In the early days, before Content Search Web Parts (CSWPs) were available in SharePoint Online, we routinely saw delays between content creation and that content showing up in search results of days or even weeks. Later the CSWP was enabled on SharePoint Online, and it is a fantastically powerful tool, far better than the Content Query Web Part (CQWP) which it nominally replaced.

But the value of any search-driven mechanisms in SharePoint is directly tied to the recency and frequency of updates to the search index. While the CQWP is quite inefficient – since it actually goes out to look for content at the source every time it runs (though there may be some caching) – the CSWP uses the search index and can thus return results using fewer server resources in some cases. (One downside is that you can only retrieve up to 50 results with the CSWP.) Since we don’t know when the search crawls run in SharePoint Online, and we often seem to not see the results we expect, we tend to blame to indexing for the problem.

There are many things that can contribute to the indexing issues. Load on the indexing servers can mean that your tenant isn’t crawled as frequently as you might want – taking hours or in the worst cases days to display items you know should be there. Unfortunately, there is no way to know if the index is the issue. Based on what I’ve heard, there are usually multiple indexing servers per tenant, and those indices can supposedly get out of sync. Search is also a very complex beast: probably way too complicated for most use cases, as it is based on the old FAST search platform. Most people simply want to be able to see content they add to a SharePoint list or library right away if they search for it. Period. So simple, yet often not what happens.

The other day, Julie and I were certain we had an ironclad instance where search indexing simply wasn’t working correctly. The scenario seems to be a very common one, and we stripped it down to as minimal an example as possible and sent it off the the Product Group.

In the example, we were making a REST call to the search API:

to retrieve all of the events in calendars across an Intranet. The use case is a very common one: we have a calendar per department, office, etc., and in some cases we’d like to promote those calendar events to display on the home page of the Intranet. Put aside the governance questions here, and just assume that everyone who can create events gets to decide whether to check the Show on the home page box. To make this all work, we have a few custom content types which inherit from Event with a few extra columns. We have some nice, fancy display mechanisms on the home page using AngularJS and on a Company Calendar page using fullcalendar. But most of that doesn’t matter: we were seeing the issue in the call to the search API.

Our query looked something like this:

This query will return all (we thought!) list items which inherit from the Event Content Type, because its ContentTypeId is 0x0102. Of course, our actual query was more complicated: we requested specific fields with selectproperties, asked for more items by setting the rowlimit to 50, etc. But again, at the core we were simply asking for a bunch of events.

But we weren’t seeing all the events we expected. We assumed that the search index wasn’t working correctly, just like most site admins would.

There was a series of meetings going on about some HR changes, and the company was giving employees a set of webinars from which they could choose one to attend. The events were at four different times during the day. In our call to the search service, we were only getting one of those events. It happened to be the first one added to the calendar; all events had been added over 24 hours before.

When we tried searching for the title of the events in the regular old search box, we still only saw one result. At least that was consistent, and it showed we weren’t doing something stupid in our REST call. I’ve had to blur a number of things out in this screenshot, but here’s what we saw on the search results page. In this case, the results were coming to us in /_layouts/15/osssearchresults.aspx for the particular subsite where the events were, but it didn’t matter if we tried using the search center.

I included Mikael Svenson (@mikaelsvenson) in my email to my Product Group friends because if there’s something about search Mikael doesn’t know, it isn’t worth knowing. I probably should have just asked him in the first place, but we truly believed we had found a bug.

[amazon_link asins=’B00RO84UBQ,B010EUVCUM’ template=’ProductCarousel’ store=’madansbl-20′ marketplace=’US’ link_id=’0ea8b883-d8e4-11e6-8774-93ebaa8aaace’]

Mikael spotted that all four of the events we expected to retrieve had the same title. This isn’t so unusual: a couple of meetings on a given day with the same title. Maybe we have five interview slots set up for a new candidate, or have several different times when the Red Cross is running a blood drive on the same day, or exactly the example we have here.

We probably should have realized something was wrong when we looked at the search results above. As you can see below – now that I’ve highlighted it – there were three plus one items shown in the histogram for the Modified date.

It turned out that because the four events were so similar, search was considering them duplicates. Of course they weren’t duplicates to us: each is a unique event with its own value to end users.

By adding trimduplicates=false to our REST call, we were able to retrieve all of the events we expected. It was a very simple fix, but given the black art of SharePoint search, not necessarily an obvious one. Perhaps we should have known better, but I don’t think this is an unusual problem. Add to that the fact that the standard SharePoint search results UI doesn’t give you any way to see the duplicates, and I think there is a significant issue.

I’ve made a suggestion on the SharePoint UserVoice to Allow easier management of the trimduplicates setting.

When we search for content, we often (some might say “usually”) need to see all results for our search query. It seems that by default, trimduplicates is set to true in SharePoint Online. This seems to be true in the search Web Parts and in the search API.

My suggestion is that we have far better and clearer ways to manage this setting, including:
* An easy toggle in the Content Search Web Part (CSWP)
* A clear way on the search results pages to choose to show duplicates where there are any. This was present in earlier versions of SharePoint, and I’m not sure when it was removed.

Deciding when duplicates are appropriate is a complex thing, and it varies greatly by use case. Giving people setting up SharePoint pages simpler control over the setting will both help build compelling user experiences on the platform and help confirm that search is indeed working properly. When someone searches for something they know is there and it doesn’t show up in search results, it undermines faith in the entire platform.

If you feel that my ideas have merit, please vote this suggestion up! Suggestions at UserVoice with enough votes truly do get the SharePoint Product Group’s attention.

If you find yourself in a situation like this, there is a tool that can be helpful to solve whatever might be going on. If you do much work with search, check out the SharePoint Search Query Tool on Codeplex. Mikael has contributed to this tool and it basically allows you to issue REST calls through a UI that “understands” SharePoint search very well.

In the screenshot below I’ve done a search against our Sympraxis tenant using the querytext='test'. That’s certainly nothing fascinating, but it points out a few of the useful aspects of the tool:

  • Simple querytext configuration
  • Easy on/off switches for the various query options; in this case unchecking Trim Duplicates was the winner.
  • An easy way to see the effect of your settings on the REST call on the URL (right at the top of the screen)
  • On the right side, a clear view of the results returned, using a number of useful formats. If you’ve written JavaScript to parse search results, you’ll know that this is really helpful.

As a little bonus, here’s the key JavaScript we use to parse the RelevantResults table from the REST call to the search API. Because we’re requesting items  which are all based on the Event Content Type, we can treat all search results the same way. In this example, we’re using AngularJS and jQuery, preparing the data for use with fullcalendar, as I mentioned above.

Hopefully this post gives you a little more insight into the inner workings of SharePoint search. To me, these little eddies and backwaters of search are what turns it into a black art. I’d love to see things get even simpler that the so-called “Quick Mode” in CSWPs.

Thanks again to Mikael and the Product Group folks who engaged on this with me. Notice that this is Part 1 – I expect to add more entries to this series.

References

Mikael pointed me to these two articles on duplicates and “shingling” in case you’d like to understand the underlying principles more fully.

Mikael’s post about duplicate trimming

SharePoint Search Query Tool

SharePoint UserVoice suggestion:

Create a Simple SharePoint 2013 Employee Directory on Office365 – Part 5 – Sorting & Refiners

This post in the series was sitting in my “outbox” for a long time. Sorry to people like Nigel (@Nigel_Price) who probably gave up on me!

In a company with 100 employees, clicking on once of the letters are the top of the page will probably narrow things down well enough that you’ll see everyone with that letter at the start of their last name on one page. If you have many more employees, you may end up with too many people to see very easily. And since we’re limited to 50 results in the Search Results Web Part, you may need to page and/or scroll a lot, depending on what properties you’ve decided to show for each person (one line vs. two lines or more, for example).

There are a couple of out of the box capabilities we can use to sort or filter our results a bit more. Because we’re using search to drive the directory, we have all of the native search capabilities that SharePoint gives us. Let’s take a look at adding some sorting and improving the refiners.

Sorting

In the Web Part Properties for the Search Results Web Part, there is a checkbox labelled “Show sort dropdown”. It’s in the Settings section (which is full of other options as well).

2015-02-26_12-04-17

By checking the box and adding some of our own JSON into the field below it, we can control what sort options are available. The JSON should be an array of sort options taking this form:

{
    "name": "Last name (A-Z)",
    "sorts": [{
        "p": "RefinableString00",
        "d": 0
    }]
}

The values break down like this:

  • name – The text value you’ll see in the dropdown
  • sorts – The parameters for the sort that apply when the option is selected
    • p – The managed property to sort on
    • d – [0 | 1] If 0, the assort is ascending, if 1 it is descending.

Note that for the Last name sort above, I’m using the RefinableString00 managed property I set up in Part 4. If we tried to do this with the LastName managed property we’d get an error because that property is not Sortable.

Here’s a pretty basic set of sorting options that should at least get you started. I’m giving you four pretty basic sort options:

  • First name (A-Z)
  • First name (Z-A)
  • Last name (A-Z)
  • Last name (Z-A)
[{"name":"First name (A-Z)","sorts":[{"p":"FirstName","d":0}]},{"name":"First name (Z-A)","sorts":[{"p":"FirstName","d":1}]},{"name":"Last name (A-Z)","sorts":[{"p":"RefinableString00","d":0}]},{"name":"Last name (Z-A)","sorts":[{"p":"RefinableString00","d":1}]}]

You should be able to take this and enhance it for your own needs.

Refiners

Next we probably want to adjust the refiners a bit. Out of the box, the Refiners Web Part will display properties that look to a machine like useful ones, but they may not be the most useful in your organization. You may want to show refiners that are part of the search results already or you may want to show refiners that aren’t visible in the results.

Changing which refiners you see and in what order works like this:

  • Edit the page
  • Put the Refinement Web Part into Edit Mode
  • Click on Choose refiners
  • Change the settings
  • Save it

The key action here is to add or subtract from the Selected refiners in the Choose refiners dialog. For each refiner, you can change the settings on the bottom of the page. Note that refiners use Display Templates as well. I’m not going to go into building Refinement Display Templates, but you can see them in _catalog/_masterpage/Display Templates/Filters.

In my case, I’ve got refiners for Department, JobTitle, PeopleKeywords, and BaseOfficeLocation. You will probably want some others which work for you.

Refinement configuration

You should see the impact of these changes immediately upon saving the page. Assuming that the properties you have selected have been indexed, that is.

What Next?

There are a number of different directions we could go from here. We could build a similar page to show a Site Directory [Nigel]. We could enhance the display of each person to show additional information about them or metrics about their recent activities. We could add as second alphabetical listing row below the first so that if you selected the letter B, you could the choose the next letter (only lit up if there are people lurking there). As with most functionality like this, coming up with the innovative ideas can be even harder than good execution on them. In this case, some of the limitations in Office 365 make it even more difficult that it should be.

I’d love to see:

  • Ability to trigger a full crawl of the User Profiles when we change the schema. It wouldn’t have to happen right away, but it would at least remove the need for Powershell.
  • Allow us to build a more intelligible set of “RefinableType” properties so that we can manage them better. If we create more than a handful of these special mappings – as most organizations will do over time – then management is going to be cumbersome at best. We can’t even search by the Alias names.

Then basic process here should work for many different use cases. As you’ve probably gleaned from this series, building good metadata and taxonomy isn’t just an academic exercise. The whole point is to make content easier to find, and a People Directory is no different: it’s a customized page that allows us to perform a certain used case in an optimal way. What “optimal” means is going to vary based upon the characteristics of your organization.

  • How big is it?
  • Why would people be looking for each other (beyond the simple phone list idea)?
  • What information is important for people to see vs. that which must be secured?
  • etc.

I hope you’ve found the series useful. If you’d like to know more or have other ideas, I’d love to hear about them in the comments below.

References

An Unpleasant Refiner Bug in SharePoint Online

As always, I don’t know where to report issues with Office 365. So here I am carping in a public channel in the hopes that the right person sees it. This also may happen with SharePoint 2013 on premises, but I can’t recall.

Unpleasant SharePoint search refinerWhen I use a refiner that is based on a Lookup column, I’ll often see something like the attached image. As you can see, there are two clients listed: Anchor Glass and 29. Anchor Glass is the title of the item in the Clients list with the ID of 29, but it still doesn’t make sense (especially to those poor users). It’s as if the indexer isn’t smart enough to handle Lookup columns properly – but only some of the time.

This happens on multiple tenants, with different Lookup columns to different lists, but not consistently, or in any pattern I’ve been able to discern. Generally it’s the Title column we’re looking up into. Using a Lookup column into a list is almost always preferable to using Managed Metadata (IMO) because we can store additional data about the values in the list. For instance, in the Clients list above, we have the client contact info, who works with the client, when we started working with them, what the billings are, etc. Managed Metadata is worthless for that.

But the bug.

Has anyone else seen this? Any ideas on how to work around it?

SharePoint Online Search: Add a Refiner for Content Type

2015-12-21_14-00-52This is yet another in the “Things that Make You Go Hmm…” category. If I only had a nickel…

I would think that one of the most common needs on a search results page would be to add a refiner by Content Type. After all, we all have robust, sensical information architectures in place to improve performance in our organizations, right?

When we create a new search results page in a Search Center, we get a few refiners out of the box that look oh-so-promising.

First there’s ContentTypeId. We all know that Microsoft uses big, ugly ids and GUIDs under the covers all the time. Sometimes they translate easily into whatever they represent. This isn’t one of those times.

2015-12-21_13-37-48

Well, what about contentclass? That looks sort of almost useful. But it isn’t. Values like STS_ListItem_DocumentLibrary won’t really help anyone.

2015-12-21_13-37-35

Oh, I know. Let’s add the ContentType Managed Property. That’s what works almost everywhere else. Well, except here, as the values don’t make sense to humans, as they are things like application/pdf Document or application/vnd.openxmlformats-officedocument.presentationml.pres… Yes, the ellipses are appropriate here. Who know what that means?

2015-12-21_13-38-05

Finally, I found a blog post from Henri Merkesdal that retrieved my sanity. There’s a property way down the list called SPContentType that does exactly what we want.
2015-12-21_13-38-20

Ahhh. That’s much better. Thanks, Henri!

2015-12-21_14-00-52