Looping Through Content in a SharePoint 2013 Site Workflow – Part 3 – High Level View of the Workflow

This entry is part 3 of 3 in the series Looping Through Content in a SharePoint 2013 Site Workflow

As I discussed in the first part of the series, my goal is to find all the sales opportunities in a Site Collection and process them based on their characteristics. To do this I’ve created a Site Workflow – SharePoint 2013 flavor – that makes a set of REST calls to traverse the Site Collection to get the items we want to process.

Site Hierarchy
I’m working in a Site Collection with a fairly simple hierarchy, most of which I inherited. There’s a root site, of course, then a number of subsites – one per business partner – and one subsite for application management. The workflow will run in the management site (only Site Owners are allowed in there) and reach out across the rest of the sites in the Site Collection.

Looking at the details of a workflow like this right off the bat is probably a little too detailed, so let’s look at a conceptual picture. If you’ve used SPServices or REST calls to traverse sites before, this may not seem too complicated for you. If not, it may look like a nice picture (maybe – the colors are a little much!), but otherwise not make a lot of sense.

Conceptual overview of the workflowWorkflow Steps

The orange boxes represent App Steps. App Steps are only available to us if we give the workflow app permissions, as we did in the previous article. We need the App Steps so that we can make REST calls into sites other than the one where the workflow is running. When I tried to run my workflow without setting up the app permissions, each of the REST calls (now contained in App Steps) failed due to an authorization issue.

Stage: Get Authorization info

The first thing we’re doing is getting a digest token, which amounts to a way to authenticate the workflow when it does writes back to any of the lists. When we read information in the workflow, we don’t need this authentication info, but we do if we want to write anything. I’ve included the step in my workflow because I’m pretty sure I’ll want to be able to update some of the sales opportunities based on their status.

Stage: Get Webs

The next stage is where we make a REST call to retrieve the list of subsites (also known as Webs) which sit directly below the root site of the Site Collection. To do this, I need to make a call to the Webs REST endpoint. This is why we needed to set the workflow up with app permissions at the Site Collection level.

Stage: Process Webs

Once we have the list of subsites we can loop through them to process them.

Loop: Process Web

This loop contains the logic I use to process anything I need within each subsite. In the first few actions in this loop, I parse out the details about each subsite (title, relative URL, etc.) into variables.

Step: Process Opportunities

Using the variables I set up in the beginning of the Process Web loop, I make a REST call into the subsite to see if the Opportunities list is there and has sales opportunities in it which meet our criteria. (For example, we might filter out the opportunities which are already sold.)

Loop: Process Opportunity

If I do get a list of opportunities back from the call above, I process each of the individual opportunities in the Process Opportunity loop. This loop is where the “meat” of the workflow lives. All of the other steps simply exist to get us here so that we can take actions on each of the opportunities that meet whatever criteria we have defined. As the business process gels more (we’re designing it as we build the workflow), we may end up with multiple loops like this. For instance, we could have one loop which sends out emails about opportunities that are late; another loop to alert the sales team about big opportunities of note; and another loop that bumps up the commission percentage based on the quality of the opportunity, etc.

Building and Debugging

Thinking through how your workflow should work at a high level like this is sometimes useful. I’m more of an iterator, though. I tend to build one part of things and test it, then bolt on some more logic and test it, etc.

One thing I’m a huge fan of is emitting some sort of debugging messages along the way. This probably harkens back to the good old days where that was the only debugging technique we really had.

In any case, each of the phases, steps, and loops above emits something into the workflow history list using the Log to History List action.

Log workflow messages

As I build up the workflow, having a clear log of what’s happening in the workflow history list is a great debugging tool. The screenshot below shows the kind of thing I tend to do.

Workflow log messages

In this case, I can see that I found 15 subsites. Then when I processed the first subsite, I found 3 opportunities to work with. For each opportunity, I can see the customer name (this is test data, of course!), and so on.

The workflow will get more and more complex, but by emitting these messages, I have a fairly decent way to see what’s going on. Debugging these workflows is not a lot of fun – it’s slow going and sometimes hard to see what’s gone wrong – but at least I can look at my log and see where things might have failed.

In the next few articles, we’ll look at each of the phases, steps, and loops I’ve outlined above to see what’s going on in more detail.

SharePoint: Enabling Knowledge Management

I had the chance recently to have chat with TechnologyAdvice’s Josh Bland (@JoshBlandTA). This interview was a part of the TechnologyAdvice Expert Interview Series. The series explores a variety of business and technology landscapes through conversations with industry leaders.

There’s plenty of great content there, including interviews with some of the speakers for the SharePoint Technology Conference – SPTechCon Austin – coming up February 21-24. If you register with code ANDERSON, you’ll get an extra $200 off any other discounts you might receive.

In this episode we discussed how I see SharePoint fitting into a successful knowledge management strategy, how modern work happens, big things that happened with SharePoint in 2015, and of course, the upcoming SPTechCon. If you enjoy this interview, you might want to check out the one I did with Josh last summer, just before SPTechCon Boston: Software Adoption: A People Problem, or A Technology Problem?

Below is an excerpt from the conversation:

Josh: What would you say, Marc? How would you sort of summarize the year? What sort of big developments occurred and what would you say were some of the highlights from 2015 in SharePoint?

Marc: One of the biggest things is that the hybrid story that Microsoft is just telling is getting richer and richer. We saw some capabilities that are available on Office 365 starting to filter back on Premises, and that’s via hooking up to a SharePoint online instant. So that you can sort of take advantage of the best of the cloud but still keep your content in-house. And of course, we’re seeing tremendous strides forward in Office 365 as well. It’s hard at this time of the year to look back and try to remember, how many of these things actually got stuffed into this one year? And it’s pretty incredible when you look at how many things have changed over the last year in the SharePoint space. We have things like Delve and groups and the video portal, and all these things that have really come into their own this year and shown that Microsoft is not sitting still, that they’re — I’m very bullish on Microsoft in what they’re doing with Office 365. And I’m not an easy one to convince [chuckles]. But I see them doing some very cool things – Planner – just all kinds of new experiences as they say, with fantastic user interfaces that really are bringing the whole platform ahead in leaps and bounds.

Josh: I want to hear what were some of the sort of difficulties that you have experienced, not only with your customers, but just in general that you think the industry has seen from SharePoint?

Marc: An ongoing challenge for anyone using these platforms is that the development story continues to evolve. And that’s not a bad thing, but for your average developer who’s trying to sort of swim with both hands tied behind their back, sometimes, keeping ahead of what the enterprise wants, it’s difficult because we look at the different– historically we’ve had features and solutions, we’ve had the sandbox model now. We have the apps crossed out, add-in model, and we’ll probably see some evolution from here. People are continually having to keep their skills fresh. Now, that to me is a damn good thing. Anybody who stops learning and thinks that what they know now is going to serve them forever is sort of letting themselves die intellectually. The fact that we have to learn new things all the time is a good thing. We’re seeing an evolution from server site code toward JavaScript and client site code to a degree that is really — it’s been a bumpy ride for some people over the last couple of years to make that mental switch. As we’re seeing better experiences coming out of Microsoft, we’re seeing these challenges for the developers inside enterprises who are trying to build bespoke functionality that is unique to that organization, or something that that organization wants that Microsoft does not provide exactly. It’s that 20% that’s actually the hard part. That’s going to be an ongoing set of challenges – understanding how this hybrid model, as you mentioned fits into the organization’s ethos – is going to be a tough one for a lot of people to get to.

Two years ago we would have been talking about the tremendous fear about security and, “Does the cloud make sense?” And, now we always talk about it as, “Which part of the cloud makes sense to me? Which parts should I take advantage of? Where am I going to get the bang for my buck?” So, we’re on a different part of the learning curve for all of that. But the development story is tough. And that hybrid aspect just adds a little bit of complexity to the whole thing.

Josh: One final question here on SPTechCon 2016. Do you see a lot of differences between Boston and Austin? All that to say, what do you think will be different about this year, Austin 2016 vs. Austin 2015 or Boston 2015?

Marc: The obvious difference is that we’ve got SharePoint 2016 coming up. So — Yeah, new software smell. There are a couple things there. One is obviously there is some new software out there; new bits. So, people will be able to hear some good information about what’s there, why would you be interested in it, how does it work and that sort of thing. One of the best things though about SPTechCon is that they’re very careful to manage the mix between: here’s the shiny new stuff and here’s that stuff that you are actually still stuck working with [chuckles]. There are still a lot of people in the crowd who use SharePoint 2010, there’s still a lot of people who are using 2013 obviously. There’s a great mix of content across that spectrum as opposed to just going for the new stuff. Go to the Microsoft conference for that. They’ll only talk about the new things. But SPTechCon has been traditionally very good about having that balance, so that everybody who comes to the conference gets a lot of depth out of it.

This podcast was created and published by TechnologyAdvice, an Inc. 5000 company looking to help buyers find the best cloud storage, payroll systems, and more. Interview conducted by Josh Bland.

Looping Through Content in a SharePoint 2013 Site Workflow – Part 2 – Setting Up App Permissions

This entry is part 2 of 3 in the series Looping Through Content in a SharePoint 2013 Site Workflow

Site Workflows are great, but if you want them to reach across into other sites you’ll probably have permission issues. By giving these workflows App Permissions, you’ll be able to let your workflow access content across your Site Collection.

Add XML

In part one of this series I described the basic task we’re trying to solve. We want to create a SharePoint 2013 site workflow that can iterate through subsites and then take some actions on each item that meets some criteria. At the end of that article, I said that the first step is that we’ll set up the workflow with proper permissions so it can traverse content across the Site Collection. So now you’re thinking, “Great. The problem is defined. Let’s pop over to SharePoint Designer and create a workflow.” Unfortunately, that isn’t the right thing to do. I know because I tried that, and I’m writing this series to pass on the lessons learned.

Before we can even think about writing the workflow, we should set up the authorization side of things. It wasn’t enough to get the digest token from the server for the REST calls. (More on this later.) When I tried to make the REST calls I needed from one subsite into the root of the Site Collection or another subsite, I got authorization errors.

Instead, there’s more heavy-duty permissions set up required to make the REST calls into other Webs (sites to normal people). We need to give our workflow App Permission, which isn’t obvious as you’re working on the workflow in SharePoint Designer (SPD), nor is it possible to do so in SPD.

Luckily I have good friends in the community. Matt Bramer (@ionline247) was the first person who replied to one of my tweets, and he gave me some good tips. Through him I found that Fabian Williams (@fabianwilliams) had written several blog posts about the trials and tribulations he went through so that I  wouldn’t have to. (I added links to Fabian’s blogs at the end of this article.) But setting up workflow App Permission is still confusing, in my opinion.

You’ll go through two two main steps to set these permissions. I found the steps generally documented in the MSDN article, Create a workflow with elevated permissions by using the SharePoint 2013 Workflow platform, but here’s my version.

Step 1 – Allow the workflow to use app permissions

Go to Site Settings in the site where you want the workflow to run and turn on the Workflows can use app permissions feature. You can do this by going to Site Settings and in the Site Actions section, choose Manage Site Features. Toward the bottom of the list of available features, you’ll see Workflows can use app permissions. To activate it, (figure 2) simply click the Activate button and wait for the activation to complete.

Workflows can use app permissions

Figure 2: In Site Settings / Site Actions / Manage Site Features you need to Activate the feature to allow workflows to read from and write to all items in the site.

Step 2 – Grant full control permission to a workflow

This step gets pretty weird. We have to give this workflow the permissions to run as if it’s an app add-in. (Though one might say that it’s an app add-in already – everything is an app add-in, right?)

In the site where you want the workflow to run, navigate to Site Settings and then under Users and Permissions, go to Site app permissions. If things have gone according to plan, you should see Workflow as one of the two or more items listed here.

What you want to grab is the yellow text in figure 3. It follows a bar character – “|” – and precedes an “@” character. I’m not sure exactly where this GUID comes from, but it’s what you need.

2016-01-12_11-05-44

Figure 3: Copy part of the Workflow App Identifier

Now you have to navigate to a hidden, super-secret page at:

http://yourtenant/sites/yoursitecollection/_layouts/15/appinv.aspx

In the example on MSDN, they send you off to the /sites/AppCatalog Site Collection for this. That’s only because they are showing an example interacting with a list there. You should go to the secret page in the Site Collection where you want the workflow to run. (Yes, I was down a rat hole on this for a while. Maybe I was just being stupid. It seemed like the good folks who wrote the MSDN article were telling me I needed to give permissions on the App Catalog site for this to work – but that wasn’t the case.)

The should look something like figure 4. The page doesn’t have a title like most SharePoint pages (hey, it’s super-secret), but the hover text for the info icon at the top of the page says “Grant permission to an app.”

Set the app permissions

Figure 4: You’ll paste the Workflow App Identifier into the Add Id field on the secret page.

Now as you can see in figure 5, you paste in the yellow text from above, and click the Lookup button. The next three fields will be filled in. You shouldn’t need to change anything in those fields.

Paste in the App ID

Figure 5: Paste in the App Id that you copied from the workflow in Site app permissions (figure 3).

Next, you need to paste some XML into the last field called Permission Request XML. Don’t try to get all smart about it like I did: paste in EXACTLY this XML— don’t make any changes.

You have two options here (the first option adds /web in line three) and I’ll explain the difference below the XML snippet:

<AppPermissionRequests>
  <AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="FullControl" />
</AppPermissionRequests>

…or…

<AppPermissionRequests>
  <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
</AppPermissionRequests>

In this case, I went with the latter; I want my workflow to have permissions on the entire Site Collection. The former would give my workflow permissions only in the subsites of the Site Collection; I needed permission on the root of the Site Collection because I wanted to iterate through the subwebs (subsites) from there. So with both the App Id and the XML pasted, the app identity and permission look like figure 6.

Add XML

Figure 6: We’ve identified the workflow’s app’s identity and indicated where it has permissions to be used.

Click Create.

Next you’ll be asked if you trust the workflow. One would think this would be a given, but click Trust It.

 Site App Permissions

At this point, you have the permissions set up that give you the possibility of looping through the subsites off the root, even though you haven’t created an actual workflow yet. (Or, if you’re like me, you beat your head against the workflow first and then realized you had to figure this part out.)

In the next article, we’ll talk through what the workflow needs to do on a high level to be useful.

References for this step

This article was also published on IT Unity on 1/25/2016. Visit the post there to read additional comments.

Looping Through Content in a SharePoint 2013 Site Workflow – Part 1 – Introduction

This entry is part 1 of 3 in the series Looping Through Content in a SharePoint 2013 Site Workflow

Site Workflows let us run logic independent of lists and libraries. By combining them with REST calls, we can traverse a Site Collection and process multiple lists and their contents by looping though them when we want. In this article, part 1 of the series, we’ll describe the problem we are trying to solve, but the solution can apply to many SharePoint scenarios.

SharePoint 2013 Site Workflow

As with most things SharePoint, setting up a workflow to loop through items across subsites seems like it should be easier. What I was trying to do seemed like a common requirement: get some data from lists in SharePoint subsites and loop through it, taking some actions on each item that meets some criteria.

I’ll warn you that this series of articles is not for the faint of heart, the new-to-workflow initiate, or the person without a lot of permissions. My solution involves reaching into Office 365’s workflow guts a bit here.

In my current case, I’m building an opportunity management solution for a sales organization. Each sales partner – these are outside companies – will have a subsite only they can access where they will enter sales opportunities as they arise. The sales folks for the host company want to be able to manage that “basket” of opportunities as well as send out emails to the partners as the opportunities move through various stages in a process. We won’t cover all of the possible permutations here as I want to focus more on the chassis of the workflow.

I need to check a specific list in a bunch of subsites for new items and send an email about each. I don’t want to create a separate List Workflow that sits in each of the lists because it would be an administrative nightmare to change the workflow down the road (we have over a dozen subsites, and the number will grow). Instead, I want to create one central SharePoint 2013 Site Workflow that finds all the subsites, finds the list in each subsite (if it exists), and iterates through the items in the list, taking different actions based on the data in each item. We’ll set this workflow to run every x minutes (probably once an hour or once a day) by adding a wait state at the end of processing.

You could accomplish this with “code” or Powershell, but that takes it out of the hands of the Site Admin who might want to change things down the road. By using a workflow, we can give them some control over things like the text of the emails without making them learn how to code to change them. As a consultant, I want to be able to leave my client with a solution they can maintain on their own as much as possible. They can still call me for the heavy lifting, but most changes should be something they can handle themselves.

On Office 365 in SharePoint Online, this feels a LOT trickier than it may have been in the past because of the complexity (nay, swamp) of authorization we must navigate. I expect that similar authentication issues may apply on premises, too, but Office 365 is more of a moving target.

In this series of articles, I’ll show you the steps I went through and let you know where I got stuck, waylaid, and confused. I believe that only one of us should have to suffer through this stuff so that everyone else can avoid it.

We’ll go through the overall concept of how the process works, as well as:

  • Setting up the workflow with the proper permissions so it can traverse content across the Site Collection
  • Initializing the workflow based on values stored in a user-maintainable list
  • Making the REST calls to discover all of the existing subsites without having to hard-wire where they are
  • Parsing the data returned from each REST call
  • Discovering whether each subsite contains the list we are looking for
  • Processing the items in each list based on a set of criteria that can change over time
  • Sending out emails appropriate for each set of criteria

This isn’t totally new ground; there are many blog posts out there that cover bits and pieces of this, and I’ll provide the links I found helpful and give credit where credit is due. I didn’t find anything that showed me how it all worked together, though.

Thanks right off the bat to Matt Bramer (@ionline247) and Fabian Williams (@fabianwilliams) for getting me up and rolling when I first start whining about this on Twitter.

This article was also published on IT Unity on 1/19/2016. Visit the post there to read additional comments.

SPServices into 2016!

SPServicesOver the break, I managed to get SPServices up and running on GitHub. You may wonder what that means and why it matters.

When I started moving SPServices away from Codeplex, it was due to several things:

  • Codeplex seems to be dying a whimpering death. Microsoft is clearly not maintaining it anymore, and its tech is falling behind – when it works.
  • I wanted to understand GitHub better. I’ve had to get too much help from my friends in the SharePoint community even just to push changes to cdnjs. Thanks Josh McCarty (@joshmcrty)!

So, mission accomplished on those goals – but it took me an embarrassingly long time to get here!

Of course, in the process I also wanted to improve SPServices. Here is what I am/was aiming at:

  • The next version will be SPServices 2.0 – This is just a number, and the current builds have this number already.
  • Make it (already is!) AMD-enabled using RequireJS
  • Convert from a monolithic file to modules (done, though perhaps more to do)
  • Enabled to take advantage of SharePoint’s REST APIs – where available – for internal calls to get list data in the value-added functions (still to do)

As you can see, I’m well on the way to meet my self-imposed goals; I think the hard work is behind me.

There are some other goals and here’s where you can help:

  • Test the “pre-alpha” builds of SPServices 2.0. If you’re familiar enough with the library to drop builds into your test environments, that would be a great help. I’ve tested using the same lists and pages I always use, but more real-world testing would be good. Report any issues you find using GitHub issues.
  • Write some tests. I’ve started writing tests with QUnit, but I’ve only scratched the surface. Writing good tests here is difficult, as we have to be sitting on top of SharePoint; in greenfield development, we can test anywhere. You’ll find some instructions for how to use the existing tests on the GitHub pages.
  • Migrate the documentation from Codeplex to GitHub – Since Codeplex is falling apart, there’s no reason to leave the documentation there, either. There are a few dozen pages (I can’t actually count the pages on Codeplex easily) of documentation, and it’s probably easiest just to move the over to GitHub manually.
  • Move the discussions off Codeplex – This one is hardest, I think. IMO, one of the big values to SPServices is the historical discussions about how to use it. But those discussions have covered many other things as well, and I’d hate to lose any of it. I’m not sure how to go about this, so if anyone has some experience moving forums like this, I’m all ears.
  • Propose improvements – I ask the community for suggestions all the time, but I don’t get a lot of them. If you’ve solved some gnarly SharePoint UI problem and would be willing to submit your code or just wish that someone would fix the darn _____, then let me know in the GitHub issues. Consider the issues our own UserVoice for SPServices.

A HUGE thank you goes out to Paul Tavares (@paul_tavares) for guiding me along this move to GitHub. He’s written most of the build code, the test chassis, etc. I couldn’t have done it without him. If you don’t know Paul, he’s one of the sharper tacks in the SharePoint community, and he only does SharePoint work on the side these days. Take a look at his SPWidgets project. I promise you – you’ll be impressed.

Thanks also to the folks over at BitHound for their code tools. If you check out the dashboard for SPServices, you’ll see that I’m doing OK, but there’s always room for improvement.

Happy 2016 to all of you, and happy coding!