Using ShareGate PowerShell to Download SharePoint Content
You might be surprised by the title of this post. Isn’t our goal to get everyone’s content *into* SharePoint? Usually that’s the case, for sure.
But many times, in the course of a migration from an older (usually on premises) version of SharePoint, we identify whole sites or branches of subsites that simply don’t have any real purpose anymore. In some cases, we migrate the content into SharePoint Online, maybe into a site called Archive or similar. But other times we just want to save the content somewhere *in case* someone needs it. Which approach you take may hinge on your organizational culture, what storage mechanisms you have available, and the relative importance of the content – among many other things.
I’ve worked on a PowerShell script for this recently and used it for several different clients. With it, I can point at the top level of a branch of sites – maybe something like /sites/HR or /departments/Finance, and the script will download the content from that site, and recursively from all its subsites, etc. into a network location of my choosing, say C:\ or Z:\Archive.
I’ve run this script with both SharePoint 2010 and SharePoint 2016 as the source version, so I expect it will work for you in most cases.
The script is pretty straightforward because I use ShareGate’s PowerShell module to do all the heavy lifting: yet another reason why ShareGate rocks! If you have ShareGate installed on your machine, you have their PowerShell module as well, even if you didn’t realize it.
Note the ShareGate module requires you to be running PowerShell 5.x, NOT the more current PowerShell 7.x.
The script comes in two pieces:
- downloadSite.ps1 – The script I actually run, after setting three variables appropriately.
- downloadSiteFunctions.psm1 – A module with functions I call above. If you want to do recursion, you’ll need functions of some sort, and using a separate module gives some flexibility for reuse.
The parameter settings I show in the downloadSite.ps1 below were what I wanted for a particular scenario. When you read through downloadSiteFunctions.psm1 below, you’ll see what the impact of those settings is.
Import-Module -Name ShareGate # Requires ShareGate to be installed on the machine
Import-Module "./PowerShell/downloadSiteFunctions.psm1" -Force
# Setup
$sourceSiteName = "Name for the downloaded folder"
$sourceSiteUrl = "https://FarmOrTenantName/siteName/"
# The downloads will end up in a folder here named $sourceSiteName
$destTop = "Z:\" # Be sure to include a trailing backslash
# Any list or library in this array will be excluded from the downloads
$exclusionLists = @(
"Content and Structure Reports",
"Master Page Gallery",
"Reusable Content",
"Style Library",
"Web Part Gallery",
"Workflow Tasks",
"Microfeed",
"Site Pages",
"Site Assets"
)
# Process root web
# Delete existing folder - we assume we want to start from scratch
Remove-Item `
-Path "$($destTop)$($sourceSiteName)" `
-Recurse `
-Force
# Create new top-level folder
$top = New-Item `
-Path "$($destTop)$($sourceSiteName)" `
-ItemType Directory -Force
# Export lists
Export-SympLists `
-ParentFolder "$($top.FullName)" `
-WebUrl $sourceSiteUrl `
-Versions $false `
-ExclusionLists $exclusionLists `
-KeepEmpty $false `
-KeepLists $true
# Process subwebs
Get-SympSubwebs `
-ParentFolder "$($top.FullName)" `
-WebUrl $sourceSiteUrl `
-Versions $false `
-ExclusionLists $exclusionLists `
-KeepEmpty $false `
-KeepLists $true
The downloadSiteFunctions.psm1 file might seem complicated, but it’s just two functions:
- Export-SympLists – Exports all list/library contents from a web using ShareGate PowerShell functions. The ShareGate function
Export-Listis a workhorse. It exports ALL lists and libraries to folders. Even better, it creates an Excel file containing the inventory with all the metadata as well as the Document (libraries) or Attachments (lists). - Get-SympSubwebs – Gets the subwebs of any web and exports their list contents using ShareGate PowerShell functions. By calling itself as a last step, the function enables recursion. Essentially, it “walks” the subsite (web) topology from the current site (web) on down.
<#
.DESCRIPTION
Exports list contents from a web using ShareGate PowerShell functions
.EXAMPLE
Export-SympLists -ParentFolder $parentFolder -WebUrl $webUrl -Versions $versions
#>
function Export-SympLists {
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Parent Folder
[string]
$ParentFolder,
# Web URL
[string]
$WebUrl,
# Versions - Should we download versions (or only the latest version) $true = all versions
[boolean]
$Versions = $false,
# ExclusionLists - Array of list names to *skip* in the download
[array]
$ExclusionLists = @(),
# KeepEmpty - Will create a folder for every library even if it is empty.
# Setting this to $false will delete the empty folders.
[boolean]
$KeepEmpty = $false,
# KeepLists - Will create a folder for every list even if it is empty.
# Setting this to $true will keep all lists, regardless of their number of items.
[boolean]
$KeepLists = $false
)
Begin {
Write-Host "Processing web $($WebUrl)"
}
Process {
# ShareGate's Connect-Site
$srcSite = Connect-Site $WebUrl
# ShareGate's Get-List
$srcLists = Get-List -Site $srcSite
# Filter out the exclusionList items, if any
foreach ($exclusionList in $ExclusionLists) {
$newLists = $srcLists | Where-Object { $_.Title -ne $exclusionList }
$srcLists = $newLists
}
# If there's something to download, do it.
if ($srcLists.length -gt 0) {
# If we want to keep versions
if ($Versions) {
$result = Export-List -List $srcLists -DestinationFolder "$($ParentFolder)"
}
else {
# Else we don't want to keep versions
$result = Export-List -List $srcLists -DestinationFolder "$($ParentFolder)" -NoVersionHistory
}
}
# If #KeepLists, then keep all lists
if ($KeepLists) {
$srcLists = $srcLists | Where-Object { $_.RootFolder -inotmatch "/Lists/" }
}
# If !$KeepEmpty, delete the folders which have no content
if (!$KeepEmpty) {
foreach ($list in $srcLists) {
$listPath = "$($ParentFolder)\$($list.Title)"
$documents = Get-Item -Path "$($listPath)\Documents\*" -ErrorAction Ignore
if ($documents.length -eq 0) {
Remove-Item -Path $listPath -Force -Confirm:$false -Recurse
}
}
}
}
End {
}
}
<#
.DESCRIPTION
Gets the subwebs of any web and exports their list contents using ShareGate PowerShell functions
.EXAMPLE
Get-SympSubwebs -ParentFolder $parentFolder -WebUrl $webUrl -Versions $versions
#>
function Get-SympSubwebs {
[CmdletBinding()]
[Alias()]
[OutputType([int])]
Param
(
# Parent Folder
[string]
$ParentFolder,
# Web URL
[string]
$WebUrl,
# Versions - Should we download versions (or only the latest version) $true = all versions
[boolean]
$Versions,
# ExclusionLists - Array of list names to skip in the download
[array]
$ExclusionLists,
# KeepEmpty - Will create a folder for every library even if it is empty.
# Setting this to $false will delete the empty folders.
[boolean]
$KeepEmpty = $false,
# KeepLists - Will create a folder for every list even if it is empty.
# Setting this to $true will keep all lists, regardless of their number of items.
[boolean]
$KeepLists = $false
)
Begin {
Write-Host "Getting subwebs of $($WebUrl)"
}
Process {
# ShareGate's Connect-Site
$siteConnection = Connect-Site $WebUrl
# ShareGate's Get-Subsite
$webs = Get-Subsite -Site $siteConnection
# Process each web
foreach ($web in $webs) {
# Remove illegal characters in the web title
$cleanTitle = $web.Title.Replace("#", "").Replace(":", " - ").Replace("/", "-").Replace("""", "'")
# Variable for the web's folder - note the leading "_"
$rootFolder = "$($ParentFolder)\_$($cleanTitle)"
# Create the web's folder
$newFolder = New-Item -Path $rootFolder -ItemType Directory -Force
# Download the lists/libraries with the provided parameters
Export-SympLists -ParentFolder $rootFolder -WebUrl "$($web.Address)" -Versions $Versions -ExclusionLists $ExclusionLists -KeepEmpty $KeepEmpty -KeepLists $KeepLists
# Get the web's subwebs - this is the recursion
Get-SympSubwebs -ParentFolder $rootFolder -WebUrl "$($web.Address)" -Versions $Versions -ExclusionLists $ExclusionLists -KeepEmpty $KeepEmpty -KeepLists $KeepLists
}
}
End {
}
}
Here’s an example. Let’s say I set the variables in downloadSite.ps1 like so:
# Setup
$sourceSiteName = "Brazil"
$sourceSiteUrl = "https://myTenant/Brazil/"
# The downloads will end up in a folder here named $sourceSiteName
$destTop = "Z:\"I end up with something like the following set of folders. Brazil is the top-level site, and it has 3 subsites: Facilities, Human Resources, and Recruitment. Notice that each subsite’s folder’s name starts with an underscore so we can easily understand the tree at a glance.
Within each subsite’s folder, all the lists and libraries within it (if requested) are represented as folders.
Each library’s folder contains the Excel file with the metadata, and a subfolder containing the documents themselves.
Each list’s folder contains the Excel file with the metadata, and a subfolder containing the attachments, if any.
This script gives me a good scaffolding and some options I can use to get content out of SharePoint in an organized and broad way. I’ve already used it with two clients, and I firmly expect I’ll use it again.