Uploading files to blob storage in Azure DevOps

Jul 11, 2022 - 3 min read

Whether using blob storage as CDN, hosting a static website, or any other purpose, the Azure file copy task can be used to upload files from Azure DevOps pipelines to blob storage in Azure. Even though these are common scenarios, there are some gotchas associated.

To start, you need to create a service connection in Azure and Azure DevOps. A Service principal requires following roles to access the storage account:

  • Storage Account Contributor (required to read storage account and create containers, Contributor also works)
  • Storage Blob Data Contributor (required to write data)

AzCopy is executed behind the scenes, and it uses either the Az or AzureRM modules depending on which is installed on the machine. There are known issues in older versions of these modules which might cause AzCopy to occasionally fail. If you are encountering upload failures, check the troubleshooting section.

Azure file copy task parameters

  • Task version: 4.* (latest as of May 2022)
  • Source: (path/to/folder/*)
  • Azure Subscription: (service connection)
  • Destination Type: Azure Blob
  • Container Name: (container name)
  • Blob Prefix: (optional, creates a parent folder for uploaded files)
  • Optional Arguments: (arguments passed to AzCopy)

Tips:

  1. When source folder path ends with ‘/*‘, the source folder is not copied to the destination; only the files contained inside the folder are copied.

  2. For CDN scenarios you can use artifact version as blob prefix:

    • Classic Release pipeline: $(Release.Artifacts.<ArtifactAlias>.BuildNumber)
    • YAML pipeline:
      • Artifact from another pipeline: $(resources.pipeline.<ArtifactAlias>.runName)
      • Current pipeline: $(Build.BuildNumber)
  3. You can pass multiple arguments to AzCopy. The example below copies all files, skips existing, and sets the max-age to 30 days:

    --recursive=true `
    --overwrite=false `
    --cache-control="public, max-age=2592000"

YAML example

- task: AzureFileCopy@4
  displayName: Copy files to storage account
  inputs:
    SourcePath: $(System.DefaultWorkingDirectory)/_artifactAlias/drop/*
    azureSubscription: My Service Connection
    Destination: AzureBlob
    storage: mystorageaccount
    ContainerName: mycontainer
    BlobPrefix: myprefix
    AdditionalArgumentsForBlobCopy: |
      --recursive=true `
      --overwrite=true

Troubleshooting

AzCopy consistently fails with non-zero exit code

Number of Transfers Failed: 1
...
Final Job Status: Failed

Upload to container: 'mycontainer' in storage account: 'mystorageaccount' with blob prefix: 'myprefix' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.'

Solution: Service principal requires “Storage Blob Data Contributor” built-in role for the storage account.

AzCopy randomly fails for some files

Number of Transfers Completed: 2
Number of Transfers Failed: 1
...
Final Job Status: CompletedWithErrors

Upload to container: 'mycontainer' in storage account: 'mystorageaccount' with blob prefix: 'myprefix' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.'

AzCopy log file might contain following:

ERR: [P#0-T#1] UPLOADFAILED: %5C%5C?\E:\Agents\1\_work\r1\a\_artifactAlias\drop\file.ext : 000 : File modified since transfer scheduled

Solution: If using a self-hosted agent, update the Az or AzureRM modules on the agent machine, as newer versions include a fix. Microsoft-hosted agents should already have the updated modules.

Alternative: If you do not have access to the agent machine, try updating file timestamps before running AzCopy:

$currentDate = Get-Date
Get-ChildItem . * -Recurse | ForEach-Object { $_.LastWriteTime = $currentDate }

Storage account not found or missing permissions to perform listKeys

Storage account: mystorageaccount not found. The selected service connection ‘Service Principal’ supports storage accounts of Azure Resource Manager type only.

or

The client ‘xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’ with object id ‘xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx’ does not have authorization to perform action ‘Microsoft.Storage/storageAccounts/listKeys/action’ over scope ‘/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myrg/providers/Microsoft.Storage/storageAccounts/mystorageaccount’ or the scope is invalid. If access was recently granted, please refresh your credentials. (in function: Get-AzRMStorageKeys)

Solution: Service principal is missing the ‘Storage Account Contributor’ built-in role for the storage account.