Continuous Integration, Continuous Delivery, workflow automation, more and more tools rely on a vibrant marketplace to share extensions that add new functionality to the systems. Unfortunately, this means you’re putting trust into a LOT of different people and many of these extensions are built by professionals and enthusiasts in their limited spare time. GitHub has such a marketplace for Actions and Azure DevOps one for Pipelines. But how do you know whether you can trust these extensions, how do you stay up to date and if you’re building these extensions yourself, how do you keep them secure and your users on the latest version? Jesse takes us through some of the best practices from the perspective of an extension author, user as well as the administrator of the system that loads these extensions.

1. Trusting the Unknown -
Ensuring Safe & Secure Extension Use in CI/CD Workflows

2. Of extensions contain some vulnerability

3. Of extensions owned by Microsoft

4. With a ⭐ rating

5. Of extensions updated in the last 3 months

6. Of Node based Azure Pipelines tasks

7. Of PowerShell Azure Pipeline Tasks…

8. That’s unfair
But it’s also a bit true…

10. Jesse Houwing
Chief Trainer & Tinkerer
xpirit.com/jesse
jessehouwing
jessehouwing
jessehouwing
hachyderm.io/jessehouwing
+31 6 41 81 333 8

13. Extensions in the context of Azure Pipelines
Manifests
Marketplace pritties
Tasks
And potentially tabs and context menu actions…

14. So what is a task?
Loads of dependencies
At least azure-pipelines-task-lib
Frozen at time of publish
Another Manifest
The actual custom code
More pretties

15. Or… using PowerShell
A lot fewer dependencies
at least VstsTaskSdk
Another Manifest
The actual custom code
More pretties

16. An extension can contain more than 1 tasks
Task A
Task A
v1

17. Where do these tasks come from?
• They’re “built-in” tasks
• Automatically updated by Microsoft in Azure DevOps
• Manually updated through Microsoft Update packs in Azure DevOps Server
• Manually updated by administrator through sideloading (see below)
• They’re from an extension from the public marketplace
• Installed through marketplace.visualstudio.com
• Automatically updated in Azure DevOps and Azure DevOps Server
• They’re from an extension from the private marketplace
• Manually uploaded to Azure DevOps Server
• They’re sideloaded directly
• Manually uploaded to Azure DevOps or Azure DevOps Server

18. 2 problems with task updates
By default, the agent:
1. Pins the major version to the one you selected
2. Updates to the latest minor version available

19. Where do these tasks come from?

20. So how do we know we can trust these tasks?
You don’t.

21. How do you know?
Well…
I downloaded 700GB of vsix files from the Azure DevOps marketplace
Extracted them and scanned them

22. So how do we know we can trust them?
Trusted publisher?
Verification?
Rating?
Number of issues and pull-requests?
Metadata provided by the author?
Recently updated?
Looks nice?

23. Is it really that bad?
Maybe not…

24. All of these potential security issues
Do not mean there is a known exploit chain

25. But it’s not pretty!

26. It’s not just the updates you should worry
about

27. So what’s the risk we’re running
• Some of these extensions turn off TLS security
• Some might overwrite arbitrary files
• Some might be used to send data to the outside world
• Some install things directly from npmjs.
• Others download 100’s of MB, extract it and execute without
verification
• Often on a host with admin permissions
There is a lot of potential

29. Why is PowerShell so much better?
It might not be but…
• Many of the dependencies are automatically updated through
Windows Update
• PowerShell relies on the .NET framework for most functionality
• There are a lot fewer dependencies to worry about

30. What is Microsoft doing?
• They’ve introduced a new Node 16 handler for Azure Pipelines
• They’re deprecating Node 6 and Node 10 for Azure Pipelines tasks.
• They’re updating all the built-in tasks for Azure DevOps.
• They’re updating all the built-in tasks for Azure DevOps Server 2022?
• They’ve added the ability to turn off Node 6 on the Pipelines Agent.
• They’re adding warnings to every workflow that runs uses Node 6
(and later 10).

31. What am I doing / have I done
• I actively update all my extensions regularly
• I automatically update my dependencies using Snyk, Dependabot and
RenovateBot
• I scan my code for vulnerabilities using Snyk Code and GitHub
advanced Security
• I maintain the Azure DevOps Extension Tasks to implement CI / CD for
Azure DevOps extensions
• I recently contributed to RenovateBot to add support for the Azure
Pipeline Marketplace. Now doing the same for dependabot-core.

32. Guidance for deprecating an extension
• Add a warning to all versions of your tasks
• Timebomb the task to fail after today+X
• Mark all tasks deprecated:true in the task.json
• Add [DEPRECATED] to your task friendly name in the task.json
• Add [DEPRECATED] to your extension name in the vss-extension.json
• Publish this last version
• Unpublish the extension
• Archive the GitHub repository

33. Guidance for deprecating task version
• Add both the old and the new version of the task to your extension
• Update the old version’s friendlyName and add [DEPRECATED] in
task.json
• Add a warning / error to the implementation to warn you users

34. What should Extension authors do?
• Employ secure password practices and use 2FA
• Deprecate extensions you’re no longer maintaining
• Update your existing extensions to Node 16 and latest dependencies
• Enable DependaBot to automatically keep your dependencies up-to-
date
• Enable GitHub Advanced Security to detect issues in *your* code
• Implement CI / CD to ship new versions with ease
• Add repo metadata to their extension manifests

35. What should Azure DevOps admins do
• Disable insecure extensions
• Implement RenovateBot to automatically
suggest updates to installed Azure
Pipelines tasks
• Disable Node 6 support on the Azure
Pipelines Agent
• Do not install every extension in the
marketplace without thought
• Provide Hosted Agents or ephemeral Scale-
set agents

36. What should Azure DevOps admins do
• Run Endpoint Security on your Azure Pipelines Agents
• For DevOps Server
• Upgrade to Azure DevOps Server 2022 and stay current
• Migrate to Azure DevOps Service
• Manually upgrade “built-in” tasks
• Be careful with internal marketplace

37. Overwrite a built-in task
npm install -g tfx-cli
tfx build tasks upload --task-zip-path Task.guid-version.zip
--service-url https://yourtfs.com/tfs/DefaultCollection
. ./script/install-task.ps1 -CollectionUrl https://yourtfs.com/tfs/DefaultCollection
-TaskZip Task.guid-version.zip
Delete all versions of a task
npm install -g tfx-cli
tfx build tasks delete --task-id ad884ca2-732e-4b85-b2d3-ed71bcbd2788
--service-url https://yourtfs.com/tfs/DefaultCollection

39. What should Azure Pipelines authors do?
• Consider whether you actually need an extension.
• Keep your pipelines current. Update your major task versions and
resolve any issues.
• Use Azure Pipelines YAML
• Pin the exact version of your tasks
• Use RenovateBot to automatically keep your pipelines up to date
• Use Hosted Agents / Ephemeral Scale-set agents whenever possible

40. What should the community do?
• Sponsor the authors of extensions you rely on.
• Submit pull requests to improve extensions.
• Submit pull requests to upgrade extensions to Node 16.
• Leave reviews on the marketplace and update them.

41. Useful resources

42. Useful Resources
• Azure DevOps Marketplace Scan
https://github.com/jessehouwing/azure-devops-marketplace-scan
https://jessehouwing.net/security-state-of-the-azure-devops-
marketplace/
• Hardening guidelines for Azure Pipelines
https://learn.microsoft.com/en-
us/azure/devops/organizations/security/security-best-
practices?view=azure-devops#secure-azure-pipelines

43. Useful resources
• Azure Pipelines Tasks Zips
https://github.com/jessehouwing/azure-pipelines-tasks-zips
• RenovateBot for Azure Pipelines
https://jessehouwing.net/azure-pipelines-enable-renovatebot/
• Azure DevOps Extension tasks
https://marketplace.visualstudio.com/items?itemName=ms-
devlabs.vsts-developer-tools-build-tasks