Azure DevOps Pipelines: Multiple Jobs in YAML

This post is going to show how to run multiple jobs out of a single YAML file from an Azure DevOps Pipeline. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core

Starting Point and the Plan

As the sample stands now we have a single Pipeline that builds two different ASP.NET Core web applications in a single job using the following YAML.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

This post is going to take this pipeline and split the build and publish of the two web applications and make each application its own job. In Pipelines a job is something that a single agent takes and runs. By splitting into multiple jobs the pipeline can run multiple jobs at the same time if you have enough build agents available. One reason to do this would be to speed up the total Pipeline run if you have parts of your build that are independent. Another example of why you would need jobs is if the different jobs need different agents such as one needing a Windows agent and another a Linux agent.

Creating the Jobs

Having different jobs means we are going to have to move things like what agent pool to use and the steps for the job under a jobs element and then declare a specific job and the details that job needs to run. As you can see in the following example the end goal is the same as the YAML from above (except it is dealing with a specific project), but the details are nested under jobs and defined under a job.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

Also notice that you can still define variables that can be used across jobs as is done above with the buildConfiguration variable. The following is the full YAML file that builds and publishes the artifacts for both web applications.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp2'
      publishLocation: 'pipeline'

After all your edits are done commit the changes to your YAML file and then run the pipeline. As you can see from the following screenshot of my sample pipeline run the pipeline has two jobs instead of one that the original YAML resulted in. Also, note that the pipeline results in two published artifacts (one per job in our case) instead of the one with the original.

Wrapping Up

As mentioned above there are a lot of reasons you might want to split up your pipeline into multiple jobs and hopefully, you now have a good idea of how that is done. Make sure and check back in the future for a post on how to take repeated tasks and make them reusable.

---

Also published on Medium.