Azure Pipelines

In this post, we are going to refactor our sample Azure DevOps Pipeline to move some of the redundant YAML to a new file and replace the redundant parts of our main YAML file. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML

Starting YAML

The following is the YAML for our current pipeline that builds two different web applications using two different jobs. Looking at the two jobs you will notice that they both have the same steps. The only difference in the steps is which project to build (WebApp1.csproj or WebApp2.csproj) and what to call the published artifact (WebApp1 or WebApp2). When developing applications we would never stand for this level of duplication and the same should apply to our pipelines.

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp2'
      publishLocation: 'pipeline'

Add a New File

To attack the duplication above we need to take the shared steps from above and move them somewhere they can be reused. We will be walking through the steps using the Azure DevOps web site and committing directly to the master branch, but these same steps could be performed locally or on the web on any branch. First, from the Repos section of the site we need to add a new file by clicking the three dots at the level we want the file added. In this case, we are adding to the root of the repo but the same option is available on any folder.

A dialog will show where you can enter the New file name, we are going to use build.yml in this case. Next, click Create to continue.

Shared YAML

Now that we have a new file we can start building the new YAML that will handle the repeated steps from the original jobs. The first thing we are going to do is define a set of parameters that this set of steps can be called with. We are going to use this to pass what project to build, which build configuration to use, and what name the published artifact. The following is the definition of our parameters.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: project
  type: string
  default: ''
- name: artifactName
  type: string
  default: ''

We can then use these parameters in the rest of the file using the ${{ parameterName }} syntax. Note that any pipeline variables are also available using the $(variableName) syntax. The following bit of YAML shows both types in the arguments line.

- task: DotNetCoreCLI@2
  displayName: 'Publish Application'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '**/${{ parameters.project }}'
    arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)'

While you can use pipeline variables I recommend passing all the values you need via parameters for the same reason that we try to avoid global variables when doing general programming. I’m using both here to show the usage of each. The following is the full YAML in our new file.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: project
  type: string
  default: ''
- name: artifactName
  type: string
  default: ''

steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }}' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: ${{ parameters.artifactName }}
      publishLocation: 'pipeline'

Finally, commit the changes to the new file.

Using Shared YAML

Not that we have the YAML that is the same between our two build jobs we can switch back over to our main YAML file, azure-pipelines.yml in the sample, and remove the steps we are wanting to replace. While the jobs will both have a steps section the only thing we will have left in them is a template call to our other YAML file, build.yml for the sample, that passes the parameters to run the other file with. The following is the resulting YAML file with the call to the shared file in both jobs highlighted.

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2

Wrapping Up

Being able to remove duplication from your YAML files should help improve the maintainability of your pipelines. I know the samples don’t show it, but the template is just a step and you could have other steps before or after it just like you would with normal tasks.

This post is going to show how to run multiple jobs out of a single YAML file from an Azure DevOps Pipeline. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core

Starting Point and the Plan

As the sample stands now we have a single Pipeline that builds two different ASP.NET Core web applications in a single job using the following YAML.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

This post is going to take this pipeline and split the build and publish of the two web applications and make each application its own job. In Pipelines a job is something that a single agent takes and runs. By splitting into multiple jobs the pipeline can run multiple jobs at the same time if you have enough build agents available. One reason to do this would be to speed up the total Pipeline run if you have parts of your build that are independent. Another example of why you would need jobs is if the different jobs need different agents such as one needing a Windows agent and another a Linux agent.

Creating the Jobs

Having different jobs means we are going to have to move things like what agent pool to use and the steps for the job under a jobs element and then declare a specific job and the details that job needs to run. As you can see in the following example the end goal is the same as the YAML from above (except it is dealing with a specific project), but the details are nested under jobs and defined under a job.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

Also notice that you can still define variables that can be used across jobs as is done above with the buildConfiguration variable. The following is the full YAML file that builds and publishes the artifacts for both web applications.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: UseDotNet@2
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: DotNetCoreCLI@2
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: DotNetCoreCLI@2
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: PublishPipelineArtifact@1
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp2'
      publishLocation: 'pipeline'

After all your edits are done commit the changes to your YAML file and then run the pipeline. As you can see from the following screenshot of my sample pipeline run the pipeline has two jobs instead of one that the original YAML resulted in. Also, note that the pipeline results in two published artifacts (one per job in our case) instead of the one with the original.

Wrapping Up

As mentioned above there are a lot of reasons you might want to split up your pipeline into multiple jobs and hopefully, you now have a good idea of how that is done. Make sure and check back in the future for a post on how to take repeated tasks and make them reusable.

This post is going to build on the Azure DevOps project we created in the last few posts and get the build pipeline to the point you have the application’s binaries. If you are just joining this series check out the previous posts to catch up.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps

Edit the Pipeline

First, we need to get back to the pipeline we were working on. From the Project menu select Pipelines.

This will land you on a page that lists your recently run pipelines. If you don’t see your pipeline list you might have to click the All option near the top of the page. Since we only have one pipeline in this project we can use the ellipsis to open a context menu and click Edit.

Publish the Application

At this point, the YAML for our pipeline looks like the following.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

The pipeline will currently tell us if the included project builds, but doesn’t provide us with the results of that build. Using the Task panel on the right search for the .NET Core task and then click the resulting task. This is the task you would want to use to invoke any of the .NET CLI commands.

Use the drop-down for Command and select publish. For this sample, the defaults for the rest of the settings will be fine. Finally, click Add to add the task to the YAML file.

The following is the resulting YAML.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true

Before we move on I want to point out the Settings link above the tasks in the YAML editor. Clicking Settings will load that task into the task panel on the right of the screen where you can make changes and then if you hit the add button it will replace your existing task with a new one with your new options selected. Be careful to not change the selection in the YAML editor as the add button is just replacing the selected text not remembering what task you click settings on. When finished click the Save button and go through the commit process. When that is finished click the Run button to execute the pipeline.

Publish Build Artifacts

The pipeline run should succeed, but we still don’t have any files we can use. Learning what variables are available in the pipeline and how to use them is one of the hardest parts of getting started with Azure Pipelines. For our example, we are trying to get the two zip files created by the publish step above which means our pipeline will need to publish artifacts to make the files available. We are going to tweak the publish command from above with an output directory using the builtin Build.ArtifactStagingDirectory variable. The following is the full task with the changes.

- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

With the file we need in the artifact staging directory we need to publish those files using the Publish Pipeline Artifact task. The following is the full task that publishes the artifact staging directory to the pipeline.

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

For reference, the following is the full YAML for the pipeline with all the above changes.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

Save and run the pipeline. When the pipeline is complete on the result page you will see 1 published for artifacts.

Click on 1 published and it will take you to a page that lists the artifacts. If you mouse over any of the rows you will see the option to download the associated file(s).

Quick Tip

As I stated above getting a handle on what directories are where can be a pain. If you ever need to see what files are where you can use the following publish task to output the full set of files the pipeline is using by publishing the pipeline’s entire workspace. This has helped me in the past to orient myself.

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'

Wrapping Up

Our pipeline is now at the point we have files we could deploy. Hopefully, this gives you a good jump start on your own build pipelines. Azure Pipelines is a huge topic and this is a very basic build so keep an eye out for more posts on this topic in the future.

After last week’s post on Azure Pipelines: Release to Azure App Service I came across a much easier way to get started using Azure DevOps and Azure App Servies. This post is going to walk through this process which is started from the Azure Portal side instead of Azure DevOps.

The names here are going to be a bit confusing. When I say Azure DevOps I am talking about the rebrand of Visual Studio Team Services which includes services for boards, repos, pipeline, etc. When I say Azure DevOps Project, or just DevOps Project, I am referring to the project type this post is going to be using from the Azure Portal side.

Create DevOps Project

From the Azure Portal click the Create a resource button.

For me, DevOps Project was in the list of popular items. If you don’t see it list you can use the search at the top to find it. Click on the DevOps Project to start the process.

On the next page, you have options to start a new application with a lot of different languages or to deploy an existing application. For this example, we are going to select .NET and click the Next button. This screen is a great example of how Microsoft is working hard to support more than just .NET.

For .NET the next choice is ASP.NET or ASP.NET Core. No surprise I’m sure that we are going to go with ASP.NET Core. There is also an option for adding a database, but we aren’t going to use it for this example. Click Next to continue.

The next step is to select which Azure Service the new application should run on. We are going to use a Linux Web App to match what last week’s sample was running on. Click Next to continue.

There are quite a few settings on this next screen, but they are all pretty clear. The first group of settings is for the Azure DevOps project and you can either use an existing account or the process will create one for you. A Project name is required.

The next group of settings is for the Azure App Service that will be created. Subscription should default in if you only have one. Web app name is going to control the URL Azure provides as well as play in the naming of the resources that are created. Click Done to start the creation process.

Deployment

After clicking down above the deployment of all the need resources will start. This process takes awhile. The portal will redirect you to a status page similar to the following while deployment is in progress. It took a little over 4 minutes for mine to complete.

When the deployment is complete click the Go to resource button.

Results

The Overview page for the project gives a great summary of the whole CI/CD pipeline that was created with links to the associated Azure DevOps pages to manage each step. The Azure resources section will have the URL you can use to access the running application.

The resulting application should look similar to the following.

Wrapping Up

This process is a much easier way to get started if you are going all in with Azure. If you read last week’s post you know there is a lot that goes into creating something close to this setup manually and even then it was missing the nice overview provided by this setup.

This post is going to use the same Azure DevOps project used in last week’s Azure Repos and Azure Pipelines post which had a build pipeline and add a release pipeline that deploys to an Azure App Service.

This walkthrough is going to assume you have an Azure account already set up using the same email address as Azure DevOps. If you don’t have an Azure account one signup for an Azure Free Account.

Build Changes

The build set up in last week’s post proved that the code built, but it didn’t actually do anything with the results of that build. In order to use the results of the build, we need to publish the resulting files. This will be needed later when setting up the release pipeline.

In order to get the results we are looking for a few steps must be added to our build.  All the changes are being made in the azure-pipelines.yml. The following is my full yaml file with the new tasks.

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  buildConfiguration: 'Release'

steps:
- task: DotNetCoreCLI@2
  displayName: Build
  inputs:
    projects: '**/EfSqlite.csproj'
    arguments: '--configuration $(BuildConfiguration)'
- task: DotNetCoreCLI@2
  displayName: Publish
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'

As you can see in the above yaml this build now has three different steps. The build (this is equivalent to what the post from last week was doing) and publish (this gets all the files in the right places) tasks are both handled using the DotNetCoreCLI@2 task.  Finally, the PublishBuildArtifacts@1 takes the results of the publish and zips them to the artifact staging directory where they can be used in a release pipeline.

Create an Azure App Service

Feel free to skip this section if you have an existing App Service to use. To create a new App Service open the Azure Portal and select App Services from the left navigation.

Next, click the Add button.

On the next page, we are going to select Web App.

Now hit the Create button.

The next page you will need to enter an App name, select the OS, and Runtime Stack before hitting the Create button. The OS and Runtime Stack should match the target of your application.

Create a Release Pipeline

On your project from the left navigation select Pipelines > Releases and then click the New pipeline button. If you already have a release pipeline setup this page will look different.

The next page has a list of template you can select from. In this example, we will be selecting Azure App Service deployment and then click the Apply button.

 

 

Artifact Set Up

After clicking Apply you will hit the pipeline overview page with to sets of information. The first is the Artifacts which for us is going to be the results of the build pipeline we set up in last week’s post. Click the Add an artifact box.

The Add an artifact dialog is where you will select where the release will get its build artifact form. We will be using the source type of build and selecting our existing build pipeline.

Once you select your build pipeline as the Source a few more controls will show up. I took the defaults on all of them. Take note of the box highlighted in the screenshot as it will give you a warning if you build is missing artifacts. Click the Add button to complete.

 

Stage Setup

 

Above we selected the Azure App Service template which is now in our pipeline as Stage 1. Notice the red exclamation, which means the stage has some issues that need to be addressed before it can be used. Click on the stage box to open it.

 

As you can see in the following screenshot the settings that are missing are highlighted in red on the Deploy Azure App Service task.  On Stage 1 click the Unlink all button and confirm. Doing this means there is more setup on the Deploy Azure App Service task, but this is the only way to use .NET Core 2.1 at the time of this writing. For some reason, the highest version available at the Stage level for Linux is .NET Core 2.0.

After clicking unlink all the options other than the name of the stage will be removed. Next, click on Deploy Azure App Service task which handles the bulk to the work will place for this pipeline. There are a lot of setting on this task. Here is a screenshot of my setup and I will call out the important bits after.

First, select your Azure subscription. You may be required to Authorize your account so if you see an Authorize button click it and go through the sign in steps for your Azure account.

Take special note of App type. In this sample, we are using Linux so it is important to select Linux App from the drop down instead of the just using Web App.

With your Azure subscription and App type selected the App Service name drop-down should only let you select Linux based App  Services that exist on your subscription.

For Image Source, I went with the Built-in Image, but it does have the option to enter use a container from a registry if the built-in doesn’t meet your needs.

For Package or folder, the default should work if you only have a single project. Since this, I have two projects I used the file browser (the … button) to select the specific zip file I want to deploy.

Runtime Stack needs to to be .NET Core 2.1 for this application.

Startup command needs to be set up to tell the .NET CLI to run the assembly that is the entry point for the application. In the example, this works out to be dotnet EfSqlite.dll.

After all the settings have been entered hit the Save button in the top right of the screen.

Execute Release Pipeline

Navigate back to Pipelines > Release and select the release you want to run. Then click the Create a release button.

On the next page, all you have to do is select the Artifact you want to deploy and then click the Create button.

Create will start the release process and send you back to the main release page. There will be a link at the top of the release page that you can click to see the status of the release.

The following is the results of my release after it has completed.

Wrapping Up

I took me more trial and error to get this setup going that I would have hoped, but once all the pieces are get up the results are awesome. At the very minimum, I recommend taking the time to at least set up a build that is triggered when code is checked into your repos. Having the feedback that a build is broken as soon as the code hits the repo versus finding out when you need a deliverable will do wonders for your peace of mind.