Azure

Azure DevOps Pipelines: Conditionals in YAML

In this week’s post, we are going to cover some ways to make tasks and jobs run conditionally. This will include options such as Pipeline variables to jobs that are dependent on other jobs. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see the build-up check out the following posts.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML
Azure DevOps Pipelines: Use YAML Across Repos

Sample YAML

The following YAML is based on the YAML from the previous posts, see links above, expanded with examples of using some ways of conditionally running some task or job. This is the full file for reference and the rest of the post will call out specific parts of the file as needed.

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: [email protected]
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  condition: and(succeeded(), eq(variables['BuildWebApp2'], 'true'))
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2
      
- job: DependentJob
  displayName: 'Build Dependent Job'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2

  steps:
  - template: [email protected]
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1Again

Job Dependencies

The more complex pipelines get the more likely the pipeline will end up with a job that can’t run until other jobs have completed. The YAML above defines three different jobs, WebApp1, WebApp2, and DependentJob. I’m sure you have guessed by now that the third job is the one that has a dependency. To make a job dependent on other jobs we use the dependsOn element and list the jobs that must complete before the job in question can run. The following is the YAML for the sample DependentJob with the dependsOn section highlighted.

- job: DependentJob
  displayName: 'Build Dependent Job'
  pool:
    vmImage: 'ubuntu-latest'

  dependsOn:
  - WebApp1
  - WebApp2

  steps:
  - template: [email protected]
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1Again

With the above setup, DependentJob will only run if both the WebApp1 and WebApp2 jobs complete successfully.

Conditions

Conditions are a way to control if a Job or Task is run. The following example is at the job level, but the same concept works at the task level. Notice the highlighted condition.

- job: WebApp2
  displayName: 'Build WebApp2'
  condition: and(succeeded(), eq(variables['BuildWebApp2'], 'true'))
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2

The above condition will cause the WebApp2 job to be skipped if the BuildWebApp2 variable isn’t true. For more details on how to use conditions see the Conditions docs.

Creating a Pipeline Variable

The rest of the post is going to walk through creating a Pipeline variable and then running some sample builds to show how depends on and the conditions defined in the YAML above affect the Pipeline results.

We are starting from an existing pipeline that is already being edited. To add (or edit) variables click the Variables button in the top right of the screen.

The Variables pop out will show. If we had existing variables they show here. Click the New variable button to add a new variable.

We are adding a variable that will control the build of WebApp2 called BuildWebApp2 that defaults to the value of true. Also, make sure and check the Let user override this value when running this pipeline checkbox to allow us to edit this variable when doing a run of the pipeline. Then click the OK button.

Back on the Variables dialog click the Save button.

Edit Variables When Starting a Pipeline

Now that our Pipeline has a variable when running the Pipeline under Advanced options you will see the Variables section showing that our Pipeline has 1 variable defined. Click Variables to view/edit the variables that will be used for this run of the Pipeline.

From the Variables section, you will see a list of the defined variables as well as an option to add new variables that will exist only for this run of the Pipeline. Click on the BuildWebApp2 variable to edit the value that will be used for this run of the Pipeline.

From the Update variable dialog, you can change the value of the variable. When done click the Update button.

Pipeline Results from Sample YAML

The following is what our sample Pipeline looks like when queued with the BuildWebApp2 variable set to false. As you can see the job will be skipped.

Next is the completed results of the Pipeline run. You can see that the Build Dependent Job was skipped as well since both Build WebApp1 and Build WebApp2 must complete successfully before it will run.

Changing the BuildWebApp2 variable back to true and running the Pipeline again results in all the jobs running successfully.

Wrapping Up

Hopefully, this has helped introduce you to some of the ways you can control your Pipelines. As with everything else Azure DevOps related things are changing a lot and new options are popping up all the time. For example, while writing this post the team just announced Runtime Parameters which look like a much better option than variables for values that frequently vary between Pipeline runs.

Azure DevOps Pipelines: Use YAML Across Repos

In last week’s post, we refactored some YAML that was reusable into a new file. This post is going to cover moving that same reusable YAML to a new repo and then using it in our existing sample repo. This post is going to build on the Azure DevOps project created in previous posts if you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML
Azure DevOps Pipelines: Reusable YAML

Create a New Repository

First, we need to create a new repository that will be used to share the YAML in question. Using the Repos section of Azure DevOps as a starting point you click the dropdown with the currently selected repo name, Playground in this example, and then click New repository.

You will be presented with a dialog where you will need to enter the Repository name and any other of the options you want to configure. In this example, we are naming the repo Shared and adding a Git ignore file for Visual Studio. When done click the Create button.

The following steps should be taken on the new Shared repo. At this point, you could clone the repo and do the rest of the steps locally and then push the changes to the repo or you can use the web interface to make all the change which is the route this post is going to show. Either way, you go it shouldn’t be too hard to adapt the steps. For the web interface to add a new file in the root of the repo click the three-dot menu to the right of the repo name and then select New and then File.

The next prompt will ask for a file name, I’m using buildCoreWebProject.yml. Click Create to continue.

You will land in the file editor. Copy and paste the code out of build.yml that we were using from the previous post into the new file. The following is the full YAML from build.yml for reference.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: project
  type: string
  default: ''
- name: artifactName
  type: string
  default: ''

steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }}' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: ${{ parameters.artifactName }}
      publishLocation: 'pipeline'

Once the code is copied in click the Commit button to save the changes to the master branch.

Switching back to our original repo, Playground in this example, we need to add the Shared repo as a resource for in our azure-pipelines.yml file. The following is the resource deliration for using another Azure DevOps repo. The official docs for check out multiple repositories also show examples with GitHub and Bitbucket. I also found this stackoverflow question helpful.

resources: 
  repositories: 
  - repository: Shared 
    name: Playground/Shared 
    type: git 
    ref: master #branch name

Shared on the repository line is the name we will be using when referencing a file out of the Shared repo. Name is the Azure DevOps project and repo name. Type is the repo type which is Git in our case. Finally, ref is the branch name from the Shared repo that we want to use. Now that we have access to the files from the Shared repo we can use its buildCoreWebProject.yml instead of the local build.yml as a template.

Before:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

After:
  - template: [email protected]
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

Notice that the only change is on the template line which changed from build.yml to [email protected]. The @Shared on the end of the filename is what tells the pipeline the file’s source is the Shared repo. The following is the full azure-pipeline.yml for reference. The frist job is using the template form the Shared repo and the second one is using a local template.

resources:      
  repositories: 
  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: [email protected]
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2

Wrapping Up

Being able to utilize YAML from different repos can help cut down on duplicated YAML and help keep your pipelines across repos cleaner. As with anything else, this is a useful tool when applied appropriately.

Azure DevOps Pipelines: Reusable YAML

In this post, we are going to refactor our sample Azure DevOps Pipeline to move some of the redundant YAML to a new file and replace the redundant parts of our main YAML file. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core
Azure DevOps Pipelines: Multiple Jobs in YAML

Starting YAML

The following is the YAML for our current pipeline that builds two different web applications using two different jobs. Looking at the two jobs you will notice that they both have the same steps. The only difference in the steps is which project to build (WebApp1.csproj or WebApp2.csproj) and what to call the published artifact (WebApp1 or WebApp2). When developing applications we would never stand for this level of duplication and the same should apply to our pipelines.

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp2'
      publishLocation: 'pipeline'

Add a New File

To attack the duplication above we need to take the shared steps from above and move them somewhere they can be reused. We will be walking through the steps using the Azure DevOps web site and committing directly to the master branch, but these same steps could be performed locally or on the web on any branch. First, from the Repos section of the site we need to add a new file by clicking the three dots at the level we want the file added. In this case, we are adding to the root of the repo but the same option is available on any folder.

A dialog will show where you can enter the New file name, we are going to use build.yml in this case. Next, click Create to continue.

Shared YAML

Now that we have a new file we can start building the new YAML that will handle the repeated steps from the original jobs. The first thing we are going to do is define a set of parameters that this set of steps can be called with. We are going to use this to pass what project to build, which build configuration to use, and what name the published artifact. The following is the definition of our parameters.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: project
  type: string
  default: ''
- name: artifactName
  type: string
  default: ''

We can then use these parameters in the rest of the file using the ${{ parameterName }} syntax. Note that any pipeline variables are also available using the $(variableName) syntax. The following bit of YAML shows both types in the arguments line.

- task: [email protected]
  displayName: 'Publish Application'
  inputs:
    command: 'publish'
    publishWebProjects: false
    projects: '**/${{ parameters.project }}'
    arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)'

While you can use pipeline variables I recommend passing all the values you need via parameters for the same reason that we try to avoid global variables when doing general programming. I’m using both here to show the usage of each. The following is the full YAML in our new file.

parameters:
- name: buildConfiguration
  type: string
  default: 'Release'
- name: project
  type: string
  default: ''
- name: artifactName
  type: string
  default: ''

steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }}' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/${{ parameters.project }}'
      arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: ${{ parameters.artifactName }}
      publishLocation: 'pipeline'

Finally, commit the changes to the new file.

Using Shared YAML

Not that we have the YAML that is the same between our two build jobs we can switch back over to our main YAML file, azure-pipelines.yml in the sample, and remove the steps we are wanting to replace. While the jobs will both have a steps section the only thing we will have left in them is a template call to our other YAML file, build.yml for the sample, that passes the parameters to run the other file with. The following is the resulting YAML file with the call to the shared file in both jobs highlighted.

trigger: none

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - template: build.yml
    parameters:
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2

Wrapping Up

Being able to remove duplication from your YAML files should help improve the maintainability of your pipelines. I know the samples don’t show it, but the template is just a step and you could have other steps before or after it just like you would with normal tasks.

Azure DevOps Pipelines: Multiple Jobs in YAML

This post is going to show how to run multiple jobs out of a single YAML file from an Azure DevOps Pipeline. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps
Azure DevOps Publish Artifacts for ASP.NET Core

Starting Point and the Plan

As the sample stands now we have a single Pipeline that builds two different ASP.NET Core web applications in a single job using the following YAML.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: [email protected]
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: [email protected]
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: [email protected]
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

This post is going to take this pipeline and split the build and publish of the two web applications and make each application its own job. In Pipelines a job is something that a single agent takes and runs. By splitting into multiple jobs the pipeline can run multiple jobs at the same time if you have enough build agents available. One reason to do this would be to speed up the total Pipeline run if you have parts of your build that are independent. Another example of why you would need jobs is if the different jobs need different agents such as one needing a Windows agent and another a Linux agent.

Creating the Jobs

Having different jobs means we are going to have to move things like what agent pool to use and the steps for the job under a jobs element and then declare a specific job and the details that job needs to run. As you can see in the following example the end goal is the same as the YAML from above (except it is dealing with a specific project), but the details are nested under jobs and defined under a job.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

Also notice that you can still define variables that can be used across jobs as is done above with the buildConfiguration variable. The following is the full YAML file that builds and publishes the artifacts for both web applications.

trigger:
- master

variables:
  buildConfiguration: 'Release'

jobs:
- job: WebApp1
  displayName: 'Build WebApp1'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp1.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp1'
      publishLocation: 'pipeline'

- job: WebApp2
  displayName: 'Build WebApp2'
  pool:
    vmImage: 'ubuntu-latest'

  steps:
  - task: [email protected]
    displayName: 'Use .NET 3.1.x'
    inputs:
      packageType: 'sdk'
      version: '3.1.x'

  - task: [email protected]
    displayName: 'Build'
    inputs:
      command: 'build'
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration)' 
  
  - task: [email protected]
    displayName: 'Publish Application'
    inputs:
      command: 'publish'
      publishWebProjects: false
      projects: '**/WebApp2.csproj'
      arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

  - task: [email protected]
    displayName: 'Publish Artifacts'
    inputs:
      targetPath: '$(Build.ArtifactStagingDirectory)'
      artifact: 'WebApp2'
      publishLocation: 'pipeline'

After all your edits are done commit the changes to your YAML file and then run the pipeline. As you can see from the following screenshot of my sample pipeline run the pipeline has two jobs instead of one that the original YAML resulted in. Also, note that the pipeline results in two published artifacts (one per job in our case) instead of the one with the original.

Wrapping Up

As mentioned above there are a lot of reasons you might want to split up your pipeline into multiple jobs and hopefully, you now have a good idea of how that is done. Make sure and check back in the future for a post on how to take repeated tasks and make them reusable.

Azure DevOps Publish Artifacts for ASP.NET Core

This post is going to build on the Azure DevOps project we created in the last few posts and get the build pipeline to the point you have the application’s binaries. If you are just joining this series check out the previous posts to catch up.

Getting Started with Azure DevOps
Pipeline Creation in Azure DevOps

Edit the Pipeline

First, we need to get back to the pipeline we were working on. From the Project menu select Pipelines.

This will land you on a page that lists your recently run pipelines. If you don’t see your pipeline list you might have to click the All option near the top of the page. Since we only have one pipeline in this project we can use the ellipsis to open a context menu and click Edit.

Publish the Application

At this point, the YAML for our pipeline looks like the following.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: [email protected]
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

The pipeline will currently tell us if the included project builds, but doesn’t provide us with the results of that build. Using the Task panel on the right search for the .NET Core task and then click the resulting task. This is the task you would want to use to invoke any of the .NET CLI commands.

Use the drop-down for Command and select publish. For this sample, the defaults for the rest of the settings will be fine. Finally, click Add to add the task to the YAML file.

The following is the resulting YAML.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: [email protected]
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: [email protected]
  inputs:
    command: 'publish'
    publishWebProjects: true

Before we move on I want to point out the Settings link above the tasks in the YAML editor. Clicking Settings will load that task into the task panel on the right of the screen where you can make changes and then if you hit the add button it will replace your existing task with a new one with your new options selected. Be careful to not change the selection in the YAML editor as the add button is just replacing the selected text not remembering what task you click settings on. When finished click the Save button and go through the commit process. When that is finished click the Run button to execute the pipeline.

Publish Build Artifacts

The pipeline run should succeed, but we still don’t have any files we can use. Learning what variables are available in the pipeline and how to use them is one of the hardest parts of getting started with Azure Pipelines. For our example, we are trying to get the two zip files created by the publish step above which means our pipeline will need to publish artifacts to make the files available. We are going to tweak the publish command from above with an output directory using the builtin Build.ArtifactStagingDirectory variable. The following is the full task with the changes.

- task: [email protected]
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

With the file we need in the artifact staging directory we need to publish those files using the Publish Pipeline Artifact task. The following is the full task that publishes the artifact staging directory to the pipeline.

- task: [email protected]
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

For reference, the following is the full YAML for the pipeline with all the above changes.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: [email protected]
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
  
- task: [email protected]
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)'

- task: [email protected]
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    publishLocation: 'pipeline'

Save and run the pipeline. When the pipeline is complete on the result page you will see 1 published for artifacts.

Click on 1 published and it will take you to a page that lists the artifacts. If you mouse over any of the rows you will see the option to download the associated file(s).

Quick Tip

As I stated above getting a handle on what directories are where can be a pain. If you ever need to see what files are where you can use the following publish task to output the full set of files the pipeline is using by publishing the pipeline’s entire workspace. This has helped me in the past to orient myself.

- task: [email protected]
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'

Wrapping Up

Our pipeline is now at the point we have files we could deploy. Hopefully, this gives you a good jump start on your own build pipelines. Azure Pipelines is a huge topic and this is a very basic build so keep an eye out for more posts on this topic in the future.

Pipeline Creation in Azure DevOps

This post is going to walk through creating a new build pipeline in Azure DevOps. This post is going to stick with a very simple example which we will build on in future posts. If you are new to this series of post check out the related posts.

Getting Started with Azure DevOps

This post will all happen from the Azure DevOps website so get logged in to your account select the project you will be working with before continuing. The project this sample is using is named Playground.

Pipeline Creation

From the project menu on the right of the site click the Pipelines option.

Since our sample project doesn’t have any pipelines setup we will see a landing page telling us to create a new pipeline. Once you have some pipelines this page is a lot more useful. Click the Create Pipeline button to continue.

The next step is to pick where our code is stored. For this sample, the code is in an Azure Repo Git repository, but as you can see Azure DevOps is pretty open about where your code is stored. As you will see from the screenshot there are a bunch of YAML based options and a very small option to use the classic editor. The classic editor is much easier to get started with, but the YAML options are getting the most attention from Microsoft and have the advantage of being stored in Git with your source so I recommend going with a YAML option even though there is more of a learning curve.

Next, select the repo this pipeline is for.

In the next step, Configure, you are given a list of templates to pick from which really helps when your new to yaml. Our sample applications are ASP.NET Core so click the Show More button and click ASP.NET Core. As you can see from the screenshot Azure DevOps can build just about anything and isn’t restricted to Microsoft based tech.

The result is the following YAML file. At this point, we aren’t doing to dive into the particulars of what the YAML is doing and go with the default. To continue to click the Save and run button.

Since the YAML is stored in the repo the save process is actually making a commit to a branch. Click the Save and run button on the commit dialog and it will save the YAML file to your branch and run the pipeline.

Pipeline Results

The following are the results from the pipeline run and it turns out to have failed. If you click the highlighted error it will take you to the detailed logs of the pipeline which will normally give you a good indication of why the pipeline failed.

The following screenshot is the result of clicking on the error. As you can see it provides a the output of the build command.

In this case line, 43 provides us with the reason the build failed. The following is the full line since the screenshot cuts it off. In this case, the issue is the agent running the build doesn’t have .NET Core 3.1 installed.

usr/share/dotnet/sdk/3.0.102/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.TargetFrameworkInference.targets(127,5): error NETSDK1045: The current .NET SDK does not support targeting .NET Core 3.1. Either target .NET Core 3.0 or lower, or use a version of the .NET SDK that supports .NET Core 3.1. [/home/vsts/work/1/s/src/WebApp1/WebApp1.csproj]

Fixing the Pipeline

Click the back button in your browser to return to the pipeline results page. Click the three dots in the top right of the results page and select Edit pipeline. This will open an edit with the YAML for the build open.

Using the Tasks helper on the right side of the screen we are going to select the Use .NET Core task which will allow us to install the version of .NET Core we need to build our applications.

Enter the version of .NET Core your application needs, 3.1 in this case and hit add.

The following is the resulting YAML. Note that the above helper isn’t required and you can hand-edit the YAML if you want.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

steps:
- task: [email protected]
  inputs:
    packageType: 'sdk'
    version: '3.1.x'

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

Hit the Save button in the top right of the page, enter a commit message and click the Save button in the bottom right of the page. This will return you back to the edit screen for your YAML. Click the Run button on the top right of the page to start the pipeline.

This round (for the sample application) the pipeline will succeed.

Wrapping Up

Hopefully, this will give you a good jumping-off point to build your first Azure DevOps Pipeline. There is a lot a depth in Pipeline some of which I will explore in some future posts.

Getting Started with Azure DevOps

I have been doing a lot of work lately with our build and release pipelines in Azure DevOps and while it is fresh on my mind I’m going to do a few posts to remind me of some of the things I have leaned and may or may not have gotten to use. This post is going to serve as a jumping-off point for the post that will follow for readers who may not have used Azure DevOps in the past.

Before moving forward make sure and sign up for a free Azure DevOps account.

Create a new Project

When you first log in to Azure DevOps you should end up at something like the following that lists all your organizations on the left, ericlanderson being the sample organization I’m using. In the man section of the page make sure you are on the Projects tab and then click the New project button.

On the next screen, all that is required is a Project name. If you do want to allow public access to the project you will have to tweak an organization policy. Under Advanced you can also tweak how you want to manage work for the project, but that is out the scope of the point of this post. When done click the Create button.

Initialize a Repo

When project creation is complete you will be forwarded to the project landing page as you can see below. Now we want to initialize a new repo for this project. Click Repos button.

Since the sample projects are going to be .NET based I opted to use a .gitignore for Visual Studio. When the options you want are selected click the Initialize button.

When the initialization process is complete you will be taken to a files view of the repo.

Connect to project and clone the repo from Visual Studio

Now that we have the Azure DevOps setup it is time to switch over to Visual Studio and interact with our new project. In Visual Studio all the interactions with Azure DevOps will happen via the Team Explorer window. The first step is to connect to the project. In the Team Explorer window hit the plug icon.

Next, click Manage Connections and then click Connect to a Project.

In the dialog that shows you will need to use the drop-down and select Add an account if you already have an account connected as I do, if not then the process might be slightly different.

After completing the login process with your Azure DevOps credentials you should see your organization listed and under it your project. In this case, the project and repo have the same name. Select the repo and click Clone.

Create Sample Applications

Now that the repo is cloned I’m creating a couple of sample applications using the .NET CLI. Open a command prompt in the root folder of the clone from above and use the following command to create a new Visual Studio solution.

dotnet new sln -n Playground

Next, use the following command to create a new web application.

dotnet new webapp -o src/WebApp1

Then add the new project to the solution file.

dotnet sln add src/WebApp1/WebApp1.csproj

I then repeated the project creation process a second mostly to give us more to work with in future posts. Here are the commands to create the second project and add it to the existing solution.

dotnet new webapp -o src/WebApp2
dotnet sln add src/WebApp2/WebApp2.csproj

Commit Code and Push to Azure DevOps

Back in Visual Studio in the Team Explorer window, we need to switch to the Changes area. There are a couple of ways to get to the changes area. If you are in the home area you can click the changes button.

Another option is to click the current area and select the new area from the drop-down menu. For example, in the following screenshot, I clicked on Branches to show the drop-down and could then click Changes.

Enter a commit message and if you use the dropdown on the Commit All button and select Commit All and Sync it will push all of the changes to the associated Azure DevOps branch.

Wrapping Up

Hopefully if you are new to Azure DevOps this will give you a good jump-off point and will provide a base for some future posts.

Azure App Service with On-premises Connection

Imagine you get a new project with the ability to use whatever cloud services you want. You jump at the change and dream up an amazing architecture using Azure. Next, you present your plan to the rest of your team and discover a new requirement of having to use an existing on-premises database.

Does this mean your grand plans are shot? Thankfully not, as Azure has multiple solutions that allow you to connect to your existing on-premises resources to enable hybrid cloud strategies.

In this post, we are going to use one of the options, Hybrid Connections, to connect from a web site hosted in an App Service to an on-premises database.

Sample Application

The base sample application we will be using for this post is a new Razor Pages App targeting .NET Core 3. We will walk through actually connecting to the database later in this post. To get started you need the new app created and running in an Azure App Service.

I have multiple walkthroughs of creating a new application and publishing them to Azure so I’m not going to rehash that here. If you need help getting your app created and published to Azure you can check out my Deploying an ASP.NET Core Application to Microsoft Azure post and instead of using the Razor template use the Web App template like the following.

dotnet new webapp

Also, note that Hybrid Connections aren’t available on the free or shared hosting plans so when you are setting up your publish profile avoid those options.

Add a Hybrid Connection to an App Service

From the Azure Portal open the App Serice you created above and under the Settings section of the menu click Networking.

In the networking details, we want to click Configure your hybrid connection endpoints.

I’m going to point out again that Hybrid Connections aren’t available at free and shared scales levels. If your App Service is on a free or shared scale use the Scale up menu option to switch to a scale level that will support Hybrid Connections.

From the Hybrid connections detail page click Download connection manager. When done this will need to be installed on the machine that is running the on-premises database you want to connect to.

Next, click Refresh and then click Add hybrid connection.

Now on the Add hybrid connection page click Create new hybrid connection.

In order to create a new hybrid connection, Azure will require some information. The key parts here are the Endpoint Host which is the name of the machine that is hosting the database you wish to communicate with and the Endpoint Port which will need to be the port that your database is configured to communicate over.

Hybrid Connection Manager on the host machine

Now that the Azure side is configured we need to use the Hybrid Connection Manager application that we installed on the target machine above to allow talk to our App Service.

After opening the Hybrid Connection Manager on the target machine click Add a new Hybrid Connection.

Now Select the Subscription the App Service is a part of. After the list of available connections, loads select the one created above and finally click Save.

After making the above changes my hybrid connection continued to show offline in Azure. After some searching, I found a blog post that suggested restating the Azure Hybrid Connection Manager Service which cleared the problem up for me.

Sample Application Changes to Connect to On-Premises Database

This is a very raw test, but it gets the point across. First, add a reference to the System.Data.SqlClient NuGet package. Next in the Index.cshtml.cs delete the OnGet function and replace it with the following.

public async Task OnGetAsync()
{
    Tables.Clear();
    var connectionString = "data source=Server;initial catalog=master; User Id=User;Password=Password"";
    await using var connection = new SqlConnection(connectionString);
    await connection.OpenAsync();
    await using var command = 
                    new SqlCommand("SELECT name FROM sys.tables", connection);
    await using var reader = command.ExecuteReader();

    while (await reader.ReadAsync())
    {
        Tables.Add(reader["name"].ToString());
    }
}

The above connects to the master database on the specified server and pulls a list of table. Note that you will need to modify the connection string to something valid for your system. It is also important to know that you can’t use integrated security with this setup so you will need to specify a user and password that exists on your SQL Server. Add the following property in the same file.

public List<string> Tables { get; set; } = new List<string>();

Add the following to the bottom of the Index.cshtml which will output the table list we pulled above to the page.

@foreach (var table in Model.Tables)
{
    @table <br/>
}

After the changes are done republish the application to your Azure App Service. After the publish is done your site should show with a list of tables that exist in the master database of your SQL Server.

Wrapping Up

Hybrid connections is a great way to take a workload to the cloud without having to actually move all your data. It is also one of those features of Azure that I had no idea that existed before a week ago.

If you need a lot of hybrid connections look closely at the pricing as the number you can use is tied to what App Service scale you are using. The number of available starts at 5 and can go up to 200 with the more expensive App Service scales.

Publish a .NET Core Worker Service to Azure

In last week’s post, .NET Core Worker Service, we created a .NET Core Worker Service and then showed how to host it as a Windows Service. This week we will be taking the application created in last week’s post and publishing it to Azure. If you haven’t read last week’s post I recommend you at least get through the application creation bits. Feel free to skip the part about making the application run as a Windows Service since we will be pushing to Azure for this post.

Publish to Azure

This post is going to assume you already have an active Azure account. If not you can sign up for a free Azure account. This post is also going to be using Visual Studio 2019 Preview for the publication process.

In Visual Studio right-click on the project and click Publish.

Since this project already has a folder publish setup from last week’s post my screenshot might look different from yours. Click the New link.

This will launch the Publish profile dialog. We will be publishing to Azure WebJobs using the Create New option. After verifying the settings are correct click Create Profile.

The next screen gets the details of the App Service that will be created. I did a new resource group instead of using the one that was defaulted in and a new hosting plan so that I could utilize the free tier, but both of these changes are optional. When you have everything set click the Create button.

Once create is clicked it takes a couple of minutes to deploy the application. This first deploy is going to be a throwaway for us. There are a few settings we need to tweak to get the application running the way we want. After the initial deployment is complete we end up back on the profile details screen. Click the Edit link.

The Publish Profile Settings dialog will show. Since this was written while .NET Core 3 is still in preview I’m using the Self-contained option for Deployment Mode. If you are doing this after the final release then you can skip this step. The second thing I am changing is the WebJob Type and I’m setting it to Continuous. This is because our Service is controlling its own work schedule and not being triggered by something else. Click Save to commit the changes and return to the publish dialog.

Now push the Publish button to push the application to Azure.

Use Azure Portal to Verify Application is Working

Now that our application is running in Azure how do we verify that it is actually executing as we expect? Since the only thing the sample application does it output some logs we are going to have to find a way to show the output of the logging. To start head over to the Azure Portal. Since we just published the application the resource we are interested in should show in your Recent resources section of the Azure Portal homepage. As you can see in the screenshot the one we are interested in is the first option and matches the name we saw above in Visual Studio. Click on the resource.

From the menu section scroll down to the Monitoring section and click App Service logs. Now turn On the option for Application Logging (Filesystem) and for the Level option select Information since that is the log level our sample application is outputting. Then click Save.

Back in the same Monitoring group of the menu select Log stream.

Wrapping Up

Running a worker in Azure ended up being pretty simple. Not that I’m surprised Microsoft tends to make all their products work well with Azure. I’m sure you will find a way to make a more useful worker than we had in this example.

Azure Function App Log Streaming

One of the things I have noticed while exploring Azure Function Apps is it is important to find ways to track what is going on during execution. While exploring the Azure Portal for one of my functions apps I noticed an option for steaming logs. This post is going to show how to get to the streaming logs for a Function App.

Log Streaming Location

From the Azure Portal open your Function App and select Platform features and then click the Log streaming link.

Log Streaming View

The following screenshot is from my sample App and has two different function executions listed.

Wrapping Up

For this small sample application, this feature isn’t critical, but on a larger application, it would become much more important. Log streaming is a quick way to get an idea of what is going on, but would only be one part of knowing what is going on.