GitHub

GitHub: Use Actions to Run Multiple Jobs

In the post, we are going to take our sample Workflow that builds two ASP.NET Core web applications and split it so each web application is built individually. This post is using the repo and Workflow built in the following posts if you need to catch up.

GitHub: Import an Azure DevOps Repo
GitHub: Use Actions to build ASP.NET Core Application
GitHub: Use Actions to Publish Artifacts

Starting Point and the Plan

Our Workflow currently contains a single job that just happens to build two ASP.NET Core web application based on the fact that the .NET CLI picks up and builds both applications. The following is the YAML for our current Workflow.

name: .NET Core

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/[email protected]
      
    - name: Setup .NET Core
      uses: actions/[email protected]
      with:
        dotnet-version: 3.1.101
 
    - name: Install dependencies
      run: dotnet restore
      
    - name: Build
      run: dotnet build --configuration Release --no-restore
      
    - name: Test
      run: dotnet test --no-restore --verbosity normal

    - name: Publish
      run: dotnet publish 
    
    - name: Upload WebApp1 Build Artifact
      uses: actions/[email protected]
      with:
        name: WebApp1
        path: /home/runner/work/Playground/Playground/src/WebApp1/bin/Debug/netcoreapp3.1/publish/
        
    - name: Upload WebApp2 Build Artifact
      uses: actions/[email protected]
      with:
        name: WebApp2
        path: /home/runner/work/Playground/Playground/src/WebApp2/bin/Debug/netcoreapp3.1/publish/

This post is going to take this Workflow and split the build and publish of the two web applications into two jobs. By splitting the Workflow into multiple jobs we open the possibility that the jobs can run in parallel. One reason to do this would be to speed up the total Workflow run time if you have parts of your build that are independent. Another example of why you would need multiple jobs is if the different jobs need different needed to run on different operating systems such as one needing to run Windows and another a Linux.

Creating the Jobs

The following is the Workflow set up along with the job to build the first web application. The first change was of the ID from build to build_web_app1 since each job has to have a unique ID. Most of the rest of the highlighted changes are related to the .NET CLI commands that are now directed at a specific project. Do also note that we changed from a hardcoded path to using expression to get the workspace path which is the ${{ github.workspace }} bit instead of /home/runner/work/Playground/Playground/. See the expression syntax docs for more info.

name: .NET Core

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build_web_app1:

    name: Build WebApp1
    runs-on: ubuntu-latest

    steps:
    - uses: actions/[email protected]
      
    - name: Setup .NET Core
      uses: actions/[email protected]
      with:
        dotnet-version: 3.1.101
 
    - name: Install dependencies
      run: dotnet restore ${{ github.workspace }}/src/WebApp1/WebApp1.csproj
      
    - name: Build
      run: dotnet build ${{ github.workspace }}/src/WebApp1/WebApp1.csproj --configuration Release --no-restore
      
    - name: Test
      run: dotnet test ${{ github.workspace }}/src/WebApp1/WebApp1.csproj --no-restore --verbosity normal

    - name: Publish
      run: dotnet publish ${{ github.workspace }}/src/WebApp1/WebApp1.csproj
    
    - name: Upload Build Artifact
      uses: actions/[email protected]
      with:
        name: WebApp1
        path: ${{ github.workspace }}/src/WebApp1/bin/Debug/netcoreapp3.1/publish/

Here is the full file with both jobs defined. As you can see the second job is basically the same thing as the first one with a different ID, name, and project.

name: .NET Core

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build_web_app1:

    name: Build WebApp1
    runs-on: ubuntu-latest

    steps:
    - uses: actions/[email protected]
      
    - name: Setup .NET Core
      uses: actions/[email protected]
      with:
        dotnet-version: 3.1.101
 
    - name: Install dependencies
      run: dotnet restore ${{ github.workspace }}/src/WebApp1/WebApp1.csproj
      
    - name: Build
      run: dotnet build ${{ github.workspace }}/src/WebApp1/WebApp1.csproj --configuration Release --no-restore
      
    - name: Test
      run: dotnet test ${{ github.workspace }}/src/WebApp1/WebApp1.csproj --no-restore --verbosity normal

    - name: Publish
      run: dotnet publish ${{ github.workspace }}/src/WebApp1/WebApp1.csproj
    
    - name: Upload Build Artifact
      uses: actions/[email protected]
      with:
        name: WebApp1
        path: ${{ github.workspace }}/src/WebApp1/bin/Debug/netcoreapp3.1/publish/
        
  build_web_app2:

    name: Build WebApp2
    runs-on: ubuntu-latest

    steps:
    - uses: actions/[email protected]
      
    - name: Setup .NET Core
      uses: actions/[email protected]
      with:
        dotnet-version: 3.1.101
 
    - name: Install dependencies
      run: dotnet restore ${{ github.workspace }}/src/WebApp2/WebApp2.csproj
      
    - name: Build
      run: dotnet build ${{ github.workspace }}/src/WebApp2/WebApp2.csproj --configuration Release --no-restore
      
    - name: Test
      run: dotnet test ${{ github.workspace }}/src/WebApp2/WebApp2.csproj --no-restore --verbosity normal

    - name: Publish
      run: dotnet publish ${{ github.workspace }}/src/WebApp2/WebApp2.csproj
       
    - name: Upload Build Artifact
      uses: actions/[email protected]
      with:
        name: WebApp2
        path: ${{ github.workspace }}/src/WebApp2/bin/Debug/netcoreapp3.1/publish/

After all the edits are done commit the changes to the repo to run the Workflow. From the results of the Workflow run, you will see that it now has two jobs and we still got the artifacts for both applications as we had before.

Wrapping Up

If your applications need it breaking them up into different jobs can be helpful not only with Workflow runtimes but it can also help your ability to reason about what each part of your build process is doing.

GitHub: Use Actions to Publish Artifacts

This post is going to take the GitHub Actions Workflow we set up in the last post and add a couple of steps that will provide us with access to our application’s binaries. If you are new to this series the following post will catch you up if needed.

GitHub: Import an Azure DevOps Repo
GitHub: Use Actions to build ASP.NET Core Application

Edit the Workflow

Our first step is to get back to the Workflow we want to edit. At the top of the repo click Actions.

On the left side of the screen select the specific Workflow, .NET Core in this case. Now that the list is filtered to just the Workflow we are interested in select the three dots on the most recent run and then click View workflow file.

On the next screen click the pencil above the Workflow file to edit the Workflow.

Add the end of the file adds a call to the .NET CLI to publish. The following is the full file and the last two lines are the publish step.

name: .NET Core

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/[email protected]
      
    - name: Setup .NET Core
      uses: actions/[email protected]
      with:
        dotnet-version: 3.1.101
 
    - name: Install dependencies
      run: dotnet restore
      
    - name: Build
      run: dotnet build --configuration Release --no-restore
      
    - name: Test
      run: dotnet test --no-restore --verbosity normal

    - name: Publish
      run: dotnet publish

Click the Start commit button and commit the change to the branch of your choice. I committed the change directly to master which triggered the Workflow to run. From the logs of the Workflow, you can see that the publish step executed successfully.

Publish Build Artifacts

Now that we have the two web applications publishing we need a way to get the file for the applications. To do this we are going to use the Upload Artifacts Action. I used the output of the Publish step above to find the path to the publish folder for each application and then used an Upload Artifacts Action for each application. The following are the two steps added to the bottom of our existing Workflow.

- name: Upload WebApp1 Build Artifact
  uses: actions/[email protected]
  with:
    name: WebApp1
    path: /home/runner/work/Playground/Playground/src/WebApp1/bin/Debug/netcoreapp3.1/publish/
    
- name: Upload WebApp2 Build Artifact
  uses: actions/[email protected]
  with:
    name: WebApp2
    path: /home/runner/work/Playground/Playground/src/WebApp2/bin/Debug/netcoreapp3.1/publish/

After checking in the changes let the Workflow run. Once complete if you click on the details of the Workflow run you will see it now has Artifacts that can be downloaded.

Wrapping Up

We now have a Workflow that results in files we could actually deploy on top of verifying that the applications builds and the tests pass. I hope this has given you a good jumping-off point to build your own Workflows. We have only scratched the surface of what can be done with GitHub Actions and I’m looking forward to seeing what else we can do.

GitHub: Use Actions to build ASP.NET Core Application

In this week’s post, we are going to use GitHub’s Actions to build one of the applications that we imported from an Azure DevOps Repo. The sample repo we are using can be found here.

Create an Action Workflow

From the repo in GitHub click the Actions option at the top center of the screen.

The Actions page will make suggestions based on the contents of the repo you are working with. In our case, the suggested .NET Core workflow is the one we are interested in. Click the Set up this workflow button.

The next screen that shows will be an editor loaded with the YAML for the .NET Core workflow we selected. For now, we are going to keep the YAML that was defaulted in and click the Start commit button. This workflow may or may not work for our repo at this point we are still exploring and can change as needed after we get a feel for how Actions work.

The next dialog is the commit details. For this initial change, we are going commit directly to master with the default commit message. Click Commit new file to continue.

View Workflow Status

Now that we have a workflow set up click on the Actions tab of the repo again to view the list of workflows and their status. As you can see in this screenshot the commit queued our new workflow to run.

The workflow finished quickly so I didn’t get to see the details while it was running, but if you click on commit title, Create dotnetcore.yml in this example, it will take you to the detail of this workflow run. From this view, you will see the jobs for the workflow listed on the left side of the screen, we only have one job which is the build. When you click on a job you will see the logs from that job. The following screenshot is the sample build job with the details of the build step expanded to show that both WebApp1 and WebApp2 were built.

Wrapping Up

Hopefully, this post will give you a good jumping-off point to create your own GitHub Actions. I was impressed with how easy it was to get started and the wide verity of languages supported especially for a feature set that has been out for less than a year. Check back in next week for more exploration of Actions.

GitHub: Import an Azure DevOps Repo

Over the last couple of months, we have been exploring some of the features of Azure DevOps around Pipelines and Repos. I thought it would be interesting to see how the same type of setup and process might look using GitHub instead of Azure DevOps. I haven’t used GitHub other than for a basic repo before so I’m not sure how much of the Azure DevOps post will carry over, but we are going to find out. In this first post, we are going to import the repo we used in the Azure DevOps series from our Azure DevOps Repo.

Importing a Repo

To start in the project click the New button in the Repositories section of your GitHub dashboard.

On the next page click the Import a repository link.

The first thing the import process wants to know is the URL of our old repository. To get this we need to head over to our Azure DevOps Repo. Once in the Azure DevOps Repo click the Clone button.

When the dialog for clone shows you will see the URL, but before copying the URL hit the Generate Git Credentials button. This will create a username and password we will also need to enter when importing the repo at GitHub.

Here is the dialog after generating credentials.

Use the button next to the URL to copy URL and head back to GitHub and paste it into the URL box.  Next, enter a name for the repo and click Begin import.

The following is the screen you will see next while GitHub works on the import. Since the repo we are importing from needs credentials it will fail after a couple of minutes and ask for a login and password. If you hit refresh it will prompt you immediately without having to wait.

Copy and paste the values we generated in Azure DevOps above into the Login and Password boxes and then click Submit.

The page will update when the process is complete. GitHub will also send you an email so don’t feel the need to keep the page open while the process is running.

GitHub Repo Cleanup

Now that our repo is in GitHub we can clean up some of the items that were specific to Azure DevOps. For the sample project, this would include the files that were used in/or defined our build pipeline. The following are the files that can be deleted.

  • azure-pipelines.yml
  • build.yml

Wrapping up

As you have seen the repo transition from Azure DevOps to GitHub is a simple process. I’m looking forward to exploring how GitHub handles some of the scenarios from the Azure DevOps series. I’m betting that GitHub has support for most of them especially since the introductions of Actions.

GitHub and Azure Pipelines: Build Triggers

In response to my post on GitHub and Azure Pipelines, I got the following question on Reddit.

Does this automatically detect branches? From your screenshot, you’re building master. If you copy to feature-A, does a new pipeline automatically get created and built?

When I initially answered this question I didn’t go deep enough. The answer to does a new pipeline automatically get created and built is no as I replied, but I think the intent of the question is do I have to go set up a new pipeline every time I create a new branch and the answer to that is also no. The existing pipeline will be triggered when any change is checked in on any branch by default. Do note that it won’t be triggered when the branch is created only when a change is checked in.

Limiting Builds

There are a couple of ways to control what branches trigger continuous integration builds. The first is by making edits to the azure-pipeline.yml file in the repo and the second is via an override in the Azure Pipeline.

YAML

The official Build pipeline triggers docs are really good, but I will cover the basic here for including branches and excluding branches. Check for docs for information on path includes/excludes as well as how to control PR validation. As an example here is the yaml file used to define a build in this repo.

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet build Sqlite --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

In order to control what branches get built, we need to add a trigger section. The smilest example is to list the branches you want to build. Ending wildcards are allowed. See the following example (trigger section taken from the official docs).

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet build Sqlite --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

trigger:
- master
- releases/*

This would build master and all branches under releases, but nothing else. The following shows how to use includes and excludes together. Again the triggers section is taken from the official docs.

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet build Sqlite --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

trigger:
  branches:
    include:
    - master
    - releases/*
    exclude:
    - releases/old*

This would build master and everything in under releases that does not start with old. Really go read the official docs on this one to see all the ins and outs.

Azure Pipelines

To override the CI build from Azure DevOp go to the build in question and click Edit.

Next, select Triggers and Continuous integration and check Override YAML.

After checking the override you will see a lot more options light up. As you can see in the following screenshot the same include and exclude options are available with the same options for wildcards.

Wrapping Up

As you can see Azure Pipelines provides a lot of flex ability in how a build gets triggered. On top of what I covered here, there are also options for setting up scheduled builds as well as trigging a build with another build is completed. If you hit a scenario that couldn’t be covered I would love to hear about it in the comments.

GitHub and Azure Pipelines

A few weeks ago Microsoft announced that Visual Studio Team Services was being replaced/rebranded by a collection of services under the brand Azure DevOps. One of the services that make up Azure DevOps is Azure Pipelines which provides a platform for continuous integration and continuous delivery for a huge number of languages on Windows, Linux, and Mac.

As part of this change, Azure Pipelines is now available on the GitHub marketplace. In this post, I am going to pick one of my existing repos and see if I can get it building from GitHub using Azure Pipelines. I’m sure Microsoft or GitHub has documentation, but I’m attempting this without outside sources.

GitHub Marketplace

Make sure you have a GitHub account with a repo you want to build. For this post, I’m going to be using my ASP.NET Core Entity Framework repo. Now that you have the basic prep out of the way head over to the GitHub Marketplace and search for Azure Pipelines or click here.

Scroll to the bottom of the page to the Pricing and setup section. There is a paid option that is the default option. Click the Free option and then click Install it for free.

On the next page, you will get a summary of your order. Click the Complete order and begin installation button.

On the next page, you can select which repos to apply the installation to. For this post, I’m going to select a single repo. After making your choice on repos click the Install button.

Azure DevOps

After clicking install you will be thrown into the account authorization/creation process with Microsoft. After getting authorized you will get to the first set up in the setup process with Azure. You will need to select an organization and a project to continue. If you don’t have these setup yet there are options to create them.

After the process complete you will be land on the New pipeline creation process where you need to select the repo to use. Clicking the repo you want to use will move you to the next step.

The next step is a template selection. My sample is an ASP.NET Core application so I selected the ASP.NET Core template. Selecting a template will move you to the next step.

The next page will show you a yaml file based on the template you selected. Make any changes your project requires (my repo had two projects so I had to change the build to point to which project I wanted to build).

Next, you will be prompted to commit the yaml file to source control. Select your options and click Save and run.

After your configuration gets saved a build will be queued. If all goes well you will see your app being built. If everything works you will see something like this build results page.

Wrapping Up

GitHub and Microsoft have done a great job on this integration. I was surprised at how smooth the setup was. It was also neat to see a project that I created on Windows being built on Linux.

If you have a public repo on GitHub and need a way to build give Azure Pipelines a try.

Identity Server: External Authentication using Twitter

This post is going to cover adding authentication using Twitter to the same project that has been used in all of my IdentityServer examples. The same basic idea would apply to almost any third party authentication setup so this should give you a good starting point for any integration. The starting point of the code can be found here.

Create Twitter App

Before any code changes create a new application on Twitter via this page. Click Create New App to begin the process.

On the Create an application page enter all the requested information. Note that the website won’t allow a localhost address. If you don’t have a real address for your application just enter a random URL as I did here. When finished click Create your Twitter application.

Now that we have an application click on the Keys and Access Tokens tab. We will need both the Consumer Key and Consumer Secret when we get to the Identity Application.

Identity Application Changes

Now that we have a Twitter application ready to go let us dive into the changes needed to the Identity Application. The first step is to add a reference to Microsoft.AspNetCore.Authentication.Twitter via NuGet.

Next in the ConfigureServices function of the Startup class after app.UseIdentityServer() add the following.

app.UseTwitterAuthentication(new TwitterOptions
{
    AuthenticationScheme = "Twitter",
    DisplayName = "Twitter",
    SignInScheme = "Identity.External",
    ConsumerKey = Configuration["Authentication:Twitter:ConsumerKey"],
    ConsumerSecret = Configuration["Authentication:Twitter:ConsumerSecret"]
});

The first three options should a straight forward enough. The next two are the values from the Twitter application I mentioned above. In this example, I am storing the values using User Secrets which get pulled out of configuration. For more details on how to set up secrets, you can see this post.

The above are all the changes required. The Identity Application will now allow users to auth using Twitter.

Logging in using Twitter

As you can see below the login page now has a button for Twitter.

When the user chooses to log in using Twitter they are shown the following page where they must approve access to their Twitter account from your application.

If this is the first time a user has logged in with Twitter they will be prompted to enter an email address to finish registration.

Wrapping up

As you can see adding external authentication is super simple. Check out the Microsoft Docs on Twitter Auth (ASP.NET Core 2.0 so look out for differences if you are not on the preview bits) and IdentityServer Docs on External Auth for more information.

The finished code can be found here.

 

Visual Studio 2017 error encountered while cloning the remote Git repository

Going through the process of getting Visual Studio 2017 installed on all my machines has been pretty smooth. The new installer works great and makes it much clearer what needs to be installed.

The issue

The one issue I have had, which was only an issue on one install, is an error when trying to clone a repo from GitHub. I say GitHub but really it would be a problem with any Git repo. The following is the error I was getting.

Error encountered while cloning the remote repository: Git failed with a fatal error.
CloneCommand.ExecuteClone

The solution

After searching I found a lot of things to try. I uninstalled and reinstalled Git for Windows multiple times using both the Visual Studio Installer and the stand alone installer. I finally stumbled onto this forum thread which had a solution that worked for me. The following is a quote of the reason for the issue and a fix posted by J Wyman who is a software engineer for Microsoft’s Developer Division.

After intensive investigation we were able to determine that Git was accidentally loading an incorrect version of libeay32.dll and ssleay32.dll due to library loading search order and precedence rules. Multiple users have confirmed that copying the binaries from the “<VS_INSTALL>\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin\” folder to the “<VS_INSTALL>\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\libexec\git-core\” folder completely resolved the issue for him.

Any other users seeing similar problem should attempt the same solution.

I hope this gets other people up and running as it did me. My only worry about this fix is what happens with Git gets updated.