DevOps Consulting: Pipeline Runners

Pipeline Runners, and the automation pipelines they run, simplify software delivery by providing a hosted mechanism to automate running tests, building packages and artifacts, and performing deployments. In other words, Pipeline Runners run Pipelines, which are the foundation of your Automated Delivery process and your overall DevOps or Cloud Engineering approach.

You’ll need to understand the role of Pipeline Runners in the delivery process for your team to implement automated continuous integration, continuous delivery, and continuous deployment.

Continue reading to learn more about Pipeline Runners!

No time to read? Hear it from the experts in DevOps consulting!

Why are Pipeline Runners Needed?

In short, pipeline runners enable automation for your continuous integration and delivery processes.

The delivery process can be challenging and time-consuming—especially with complex, distributed systems developed by multiple teams. Automation pipelines for testing, build processes, compliance checks, code quality checks, secrets management, and multi-environment deployments all contribute to delivery complexity. It is challenging to ensure consistency across all development efforts without some form of automation.

Pipeline runners provide a consistent environment for all aspects of software delivery. Most pipeline runners support adding pipeline configuration directly into project repositories so you can keep your delivery automation logic with the code your pipeline is automating.

Many pipeline runners even provide a way to share automation tasks across projects to mitigate duplication of automation scripts and actions. For example, GitHub Actions can be published as separate repositories and included in any project’s automation workflows.

What is a Pipeline Runner?

Pipeline runners are essentially engines that automatically perform tasks, or run pipelines, whenever code or configurations change. You can think of them as computers that run your delivery scripts (i.e., test, build, publish, and deploy), so you don’t have to run scripts on your local computer.

Pipeline runner compute types are typically one or more of the following: Virtual Machines, Docker Containers, or Serverless Functions.

Depending on your use case, you may use different products for your Continuous Integration pipeline than you do for your Deployment/Delivery pipelines. Products designed specifically for Deployments, such as Harness, Spinnaker, ArgoCD, or FluxCD, often have additional features that provide advanced logging, auditing, and reporting capabilities that projects may require.

How do Pipeline Runners Work?

Most pipeline runners allow you to provide pipeline configuration directly within your project repositories.

For example, with GitHub Actions, you specify .yaml files in .github/workflows. The following table shows some locations to specify workflows by some common Pipeline Runner services:

Pipeline Runner configuration file syntax will vary by service but will commonly follow this structure:

• Define workflows
• Define actions within a workflow
• Define steps within an action

The actual compute type is often abstracted from users so that you can focus on the automation instead of the infrastructures. Compute type abstraction is common with Source Control Management (SCM) software, also called Version Control Systems (VCS), like Software-as-a-Service (SaaS) GitLab and GitHub. However, even in cases like these, you can typically specify the operating system or container with which you want your tasks to run.

It’s also common for SaaS-based SCM systems to allow you to “bring your own infrastructure” by configuring compute instances that you own to run tasks. Providing your own infrastructure is especially useful if you don’t have the capacity or need to host the entire SCM system internally, but still need to run automations within a private network (connecting to an internal database, accessing 3rd party tools, or performing deployments to internal resources, for example) or have specific compute requirements for your pipelines that aren’t met by the available runner types.

Compute types for Pipeline Runners and how they are used will vary by service. Some common approaches to Pipeline Runner compute are:

• A Virtual Machine that is always running and ready to accept new automation workloads (whether scripts are executed directly on the VM or within a container inside the VM). This approach allows for a single VM to handle workloads from multiple projects at a time, and it lowers the latency of how long it takes for an automation to start.
• On-Demand Virtual Machines that only start up when a workflow automation trigger happens. This approach saves on cost because you do not have to pay for a running VM when you’re not using it, but it increases the latency for an automation to start.
• On-Demand containers within a container orchestrator like Kubernetes. This approach lowers the startup latency because containers are generally faster to start than VMs but requires a container orchestrator running all the time and the know-how to manage it. Container-only-based Pipeline Runners also have additional considerations. If you want to, for example, run Docker commands within a Docker container (Docker-in-Docker), there are potential networking hurdles to overcome.

It’s worth noting that some SaaS or UI-driven pipeline runners, such as Harness or Spinnaker, often further abstract the underlying compute so you can focus more on pipeline logic, though the compute will likely still follow one of the above approaches.

What options are available for Pipeline Runners?

Cloud engineering and DevOps often require careful research and selection of tooling, and selecting the right Pipeline Runner is no exception as there are many options to choose from. Some common runners include:

• GitHub Actions
• GitLab Pipelines
• CircleCI
• TravisCI
• Bitbucket Pipelines
• Buddy.works
• Jenkins
• AWS CodeBuild and AWS Code Pipeline
• Azure Pipelines
• Argo
• Flux
• Team City
• Harness.io

This is certainly not an exhaustive list, and which pipeline runner you choose will depend on multiple factors, such as:

• Which SCM system your team or organization is using
• Whether or not there is expertise to support customizations and infrastructure, if applicable
• If you have audit or regulatory requirements that require advanced logging and/or approvals within the deployment process.
• Cost

Most pipeline runners will provide enhanced features (such as ease of triggering downstream pipelines, observability dashboards, or easily shareable actions), but in general, they have the same general pattern and features:

• Specify rules to trigger pipelines, such as commits to specific branches, tags and versions, pull requests, and manual triggers
• Define what operating system or container image to run a task inside
• Define secrets for secure automations
• Provide an interface to view the automation execution

You can rest assured that regardless of which pipeline runner you use, your CI/CD pipelines can typically be implemented within it. You’ll just need to decide which one works best for your team(s).

How much does a Pipeline Runner cost?

Asking how much a Pipeline Runner will cost is like asking, “How much will it cost to go to the store?”. The answer depends on many factors, such as:

• Are you taking public transportation, or are you walking?
• Do you own your own vehicle? If so, what kind of vehicle?
• How much does fuel cost in your area?
• How far away is the store?

Many Pipeline Runners provide a SaaS-based option, and they’re often free to start. Some Pipeline Runners have no SaaS option and require you to provide the infrastructure, which will incur hosting, maintenance, and support costs. Many Pipeline Runners provide a per-minute cost approach which means that you are charged per minute of pipeline execution, and some have a tiered pricing model for advanced features.

To determine how much a Pipeline Runner will cost for you, you’ll first need to determine how it will be used in your ecosystem by answering questions like:

• How many pipelines will be run in a given time frame?
• How many pipelines will need to run concurrently?
• How big should the pipeline runners be in terms of CPU and memory?
• Which advanced pipeline runner features will my team need to leverage?
• Will we use SaaS-hosted runners, or will we need infrastructure to host our own?
• Are the pipelines going to be running Open Source (OS) projects? (some Pipeline Runner services offer perks for OS projects)

Answers to these questions (and more like them) will narrow down the scope of which Pipeline Runner is right for you.

Here is an extremely high-level breakdown of the costs of 4 common pipeline runners to give you a rough idea of what to expect:

Again, this is a high-level overview of pricing, and the numbers listed are a bit oversimplified to focus on Pipeline Runners. For example, GitLab Ultimate does allow for a higher initial limit of minutes per month, but it also comes with additional features such as advanced security testing, vulnerability management, and compliance pipelines, and GitHub Actions have different pricing multipliers for different operating systems.

How do I get started with Pipeline Runners?

Depending on where your code is located and how big your project and team is, there are only a few steps to get started writing pipelines and utilizing Pipeline Runners. Here is a scenario for smaller projects with two common Pipeline Runners. Assume that you have a repository with a JavaScript app with tests, and you want to run those tests on commits to the main branch.

GitHub Actions

For GitHub Actions, the first thing you’ll need to do is define a pipeline configuration file:

.github/workflows/test.yaml

# Name the workflow
name: Run Tests
# Define triggers
on:
  push:
    branches: [main]
# Define the jobs
jobs:
  # Test job
  test:
    # Define Runner
    runs-on: ubuntu-latest
    # Define the steps
    steps:
      # Checkout the repository code
      - name: checkout repo
        uses: actions/checkout@v2
      # Run the tests
      - name: Run Tests
        run: |
          npm install
          npm run test

This workflow configuration will run on commits to the main branch and will run npm run test against the repository code in an Ubuntu runner.

Once you have contents in your workflow configuration file, commit the changes and push to the base branch to trigger your pipeline.

Here are some resources to help you get started with GitHub Actions:

• Quickstart for GitHub Actions
• GitHub Actions Marketplace

GitLab Pipelines

Similarly, for GitLab pipelines, you’ll define a pipeline configuration file:

.gitlab-ci.yml

# Define runner image
image: node:latest

# Name the workflow
Run Tests:
  stage: test
  # Define triggers
  rules:
    - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
  when: always
  # run the test
  script:
    - npm install
    - npm run test

This workflow configuration will run on commits to the base branch (i.e. main) and will run npm run test against the repository code in a Node image.

Once you have contents in your workflow configuration file, commit the changes and push to the base branch to trigger your pipeline.

A great place to help you get started with GitLab Pipelines is the GitLab CI/CD Quick Start guide.

Next Steps

You should now have a decent idea about the role Pipelines and Pipeline Runners play in automated software delivery, including why they’re needed, what they are, how they work, some options available to you, roughly how much they cost, and how to get started.

Having a high-level understanding is only the tip of the iceberg for Pipeline Runners, however, and there are many features, strategies, and patterns that you and your teams can leverage to take full advantage of Pipeline Runners and enhance your Software Delivery Automation and DevOps approach.

DevOps Consulting Services

Regardless of where you are on your DevOps maturity journey, you should consider leveraging the expertise of 3rd parties or vendors, like Bitovi DevOps Consulting. You can easily bring DevOps and operations expertise to your projects to bootstrap your infrastructure, automations, DevOps culture, and maturity plan.