Skip to main content
Back to blog
DevOps#azure-devops#azure-pipelines#automated-testing#yaml-pipeline#test-automation#ci-cd

How to Create Azure DevOps Pipeline for Automated Testing

Step-by-step guide to creating an Azure DevOps pipeline for automated testing. Covers pipeline creation, YAML configuration, test result publishing, artifact storage, and pipeline triggers — with complete working examples.

InnovateBits4 min read
Share

Creating a pipeline for automated testing in Azure DevOps involves three decisions: what triggers the pipeline, what steps it runs, and how it reports results. This guide walks through each decision with complete, working YAML.

Creating the pipeline

Option A: From the UI (YAML)

  1. Go to Pipelines → + New pipeline
  2. Select your source: Azure Repos Git (or GitHub)
  3. Select your repository
  4. Choose Starter pipeline (edits in the browser) or Existing Azure Pipelines YAML file (if you already have a YAML file)

Option B: Commit azure-pipelines.yml

Create the file at the root of your repository:

touch azure-pipelines.yml
git add azure-pipelines.yml
git commit -m "Add CI pipeline"
git push

Azure DevOps detects this file automatically when you create the pipeline.

Pipeline structure for testing

A well-structured testing pipeline has three stages:

Stage 1: Validate       (fast checks — linting, compilation)
Stage 2: Unit Tests     (fast, isolated tests — run first)
Stage 3: Integration/E2E (slower, environment-dependent tests)

Failing fast in Stage 1 saves agent minutes and developer waiting time.

Complete pipeline YAML

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - release/*
  paths:
    exclude:
      - docs/**
      - '*.md'
 
pr:
  branches:
    include:
      - main
 
pool:
  vmImage: ubuntu-latest
 
variables:
  - group: test-environment-vars   # Variable group from Library
 
stages:
  # ── Stage 1: Validate ─────────────────────────────────────────────────────
  - stage: Validate
    displayName: Validate
    jobs:
      - job: Lint
        displayName: Lint and type check
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '20.x'
 
          - script: npm ci
            displayName: Install dependencies
 
          - script: npm run lint
            displayName: ESLint
 
          - script: npm run typecheck
            displayName: TypeScript check
 
  # ── Stage 2: Unit Tests ───────────────────────────────────────────────────
  - stage: UnitTests
    displayName: Unit Tests
    dependsOn: Validate
    jobs:
      - job: Unit
        displayName: Run unit tests
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '20.x'
 
          - script: npm ci
            displayName: Install
 
          - script: |
              npm run test:unit -- \
                --reporter=junit \
                --outputFile=results/unit-results.xml \
                --coverage
            displayName: Run unit tests
 
          - task: PublishTestResults@2
            displayName: Publish unit results
            inputs:
              testResultsFormat: JUnit
              testResultsFiles: results/unit-results.xml
              testRunTitle: Unit Tests — $(Build.BuildNumber)
              mergeTestResults: true
            condition: always()
 
          - task: PublishCodeCoverageResults@1
            displayName: Publish coverage
            inputs:
              codeCoverageTool: Cobertura
              summaryFileLocation: coverage/cobertura-coverage.xml
            condition: always()
 
  # ── Stage 3: E2E Tests ────────────────────────────────────────────────────
  - stage: E2ETests
    displayName: E2E Tests
    dependsOn: UnitTests
    condition: succeeded()
    jobs:
      - job: Playwright
        displayName: Playwright E2E
        timeoutInMinutes: 30
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '20.x'
 
          - script: npm ci
            displayName: Install
 
          - script: npx playwright install --with-deps chromium firefox
            displayName: Install browsers
 
          - script: npx playwright test --reporter=junit,html
            displayName: Run Playwright tests
            env:
              BASE_URL: $(STAGING_URL)
              TEST_EMAIL: $(TEST_USER_EMAIL)
              TEST_PASSWORD: $(TEST_USER_PASSWORD)
            continueOnError: false
 
          - task: PublishTestResults@2
            displayName: Publish E2E results
            inputs:
              testResultsFormat: JUnit
              testResultsFiles: playwright-results/results.xml
              testRunTitle: E2E Tests — $(Build.BuildNumber)
            condition: always()
 
          - task: PublishPipelineArtifact@1
            displayName: Upload HTML report
            inputs:
              targetPath: playwright-report
              artifact: playwright-html-report
              publishLocation: pipeline
            condition: always()
 
          - task: PublishPipelineArtifact@1
            displayName: Upload screenshots on failure
            inputs:
              targetPath: test-results
              artifact: test-screenshots
            condition: failed()

Setting up variables and secrets

Never hardcode credentials in YAML. Use variable groups:

  1. Go to Pipelines → Library → + Variable group
  2. Name: test-environment-vars
  3. Add variables: 
    STAGING_URL          = https://staging.yourapp.com
TEST_USER_EMAIL      = testuser@example.com
TEST_USER_PASSWORD   = [mark as secret] ****
  4. Reference the group in pipeline YAML: - group: test-environment-vars

Secret variables are masked in logs automatically.

Pull request validation

To block merges when tests fail:

  1. Go to Repos → Branches → [main branch] → Branch Policies
  2. Click + Add build validation
  3. Select your pipeline
  4. Set: Required (not Optional)
  5. Trigger: Automatic on PR updates

Now PRs to main cannot be merged until the pipeline passes.

Common errors and fixes

Error: Pipeline runs but no test results appear in the Tests tab Fix: The PublishTestResults task must find actual XML files. Add a debug step: - script: find $(System.DefaultWorkingDirectory) -name "*.xml" -type f to see what's generated.

Error: "The pipeline is not valid. Job 'E2E' has a dependency on 'UnitTests' which doesn't exist" Fix: dependsOn references the stage name (not displayName). Match it exactly: dependsOn: UnitTests.

Error: E2E tests time out after 10 minutes Fix: Add timeoutInMinutes: 30 (or appropriate value) to the job definition. Default timeout is 60 minutes for Microsoft-hosted agents, but PR validation pipelines have a 10-minute default.

Error: Tests fail with "ECONNREFUSED" — can't reach the application Fix: The staging environment URL is unreachable from the pipeline agent. Check if staging is behind a VPN or firewall that blocks Azure DevOps agents. Use self-hosted agents inside the network if needed.

Error: Variable group not found in pipeline Fix: Go to Library → [Variable group] → Pipeline permissions and authorize the pipeline to use the group.

Free newsletter

Stay ahead in AI-driven QA

Get practical tutorials on test automation, AI testing, and quality engineering — straight to your inbox. No spam, unsubscribe any time.

Discussion

Sign in with GitHub to comment · powered by Giscus