Matrix Testing

Learn how to test multiple services in parallel using GitHub Actions matrix strategy with isolated Docker environments.

Overview

Matrix testing allows you to run HTTPTests for multiple services simultaneously, each in its own isolated environment, providing faster feedback and better parallelization.

The Problem

When you have multiple services to test:

your-repo/
├── service-a/.httptests/
├── service-b/.httptests/
├── service-c/.httptests/
├── service-d/.httptests/
└── service-e/.httptests/

Sequential approach:

yaml

# ❌ Slow - runs one after another
- uses: serviceguards-com/httptests-action@latest
  with:
    httptests-directory: ./service-a
- uses: serviceguards-com/httptests-action@latest
  with:
    httptests-directory: ./service-b
# ... and so on

Problems:

❌ Slow - tests run sequentially
❌ Long feedback cycles
❌ One failure blocks others
❌ Poor resource utilization

The Solution: Matrix Strategy

Run all services in parallel with GitHub Actions matrix strategy:

yaml

name: HTTPTests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          - service-a
          - service-b
          - service-c
          - service-d
          - service-e
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./${{ matrix.service }}

Benefits:

✅ Tests run in parallel
✅ Faster overall execution
✅ Independent failure isolation
✅ Better GitHub Actions UI
✅ Isolated Docker environments

Basic Matrix Configuration

Simple Service List

yaml

name: HTTPTests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        test-suite:
          - ./api-gateway
          - ./user-service
          - ./order-service
          - ./payment-service
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ${{ matrix.test-suite }}

With Job Names

yaml

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          - name: API Gateway
            path: ./api-gateway
          - name: User Service
            path: ./user-service
          - name: Order Service
            path: ./order-service
    
    name: Test ${{ matrix.service.name }}
    
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ${{ matrix.service.path }}

Advanced Configurations

Fail-Fast vs Continue on Error

Fail-Fast (Default)

yaml

strategy:
  matrix:
    service: [service-a, service-b, service-c]
  # fail-fast: true (default)

Behavior: If one service fails, all others are cancelled.

Use when: You want quick feedback and failing fast saves time.

Continue on Error

yaml

strategy:
  matrix:
    service: [service-a, service-b, service-c]
  fail-fast: false

Behavior: All services continue testing even if one fails.

Use when: You want to see all failures, not just the first one.

Timeout Configuration

yaml

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 10  # Per service timeout
    strategy:
      matrix:
        service: [service-a, service-b, service-c]
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./${{ matrix.service }}

Conditional Matrix

Run only changed services:

yaml

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      services: ${{ steps.filter.outputs.changes }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            service-a:
              - 'services/service-a/**'
            service-b:
              - 'services/service-b/**'
            service-c:
              - 'services/service-c/**'

  test:
    needs: detect-changes
    if: ${{ needs.detect-changes.outputs.services != '[]' }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJSON(needs.detect-changes.outputs.services) }}
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./services/${{ matrix.service }}

Complete Examples

Example 1: Microservices Monorepo

yaml

name: Microservices Tests
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    strategy:
      fail-fast: false
      matrix:
        service:
          - api-gateway
          - user-service
          - order-service
          - payment-service
          - notification-service
    
    name: Test ${{ matrix.service }}
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Run HTTPTests
        uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./services/${{ matrix.service }}
          python-version: '3.11'

Example 2: Different Service Categories

yaml

name: HTTPTests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        include:
          - category: "API"
            service: api-gateway
            timeout: 10
          - category: "API"
            service: rest-api
            timeout: 10
          - category: "Proxy"
            service: nginx-proxy
            timeout: 5
          - category: "Proxy"
            service: load-balancer
            timeout: 5
    
    name: Test ${{ matrix.category }} - ${{ matrix.service }}
    timeout-minutes: ${{ matrix.timeout }}
    
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./${{ matrix.service }}

Example 3: Environment-Specific Testing

yaml

name: Multi-Environment Tests
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: [api-gateway, user-service]
        environment: [staging, production]
    
    name: Test ${{ matrix.service }} (${{ matrix.environment }})
    
    steps:
      - uses: actions/checkout@v4
      - uses: serviceguards-com/httptests-action@latest
        with:
          httptests-directory: ./${{ matrix.service }}-${{ matrix.environment }}

Isolated Docker Environments

Each matrix job runs in its own runner with isolated Docker:

yaml

# Job 1: service-a
🚀 Starting environment...
docker compose -p httptests-service-a up -d

# Job 2: service-b (parallel)
🚀 Starting environment...
docker compose -p httptests-service-b up -d

# Job 3: service-c (parallel)
🚀 Starting environment...
docker compose -p httptests-service-c up -d

Key features:

✅ Separate Docker networks per service
✅ No port conflicts
✅ Independent container lifecycle
✅ Automatic cleanup per job

GitHub Actions UI Benefits

Matrix testing provides clear UI visualization:

✅ test (api-gateway)        - Passed in 2m 34s
✅ test (user-service)       - Passed in 1m 52s
❌ test (order-service)      - Failed in 3m 12s
✅ test (payment-service)    - Passed in 2m 03s
✅ test (notification-service) - Passed in 1m 28s

Benefits:

See status of each service independently
Quick identification of which service failed
Individual logs per service
Re-run only failed jobs

Best Practices

1. Use Descriptive Job Names

yaml

# Good
strategy:
  matrix:
    service:
      - name: API Gateway
        path: ./api-gateway
      - name: User Service
        path: ./user-service

name: Test ${{ matrix.service.name }}

# Bad
strategy:
  matrix:
    service: [./api-gateway, ./user-service]
name: Test ${{ matrix.service }}  # Shows "./api-gateway"

2. Set Appropriate Timeouts

yaml

jobs:
  test:
    timeout-minutes: 15  # Prevent hanging jobs
    strategy:
      matrix:
        service: [...]

3. Use fail-fast Appropriately

yaml

# Development - fail fast to save time
strategy:
  matrix:
    service: [...]
  fail-fast: true

# CI/CD - see all failures
strategy:
  matrix:
    service: [...]
  fail-fast: false

4. Limit Concurrency for Resource Constraints

yaml

strategy:
  max-parallel: 3  # Run max 3 jobs at once
  matrix:
    service: [s1, s2, s3, s4, s5, s6]

yaml

# Separate workflows for different layers
name: API Layer Tests
strategy:
  matrix:
    service: [api-gateway, rest-api, graphql-api]

---

name: Service Layer Tests
strategy:
  matrix:
    service: [user-service, order-service, payment-service]

Comparison: Matrix vs Separate Jobs

Matrix Strategy (Recommended)

yaml

jobs:
  test:
    strategy:
      matrix:
        service: [a, b, c]

Pros:

✅ Less code duplication
✅ Easier to add/remove services
✅ Better GitHub UI
✅ Consistent configuration

Cons:

⚠️ All jobs use same configuration
⚠️ Harder to customize per service

Separate Jobs

yaml

jobs:
  test-a:
    steps: [...]
  test-b:
    steps: [...]
  test-c:
    steps: [...]

Pros:

✅ Full control per job
✅ Easy to customize each service
✅ Can have dependencies between jobs

Cons:

❌ More code duplication
❌ Harder to maintain
❌ More verbose

Troubleshooting

Tests Run Sequentially Instead of Parallel

Issue: Jobs appear to run one after another

Cause: GitHub Actions concurrent job limits

Solutions:

Check your GitHub plan's concurrency limits
Use max-parallel to adjust
Check organization/repository settings

Resource Exhaustion

Issue: Jobs fail with "Docker daemon not responding"

Cause: Too many parallel Docker operations

Solutions:

yaml

strategy:
  max-parallel: 5  # Limit concurrent jobs
  matrix:
    service: [...]

Port Conflicts

Issue: "Port already in use" errors

Cause: Matrix jobs sharing ports (shouldn't happen)

Solutions:

Each job gets isolated runner - this shouldn't occur
If it does, check for host port mappings in docker-compose
HTTPTests uses isolated project names automatically

Performance Tips

1. Optimize Docker Builds

dockerfile

# Use specific base image versions
FROM nginx:1.25-alpine

# Minimize layers
COPY nginx.conf /etc/nginx/nginx.conf

# No unnecessary packages

2. Cache Dependencies

yaml

- name: Cache Python dependencies
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}

3. Parallelize Appropriately

yaml

# If you have 20 services, don't run all at once
strategy:
  max-parallel: 10  # Balance speed vs resources

Installation - Basic workflow setup
Microservices Example - Real-world usage
GitHub Actions Documentation - Official matrix docs

Matrix Testing ​

Overview ​

The Problem ​

The Solution: Matrix Strategy ​

Basic Matrix Configuration ​

Simple Service List ​

With Job Names ​

Advanced Configurations ​

Fail-Fast vs Continue on Error ​

Fail-Fast (Default) ​

Continue on Error ​

Timeout Configuration ​

Conditional Matrix ​

Complete Examples ​

Example 1: Microservices Monorepo ​

Example 2: Different Service Categories ​

Example 3: Environment-Specific Testing ​

Isolated Docker Environments ​

GitHub Actions UI Benefits ​

Best Practices ​

1. Use Descriptive Job Names ​

2. Set Appropriate Timeouts ​

3. Use fail-fast Appropriately ​

4. Limit Concurrency for Resource Constraints ​

5. Group Related Services ​

Comparison: Matrix vs Separate Jobs ​

Matrix Strategy (Recommended) ​

Separate Jobs ​

Troubleshooting ​

Tests Run Sequentially Instead of Parallel ​

Resource Exhaustion ​

Port Conflicts ​

Performance Tips ​

1. Optimize Docker Builds ​

2. Cache Dependencies ​

3. Parallelize Appropriately ​

Related Topics ​

Matrix Testing

Overview

The Problem

The Solution: Matrix Strategy

Basic Matrix Configuration

Simple Service List

With Job Names

Advanced Configurations

Fail-Fast vs Continue on Error

Fail-Fast (Default)

Continue on Error

Timeout Configuration

Conditional Matrix

Complete Examples

Example 1: Microservices Monorepo

Example 2: Different Service Categories

Example 3: Environment-Specific Testing

Isolated Docker Environments

GitHub Actions UI Benefits

Best Practices

1. Use Descriptive Job Names

2. Set Appropriate Timeouts

3. Use fail-fast Appropriately

4. Limit Concurrency for Resource Constraints

5. Group Related Services

Comparison: Matrix vs Separate Jobs

Matrix Strategy (Recommended)

Separate Jobs

Troubleshooting

Tests Run Sequentially Instead of Parallel

Resource Exhaustion

Port Conflicts

Performance Tips

1. Optimize Docker Builds

2. Cache Dependencies

3. Parallelize Appropriately

Related Topics