Upstream Target Tracking

Learn how HTTPTests automatically tracks and validates exact proxy destinations, ensuring your nginx configuration routes to the correct upstream services.

The Problem

When testing proxy configurations, how do you ensure requests route to the exact correct upstream?

Consider this scenario:

# Multiple possible upstreams exist in your Docker network
mock:
  additional_ports:
    - 5001
    - 9999
  network_aliases:
    - backend
    - different-host

Your nginx config proxies to one:

location /api/ {
    proxy_pass http://backend:5001/;
}

The challenge: All these upstreams could respond to HTTP requests:

• backend:5001 ✅ (correct)
• backend:9999 ❌ (wrong port)
• different-host:5001 ❌ (wrong host)

The question: How do you test that nginx proxies to backend:5001 specifically, and not any of the alternatives?

The Solution

HTTPTests automatically adds X-Upstream-Target headers to your nginx configurations, enabling precise upstream validation.

How It Works

1. Automatic Injection: Before building containers, HTTPTests scans your nginx configs
2. Detects proxy_pass: Finds all proxy_pass directives
3. Adds Header: Injects proxy_set_header X-Upstream-Target "<upstream>";
4. Preserves Format: Maintains indentation and formatting
5. Idempotent: Safe to run multiple times

Example Transformation

Before (your original nginx.conf):

location /api/ {
    proxy_pass http://backend:5001/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

After (automatic injection):

location /api/ {
    proxy_pass http://backend:5001/;
    proxy_set_header X-Upstream-Target "backend:5001";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

The header is added immediately after proxy_pass, before any other proxy_set_header directives.

Testing Upstream Destinations

Once the header is injected, validate it in your tests:

{
    "paths": ["/api/users"],
    "method": "GET",
    "expectedStatus": 200,
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend:5001"]
    ]
}

This test will only pass if:

• The request reaches the upstream service
• The upstream is specifically backend:5001
• Not backend:9999 or different-host:5001

Why This Matters

Scenario: Multiple Microservices

mock:
  additional_ports:
    - 5001  # User service
    - 5002  # Order service
    - 5003  # Payment service
  network_aliases:
    - user-service
    - order-service
    - payment-service

location /users/ {
    proxy_pass http://user-service:5001/;
}

location /orders/ {
    proxy_pass http://order-service:5002/;
}

location /payments/ {
    proxy_pass http://payment-service:5003/;
}

Without upstream tracking: Tests might pass even if routes are misconfigured.

With upstream tracking: Each route is validated precisely:

{
    "paths": ["/users/profile"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "user-service:5001"]
    ]
},
{
    "paths": ["/orders/12345"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "order-service:5002"]
    ]
},
{
    "paths": ["/payments/status"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "payment-service:5003"]
    ]
}

Scenario: Load Balancer Testing

upstream backend_cluster {
    server backend-1:5001;
    server backend-2:5002;
    server backend-3:5003;
}

location /api/ {
    proxy_pass http://backend_cluster/;
}

After injection:

location /api/ {
    proxy_pass http://backend_cluster/;
    proxy_set_header X-Upstream-Target "backend_cluster";
}

Test validates load balancer usage:

{
    "paths": ["/api/users"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend_cluster"]
    ]
}

How the Injection Works

The Script: add_upstream_headers.py

HTTPTests runs this script automatically before building containers:

python add_upstream_headers.py ./your-service/

What it does:

1. Scans all .conf files in the directory
2. Uses regex to find proxy_pass directives
3. Extracts the upstream URL (e.g., http://backend:5001/)
4. Checks if X-Upstream-Target already exists
5. If not, adds proxy_set_header X-Upstream-Target "backend:5001";
6. Preserves indentation and formatting

Supported Patterns

The script handles various nginx patterns:

# Basic proxy_pass
proxy_pass http://backend:80/;
# → X-Upstream-Target: "backend:80"

# With path
proxy_pass http://backend:80/api/v1/;
# → X-Upstream-Target: "backend:80"

# Upstream block
proxy_pass http://backend_cluster;
# → X-Upstream-Target: "backend_cluster"

# With variables
proxy_pass http://$backend_host:$backend_port/;
# → X-Upstream-Target: "$backend_host:$backend_port"

# HTTPS
proxy_pass https://secure-backend:443/;
# → X-Upstream-Target: "secure-backend:443"

Idempotency

Running the script multiple times is safe:

# First run - adds header
python add_upstream_headers.py ./service/

# Second run - skips (header exists)
python add_upstream_headers.py ./service/

# Third run - still skips
python add_upstream_headers.py ./service/

The script checks for existing X-Upstream-Target headers and skips if found.

Manual Header Addition

If you prefer manual control, you can add the header yourself:

location /api/ {
    proxy_pass http://backend:5001/;
    proxy_set_header X-Upstream-Target "backend:5001";  # Manual
    proxy_set_header Host $host;
}

The automatic script will detect this and skip injection.

Advanced Use Cases

Testing Failover

location /api/ {
    proxy_pass http://backend:5001/;
    proxy_next_upstream error timeout http_502;
}

Test that primary upstream is attempted first:

{
    "paths": ["/api/users"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend:5001"]
    ]
}

Testing Path Rewrites

location /old-api/ {
    proxy_pass http://backend:80/new-api/;
}

Validate routing to new API path:

{
    "paths": ["/old-api/users"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend:80"]
    ]
}

Testing Conditional Routing

location /api/ {
    if ($http_x_version = "v2") {
        proxy_pass http://backend-v2:80/;
    }
    proxy_pass http://backend-v1:80/;
}

Test both routes:

{
    "paths": ["/api/users"],
    "method": "GET",
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend-v1:80"]
    ]
},
{
    "paths": ["/api/users"],
    "method": "GET",
    "additionalRequestHeaders": {
        "X-Version": "v2"
    },
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend-v2:80"]
    ]
}

Troubleshooting

Header Not Found

Issue: Test fails with "Expected header 'X-Upstream-Target' not found"

Solutions:

1. Check automatic injection ran:

🔧 Adding X-Upstream-Target headers to nginx configs...

Should appear in logs.

2. Verify nginx syntax:

# Correct
proxy_pass http://backend:80/;

# Incorrect (might not be detected)
proxy_pass    http://backend:80/;  # Extra spaces

3. Check file location: Script scans .conf files in the httptests directory.

4. Manual addition: Add the header manually if automatic injection fails.

Wrong Upstream Value

Issue: Test expects backend:5001 but receives backend:80

Solutions:

1. Check nginx configuration:

location /api/ {
    # Make sure this matches what you expect
    proxy_pass http://backend:5001/;
}

2. Update test:

{
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "backend:5001"]  // Must match nginx config
    ]
}

Best Practices

1. Always Validate Critical Routes

{
    "paths": ["/api/users", "/api/orders", "/api/payments"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "critical-service:5001"]
    ]
}

2. Test Each Microservice Route

// User service
{
    "paths": ["/users/profile"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "user-service:5001"]
    ]
},
// Order service
{
    "paths": ["/orders/12345"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "order-service:5002"]
    ]
}

3. Document Expected Upstreams

# config.yml
mock:
  additional_ports:
    - 5001  # User service - handles /users/*
    - 5002  # Order service - handles /orders/*
  network_aliases:
    - user-service
    - order-service

4. Validate Load Balancer Usage

{
    "paths": ["/api/users"],
    "expectedRequestHeadersToUpstream": [
        ["X-Upstream-Target", "user_service_cluster"]
    ]
}

Related Topics

• Collection Headers - Reduce test duplication
• test.json Reference - Complete test configuration
• Nginx Proxy Example - Practical examples