Phase 9: Testing Infrastructure

1. Testing Frameworks per Language

flowchart TD
  SDK["copilot-sdk"] --> TS["TypeScript\nVitest"]
  SDK --> PY["Python\npytest + pytest-asyncio"]
  SDK --> GO["Go\ngo test"]
  SDK --> CS["C#\nxUnit"]
            

TypeScript Vitest

Configuration: nodejs/vitest.config.ts

export default defineConfig({
  test: {
    globals: true,
    environment: "node",
    testTimeout: 30000,
    hookTimeout: 30000,
    teardownTimeout: 10000,
    isolate: true,
    pool: "forks",
    exclude: [
      "**/node_modules/**",
      "**/dist/**",
      "**/*.d.ts",
      "**/basic-test.ts",
    ],
  },
});

Key choices: 30-second timeout (generous for E2E), process forking for isolation, global test APIs enabled.

Python pytest + pytest-asyncio

Configuration: python/pyproject.toml (lines 81-86)

[tool.pytest.ini_options]
testpaths = ["."]
python_files = "test_*.py"
python_classes = "Test*"
python_functions = "test_*"
asyncio_mode = "auto"

Dev dependencies: pytest>=7.0.0, pytest-asyncio>=0.21.0, pytest-timeout>=2.0.0, httpx>=0.24.0.

Go go test

Standard go test framework. No special configuration — tests follow Go conventions with _test.go suffixes and testing.T parameters.

C# xUnit

Configuration: dotnet/test/Harness/E2ETestBase.cs

public abstract class E2ETestBase
  : IClassFixture<E2ETestFixture>,
    IAsyncLifetime

Uses IClassFixture<T> for shared context, IAsyncLifetime for async setup/teardown, and Fact/Theory attributes.

2. Test Structure — Unit Tests vs E2E Tests

Directory Layout

copilot-sdk/
  nodejs/
    test/
      client.test.ts              # Unit tests
      e2e/
        harness/
          CapiProxy.ts            # Per-SDK proxy client
          sdkTestContext.ts        # Test context
          sdkTestHelper.ts        # Utility functions
        session.test.ts           # E2E tests
        hooks.test.ts
        ... (20+ E2E test files)
  python/
    test_client.py                # Unit tests
    test_jsonrpc.py
    test_event_forward_compatibility.py
    test_rpc_timeout.py
    e2e/
      conftest.py                 # Shared fixtures
      testharness/
        context.py / helper.py / proxy.py
      test_session.py / test_hooks.py  ... (13+ files)
  go/
    client_test.go                # Unit tests
    definetool_test.go / session_test.go / types_test.go
    internal/e2e/
      testharness/
        context.go / helper.go / proxy.go
      session_test.go / hooks_test.go  ... (14+ files)
  dotnet/
    test/
      Harness/
        E2ETestBase.cs / E2ETestContext.cs
        E2ETestFixture.cs / TestHelper.cs
      SessionTests.cs / HooksTests.cs  ... (17+ files)
  test/                           # SHARED cross-language
    harness/
      server.ts / capturingHttpProxy.ts
      replayingCapiProxy.ts / util.ts
    snapshots/                    # YAML snapshot files
    scenarios/                    # Polyglot scenario tests

Unit vs E2E Distinction

3. Test Harness Architecture

The test harness is a layered system with a shared TypeScript server at the base and per-language wrapper clients on top.

block-beta
  columns 1
  L5["Layer 5: Test Files\n22 E2E Vitest - 14 E2E pytest - 13 E2E go test - 17 E2E xUnit"]
  L4["Layer 4: Per-Language Wrappers\nCapiProxy.ts - proxy.py - proxy.go - E2ETestContext.cs"]
  L3["Layer 3: ReplayingCapiProxy\ntest/harness/replayingCapiProxy.ts lines 52-1059\nRecord/replay, YAML snapshots, normalization"]
  L2["Layer 2: CapturingHttpProxy\ntest/harness/capturingHttpProxy.ts lines 11-206\nTransparent HTTP proxy: forwards and records"]
  L1["Layer 1: Shared Proxy Server\ntest/harness/server.ts - Node.js HTTP server, random port\nEndpoints: POST /config - GET /exchanges - POST /stop"]
  L5 --> L4 --> L3 --> L2 --> L1
  style L5 fill:#7c3aed,color:#fff
  style L4 fill:#6d28d9,color:#fff
  style L3 fill:#5b21b6,color:#fff
  style L2 fill:#4c1d95,color:#fff
  style L1 fill:#3b0764,color:#fff
            

Layer 1: Shared Proxy Server

// test/harness/server.ts (lines 1-13)
const proxy = new ReplayingCapiProxy("https://api.githubcopilot.com");
const proxyUrl = await proxy.start();
console.log(`Listening: ${proxyUrl}`);

Launched as a child process by each language's CapiProxy wrapper. Listens on random local port, prints Listening: http://127.0.0.1:<port> to stdout.

Control Endpoints

Layer 2: CapturingHttpProxy

test/harness/capturingHttpProxy.ts (lines 11-206) — Base class that acts as a transparent HTTP proxy:

• Starts http.Server on 127.0.0.1:0
• Forwards requests to target URL (https://api.githubcopilot.com)
• Records all request/response pairs as CapturedExchange objects
• Handles streaming responses by forwarding chunks as they arrive

Layer 3: ReplayingCapiProxy

test/harness/replayingCapiProxy.ts (lines 52-1059) — Core of the entire test infrastructure.

flowchart TD
  A["Incoming Request"] --> B["Check snapshot cache"]
  B --> C{"Match found?"}
  C -->|Yes| D["Replay from YAML"]
  C -->|No| E["Forward & Record"]
            

Layer 4: Per-Language Wrappers

flowchart LR
  subgraph TypeScript
    TS1["CapiProxy.ts\nSpawns via npx tsx, parses port"]
    TS2["sdkTestContext.ts\nTemp dirs, config, proxy lifecycle"]
  end
  subgraph Python
    PY1["proxy.py\nSpawns via subprocess, httpx"]
    PY2["context.py\nAsync context manager, fixtures"]
  end
  subgraph Go
    GO1["proxy.go\nSpawns via exec.Command, net/http"]
    GO2["context.go\nTestMain setup, t.Cleanup teardown"]
  end
  subgraph CSharp["C#"]
    CS1["E2ETestContext.cs\nSpawns via Process.Start, HttpClient"]
    CS2["E2ETestBase.cs\nIAsyncLifetime fixture pattern"]
  end
            

4. E2E Test Pattern

flowchart TD
  A["Test starts -- context.configureForTest#40;category, name#41;"] --> B["POST /config -- proxy loads snapshot YAML"]
  B --> C["CopilotClient.createSession#40;#41; with proxy URL"]
  C --> D["session.sendAndWait#40;prompt#41;"]
  D --> E["CLI calls LLM API -- proxy intercepts & replays"]
  E --> F["Assert on response / events / tool calls"]
  F --> G["Teardown: session.disconnect#40;#41;, proxy.stop#40;#41;"]
            

Environment Configuration

5. Scenario Testing

The test/scenarios/ directory contains 35 polyglot scenario tests, each implemented in all four languages.

Scenario Structure

test/scenarios/
  01-basic-conversation/
    typescript/ (package.json, index.ts)
    python/     (main.py)
    go/         (main.go)
    dotnet/     (Program.cs, *.csproj)
  02-tool-use/
    typescript/ python/ go/ dotnet/
  ...
  35-advanced-hooks/
    typescript/ python/ go/ dotnet/

Scenario Runner

test/scenarios/verify.sh — Runs all scenarios against a live CLI with real GitHub tokens. Not wired into CI.

6. Snapshot Testing

Snapshot File Format

Each test has a corresponding YAML snapshot in test/snapshots/:

# test/snapshots/session/basic_conversation.yaml
exchanges:
  - request:
      method: POST
      url: /chat/completions
      headers:
        content-type: application/json
      body:
        messages:
          - role: system
            content: "..."
          - role: user
            content: "What is 2+2?"
    response:
      status: 200
      headers:
        content-type: application/json
      body:
        choices:
          - message:
              role: assistant
              content: "2+2 equals 4."

Snapshot Naming Convention

test/snapshots/{category}/{test_name}.yaml
# Examples:
test/snapshots/session/basic_conversation.yaml
test/snapshots/hooks/pre_tool_use_deny.yaml
test/snapshots/permissions/approve_all.yaml

Normalization Pipeline

flowchart TD
  A["Raw HTTP request/response"] --> B["Strip volatile headers #40;Date, X-Request-Id#41;"]
  B --> C["Normalize paths #40;OS-specific to forward slashes#41;"]
  C --> D["Replace tool call IDs with deterministic placeholders"]
  D --> E["Normalize timestamps to epoch"]
  E --> F["Normalize shell names #40;PowerShell / bash#41;"]
  F --> G["Deterministic, cross-platform snapshot"]
            

Request Matching Logic

Corruption Prevention

7. Test Categories

Node.js E2E Tests (22 files)

Python E2E Tests (14 files)

Go E2E Tests (13 files) & C# E2E Tests (17 files)

Both SDKs mirror the same test categories as Node.js and Python, with language-appropriate implementations.

8. CI Integration

Six GitHub Actions workflows orchestrate the test suite:

1. Node.js SDK Tests (nodejs-sdk-tests.yml)

2. Python SDK Tests (python-sdk-tests.yml)

3. Go SDK Tests (go-sdk-tests.yml)

4. .NET SDK Tests (dotnet-sdk-tests.yml)

5. Scenario Build Verification (scenario-builds.yml)

Triggers on PRs/pushes touching test/scenarios/** or SDK source. Four parallel jobs:

flowchart LR
  TS["TS\nnpm install per scenario"]
  PY["Py\npy_compile + import copilot"]
  GO["Go\ngo build ./... per scenario"]
  CS["C#\ndotnet build per scenario"]
            

6. Codegen Check (codegen-check.yml)

Validates that generated code is up-to-date.

Common CI Patterns

9. Test Utilities

Node.js Helpers

nodejs/test/e2e/harness/sdkTestHelper.ts

Python Helpers

python/e2e/testharness/helper.py

Go Helpers

go/internal/e2e/testharness/helper.go

C# Helpers

dotnet/test/Harness/TestHelper.cs

Shared Utilities

test/harness/util.ts (lines 1-35)

Python conftest.py Fixtures

python/e2e/conftest.py (lines 1-47)

C# Test Base

dotnet/test/Harness/E2ETestBase.cs (lines 13-79) — The E2ETestBase abstract class provides:

• Ctx and Client properties from the shared fixture
• InitializeAsync() that calls ConfigureForTestAsync
• CreateSessionAsync() / ResumeSessionAsync() convenience methods (default approve-all)
• GetSystemMessage() / GetToolNames() helpers for exchange inspection

dotnet/test/Harness/E2ETestFixture.cs (lines 10-30) — xUnit IAsyncLifetime fixture:

• Creates E2ETestContext and CopilotClient on initialize
• Calls ForceStopAsync() and DisposeAsync() on teardown

10. Coverage Analysis

Well-Tested Areas

• Session lifecycle: Comprehensive coverage across all 4 SDKs (create, resume, abort, delete, stateful conversation)
• Permission handling: Approve, deny, async handlers, error cases, resume with permissions
• Tool use hooks: Pre/post tool use, deny behavior, both hooks combined
• Client construction and validation: URL parsing, auth options, mutual exclusivity checks
• Custom tools and agents: Registration, MCP server configuration, custom agent setup
• Multi-client scenarios: Tool registration across clients, permission handling, event visibility
• Streaming and event fidelity: Event ordering, field correctness, streaming chunk accuracy
• Cross-platform: All E2E tests run on Linux, macOS, and Windows in CI
• Cross-SDK snapshot sharing: All 4 languages share the same YAML snapshots

Potential Gaps

Cross-SDK Test Parity Summary

Legend: Y = has dedicated test file, – = no dedicated test file (may be partially covered elsewhere)

11. Architecture Diagram

flowchart TD
  SNAP["test/snapshots/*.yaml\n#40;shared across all SDKs#41;"]
  SNAP --> HARNESS["test/harness/\nserver.ts - replayingCapiProxy.ts"]
  SNAP --> SCENARIOS["test/scenarios/\n35 scenarios x 4 langs"]

  HARNESS --> TS_W["Node.js\nCapiProxy - Context - Helper"]
  HARNESS --> PY_W["Python\nproxy - context - helper"]
  HARNESS --> GO_W["Go\nproxy - context - helper"]
  HARNESS --> CS_W["C#\nContext - Base - Helper"]

  TS_W --> TS_E["22 E2E #40;Vitest#41;"]
  PY_W --> PY_E["14 E2E #40;pytest#41;"]
  GO_W --> GO_E["13 E2E #40;go test#41;"]
  CS_W --> CS_E["17 E2E #40;xUnit#41;"]
            

12. Key Design Decisions