Mcp-Release-Qa

# MCP Release QA

Test the server that users will run. A schema review or a passing unit test is

not runtime evidence.

This skill complements security review. It focuses on protocol behavior,

published-contract drift, transport correctness, and reproducible release

evidence.

Rules

- Run checks against a fresh server process built from the candidate revision.

- Keep `initialize`, `notifications/initialized`, discovery, and invocation in

the same session. A new process is a new STDIO session.

- Treat source registrations as implementation truth and public documentation

as a contract that must match it.

- Record exact commands and raw responses. Do not replace missing evidence with

"looks correct."

- Do not invoke mutation-capable tools against production data. Use fixtures, a

sandbox, or stop and name the missing safe test environment.

- Derive the expected capability inventory from the candidate source on every

run.

1. Establish the release surface

Identify:

- candidate commit and build command;

- server entry point and transport: STDIO, Streamable HTTP, or SSE;

- supported MCP protocol versions;

- source files that register tools, resources, resource templates, and prompts;

- generated catalogs, manifests, README tables, and install instructions;

- existing protocol, integration, and smoke-test commands.

Prefer repository-native commands. Inspect `package.json`, `pyproject.toml`,

`Makefile`, CI workflows, and contributor instructions before inventing a test

harness.

2. Start a clean server

Build the candidate and start the documented entry point with test-safe

configuration. Capture:

- the exact command;

- commit SHA;

- environment variable names, with values redacted;

- stdout, stderr, and exit status;

- the endpoint or child-process transport used by the client.

For STDIO, stdout is protocol-only. Logs, banners, and stack traces belong on

stderr. For HTTP transports, record the status, relevant MCP headers, and

session identifier handling without printing credentials.

If the server cannot start from its documented instructions, report that as a

release failure and preserve the startup error verbatim.

3. Exercise one complete session

Run this sequence through a real MCP client or the repository's integration

harness:

1. `initialize` with a protocol version the server claims to support.

2. Confirm the negotiated version and advertised capabilities.

3. Send `notifications/initialized`.

4. Call `ping`.

5. Call each supported discovery method:

- `tools/list`

- `resources/list`

- `resources/templates/list`

- `prompts/list`

6. Exercise at least one representative read-only item from every advertised

capability class.

7. Follow pagination until no cursor remains when a list method is paginated.

Do not send post-initialization requests through separate one-shot processes.

That accidentally tests several incomplete sessions instead of one valid

session.

4. Prove inventory parity

Build four inventories from current evidence:

| Surface | Evidence |

|---|---|

| Source | Registered tool, resource, template, and prompt definitions |

| Runtime | Results from the live discovery methods |

| Generated metadata | Catalogs, manifests, or generated indexes |

| Documentation | README, reference pages, and install output |

Compare by stable identifier. Report:

- source entries missing at runtime;

- runtime entries absent from metadata or documentation;

- stale names, descriptions, arguments, URIs, or prompt parameters;

- documented install commands that do not start the candidate server.

Regenerate derived files with the repository's own build command, then fail if

the working tree still contains unexplained generated changes.

5. Check published contracts

For every discovered item, verify the runtime definition against its source:

Tools

- Name and description are stable and specific.

- `inputSchema` defines types, required fields, enums, and bounds where needed.

- Unknown prop