# ADR-0009: Per-Type Response Validation Layer

**Date:** 2026-04-10
**Last updated:** 2026-04-13 (rev 2)
**Status:** Implemented
**Deciders:** Rafael Ruiz (CTO)
**Related ADRs:** ADR-0007 (MSVL for RAG Evaluation), ADR-0008 (Adaptive Query Routing), ADR-0003 (Hybrid Retrieval RRF)

---

## Context

### Problem 1 — Syntactically invalid code reaches users

ADR-0007 documents that 10–16% of `CODE_GENERATION` responses contain syntactically invalid AVAP code — foreign language injection (Go, Python, JavaScript) or hallucinated commands (`getSHA256`, `readParam`, `returnResult`). The LLM judge used in `EvaluateRAG` does not detect these failures because it evaluates semantic coherence, not syntactic validity.

This problem exists in production today. A user receiving a `CODE_GENERATION` response has no indication that the generated code would fail on the PLATON kernel.

The AVAP Parser REST HTTP service (Tornado, port 8888) — established in ADR-0007 as a hard dependency of the evaluation pipeline — is already available in the stack. It returns not just `VALID / INVALID` but a **complete line-by-line execution trace** on failure:

```
Line 3: unknown command 'getSHA256' — expected known identifier
Line 7: unexpected construct 'for i in range(...)' — AVAP loop syntax required
Line 12: 'returnResult' not defined — did you mean 'addResult'?
```

This trace is structured, specific, and directly actionable by the LLM. A retry informed by the parser trace is fundamentally different from a blind retry — the model knows exactly what failed and where.

### Problem 2 — Context relevance is not evaluated pre-generation

The engine retrieves 8 chunks from Elasticsearch for every `RETRIEVAL` and `CODE_GENERATION` query without checking whether those chunks actually answer the question. `CONFIDENCE_PROMPT_TEMPLATE` has been scaffolded in `prompts.py` since the initial implementation but is not wired into the graph.

Undetected low-relevance retrieval produces responses that are semantically fluent but factually ungrounded — the model generates plausible-sounding AVAP explanations or code not supported by the retrieved documentation.

### Why these are one ADR

Both problems share the same architectural response: adding validation nodes to the production graph with type-specific logic and feedback-informed retry. The decision is one — **add a per-type validation layer** — and the implementation shares the same graph positions (post-retrieve, post-generate), the same retry contract (maximum 1 retry per request), and the same rationale (the engine must not silently return responses it has evidence to question).

Splitting into separate ADRs would produce two documents that cannot be understood independently.

---

## Decision

Add a **Per-Type Response Validation Layer (PTVL)** to the production LangGraph pipeline. Each query type has a distinct validation strategy matching its failure modes.

### Validation contract by type

| Type | When | What | Mechanism |
|---|---|---|---|
| `CODE_GENERATION` | Post-generation | Syntax + execution + assertion correctness | AVAP Parser REST HTTP — `/api/v1/execute` with `/parse` fallback |
| `RETRIEVAL` | Pre-generation | Relevance of retrieved context to the query | LLM relevance check — `CONFIDENCE_PROMPT_TEMPLATE` |
| `CONVERSATIONAL` | None | — | No retrieval, no code generated |
| `PLATFORM` | None | — | No retrieval, no code generated |

---

### Decision 1 — CODE_GENERATION: three-level validation with trace-guided retry

Validation operates at three levels in order of depth:

1. **Syntax** — can the code be parsed into a valid AST?
2. **Execution** — does the code run without runtime errors?
3. **Assertions** — does the code produce the expected output?

#### Flow

```
generate_code node
    │
    ▼
generate_tests node
    [LLM generates test_inputs + test_list from code + user request]
    │
    ▼
validate_code node
    │
    ▼
[V1] AVAP Parser  POST /api/v1/execute
     {"code": "...", "test_inputs": {...}, "test_list": [...]}
    │
    ├── success=true + assertion_result=true ──► return response
    │
    ├── success=false (runtime error) ──────────┐
    │                                            │
    └── assertion_result=false ─────────────────┤
                                                 │
                                         [inject error into retry prompt]
                                                 │
                                                 ▼
                                      generate_code_retry node (1 attempt only)
                                                 │
                                                 ▼
                                      validate_code_after_retry
                                                 │
                                      [V2] AVAP Parser /api/v1/execute
                                                 │
                                         ├── VALID ──────► return response
                                         └── INVALID ────► return response + validation_status flag
```

**Fallback path:** If `/api/v1/execute` returns 404 (endpoint not yet deployed), validation falls back to `POST /parse` for AST-only syntax checking. The fallback is transparent — no configuration change required.

#### generate_tests node

Before validation, a dedicated LLM call generates test cases from the user's original request and the generated code:

```python
# Input to TEST_GENERATION_PROMPT
user_request:   "como seria una api que reciba un parametro y lo devuelva?"
generated_code: "addParam(\"client_id\", id_interno)\naddResult(id_interno)"

# Output
{
  "test_inputs": {"client_id": "12345"},
  "test_list":   ["re.match(r'^\\d+$', str(id_interno))"]
}
```

`test_inputs` are injected as request variables when the parser executes the code. `test_list` items are regex assertions evaluated against the output variables after execution.

**Variable naming contract** (enforced by `TEST_GENERATION_PROMPT`):

- `addParam("request_param_name", avap_var)` — `test_inputs` key is `"request_param_name"`; assertion variable is `avap_var`.
- `addResult(x)` — adds `x` to the HTTP response body. **Does NOT create a variable.** Assertions must never reference a name that only appears inside `addResult()`.
- Only variables on the left side of `=`, the second arg of `addParam`, or the first arg of `addVar` are valid assertion targets.

If the LLM call fails or times out (15s hard limit), `generate_tests` returns empty `test_inputs` and `test_list` — validation continues using execution-only (no assertions). This keeps the node non-blocking.

The core generation logic is implemented in `_run_generate_tests(user_request, generated_code, llm)` — a module-level function in `graph.py`. Both the `generate_tests` LangGraph node and the `AskAgentStream` streaming state machine delegate to it.

#### Assertion KeyError guard in `_call_parser`

If the parser returns `{'success': False, 'error': "'variable_name'"}` — a Python `KeyError` pattern — it means an assertion referenced a variable that does not exist in the execution scope (a test generation defect, not a code defect). In this case `_call_parser`:

1. Logs the event as `[ptvl] assertion KeyError: variable '...' not in scope — dropping affected assertions`.
2. Filters out any assertion in `test_list` that references `str(bad_var)`.
3. Retries `_call_parser` with the filtered list.
4. If all assertions were filtered out, returns `(True, "")` — the code itself executed successfully.

This prevents the LLM fix path from being triggered on a test generation error, which would incorrectly modify correct code.

#### Trace-guided retry

The parser error is injected into the retry prompt as a structured correction context:

```
<parser_feedback>
The previous attempt failed when the AVAP code was executed.
The execution engine reported the following error:

[error from /api/v1/execute logs]

Rules to fix it:
- Read the error carefully — it identifies the exact command or line that failed.
- Do NOT repeat the same construct that caused the error.
- Only use commands from <avap_syntax_reminder> or <context>.
- If the error mentions an unknown command, replace it with the correct AVAP equivalent.
- If the error mentions a variable, make sure it is declared before use.
</parser_feedback>
```

This is not a blind retry. The LLM receives the exact runtime failure and can target its corrections.

#### qwen3 thinking stripping

The engine strips `<think>...</think>` blocks from all LLM outputs before passing code to the parser. `qwen3` models emit thinking content by default; without stripping, the parser receives the full response including reasoning text and fails to find valid AVAP constructs.

```python
def _strip_thinking(text: str) -> str:
    text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
    text = re.sub(r"</think>", "", text)  # lone closing tag
    return text.strip()
```

This runs before code block extraction in `_extract_avap_code` and before JSON parsing in `generate_tests`.

#### Parser protocol

The AVAP Parser exposes a **REST HTTP API** (Tornado, port 8888). Two endpoints are used:

**Primary — `/api/v1/execute`** (execution + assertions):

```
POST {AVAP_PARSER_URL}/api/v1/execute
Content-Type: application/json

{
  "code":        "<AVAP code>",
  "test_inputs": {"param": "value"},
  "test_list":   ["re.match(r'^pattern$', str(variable))"]
}
```

Response:
```json
{
  "success":          true,
  "result":           [...],
  "variables":        {"param": "value", ...},
  "assertion_result": true,
  "logs":             [{"command": "...", "duration_ms": 1.2, "success": true}]
}
```

**Fallback — `/parse`** (AST-only, when `/api/v1/execute` returns 404):

```
POST {AVAP_PARSER_URL}/parse
Content-Type: application/json

{"code": "<AVAP code>"}
```

Response:
```json
{"valid": true,  "ast": {...}}
{"valid": false, "error": "..."}
```

Code is extracted from the LLM response by scanning for the first markdown code block (` ```avap ` or generic ` ``` `). `<think>` content is stripped before extraction. The fence markers are stripped before sending to the parser.

#### Parser SLA

≤2 seconds per call (`AVAP_PARSER_TIMEOUT`). **Silent fallback is permitted in production** (unlike evaluation, where ADR-0007 mandates abort). The distinction is that evaluation scores must be trustworthy; production responses degrade gracefully.

#### Parser availability — circuit breaker

A single timeout or connection error does not mean the parser is down. A sustained outage does. Hammering an unavailable REST service on every `CODE_GENERATION` request adds latency to every user request with zero benefit.

The engine implements a **circuit breaker** with three states:

```
CLOSED  ──[N consecutive failures]──► OPEN
OPEN    ──[cooldown expires]────────► HALF-OPEN
HALF-OPEN ──[probe succeeds]────────► CLOSED
HALF-OPEN ──[probe fails]───────────► OPEN
```

| Parameter | Default | Env var |
|---|---|---|
| Failure threshold to open | 3 consecutive failures | `PARSER_CB_THRESHOLD` |
| Cooldown before half-open | 30 seconds | `PARSER_CB_COOLDOWN` |
| Timeout per call | 2 seconds | `AVAP_PARSER_TIMEOUT` |

While the circuit is **OPEN**, `CODE_GENERATION` responses are returned immediately with `validation_status = PARSER_UNAVAILABLE` — no HTTP call is attempted. The cooldown prevents thundering-herd reconnection attempts.

```
[ptvl] circuit OPEN — skipping parser, returning unvalidated
[ptvl] circuit HALF-OPEN — probing parser availability
[ptvl] circuit CLOSED — parser reachable
```

Setting `AVAP_PARSER_TIMEOUT=0` permanently opens the circuit — disables parser validation entirely. Useful during development when the parser service is not deployed.

#### New environment variables

```
AVAP_PARSER_URL=http://...    # URL of AVAP Parser REST service (e.g. http://45.77.193.144:8888)
AVAP_PARSER_TIMEOUT=2         # seconds per call; 0 = disable validation entirely
PARSER_CB_THRESHOLD=3         # consecutive failures before circuit opens
PARSER_CB_COOLDOWN=30         # seconds before circuit attempts half-open probe
```

---

### Decision 2 — RETRIEVAL: context relevance check with reformulation retry

#### Flow

```
retrieve node
    │
    ▼
[V1] CONFIDENCE_PROMPT_TEMPLATE
    YES / NO
    │
    ├── YES ────────────────────────────────► generate node (normal path)
    │
    └── NO
            │
            ▼
        reformulate_with_hint node
        [reformulate query signalling context was insufficient]
            │
            ▼
        retrieve (retry)
            │
            ▼
        generate node  (regardless of second retrieval result)
```

On second retrieval, generation proceeds unconditionally. The retry is a single best-effort improvement, not a gate. The model generates with whatever context is available — the user receives a response, not a refusal.

#### Relevance check

`CONFIDENCE_PROMPT_TEMPLATE` evaluates whether the retrieved context contains at least one passage relevant to the question. It returns `YES` or `NO` — no graduated score.

The relevance check is only applied to `RETRIEVAL` queries here. `CODE_GENERATION` has the parser as a stronger post-generation signal; the pre-generation relevance check would add latency with lower signal value than the parser provides.

#### Reformulation hint

When context is insufficient, the reformulation node receives a hint that standard context was not found. This produces a semantically different reformulation — broader synonyms, alternative phrasings — rather than a near-duplicate of the original query.

```
[CONTEXT_INSUFFICIENT]
The previous retrieval did not return relevant context for: "{original_query}"
Reformulate this query using broader terms or alternative phrasing.
```

---

### Decision 3 — AskAgentStream: streaming state machine for CODE_GENERATION

The `AskAgentStream` path streams tokens directly to the client. Post-generation validation (Decision 1) cannot run here because tokens are already yielded before the response is complete.

**Decision:** implement a **streaming state machine** that operates inline on the token stream. The machine has two states: TEXT and CODE.

#### Flow

```
LLM token stream
    │
    ▼
STATE: TEXT
  → yield token to client immediately
  → detect ``` fence in lookahead buffer (2-char safety window for split tokens)
    │ ``` detected
    ▼
STATE: CODE  (buffering — nothing yielded to client)
  → accumulate tokens in code_buffer
  → detect closing ``` after first newline
    │ closing ``` detected
    ▼
_extract_avap_code(complete_block)
    │
    ▼
_run_generate_tests(query, avap_code, llm)   ← same helper as build_graph path
    [produces test_inputs + test_list, 15s timeout]
    │
    ▼
_call_parser(complete_block, test_inputs, test_list)
    │
    ├── VALID ──────────────────────────────► yield code block → back to TEXT
    │
    ├── INVALID
    │       │
    │       ▼
    │   LLM fix call (fix only the code block, not the full response)
    │   "Fix this AVAP code: {trace}"
    │       │
    │       ▼
    │   _call_parser(fixed_block, test_inputs, test_list)   ← same tests reused
    │       ├── VALID ──────────────────────► yield fixed block → back to TEXT
    │       ├── INVALID ────────────────────► yield fixed block + INVALID_UNRESOLVED → back to TEXT
    │       └── UNAVAILABLE ────────────────► yield fixed block + PARSER_UNAVAILABLE → back to TEXT
    │
    └── UNAVAILABLE ────────────────────────► yield block as-is + PARSER_UNAVAILABLE → back to TEXT
```

#### Key properties

- Text before and after the code block streams to the client without delay.
- Only the code block itself introduces latency (one `_run_generate_tests` LLM call + one parser call, optionally one LLM fix call + one additional parser call).
- The fix call asks the LLM to correct **only the code block**, not the full response — the text already streamed to the client remains valid.
- If the response contains multiple code blocks, each block is processed independently in sequence. `validation_status` in the final `is_final=True` message reflects the last block validated.
- If the stream ends while still in CODE mode (malformed response without closing fence), the buffer is flushed as-is.

#### Difference from AskAgent path

| Property | AskAgent | AskAgentStream |
|---|---|---|
| Validation point | Post-generation, pre-delivery | Inline, at code block boundary |
| Test generation | `generate_tests` LangGraph node | `_run_generate_tests()` called directly before `_call_parser` |
| Execution + assertion validation | `validate_code` node → `/api/v1/execute` with `test_inputs`/`test_list` | `_call_parser()` with `test_inputs`/`test_list` |
| Retry mechanism | Full `generate_code_retry` LangGraph node | Targeted LLM fix call (code block only) |
| Retry reuses same tests | Yes (state carries `test_inputs`/`test_list`) | Yes (`_ti`/`_tl` captured before first call) |
| Text streaming | N/A (non-streaming) | Uninterrupted |
| `validation_status` delivery | In `AgentResponse` (only response) | In final `AgentResponse` (`is_final=True`) |

---

## Graph changes

### New nodes

| Node | Graph | Trigger |
|---|---|---|
| `generate_tests` | `build_graph` | After `generate_code` — generates `test_inputs` + `test_list` |
| `validate_code` | `build_graph` | After `generate_tests` |
| `generate_code_retry` | `build_graph` | After `validate_code` when INVALID |
| `validate_code_after_retry` | `build_graph` | After `generate_code_retry` |
| `check_context_relevance` | `build_graph` + `build_prepare_graph` | After `retrieve`, RETRIEVAL only |
| `reformulate_with_hint` | `build_graph` + `build_prepare_graph` | After `check_context_relevance` when NO |
| `retrieve_retry` | `build_graph` + `build_prepare_graph` | After `reformulate_with_hint` |

### AskAgentStream

No new LangGraph nodes. The validation logic runs as a **streaming state machine** inline in `server.py:AskAgentStream`. See Decision 3.

Test generation and execution-level validation now have full parity with the `AskAgent` path: `_run_generate_tests` is called after each code block is buffered, and `_call_parser` receives the resulting `test_inputs`/`test_list`. Both paths share the same module-level `_run_generate_tests` implementation.

### Updated flow — `build_graph`

```mermaid
flowchart TD
    START([start]) --> CL[classify]

    CL -->|RETRIEVAL| RF[reformulate]
    CL -->|CODE_GENERATION| RF
    CL -->|CONVERSATIONAL| RC[respond_conversational]
    CL -->|PLATFORM| RP[respond_platform]

    RF --> RT[retrieve]

    RT -->|CODE_GENERATION| GC[generate_code]
    RT -->|RETRIEVAL| CR{check_context\nrelevance}

    CR -->|YES| GE[generate]
    CR -->|NO| RH[reformulate_with_hint]
    RH --> RT2[retrieve retry]
    RT2 --> GE

    GC --> GT[generate_tests\nLLM → test_inputs + test_list]
    GT --> VC{validate_code\n/api/v1/execute}
    VC -->|VALID + assertions pass| END([end])
    VC -->|runtime error or assertion fail| GCR[generate_code_retry\ntrace-guided]
    GCR --> VC2{validate_code_after_retry\n/api/v1/execute}
    VC2 -->|VALID| END
    VC2 -->|INVALID| END

    GE --> END
    RC --> END
    RP --> END
```

---

## New AgentState fields

```python
class AgentState(TypedDict):
    ...
    # PTVL fields
    parser_trace:       str   # raw parser error from first validation attempt (empty if valid)
    validation_status:  str   # see validation status values below
    context_relevant:   bool  # result of CONFIDENCE_PROMPT check (RETRIEVAL only)
    # Test generation fields (set by generate_tests node)
    test_inputs:        dict  # variables injected when executing generated code
    test_list:          list  # regex assertions validated against output variables after execution
```

### Validation status values

| Value | Meaning | When set |
|---|---|---|
| `""` (empty) | Valid — no issues detected | Parser returned VALID on first or second attempt |
| `INVALID_UNRESOLVED` | Parser ran, code failed both attempts | Two parser calls made, both returned INVALID |
| `PARSER_UNAVAILABLE` | Parser was unreachable or circuit is open | No parser call was made or all calls timed out |

These are semantically distinct signals. `INVALID_UNRESOLVED` means the engine has evidence the code is wrong. `PARSER_UNAVAILABLE` means the engine has no evidence either way — the code may be correct. Clients must not treat them equivalently.

`validation_status` is surfaced to the client via `AgentResponse`:

```protobuf
message AgentResponse {
  string text              = 1;
  string avap_code         = 2;
  bool   is_final          = 3;
  string validation_status = 4;  // "" | "INVALID_UNRESOLVED" | "PARSER_UNAVAILABLE"
}
```

**Implementation status: complete (2026-04-13).** Field 4 added to `brunix.proto`. Populated in both `AskAgent` (from `final_state`) and `AskAgentStream` (in the `is_final=True` message at end of stream). Full test generation + assertion validation parity between both paths achieved 2026-04-13.

Clients that do not read `validation_status` are unaffected — the field defaults to empty string.

---

## Routing contract additions (RC-07, RC-08)

These rules extend the contract defined in ADR-0008.

### RC-07 — Parser validation gate (priority: high)

Every `CODE_GENERATION` response **MUST** be submitted to the AVAP Parser before delivery to the client, unless `AVAP_PARSER_TIMEOUT=0` or the parser service is unreachable.

```
route(q) = CODE_GENERATION → parser_validate(response) before yield
```

A `CODE_GENERATION` response returned without parser validation due to parser unavailability **MUST** be logged as `[ptvl] parser unavailable — returning unvalidated`.

### RC-08 — Retry budget (priority: medium)

Each request has a maximum of **1 retry** regardless of type. A `CODE_GENERATION` request that fails parser validation twice returns the second attempt with `validation_status=INVALID_UNRESOLVED`. A `RETRIEVAL` request whose context is insufficient reformulates once and generates unconditionally on the second retrieval.

No request may enter more than one retry cycle.

---

## Consequences

### Positive

- Syntactically invalid AVAP code no longer reaches users silently. `validation_status` gives the client a typed signal: `INVALID_UNRESOLVED` (evidence of bad code) vs `PARSER_UNAVAILABLE` (no evidence either way) — clients can respond differently to each.
- Execution validation (`/api/v1/execute`) catches runtime errors invisible to the AST parser — undefined variables, unsupported commands, type mismatches.
- Assertion validation verifies the code produces the expected output, not just that it runs.
- The parser error trace makes retries targeted rather than blind — the LLM corrects specific runtime failures, not the whole response.
- Circuit breaker prevents parser outages from adding latency to every `CODE_GENERATION` request. After 3 consecutive failures the engine stops trying for 30 seconds.
- Context relevance check catches retrievals that return topically adjacent but non-answering chunks, reducing fluent-but-ungrounded responses.
- `AVAP_PARSER_TIMEOUT=0` allows development without the parser service — no hard dependency at startup.
- Automatic fallback from `/api/v1/execute` to `/parse` — new parser endpoint can be deployed without a coordinated engine restart.

### Negative / Trade-offs

- **`CODE_GENERATION` latency**: +1 `generate_tests` LLM call + 1 parser execution call per request. If `generate_tests` fails (timeout/error), the node returns empty tests and validation continues — no blocking.
- **`RETRIEVAL` latency**: +1 LLM call (relevance check) on every request. At `qwen3:1.7b` local inference, this adds ~300–500ms to every RETRIEVAL request — not negligible.
- The parser becomes a **soft production dependency** for CODE_GENERATION. Parser outages degrade validation silently; monitoring must alert on sustained `parser unavailable` log volume.
- The context relevance check is a **generative model doing a binary classification task** — the same architectural mismatch noted in ADR-0008 for the classifier. It is the correct interim solution while no discriminative relevance model exists.
- **`registerEndpoint` suppressed by default** — `CODE_GENERATION_PROMPT` rule 7 omits `registerEndpoint` unless the user explicitly requests endpoint registration. This prevents over-engineering simple code responses but requires the user to be explicit when endpoint scaffolding is needed.

### Open questions

1. **`RETRIEVAL` latency budget**: The +300–500ms from the relevance LLM call may be unacceptable for the VS Code extension use case where streaming latency is user-visible. A discriminative relevance model (embedding similarity between query vector and context vector, cosine threshold) would be ~1ms and eliminate this cost entirely. Deferred to a future amendment. **Trigger:** when the RETRIEVAL validation LLM call appears as a measurable contribution in Langfuse traces.

2. **`validation_status` UX**: Proto field 4 is defined and populated. What the VS Code extension or AVS Platform displays when `validation_status=INVALID_UNRESOLVED` or `PARSER_UNAVAILABLE` is not yet specified. Requires a product decision outside this ADR's scope. **Open.**

3. **Parser version pinning**: Inherited from ADR-0007 open question 2. Parser upgrades may alter what is considered valid AVAP. A policy for handling parser version changes in the production pipeline has not been defined. **Open.**

4. **`validation_status` across multiple code blocks (AskAgentStream)**: When a response contains more than one code block, the `validation_status` in the final `is_final=True` message reflects only the last block validated. If an earlier block was `INVALID_UNRESOLVED` and the last was valid, the client receives `""`. In practice AVAP responses contain one code block. A future amendment may accumulate the worst-case status across all blocks. **Low priority.**

---

## Future Path

The context relevance check for `RETRIEVAL` (Decision 2) uses a generative LLM for a discriminative task — the same pattern that ADR-0008 identified as tactical debt for the classifier. The correct steady-state implementation is a cosine similarity threshold between the query embedding vector and the average context embedding vector:

```
relevance_score = cosine(embed(query), mean(embed(chunks)))
if relevance_score < RELEVANCE_THRESHOLD:
    reformulate_with_hint()
```

This runs in microseconds using the `bge-m3` embeddings already computed during retrieval. It replaces the `CONFIDENCE_PROMPT_TEMPLATE` LLM call entirely and eliminates the +300–500ms latency penalty on every RETRIEVAL request.

**Trigger for this upgrade:** once the RETRIEVAL validation LLM call appears as a measurable latency contribution in Langfuse traces.
