feat: add extraction schema, sidebar nav, few-shot prompting, and prompt settings

Overhaul extraction pipeline with new TradeItem model, conversation flow,
and dedicated extraction endpoint. Add sidebar navigation with NavMenu
component and landing page. Introduce few-shot prompting service and
tests. Add prompt settings and email upload specs. Update OpenSpec
tooling with improved export-spec and extract-feature commands. Archive
completed changes and export full specs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-06

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/design.md

@@ -0,0 +1,69 @@
+## Context
+
+The extraction agent (Semantic Kernel with auto-invoke tools) needs few-shot examples to reliably map sales emails to TradeItem JSON. The `update-extraction-schema` change provides the real schema and tools. This change adds the prompting infrastructure and a dedicated endpoint.
+
+Up to 100 input/output example pairs are available. Research shows 3-5 well-chosen examples are optimal for few-shot prompting — more can degrade performance by consuming context and introducing noise.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Load curated few-shot examples from disk and assemble a reusable ChatHistory prefix
+- Provide a fixed instruction template for extraction (not user-editable)
+- Create a dedicated extraction endpoint with the correct prompt and tools
+- Keep the general chat endpoint unchanged
+
+**Non-Goals:**
+- Dynamic example selection (RAG-like similarity matching) — future enhancement
+- Email upload UI (separate change: `email-upload-ux`)
+- Building or curating the actual example content (user provides these)
+- Evaluation pipeline for the ~95 non-few-shot examples (future work)
+
+## Decisions
+
+### 1. Examples as conversation turns (not system prompt string)
+
+**Decision:** Inject few-shot examples as alternating User/Assistant messages in the ChatHistory, after the system message and before the real email.
+
+**Why:** Chat models treat conversation turns as prior context — the model "sees" the examples as things it already did correctly. This is more effective than embedding examples in the system prompt string, where they're treated as instructions rather than demonstrated behavior.
+
+### 2. Examples loaded once at startup, cached as ChatHistory prefix
+
+**Decision:** `FewShotService` reads example files at startup, builds a `ChatHistory` prefix (system message + example turns), and caches it as a singleton. Each extraction request clones this prefix and appends the real email.
+
+**Why:** Example files don't change at runtime. Loading once avoids repeated disk I/O. Cloning the cached prefix is cheap (ChatHistory is a list of message objects).
+
+**Alternative considered:** Load examples per-request. Rejected — unnecessary I/O for static data.
+
+### 3. Instruction template as an embedded text file
+
+**Decision:** Store the extraction instruction template as a text file at `examples/extraction/instruction-template.txt`, loaded by FewShotService alongside the examples.
+
+**Why:** Keeps the prompt text editable without recompilation. Co-located with the examples it references. Not in appsettings.json because it's multi-line prose, not configuration.
+
+### 4. Separate extraction endpoint, not a mode flag on /api/chat
+
+**Decision:** `POST /api/chat/extract` as a new controller action, separate from `POST /api/chat`.
+
+**Why:** The extraction path uses a completely different ChatHistory (few-shot prefix, not user system prompt), different tools (extraction plugins only), and a different request DTO. A mode flag on the existing endpoint would add branching complexity. Separate endpoints make each path clear.
+
+**Alternative considered:** Mode flag on ChatRequest (e.g. `"mode": "extract"`). Rejected — the request shapes diverge enough to warrant separate DTOs and endpoints.
+
+### 5. ExtractionRequest includes conversation messages for follow-up
+
+**Decision:** `ExtractionRequest` contains `EmailHtml` (string, the email to extract) plus `Messages` (list, optional follow-up conversation for disambiguation).
+
+**Why:** After initial extraction, the agent may ask disambiguation questions. Follow-up user replies need to be sent back with the full conversation context so the agent can continue. The first request has only `EmailHtml`; subsequent requests include the growing `Messages` list.
+
+### 6. Example folder uses numbered subdirectories
+
+**Decision:** `examples/extraction/few-shot/01/input.html + output.json`, `02/`, etc.
+
+**Why:** Numbered prefixes control ordering in the ChatHistory. Each subdirectory is one example, keeping input and output together. Easy to add/remove/reorder examples by renaming directories.
+
+## Risks / Trade-offs
+
+**[Example quality determines extraction quality]** → Poorly chosen few-shot examples will mislead the model. Mitigation: document selection criteria (diversity of swap structures, currencies, breakclause values). The user curates the 3-5 examples.
+
+**[Instruction template drift]** → If the schema changes, the instruction template must be updated manually. Mitigation: the template references the TradeItem field names explicitly, making it obvious when they're out of sync.
+
+**[ChatHistory size with few-shot examples]** → Each example adds ~2-5KB of tokens. With 5 examples that's ~10-25KB, well within model context limits. Not a risk at current scale but would be if dynamic selection adds more examples later.

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/proposal.md

@@ -0,0 +1,31 @@
+## Why
+
+The extraction agent needs few-shot examples to reliably produce correct structured output from sales emails. Without examples, the agent relies entirely on the instruction template and tool descriptions, which cannot fully convey the implicit mapping conventions (date parsing from "OB" prefix, flattening swap legs, currency symbol mapping, breakclause defaults). A curated set of 3-5 input/output examples injected as conversation turns in the ChatHistory dramatically improves extraction accuracy. The remaining ~95 available examples serve as an evaluation set for offline quality testing.
+
+Additionally, the extraction workflow needs a dedicated API endpoint separate from general chat, since it uses a different system prompt, different tools, and the few-shot ChatHistory prefix.
+
+## What Changes
+
+- **Create examples folder structure** at `examples/extraction/few-shot/` with numbered subdirectories, each containing `input.html` (email) and `output.json` (expected ExtractionResult)
+- **Create extraction instruction template** — a fixed system prompt defining the extraction task, schema, and mapping rules (separate from the user-editable system prompt)
+- **Create a FewShotService** that loads examples from disk at startup and pre-assembles a ChatHistory prefix (system message + alternating user/assistant turns)
+- **Add `POST /api/chat/extract` endpoint** that uses the few-shot ChatHistory, appends the real email, and streams the extraction response via SSE
+- **Create `ExtractionRequest` DTO** for the extraction endpoint (email content + optional follow-up messages for disambiguation)
+- **Update client `ChatApiClient`** with a method for the extraction endpoint
+
+## Capabilities
+
+### New Capabilities
+- `few-shot-prompting`: Defines the example folder structure, loading mechanism, ChatHistory assembly, and instruction template for few-shot extraction prompting
+- `extraction-endpoint`: Defines the dedicated extraction API endpoint, its request/response contract, and how it differs from the general chat endpoint
+
+### Modified Capabilities
+- `chat-streaming`: Add the extraction endpoint alongside the existing chat endpoint, sharing the same SSE streaming contract
+
+## Impact
+
+- **New files**: examples folder, FewShotService, instruction template, ExtractionRequest DTO, extraction controller action
+- **Configuration**: example folder path in appsettings.json
+- **API surface**: new `POST /api/chat/extract` endpoint
+- **Client**: new method on ChatApiClient (no UI changes — that's the email-upload-ux change)
+- **Depends on**: `update-extraction-schema` (needs TradeItem schema for examples and validation tools)

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/specs/chat-streaming/spec.md

@@ -0,0 +1,25 @@
+## MODIFIED Requirements
+
+### Requirement: Chat endpoint proxies to Responses API
+
+The API backend SHALL expose `POST /api/chat` that accepts a `ChatRequest` containing messages, an optional system prompt, and optional model settings. The request is processed using a Semantic Kernel chat completion service. When a system prompt is provided, it SHALL be added as the first system message in the ChatHistory. When model settings are provided, non-null values SHALL be applied to the execution settings. A separate `POST /api/chat/extract` endpoint SHALL handle extraction-specific requests with few-shot prompting.
+
+#### Scenario: Successful chat request with system prompt
+
+- **WHEN** the client sends a POST to `/api/chat` with messages and a system prompt
+- **THEN** the API creates a ChatHistory with the system prompt as the first message, followed by the conversation messages, and processes them through Semantic Kernel
+
+#### Scenario: Successful chat request with model settings
+
+- **WHEN** the client sends a POST to `/api/chat` with messages and model settings (e.g., Temperature=0.3)
+- **THEN** the API applies the settings to OpenAIPromptExecutionSettings before calling the Semantic Kernel
+
+#### Scenario: Successful chat request without optional fields
+
+- **WHEN** the client sends a POST to `/api/chat` with only messages (no system prompt, no settings)
+- **THEN** the API processes the request with default behavior (no system message, default execution settings)
+
+#### Scenario: Extraction request routed to dedicated endpoint
+
+- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML
+- **THEN** the API uses the few-shot ChatHistory prefix and extraction tools instead of the general chat configuration

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/specs/extraction-endpoint/spec.md

@@ -0,0 +1,43 @@
+## ADDED Requirements
+
+### Requirement: Extraction API endpoint
+
+The API SHALL expose `POST /api/chat/extract` that accepts an `ExtractionRequest` containing the email HTML content and optional follow-up conversation messages. The endpoint SHALL use the few-shot ChatHistory prefix (not the user-editable system prompt) and load extraction-specific SK plugins.
+
+#### Scenario: Initial extraction request
+
+- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML and no follow-up messages
+- **THEN** the API assembles the few-shot ChatHistory, appends the email as the final user message, and streams the extraction response via SSE
+
+#### Scenario: Follow-up disambiguation request
+
+- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML and follow-up messages (e.g., user selecting a counterparty)
+- **THEN** the API assembles the few-shot ChatHistory, appends the email, appends all follow-up messages, and streams the continuation response via SSE
+
+#### Scenario: SSE streaming contract
+
+- **WHEN** the extraction endpoint streams a response
+- **THEN** it uses the same SSE format as `/api/chat`: `data: {"text":"..."}\n\n` for deltas and `data: [DONE]\n\n` for completion
+
+### Requirement: ExtractionRequest DTO
+
+The system SHALL define an `ExtractionRequest` class with `EmailHtml` (string, required) and `Messages` (List<ChatMessage>, optional) for follow-up conversation context.
+
+#### Scenario: First request has email only
+
+- **WHEN** the user uploads an email for the first time
+- **THEN** the ExtractionRequest contains `EmailHtml` with the email content and an empty `Messages` list
+
+#### Scenario: Follow-up request includes conversation
+
+- **WHEN** the user replies to a disambiguation question
+- **THEN** the ExtractionRequest contains the original `EmailHtml` plus `Messages` with the full assistant/user exchange since the extraction started
+
+### Requirement: Extraction endpoint uses extraction tools only
+
+The extraction endpoint SHALL import only the extraction-specific SK plugins (counterparty lookup, trade validation, currency validation, schema validation). General chat tools (if any) SHALL NOT be loaded for extraction requests.
+
+#### Scenario: Tool isolation
+
+- **WHEN** the extraction endpoint processes a request
+- **THEN** only extraction-related KernelFunctions are available to the LLM

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/specs/few-shot-prompting/spec.md

@@ -0,0 +1,52 @@
+## ADDED Requirements
+
+### Requirement: Few-shot example folder structure
+
+The system SHALL store few-shot examples at `examples/extraction/few-shot/` with numbered subdirectories (e.g., `01/`, `02/`). Each subdirectory SHALL contain `input.html` (the example email) and `output.json` (the expected ExtractionResult JSON).
+
+#### Scenario: Example folder layout
+
+- **WHEN** the application starts
+- **THEN** it reads example pairs from `examples/extraction/few-shot/` in numeric directory order
+
+#### Scenario: Adding a new example
+
+- **WHEN** a new subdirectory (e.g., `04/`) is added with `input.html` and `output.json`
+- **THEN** the new example is included in the few-shot ChatHistory prefix after the next application restart
+
+### Requirement: Extraction instruction template
+
+The system SHALL load a fixed instruction template from `examples/extraction/instruction-template.txt` that defines the extraction task, the TradeItem schema, and the mapping rules (date parsing, leg flattening, currency mapping, breakclause defaults). This template is NOT the user-editable system prompt.
+
+#### Scenario: Template loaded at startup
+
+- **WHEN** the application starts
+- **THEN** the instruction template is loaded from disk and used as the system message in the extraction ChatHistory
+
+#### Scenario: Template content
+
+- **WHEN** the instruction template is loaded
+- **THEN** it contains the TradeItem field definitions, expected JSON output format, and explicit mapping rules
+
+### Requirement: ChatHistory assembly with few-shot examples
+
+The system SHALL provide a `FewShotService` that assembles a reusable ChatHistory prefix at startup: the instruction template as a system message, followed by alternating User (input.html) and Assistant (output.json) messages for each example. Each extraction request SHALL clone this prefix and append the real email as the final user message.
+
+#### Scenario: ChatHistory prefix structure
+
+- **WHEN** the service assembles the prefix with 3 examples
+- **THEN** the ChatHistory contains: 1 system message + 3 user messages + 3 assistant messages (7 messages total)
+
+#### Scenario: Prefix cached and cloned per request
+
+- **WHEN** an extraction request arrives
+- **THEN** the service clones the cached prefix (not re-reading from disk) and appends the email content as a new user message
+
+### Requirement: Evaluation example folder
+
+The system SHALL support an `examples/extraction/evaluation/` folder for bulk examples used in offline testing. This folder is NOT loaded at startup and NOT used in the few-shot prompt.
+
+#### Scenario: Evaluation folder ignored at runtime
+
+- **WHEN** the application starts
+- **THEN** it does not load examples from `examples/extraction/evaluation/`

openspec/changes/archive/2026-04-06-few-shot-prompt-infrastructure/tasks.md

@@ -0,0 +1,36 @@
+## 1. Example Folder Structure
+
+- [x] 1.1 Create directory structure: `examples/extraction/few-shot/` and `examples/extraction/evaluation/`
+- [x] 1.2 Add placeholder example `01/` with `input.html` (sample email HTML) and `output.json` (sample ExtractionResult JSON matching TradeItem schema). Use realistic but anonymized data.
+- [x] 1.3 Add placeholder example `02/` with a different email pattern (e.g., single swap, different currency)
+- [x] 1.4 Add placeholder example `03/` with an edge case (e.g., breakclause = "Y", or unusual counterparty name)
+
+## 2. Instruction Template
+
+- [x] 2.1 Create `examples/extraction/instruction-template.txt` with the fixed extraction system prompt: task description, TradeItem schema definition (all 7 fields with types), mapping rules (date parsing, leg flattening, currency symbol → ISO code, breakclause default), and expected JSON output format
+
+## 3. FewShotService
+
+- [x] 3.1 Create `FewShotService.cs` in the API project — constructor loads instruction template and all few-shot examples from disk, assembles a ChatHistory prefix (system message + alternating user/assistant turns)
+- [x] 3.2 Add `CloneWithEmail(string emailHtml)` method that clones the cached ChatHistory prefix and appends the email as a user message
+- [x] 3.3 Add `CloneWithEmailAndMessages(string emailHtml, List<ChatMessage> messages)` method for follow-up disambiguation requests — clones prefix, appends email, appends follow-up messages
+- [x] 3.4 Register `FewShotService` as singleton in `Program.cs`
+- [x] 3.5 Add `Examples:FewShotPath` configuration in `appsettings.json` pointing to the examples folder
+
+## 4. ExtractionRequest DTO
+
+- [x] 4.1 Create `ExtractionRequest.cs` in Shared/Models with `EmailHtml` (string, required) and `Messages` (List<ChatMessage>, optional)
+
+## 5. Extraction Endpoint
+
+- [x] 5.1 Add `Extract` action to `ChatController` (or new `ExtractionController`) — `POST /api/chat/extract` accepting `ExtractionRequest`
+- [x] 5.2 In the Extract action: get ChatHistory from `FewShotService.CloneWithEmail()` or `CloneWithEmailAndMessages()` based on whether Messages are present
+- [x] 5.3 Import extraction-specific plugins only (not general chat plugins)
+- [x] 5.4 Stream response via SSE using the same format as the existing chat endpoint
+- [x] 5.5 Update `ChatApiClient` on the client side — add `SendExtractionStreamingAsync(ExtractionRequest request)` method mirroring the existing streaming pattern
+
+## 6. Build and Verify
+
+- [x] 6.1 Build the solution (`dotnet build`) and confirm no compilation errors
+- [x] 6.2 Run all tests (`dotnet test`) and confirm they pass
+- [x] 6.3 Add unit test for `FewShotService` — verify it loads examples and assembles correct ChatHistory structure (message count, roles, ordering)