feat: add extraction schema, sidebar nav, few-shot prompting, and prompt settings

Overhaul extraction pipeline with new TradeItem model, conversation flow,
and dedicated extraction endpoint. Add sidebar navigation with NavMenu
component and landing page. Introduce few-shot prompting service and
tests. Add prompt settings and email upload specs. Update OpenSpec
tooling with improved export-spec and extract-feature commands. Archive
completed changes and export full specs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
local
2026-04-06 23:39:23 +01:00
parent 7a5c22593a
commit 5b027eb0db
83 changed files with 4242 additions and 296 deletions

View File

@@ -0,0 +1,25 @@
## MODIFIED Requirements
### Requirement: Chat endpoint proxies to Responses API
The API backend SHALL expose `POST /api/chat` that accepts a `ChatRequest` containing messages, an optional system prompt, and optional model settings. The request is processed using a Semantic Kernel chat completion service. When a system prompt is provided, it SHALL be added as the first system message in the ChatHistory. When model settings are provided, non-null values SHALL be applied to the execution settings. A separate `POST /api/chat/extract` endpoint SHALL handle extraction-specific requests with few-shot prompting.
#### Scenario: Successful chat request with system prompt
- **WHEN** the client sends a POST to `/api/chat` with messages and a system prompt
- **THEN** the API creates a ChatHistory with the system prompt as the first message, followed by the conversation messages, and processes them through Semantic Kernel
#### Scenario: Successful chat request with model settings
- **WHEN** the client sends a POST to `/api/chat` with messages and model settings (e.g., Temperature=0.3)
- **THEN** the API applies the settings to OpenAIPromptExecutionSettings before calling the Semantic Kernel
#### Scenario: Successful chat request without optional fields
- **WHEN** the client sends a POST to `/api/chat` with only messages (no system prompt, no settings)
- **THEN** the API processes the request with default behavior (no system message, default execution settings)
#### Scenario: Extraction request routed to dedicated endpoint
- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML
- **THEN** the API uses the few-shot ChatHistory prefix and extraction tools instead of the general chat configuration

View File

@@ -0,0 +1,43 @@
## ADDED Requirements
### Requirement: Extraction API endpoint
The API SHALL expose `POST /api/chat/extract` that accepts an `ExtractionRequest` containing the email HTML content and optional follow-up conversation messages. The endpoint SHALL use the few-shot ChatHistory prefix (not the user-editable system prompt) and load extraction-specific SK plugins.
#### Scenario: Initial extraction request
- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML and no follow-up messages
- **THEN** the API assembles the few-shot ChatHistory, appends the email as the final user message, and streams the extraction response via SSE
#### Scenario: Follow-up disambiguation request
- **WHEN** the client sends a POST to `/api/chat/extract` with email HTML and follow-up messages (e.g., user selecting a counterparty)
- **THEN** the API assembles the few-shot ChatHistory, appends the email, appends all follow-up messages, and streams the continuation response via SSE
#### Scenario: SSE streaming contract
- **WHEN** the extraction endpoint streams a response
- **THEN** it uses the same SSE format as `/api/chat`: `data: {"text":"..."}\n\n` for deltas and `data: [DONE]\n\n` for completion
### Requirement: ExtractionRequest DTO
The system SHALL define an `ExtractionRequest` class with `EmailHtml` (string, required) and `Messages` (List<ChatMessage>, optional) for follow-up conversation context.
#### Scenario: First request has email only
- **WHEN** the user uploads an email for the first time
- **THEN** the ExtractionRequest contains `EmailHtml` with the email content and an empty `Messages` list
#### Scenario: Follow-up request includes conversation
- **WHEN** the user replies to a disambiguation question
- **THEN** the ExtractionRequest contains the original `EmailHtml` plus `Messages` with the full assistant/user exchange since the extraction started
### Requirement: Extraction endpoint uses extraction tools only
The extraction endpoint SHALL import only the extraction-specific SK plugins (counterparty lookup, trade validation, currency validation, schema validation). General chat tools (if any) SHALL NOT be loaded for extraction requests.
#### Scenario: Tool isolation
- **WHEN** the extraction endpoint processes a request
- **THEN** only extraction-related KernelFunctions are available to the LLM

View File

@@ -0,0 +1,52 @@
## ADDED Requirements
### Requirement: Few-shot example folder structure
The system SHALL store few-shot examples at `examples/extraction/few-shot/` with numbered subdirectories (e.g., `01/`, `02/`). Each subdirectory SHALL contain `input.html` (the example email) and `output.json` (the expected ExtractionResult JSON).
#### Scenario: Example folder layout
- **WHEN** the application starts
- **THEN** it reads example pairs from `examples/extraction/few-shot/` in numeric directory order
#### Scenario: Adding a new example
- **WHEN** a new subdirectory (e.g., `04/`) is added with `input.html` and `output.json`
- **THEN** the new example is included in the few-shot ChatHistory prefix after the next application restart
### Requirement: Extraction instruction template
The system SHALL load a fixed instruction template from `examples/extraction/instruction-template.txt` that defines the extraction task, the TradeItem schema, and the mapping rules (date parsing, leg flattening, currency mapping, breakclause defaults). This template is NOT the user-editable system prompt.
#### Scenario: Template loaded at startup
- **WHEN** the application starts
- **THEN** the instruction template is loaded from disk and used as the system message in the extraction ChatHistory
#### Scenario: Template content
- **WHEN** the instruction template is loaded
- **THEN** it contains the TradeItem field definitions, expected JSON output format, and explicit mapping rules
### Requirement: ChatHistory assembly with few-shot examples
The system SHALL provide a `FewShotService` that assembles a reusable ChatHistory prefix at startup: the instruction template as a system message, followed by alternating User (input.html) and Assistant (output.json) messages for each example. Each extraction request SHALL clone this prefix and append the real email as the final user message.
#### Scenario: ChatHistory prefix structure
- **WHEN** the service assembles the prefix with 3 examples
- **THEN** the ChatHistory contains: 1 system message + 3 user messages + 3 assistant messages (7 messages total)
#### Scenario: Prefix cached and cloned per request
- **WHEN** an extraction request arrives
- **THEN** the service clones the cached prefix (not re-reading from disk) and appends the email content as a new user message
### Requirement: Evaluation example folder
The system SHALL support an `examples/extraction/evaluation/` folder for bulk examples used in offline testing. This folder is NOT loaded at startup and NOT used in the few-shot prompt.
#### Scenario: Evaluation folder ignored at runtime
- **WHEN** the application starts
- **THEN** it does not load examples from `examples/extraction/evaluation/`