Architecture

Provider Pattern

Norman Engine uses a factory pattern to abstract LLM providers. The createProvider() function returns a provider instance based on the LLM_PROVIDER environment variable.

// providers/factory.ts
const provider = createProvider(); // Returns OpenAI or Bedrock provider
 
// Both implement the same interface:
interface LLMProvider {
  chat(options: ChatOptions): Promise<ChatResponse>;
  chatStream(options: ChatOptions): AsyncGenerator<string>;
  getModels(): Promise<Model[]>;
}

This means switching providers is a config change — no code modifications needed.

Request Flow

Client → Express Router → Auth Middleware → Provider Factory → LLM API
                                                    ↓
                                            Token Service
                                            (count + record)

Streaming (SSE)

1. Client sends POST /api/chat with stream: true
2. Response headers set for SSE (text/event-stream)
3. Provider's chatStream() yields chunks
4. Each chunk written as SSE data: frame
5. Final data: [DONE] sent on completion
6. Token usage recorded asynchronously after stream ends

Non-Streaming

1. Client sends POST /api/chat or POST /api/complete
2. Provider's chat() returns complete response
3. JSON response sent immediately
4. Usage recorded in background (non-blocking)

Token Service

Every request is tracked:

• Usage Recording: Per-user, per-model token counts stored in MongoDB
• Chat Logging: Full request/response logged with latency metrics
• Aggregation: getUserUsage() provides daily/weekly/monthly rollups