Requirements

Functional Requirements:

• Ingest live tick-by-tick market data (full order-book updates) with minimal delay by connecting to exchange data feeds for real-time price data, order book updates, and trades.
• Expose a plug-in API so quantitative developers can load custom strategies that transform market data into trade orders.
• Apply real-time pre-trade risk checks on every order, including position limits, notional caps, and order-rate rules, before it is transmitted to the exchange.
• Provide an integrated backtesting harness that replays historical tick data so strategies can be validated offline under realistic fills before deployment.

Non-Functional Requirements:

Performance

• Market data ingestion latency: sub-millisecond to low single-digit milliseconds depending on exchange connectivity.
• Pre-trade risk check latency: ideally under 100 microseconds in hot path.
• Order submission latency: low milliseconds or below for colocated HFT setups.
• Support millions of market-data events per second during market spikes.
• Support high order throughput while enforcing per-strategy, per-user, and exchange rate limits.

Reliability

• No silent order loss.
• Every order must have a durable lifecycle record.
• Market data gaps must be detected using sequence numbers.
• Strategy outputs must be deterministic in backtesting and explainable in production.

Availability

• Market data ingestion, risk checks, and order routing should remain available during partial component failures.
• Use active-active or active-passive regional deployment depending on exchange colocation constraints.
• Target availability:
  • Trading hot path: 99.99% or higher during market hours.
  • Backtesting and analytics: 99.9% or lower is acceptable.

Scalability

• Horizontally scale market-data consumers by symbol partition.
• Horizontally scale strategy runners by strategy ID, account ID, or symbol shard.
• Store historical tick data in partitioned object storage by venue, asset class, symbol, and date.
• Separate hot-path trading systems from cold-path analytics and backtesting systems.

Resilience and Fault Tolerance

• Use sequence numbers and replay buffers for market-data recovery.
• Use idempotency keys for order placement, cancel, and replace requests.
• Use circuit breakers for extreme volatility, exchange instability, or internal degradation.
• Risk engine must fail closed by default: if risk cannot be evaluated, reject the order.
• Use kill switches at strategy, account, asset, venue, and global levels.

Security

• Strong authentication and authorization for users and services.
• Hardware-backed key storage or HSM for exchange credentials.
• Signed strategy artifacts and sandboxed execution.
• Immutable audit logs.
• Network segmentation between public APIs, strategy runtime, risk engine, and exchange gateway.
• Strict separation between research/backtesting and production trading credentials.

API Design

2.1 Strategy Plug-in API

Strategies should not directly talk to exchanges. They receive normalized market data and emit order intents.

Example order intent:

2.2 Market Data API

Prefer binary streaming protocols for production hot paths.

2.3 Order API

Internal strategy-facing API:

Important fields:

2.4 Risk Configuration API

2.5 Backtesting API

High-Level Design

The platform should be split into a latency-sensitive trading hot path and a scalable analytics/backtesting cold path.

3.1 Hot Path

The hot path handles live trading:

1. Exchange market-data feeds send tick data.
2. Market Data Gateway normalizes events and validates sequence numbers.
3. In-memory order book builder maintains full depth per symbol.
4. Event bus distributes normalized events to strategy runners.
5. Strategy runners emit order intents.
6. Risk Engine performs pre-trade checks.
7. Order Management System persists order state.
8. Exchange Gateway sends orders to venues.
9. Execution reports update positions, PnL, and strategy state.

The hot path should avoid heavy databases, distributed joins, synchronous analytics queries, and non-deterministic dependency calls.

3.2 Cold Path

The cold path handles backtesting, research, analytics, and reporting:

1. Raw market data is written to immutable storage.
2. Normalized market data is stored in a queryable historical tick store.
3. Backtesting workers replay ticks to the same strategy interface used in production.
4. Simulation engine models latency, fills, fees, rejects, and market impact.
5. Results are stored for analysis and comparison.

3.3 Core Design Principle

Production trading and backtesting must share the same strategy API and market-data event model. Otherwise, strategies may perform well in backtests but fail in production due to mismatched assumptions.

Detailed Component Design

5.1 Market Data Gateway

Responsible for connecting to exchange feeds and normalizing tick data.

Responsibilities

• Maintain TCP, UDP multicast, WebSocket, or proprietary feed connections.
• Decode venue-specific binary protocols.
• Normalize events into internal schemas.
• Validate sequence numbers.
• Detect packet loss, feed gaps, crossed books, stale books, and out-of-order messages.
• Publish normalized events to the low-latency event bus.
• Persist raw feed data for audit and backtesting.

Reliability Design

• Use active/passive feed handlers per exchange.
• Use exchange-provided sequence numbers to detect missing events.
• Maintain replay buffers for short-term recovery.
• If gaps cannot be recovered, mark affected symbols as stale and block strategy trading for those symbols.

Performance Design

• Use zero-copy parsing where possible.
• Use kernel bypass or optimized networking for HFT environments.
• Pin hot-path processes to dedicated CPU cores.
• Avoid garbage-collected runtimes in the tightest market-data path if ultra-low latency is required.
• Partition processing by venue and symbol.

5.2 Order Book Builder

Maintains real-time full-depth order books.

Responsibilities

• Apply incremental order-book updates.
• Build top-of-book and full-depth views.
• Publish book snapshots to strategies.
• Detect invalid book states.

Edge Cases

Out-of-order update

Action:

• Buffer briefly if sequence gap is small.
• Request replay or snapshot recovery.
• Mark symbol as non-tradable until recovered.

Stale order book

Action:

• If no update is received within configured threshold, mark symbol stale.
• Risk engine blocks new orders for stale symbols.

5.3 Low-Latency Event Bus

The event bus distributes market events to strategies.

Options

• For ultra-low latency: Aeron, Chronicle Queue, shared memory, or custom ring buffer.
• For scalable non-HFT streaming: Kafka, Redpanda, Pulsar, or NATS.

Recommended Architecture

Use two buses:

1. Hot-path bus for live trading.
2. Durable event stream for audit, analytics, and replay.

This prevents analytics workloads from affecting trading latency.

5.4 Strategy Runtime

Runs user-defined trading algorithms.

Responsibilities

• Load signed strategy artifacts.
• Provide normalized market-data callbacks.
• Emit order intents.
• Track strategy-level state.
• Enforce CPU, memory, and outbound order budgets.
• Support dry-run, paper-trading, backtest, shadow, and live modes.

Isolation

Strategies should run in isolated sandboxes:

• Process-level isolation for low-latency strategies.
• Container isolation for medium-latency strategies.
• WebAssembly or restricted runtime for safer plug-ins.
• No direct network access to exchanges.
• No direct database mutation permissions.

Deployment Modes

5.5 Pre-Trade Risk Engine

The Risk Engine is the most critical safety component.

Checks Per Order

• Max quantity per order.
• Max notional per order.
• Max position per symbol.
• Max gross exposure.
• Max net exposure.
• Max daily loss.
• Max open orders.
• Max order rate.
• Allowed symbol list.
• Allowed venue list.
• Price collar checks.
• Fat-finger protection.
• Self-trade prevention.
• Stale market-data protection.
• Trading halt protection.
• Strategy kill-switch status.

Design Choice

Risk checks should be performed in-memory on the hot path.

Use strongly consistent state for:

• Current positions.
• Open orders.
• Pending orders.
• Recently sent orders.
• Per-strategy counters.
• Per-account exposure.

Failure Behavior

The Risk Engine should fail closed.

If position state, limits, market data, or account state is unavailable, reject the order.

5.6 Order Management System

The OMS tracks every order from creation to terminal state.

Responsibilities

• Assign internal order ID.
• Deduplicate requests by idempotency key.
• Persist order state transitions.
• Route orders to the correct exchange gateway.
• Handle cancel and replace workflows.
• Reconcile exchange execution reports.
• Update positions and PnL.
• Publish order updates back to strategies.

Order State Machine

Idempotency Case

Problem:

A strategy retries SubmitOrder because it did not receive an acknowledgement.

Solution:

• Every request must include an idempotency key.
• OMS stores idempotency key with request hash and resulting order ID.
• If the same key is retried with the same payload, return the original result.
• If the same key is retried with a different payload, reject as an idempotency conflict.

Example:

This prevents duplicate orders during client retries, network timeouts, OMS failover, or exchange gateway reconnection.

5.7 Exchange Gateway

Responsible for venue-specific order transmission.

Responsibilities

• Translate internal order format to exchange protocol.
• Maintain exchange sessions.
• Handle heartbeats.
• Track sequence numbers.
• Resend messages when required by protocol.
• Parse execution reports.
• Enforce exchange-specific throttles.
• Handle disconnects and session recovery.

Rate Limit Case

Problem:

Exchange allows only a certain number of order messages per second.

Solution:

• Maintain per-venue token buckets.
• Maintain per-account and per-strategy token buckets.
• Prioritize cancels over new orders during stress.
• Reject or queue low-priority orders when limits are reached.
• Alert when sustained throttling occurs.
• Feed rate-limit state back to strategies.

Recommended behavior:

5.8 Position and PnL Service

Tracks real-time trading state.

Responsibilities

• Maintain current positions.
• Maintain realized and unrealized PnL.
• Maintain open exposure.
• Feed state to the Risk Engine.
• Reconcile internal state with exchange drop copies, clearing data, and broker reports.

Consistency

The Risk Engine should use the freshest in-memory position state, while slower reconciliation systems verify correctness asynchronously.

5.9 Backtesting Engine

The backtesting engine should reuse production strategy interfaces.

Responsibilities

• Load historical tick data.
• Replay market data deterministically.
• Simulate exchange matching behavior.
• Apply the same risk checks used in production.
• Produce performance reports.
• Compare strategy versions.
• Support parameter sweeps.

Realistic Simulation Features

• Queue position modeling.
• Partial fills.
• Market impact.
• Slippage.
• Fees and rebates.
• Latency model.
• Exchange rejects.
• Trading halts.
• Rate-limit simulation.
• Missing-data simulation.
• Volatility spike simulation.

Backtesting Metrics

5.10 Historical Data Store

Storage Layers

Partitioning

Partition by:

• Asset class.
• Venue.
• Symbol.
• Date.
• Feed type.

Example path:

5.11 Security Design

Authentication

• SSO for human users.
• mTLS for service-to-service communication.
• Short-lived service tokens.
• Hardware-backed key storage for exchange credentials.

Authorization

Use role-based and attribute-based access control.

Examples:

Strategy Security

• Strategies must be signed before deployment.
• Production deployment requires approval workflow.
• Strategy runtime has no direct exchange credentials.
• Strategy runtime has restricted filesystem and network permissions.
• All strategy artifacts are versioned.

5.12 Observability and Performance Metrics

Hot-Path Metrics

Reliability Metrics

Business Metrics

5.13 Thundering Herd Scenario

Scenario

Market volatility spikes after major news. Thousands of symbols update simultaneously. Many strategies react at the same time and submit a burst of orders.

Risks

• Event bus overload.
• Risk Engine saturation.
• Exchange rate-limit violations.
• OMS queue buildup.
• Duplicate retries.
• Cascading latency.
• Strategies trading on stale data.

Mitigations

1. Use backpressure between market data, strategy runtime, risk, and OMS.
2. Partition workloads by symbol, account, and venue.
3. Use bounded queues, not unbounded queues.
4. Drop or coalesce non-critical analytics events, never critical order events.
5. Use priority lanes:
   • cancel orders
   • reduce-risk orders
   • new orders
   • analytics
6. Apply adaptive throttling per strategy.
7. Trigger circuit breakers if latency exceeds safe thresholds.
8. Use jittered retries to avoid synchronized retry storms.
9. Reject new orders if market data is stale.
10. Keep kill switch independent from overloaded strategy infrastructure.

5.14 Idempotent Case

Scenario

A strategy sends an order. The OMS processes it, but the network response times out. The strategy retries the same request.

Required Behavior

The platform must not create two orders.

Solution

• Require idempotency_key on every order, cancel, and replace request.
• Store the key with request hash and response.
• On retry:
  • Same key and same payload: return original response.
  • Same key and different payload: reject.
  • New key: treat as a new request.

Additional Protection

• Use unique client_order_id.
• Deduplicate exchange execution reports.
• Reconcile OMS state against exchange state.
• Make order state transitions append-only and auditable.

5.15 Rate Limit Case

Scenario

A strategy tries to send 10,000 orders per second, but the exchange allows 1,000 messages per second.

Required Behavior

The platform must protect the exchange session, account, and market integrity.

Solution

• Use token-bucket rate limiters.
• Enforce limits at multiple layers:
  • Strategy Runtime
  • Risk Engine
  • OMS
  • Exchange Gateway
• Prioritize cancels and risk-reducing orders.
• Reject excess new orders with explicit reason codes.
• Notify strategy of throttling state.
• Alert operators if rate-limit pressure persists.

Example rejection:

5.16 Availability and Failover Strategy

Market Data Failover

• Use redundant feed handlers.
• Detect sequence gaps.
• Recover using replay or snapshot.
• Mark affected instruments stale until recovered.

OMS Failover

• Persist order state transitions.
• Use leader election for active OMS shard ownership.
• On failover, replay event log to reconstruct state.
• Reconcile with exchange open orders before accepting new orders.

Exchange Gateway Failover

• Maintain hot standby sessions where supported.
• Handle exchange protocol sequence recovery.
• Block duplicate order replay after reconnect.
• Prioritize cancel-on-disconnect policies where supported.

Risk Engine Failover

• Risk state must be reconstructable from:
  • positions
  • open orders
  • pending orders
  • fills
  • limits
  • recent order counters

If risk state cannot be reconstructed, reject new orders until safe.

5.17 Scalability Strategy

Market Data

• Shard by venue and symbol.
• Use dedicated ingestion workers per feed.
• Use compact binary event format.
• Use separate hot and cold pipelines.

Strategy Runtime

• Shard by strategy ID, account ID, and symbol universe.
• Use autoscaling for non-HFT strategies.
• Use pinned colocated instances for HFT strategies.
• Prevent noisy-neighbor impact with quotas.

Risk Engine

• Shard by account or trading book.
• Keep hot state in memory.
• Use lock-free or single-writer-per-shard design where possible.
• Replicate state asynchronously for failover.

Backtesting

• Scale horizontally using distributed workers.
• Partition historical tick data by date and symbol.
• Use batch scheduling for large simulations.
• Cache normalized datasets and computed features.

5.18 Recommended Data Stores

5.19 Critical Tradeoffs

Latency vs Durability

For HFT, synchronous database writes in the order hot path may be too slow. A common solution is to write to a durable append-only log optimized for sequential writes, then asynchronously project state into query databases.

Strategy Flexibility vs Safety

Allowing arbitrary plug-ins is powerful but dangerous. Use sandboxing, artifact signing, resource limits, and staged deployment.

Throughput vs Ordering

Strict global ordering does not scale. Use per-symbol, per-account, or per-strategy ordering where correctness requires it.

Backtest Accuracy vs Speed

Tick-level queue-position-aware simulation is expensive but more realistic. Use faster approximate backtests for exploration and slower high-fidelity backtests before deployment.

5.20 Final Architecture Summary

The platform should use a low-latency event-driven architecture with strict separation between live trading and offline research. The hot path should be optimized for deterministic behavior, bounded latency, risk enforcement, and fault isolation. The cold path should optimize for scalable storage, replayability, analytics, and high-fidelity backtesting.

The most important design principles are:

1. Never send an order without risk approval.
2. Never lose or duplicate an order silently.
3. Never trade on stale or corrupted market data.
4. Always make order decisions auditable.
5. Keep backtesting and production behavior as similar as possible.
6. Fail closed when safety-critical dependencies are unavailable.
7. Use circuit breakers, kill switches, and rate limits to survive volatility.