The Database of a Trading Site Processes Continuous Market Data to Update Financial Asset Prices

Core Architecture for Real-Time Data Ingestion

Modern trading site platforms rely on specialized database systems that handle streams of tick-level data from exchanges. These systems must ingest thousands of price updates per second without bottlenecks. The typical architecture uses a time-series database (like InfluxDB or ClickHouse) optimized for high-frequency writes. Each trade, bid, and ask generates a record containing timestamp, asset identifier, price, and volume. The database partitions this data by asset and time window, enabling fast queries for recent price changes. A caching layer, often Redis, stores the latest prices for each asset to serve user dashboards instantly.

Data arrives via WebSocket connections from liquidity providers. The database processes these streams through a pipeline that normalizes format, validates integrity (checking for outliers or stale data), and applies business logic such as spread calculation. For example, if the raw feed shows a stock at $100.05 bid and $100.10 ask, the system computes the mid-price ($100.075) and updates the asset’s current price field in the database. This entire cycle-from ingestion to update-must complete in under 10 milliseconds to maintain a competitive edge.

Handling Data Volume and Velocity

High-frequency markets generate terabytes daily. The database uses sharding across multiple nodes, with each shard responsible for a subset of assets. Write-ahead logging ensures no data loss during crashes. Compression algorithms like Gorilla reduce storage footprint by encoding timestamps and values efficiently. For volatile assets like cryptocurrencies, the system may batch writes every 50 milliseconds to balance throughput and latency, rather than writing every single tick.

Updating Financial Asset Prices with Accuracy

Price updates are not simple overwrites. The database maintains a versioned history to support audit trails and technical analysis. When new market data arrives, the system inserts a new row into the price_history table and updates the current_price column in the assets table. Atomic operations prevent race conditions-two simultaneous trades for the same asset are processed sequentially via row-level locking or optimistic concurrency control. This ensures the displayed price always reflects the most recent trade or market consensus.

Some platforms use a “last traded price” versus “mark price” distinction. The database calculates the mark price as a weighted average of multiple exchange feeds to resist manipulation. For instance, if a sudden spike occurs on one exchange, the system ignores it if other sources disagree. This logic runs inside stored procedures or stream processors (like Apache Flink) that query the database and write back the adjusted price. The result is a robust, manipulation-resistant price feed for traders.

Latency Optimization Techniques

To minimize delay, databases use in-memory tables for active assets. Disk writes happen asynchronously. Indexing on (asset_id, timestamp) accelerates range queries for charts. Connection pooling and prepared statements reduce overhead for frequent update queries. Some systems implement a “hot” path where price updates bypass SQL parsing and directly modify memory-mapped files, achieving microsecond-level latency.

Challenges and Solutions in Continuous Processing

Network latency from data sources can cause out-of-order arrivals. The database sequences events using a hybrid logical clock (HLC) to resolve conflicts. If a trade with timestamp T1 arrives after a later trade with T2, the system still inserts both but marks T1 as historical. Duplicate detection via unique trade IDs prevents double-counting. Another challenge is maintaining data consistency during a market crash when volumes spike 100x. Auto-scaling database clusters add read replicas on demand, while write nodes prioritize critical updates over non-essential analytics.

Regulatory compliance requires storing all price data for years. The database tiered storage moves data older than 30 days to cheaper object storage (e.g., S3) with compressed formats. Queries for historical data are slower but acceptable for backtesting. For real-time needs, only recent data resides in fast storage. This hybrid approach keeps costs manageable while meeting retention rules.

FAQ:

How does the database handle price updates from multiple exchanges simultaneously?

The system uses a consensus algorithm that compares timestamps and trade volumes. If feeds conflict, it applies a weighted average from trusted sources, then updates the database atomically.

What happens if the database receives a price that is clearly erroneous, like a zero value?

Validation rules reject outliers beyond a standard deviation threshold. The database logs the error and retains the previous valid price until a corrected feed arrives.

Can the database support backtesting with historical tick data?

Yes. The time-series structure allows efficient range scans. Queries for specific date ranges retrieve compressed data from cold storage, then decompress on the fly.

How does the system prevent data loss during a server crash?

Write-ahead logging (WAL) records every operation before applying it. On restart, the database replays the WAL to restore the last consistent state.

Is there a limit to how many asset prices can be updated per second?

Modern setups handle 100,000+ updates per second per node. Clusters scale horizontally to absorb spikes, with load balancers distributing the stream.

Reviews

Marcus L.

I run a small trading firm. The database updates prices so fast that our arbitrage bot never misses a spread. Latency is below 5ms consistently.

Sarah K.

As a developer, I appreciate the clean API and atomic updates. We integrated our own risk checks easily. No data corruption even during flash crashes.

David R.

The tiered storage saved us 60% on costs. Historical data queries are slower but acceptable for monthly reports. Real-time performance is rock solid.