Documentation

Block Indexers

Overview

Indexers are services that continuously monitor the blockchain, processing each block to extract and store relevant data in a queryable format. A well-designed indexer maintains a processing pipeline where block fetching, transaction parsing, event decoding, and database persistence operate as independent stages. This architecture allows each stage to proceed without waiting for others, maximizing throughput and minimizing latency.

Threading Considerations

To achieve optimal indexer performance, threading is highly recommended. Block processing involves multiple operations that benefit from concurrent execution:

  • CPU-intensive processing: Parsing transactions, decoding events, and updating state.
  • I/O-bound operations: Fetching blocks from the network while processing others.
  • Database writes: Writing indexed data without blocking fetch operations.
  • Parallel transaction processing: Transactions within a block can often be processed concurrently.
Tip

A typical threading model separates block fetching into a dedicated thread that populates a queue, while worker threads consume blocks from the queue and handle parsing and storage. This separation ensures that network latency does not bottleneck data processing, and database writes do not delay block retrieval.

Recommendations

Building on these threading concepts, the following recommendations help ensure a robust and efficient indexer implementation:

  • Separate fetching from processing: Use the main thread for I/O operations and delegate CPU-intensive work to worker threads.
  • Batch block fetches: Retrieve multiple blocks in a single request using getBlocks() to reduce network overhead.
  • Use worker pools: Distribute block processing across multiple threads to maximize CPU utilization.
  • Implement queue management: Apply backpressure mechanisms to prevent memory exhaustion when processing falls behind fetching.
  • Database connection pooling: Assign each worker its own database connection to avoid contention and maximize write throughput.
  • Handle reorganizations: Implement rollback logic to revert indexed state when chain reorganizations occur.

Typical Architecture

Conceptual indexer architecture with worker threads
typescript
import { Worker, isMainThread, parentPort, workerData } from 'worker_threads';

// Main thread: fetches blocks and dispatches to workers
// Worker threads: process blocks, decode transactions, index data

interface IndexerConfig {
    workerCount: number;      // Number of worker threads (recommend: CPU cores - 1)
    batchSize: number;        // Blocks per batch
    queueSize: number;        // Max blocks in processing queue
}

// Fetch blocks in main thread
async function fetchBlocks(provider: JSONRpcProvider, start: bigint, end: bigint) {
    const blockNumbers: number[] = [];
    for (let i = start; i <= end; i++) {
        blockNumbers.push(Number(i));
    }
    return provider.getBlocks(blockNumbers);
}

// Process blocks in worker threads
// Each worker handles: transaction parsing, event decoding, database writes
On this page