Blockchain Reorg Handling – Streams | Quicknode Docs

Blockchain Reorg Handling – Streams

What is a Blockchain Reorg?

A blockchain reorg occurs when a node discovers a new chain segment that is longer than the one previously considered the main chain. The node will switch to the longer chain, making it the new canonical chain. This process can lead to transactions that were considered finalized on the old chain being omitted from the new chain, necessitating a robust mechanism to handle such events.

To learn more, take a look at this in-depth blog post: Understanding Blockchain Reorgs: Why Block Numbers Don't Matter as Much as You Think

How Streams Detects and Handles Reorgs

Streams is designed to deliver blockchain data in a sequential and continuous manner, which helps manage blockchain reorganizations effectively.

Detection Process

Step	Description
Continuous flow	Streams delivers blockchain data block by block in batches in sequential order, ensuring you receive data in the exact order it appears on the blockchain
Monitoring blocks	Each block is monitored to ensure its validity and position in the blockchain sequence
Reorg identification	A reorg is detected when a new block's parent hash does not match the hash of the previous block that was streamed
Immediate response	Upon detecting a reorg, the system immediately takes action to reconcile the data

Reconciliation Process

When a reorg is detected, Streams reconciles the data:

Reconciliation with blockchain — The blockchain is polled directly to fetch the most accurate and up-to-date information about the blocks.
Connecting the chain — Working backwards from the point of discrepancy, the streamed blocks are compared with the data obtained from the blockchain. The goal is to find the last block where both datasets match, correctly identifying the point of divergence in the chain.

Enabling Restream on Reorg

All paid plans have access to the "Restream on reorg" option for at-the-tip streams.

With this feature enabled, once the chain is reconciled and the correct sequence of blocks is identified, streaming resumes. This involves re-streaming some blocks to correct any discrepancies caused by the reorg. In the Logs of your stream, you may find entries indicating where a reorg was detected and how the chain is connected once the canonical hash order is identified.

To enable via REST API, set fix_block_reorgs: 1 when creating or updating a stream.

Reorg Metadata in Payloads

When a chain reorganization occurs, the affected blocks are re-delivered with updated data. The metadata includes two fields to help you identify and handle reorged blocks:

Field	Type	Description
reorgs	array \| null	Detailed reorg info including block number, hash, and timestamp of the displaced (orphaned) block
blocks_reorged	array \| null	Block numbers delivered as a result of a chain reorganization

For the full payload structure and examples, see Data Sources — Payload Structure.

Handling Restreamed Data by Destination

If you notice that a block is streamed with a number that is not increased sequentially, it indicates that re-streaming is in progress to reconcile the chain post-reorg. This process ensures the integrity and accuracy of the data you receive, aligning it with the most current state of the blockchain.

Webhook Destinations

Utilizing webhooks allows users to directly monitor incoming stream messages. The deterministic "Restream on Reorg" method seamlessly integrates with webhook-based systems, ensuring users automatically receive updated data segments following a reorg.

How to Handle

For every incoming batch_start_range that is not sequentially increasing than previously processed batch_end_range, consider overwriting the data you previously processed.

S3 Destinations

Users storing data in S3 can leverage both approaches to enhance data integrity:

Without Versioning: Files will be overwritten with the most recent data.
With Versioning: Users should focus on the last file version but pay extra attention to file versions that mismatch in data size. A size discrepancy could indicate a reorg event, necessitating a review of using affected files for post-processing.

PostgreSQL Destinations

When using PostgreSQL as your destination, reorgs require careful handling of duplicate block data. During a reorg, Streams will attempt to re-deliver blocks that were previously delivered.

Primary Key Conflicts: If your table uses from_block_number and to_block_number as part of the primary key, re-streamed blocks may conflict with existing rows. Implement an upsert strategy (e.g., ON CONFLICT ... DO UPDATE) in your ingestion logic to handle this gracefully.
Data Consistency: When you receive a block with a batch_start_range that overlaps previously processed data, treat it as a correction and overwrite the existing records for that block range.

Azure Blob Storage Destinations

Azure Blob Storage behaves similarly to S3 during reorgs:

Default Behavior: Blobs for re-streamed blocks will overwrite existing blobs with the same name, ensuring your storage reflects the canonical chain.
With Versioning/Soft Delete: If blob versioning or soft delete is enabled, previous versions are retained. Review version history for blobs with unexpected size changes, as these may indicate reorg corrections.

Verifying Block Hash Continuity

Regardless of the chosen method or destination, implementing a robust monitoring system is essential. This system should verify the hash-connectedness of block data, ensuring the streamed data's integrity and relevance.

Practice	Description
Continuity Check	Independently verify the continuity of block hashes. Ensure each received block's previous hash matches the hash of the block that was received immediately prior.
Ensuring Chain Integrity	By verifying hash continuity, you can independently confirm that you are working with a consistent and unbroken chain of blocks.
Handling Discrepancies	If you detect a mismatch in hash continuity, pause your processes that depend on this data and verify against the blockchain. This could indicate a reorg being processed or an anomaly that needs attention.
Proactive Data Management	While Streams manages reorgs and delivers accurate data seamlessly, user-side verification forms an additional layer of data integrity.

Reducing Reorg Impact with Latest Block Delay

The "Latest Block Delay" feature offers a probabilistic way to mitigate the risks associated with reorgs. By setting a delay from the chain tip, you can stream data that is a specified number of blocks behind the most recent one.

This approach reduces the likelihood of processing blocks that may not be part of the hash-connected canonical chain. Although this method does not guarantee immunity from reorgs, it significantly decreases the chances of being affected by them, providing a safer yet near-real-time data stream.

To configure via REST API, use the keep_distance_from_tip parameter when creating or updating a stream.