Blockchain Data Backfilling – Streams

Updated on
Jul 20, 2026

On this page

Backfilling is the process of retrieving historical blockchain data to populate databases, analyze past trends, or audit transactions. Whether you need to index an entire chain from the Genesis block or just catch up on the last 24 hours of activity, retrieving historical data efficiently is a critical infrastructure challenge.

Streams makes backfilling simple with one-click templates, server-side filtering, batching and compression options, and guaranteed delivery to your preferred destination. Instead of writing complex scripts to poll RPC endpoints, you configure a Stream, set your block range, optionally apply filters to extract only the data you need, and Streams pushes the data to your destination (e.g., Webhook, S3, PostgreSQL, Azure Storage, etc.) reliably and at scale.

Why Use Streams for Backfilling?

Streams handles the infrastructure side such as retries, block ordering, and error handling, letting you focus exclusively on your application's data logic.

Estimating Backfill Costs

Backfilling historical blockchain data consumes API credits based on the number of blocks processed, using network dataset multipliers. Streams uses the same shared API credit pool as RPC.

Use the API Credits Calculator to estimate your backfill costs before starting. The calculator shows:

• Total API credits needed for backfilling available historical blocks
• Credits per block based on your selected dataset and network
• Which networks support backfilling (Solana networks, for example, do not currently support backfill)

How to Backfill Data

1. Select your chain and network. See the supported chains for Streams.
2. Define your block range. You can choose to start from the Genesis block or a specific block height. For the end block, you can set it to a specific height or choose continuous streaming to keep receiving new blocks.
3. Select your dataset. Streams provides different datasets such as Block, Block with Receipts, Transactions, Logs, etc. Choose the one that fits your use case. See Backfill Data by Ecosystem for chain-specific datasets.
4. Apply filters ( optional). Use server-side filtering to narrow down the data you want to receive.
5. Choose your destination. Configure where you want the data to be sent, such as Webhooks, S3, PostgreSQL, etc. Check out the Destinations documentation for more details.
6. Check the connection and send a test payload in the Stream configuration page to ensure everything is set up correctly.
7. Start the Stream.

Backfill Data by Ecosystem

Streams can be used to backfill data for any ecosystem, including Ethereum, Bitcoin, Solana, and more. Since they have different data structures and formats, select your ecosystem below for chain-specific datasets, example filters, and responses.

• Ethereum and EVM Chains
• Solana
• Bitcoin
• XRP Ledger

Ethereum and EVM Chains

Streams support many EVM chains, including Ethereum, Base, Arbitrum, and BNB Smart Chain. All EVM chains share a similar architecture and data structure, so filters and payload formats are generally the same across all chains, while some chain-specific differences may exist.

Decoding EVM Data

When working with EVM-compatible chains, you can verify and parse data more easily using the decodeEVMReceipts function. This utility transforms raw hex data into human-readable formats by taking raw transaction receipts and your contract ABIs as inputs.

The decoding process automatically:

• Matches event signatures in transaction logs with the provided ABIs.
• Decodes parameters according to their types (addresses, integers, strings, etc.).
• Returns structured data with named parameters in a decodedLogs object.

Learn more about this function: Decoding EVM Data

Available Data Sources

For EVM chains such as Ethereum, Base, Arbitrum, and BNB Smart Chain, you can use the following datasets to backfill data.

note Data sources availability varies by chain. Some datasets (specifically Traces) may not be supported on all EVM networks. Check the Data Sources page for the current support matrix.

Example: ERC-20 Token Transfers

The sample function below filters a block of transactions to identify and decode standard ERC-20 token transfers by checking the input data for the transfer method signature.

// Chain: Ethereum
// Dataset: Transactions
// Test with block: 23977403

type Payload struct {

Data     [][]Tx                 `json:"data"`

Metadata map[string]interface{} `json:"metadata"`

}

type Tx struct {

Hash        string `json:"hash"`

From        string `json:"from"`

To          string `json:"to"`

Input       string `json:"input"`

BlockNumber string `json:"blockNumber"`

}

func Filter(qn *qn.QNContext, payload Payload) interface{} {

// The standard ERC-20 transfer(address,uint256) method signature

const transferMethodID = "0xa9059cbb"

var filteredTransactions []map[string]interface{}

for _, transactions := range payload.Data {
        for _, tx := range transactions {
            if !strings.HasPrefix(tx.Input, transferMethodID) || len(tx.Input) < 138 {
                continue
            }
            toAddress := "0x" + tx.Input[34:74]
            amount := hexToDecimal(tx.Input[74:])
            filteredTransactions = append(filteredTransactions, map[string]interface{}{
                "txHash":        tx.Hash,
                "fromAddress":   tx.From,
                "toAddress":     toAddress,
                "amount":        amount,
                "tokenContract": tx.To,
                "blockNumber":   tx.BlockNumber,
            })
        }
    }
    if len(filteredTransactions) == 0 {
        return nil
    }
    return map[string]interface{}{"transactions": filteredTransactions}
}
// hexToDecimal function omitted for brevity.

// Chain: Ethereum
// Dataset: Transactions
// Test with block: 23977403

function main(payload) {
    const filteredTransactions = [];
    const transferMethodId = "0xa9059cbb";
    for (const transactions of payload.data) {
        for (const transaction of transactions) {
            if (typeof transaction === 'object' && transaction !== null && typeof transaction.input === 'string') {
                if (transaction.input.startsWith(transferMethodId)) {
                    const toAddress = "0x" + transaction.input.substr(34, 40);
                    const value = BigInt("0x" + transaction.input.substr(74));
                    filteredTransactions.push({
                        txHash: transaction.hash,
                        fromAddress: transaction.from,
                        toAddress: toAddress,
                        amount: value.toString(),
                        tokenContract: transaction.to,
                        blockNumber: transaction.blockNumber,
                    });
                }
            }
        }
    }
    return filteredTransactions.length > 0 ? { transactions: filteredTransactions } : null;
}

{
  "transactions": [
    {
      "amount": "80007920",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0xa80f9793051cd1f428ad61b276d431f30dd59b6a",
      "toAddress": "0xabd22d07c199a56bafa5e2add3cde1127bc98292",
      "tokenContract": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
      "txHash": "0x013a9cdd1de2ade97a1d79178fe59f2025e41495db050aab0cc706cd2be3b2fe"
    },
    {
      "amount": "71839000",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0x828ee64b59f33e6c3a6b8d4ad8298aeb65421445",
      "toAddress": "0xbb03a0b5159d985c304ab183b80e884ae38c9c89",
      "tokenContract": "0xdac17f958d2ee523a2206206994597c13d831ec7",
      "txHash": "0x2ae18c8918de4744df1d93729e75f89f402723630982224654d18cae047ec74b"
    }
  ]
}

Solana

Streams supports backfilling historical slot ranges on Solana. Due to the network's high throughput and massive data volume, backfills are typically used to index specific windows of time (e.g., retrieving the last 1-2 weeks of data) rather than the entire ledger history from Genesis.

Available Data Sources

Example: Track Account Balance Changes

// Chain: Solana
// Dataset: Block
// Test with slot: 282164688

type Payload struct {
    Data     []SolanaBlock          `json:"data"`
    Metadata map[string]interface{} `json:"metadata"`
}

type SolanaBlock struct {
    ParentSlot   int64      `json:"parentSlot"`
    BlockTime    int64      `json:"blockTime"`
    Transactions []SolanaTx `json:"transactions"`
}

type SolanaTx struct {
    Meta struct {
        Err          interface{} `json:"err"`
        PreBalances  []int64     `json:"preBalances"`
        PostBalances []int64     `json:"postBalances"`
    } `json:"meta"
}

// Filter implementation omitted for brevity.

// Chain: Solana
// Dataset: Block
// Test with slot: 282164688
const FILTER_CONFIG = {
  accountId: '9kwU8PYhsmRfgS3nwnzT3TvnDeuvdbMAXqWsri2X8rAU',
  skipFailed: true,
};

function main(payload) {
  const matchedTransactions = [];
  for (const block of payload.data) {
    for (const tx of block.transactions) {
      const result = processTransaction(tx, block);
      if (result) {
        matchedTransactions.push(result);
      }
    }
  }
  return matchedTransactions.length > 0 ? { matchedTransactions } : null;
}

// processTransaction implementation omitted for brevity.

Bitcoin

Streams supports backfilling for Bitcoin and Bitcoin Cash. These chains use the UTXO (Unspent Transaction Output) model, which differs from account-based blockchains.

Available Data Sources

Example: Track High-Value Transactions

// Chain: Bitcoin
// Dataset: Block
// Test with block: 927171
function main(payload) {
  const MIN_VALUE = 100000000;
  const results = [];
  for (const block of data) {
    for (const tx of block.txs) {
      if (parseInt(tx.value) >= MIN_VALUE) {
        results.push({
          txid: tx.txid,
          blockHeight: tx.blockHeight,
          blockTime: tx.blockTime,
          valueBTC: parseInt(tx.value) / 100000000,
          fees: tx.fees
        });
      }
    }
  }
  return results.length ? results : null;
}

[
  {
    "blockHeight": 927171,
    "blockTime": 1765316511,
    "fees": "0",
    "txid": "fbc62d0e65f2bed9e193480b33adc6f117ba44c839928becadfb61f817f6e7fb",
    "valueBTC": 3.14554746
  },
  {
    "blockHeight": 927171,
    "blockTime": 1765316511,
    "fees": "5640",
    "txid": "494ff35569b01a799db0ef0975923844faa46bd1a57a8625cd44b152ded3b661",
    "valueBTC": 6.92443529
  }
]

XRP Ledger

Streams allows you to retrieve historical data from the XRP Ledger (XRPL), delivering complete ledger objects to your destination.

Available Data Sources

Example: Filter Successful Payments

// Chain: XRP Ledger
// Dataset: Ledger
// Test with ledger: 100768299
function main(payload) {
  const results = [];
  for (const item of data) {
    const ledger = item.ledger;
    for (const tx of ledger.transactions) {
      if (tx.TransactionType === "Payment" && tx.metaData?.TransactionResult === "tesSUCCESS") {
        results.push({
          hash: tx.hash,
          ledgerIndex: ledger.ledger_index,
          closeTime: ledger.close_time_iso,
          account: tx.Account,
          destination: tx.Destination,
          fee: tx.Fee
        });
      }
    }
  }
  return results.length ? results : null;
}

[
  {
    "account": "rUg8ac5ikpTaWk5RPei8xuYkNEyUs53G1i",
    "closeTime": "2025-12-09T21:49:31Z",
    "destination": "rNxp4h8apvRis6mJf9Sh8C6iRxfrDWN7AV",
    "fee": "12",
    "hash": "0059A19205DA30084FB42C54C343BED150D8F3EB5443FA2C77B87F540A17D6D7",
    "ledgerIndex": "100768299"
  },
  {
    "account": "rUg8ac5ikpTaWk5RPei8xuYkNEyUs53G1i",
    "closeTime": "2025-12-09T21:49:31Z",
    "destination": "rMqfygR9sbZvWMRqStzUunBXH8Ut5DLfxs",
    "fee": "12",
    "hash": "099243DEE9C12B90A6080A99FC10EB620CDBB713BF17CC6E5D674176A4308A0A",
    "ledgerIndex": "100768299"
  }
]

Tips for Backfilling

Batching for Faster Backfills

By default, Streams delivers one block at a time. For historical backfills, increasing the batch size (e.g., 10 or 100 blocks per delivery) reduces overhead and speeds up ingestion. Adjust this in your Stream settings based on your destination's capacity.

Consider using Elastic Batch for automatic batch size adjustment if you continue to real-time streaming.

Performance Optimization

Combining Filtering with Compression improves performance during large-scale backfills.

Debugging Filters

Log from your filter function to debug during development. Logs appear in the Logs tab next to the Results tab in the Stream Filter Editor.

In JavaScript, use console.log(). In Go, use fmt.Println.

Testing with Raw Data

When writing a new filter, you can download the raw data for a specific test block from the Stream configuration UI. This helps you understand the exact payload structure before writing filter logic.

Security and Authentication

Streams includes a security token with each delivery that you can use to verify requests originated from Quicknode. You can also configure custom headers in your destination settings. For webhook destinations, see How to Validate Incoming Streams Webhook Messages for implementation details.