Historical Data for Streams

Suggest Edits

📘

Streams by default operate with real-time data, but can be configured to handle both real-time and historical data. Allowing you to query the full history of any blockchain event that ever happened for a given stream.

Overview

The ability to access and query blockchain data is an essential part of many applications and systems. Streams provide a convenient way to handle and process this data, allowing users to easily subscribe to and consume real-time updates from the blockchain.

Stream Backfills allows streams to also handle historical data. With this feature, streams can be configured to handle both real-time and historical data. This allows users to query the full history of any blockchain event by a given stream, providing a more comprehensive and powerful way to access and analyze blockchain data. This can be useful in situations where it is necessary to access and analyze data from past events, such as when performing retroactive analysis or when trying to recreate the state of the blockchain at a specific point in time.

Current Limitations

  • Ethereum is the only supported network at this time.
  • Currency conversion is not supported inside the blueprint filters.
  • Either From address or To address must be set for blueprints where the source type is "Transactions".

You can check if your stream/blueprint is supported on the Blueprint Create Page:

1592

or on the Stream Page:

2200

Getting Started

Prerequisites

  • You have a subscription plan that has Stream Backfills
  • You have a stream deployed and running Learn how
  • You have an API key Learn how
  • You have validated that your selected stream can be backfilled (check the Current Limitations)

Add Historical Data To Your Streams

There are 2 ways to add historical data to your streams:

Option 1: via the UI

Navigate to the stream's page you want to be backfilled with historical data and click "Backfill historical data".

2634

Once you have clicked "Backfill historical data", the backfill process will begin. This process may take some time, depending on the amount of data being backfilled.

2216

Option 2: via the API

This functionality is also exposed from our API. To start the backfilling make the following request:

curl --request POST \
     --url https://api.hal.xyz/v1/streams/id/backfilling \
     --header 'content-type: application/json'

Once you have made the request, the backfill process will begin. This process may take some time, depending on the amount of data being backfilled. You can check the status of the backfill by making a GET request to the /streams/id/backfilling endpoint. This endpoint will return a JSON object containing information about the backfill process, including its status and progress.

Consume Historical Data via API

2194

Once the backfill process is complete, you can now consume the historical data via the Stream Events API. The historical data will be merged with the real-time data and sorted on the block number by default.

Updated 10 months ago