Data and analytics

Last edit: @wackerow(opens in a new tab), July 3, 2024

Introduction

As utilization of the network continues to grow, an increasing amount of valuable information will exist in the on-chain data. As the volume of data rapidly increases, calculating and aggregating this information to report upon or drive a dapp can become a time and process heavy endeavor.

Leveraging existing data providers can expedite development, produce more accurate results, and reduce ongoing maintenance efforts. This will enable a team to concentrate on the core functionality their project is trying to provide.

Prerequisites

You should understand the basic concept of Block Explorers in order to better understand using them in the data analytics context. In addition, familiarize yourself with the concept of an index to understand the benefits they add to a system design.

In terms of architectural fundamentals, understanding what an API(opens in a new tab) and REST(opens in a new tab) are, even in theory.

Block explorers

Many Block Explorers offer RESTful(opens in a new tab) API(opens in a new tab) gateways that will provide developers visibility into real-time data on blocks, transactions, miners, accounts, and other on-chain activity.

Developers can then process and transform this data to give their users unique insights and interactions with the blockchain. For example, Etherscan(opens in a new tab) provides execution and consensus data for every 12s slot.

The Graph

The Graph Network(opens in a new tab) is a decentralized indexing protocol for organizing blockchain data. Instead of building and managing off-chain and centralized data stores to aggregate on-chain data, with The Graph, developers can build serverless applications that run entirely on public infrastructure.

Using GraphQL(opens in a new tab), developers can query any of the curated open APIs, known as sub-graphs, to acquire the necessary information they need to drive their dapp. By querying these indexed sub-graphs, Reports and dapps not only get performance and scalability benefits but also the built in accuracy provided by network consensus. As new improvements and/or sub-graphs are added to the network, your projects can rapidly iterate to take advantage of these enhancements.

Client diversity

Client diversity is important for the overall health of the Ethereum network because it provides resilience to bugs and exploits. There are now several client diversity dashboards including clientdiversity.org(opens in a new tab), rated.network(opens in a new tab), supermajority.info(opens in a new tab) and Ethernodes(opens in a new tab).

Dune Analytics

Dune Analytics(opens in a new tab) pre-processes blockchain data into relational database (PostgreSQL and DatabricksSQL) tables, allows users to query blockchain data using SQL and build dashboards based on query results. On-chain data are organized into 4 raw tables: blocks, transactions, (event) logs and (call) traces. Popular contracts and protocols have been decoded, and each has its own set of event and call tables. Those event and call tables are processed further and organized into abstraction tables by the type of protocols, for example, dex, lending, stablecoins, etc.

SubQuery Network

SubQuery(opens in a new tab) is a leading data indexer that gives developers fast, reliable, decentralized, and customized APIs for their web3 projects. SubQuery empower developers from over 165+ ecosystems (including Ethereum) with rich indexed data to build an intuitive and immersive experiences for their users. The SubQuery Network powers your unstoppable apps with a resilient and decentralized infrastructure network. Use SubQuery's blockchain developer toolkit to build the web3 applications of the future, without spending time building a custom backend for data processing activities.

To start, visit the Ethereum quick start guide(opens in a new tab) to start indexing Ethereum blockchain data in minutes in a local Docker environment for testing before going live on a SubQuery's managed service(opens in a new tab) or on SubQuery's decentralised network(opens in a new tab).

Ethernow - Mempool Data Program

Blocknative(opens in a new tab) provides open access to its Ethereum historical mempool data archive(opens in a new tab). This enables researchers and community good projects to explore the pre-chain layer of Ethereum Mainnet. The data set is actively maintained and represents the most comprehensive historical record of mempool transaction events within the Ethereum ecosystem. Learn more at Ethernow(opens in a new tab).