Streamlining The Graph: An Introduction to Substreams in Web3

Athsrueas.eth | Thomas Freestone

If you want data from an Ethereum network to use it you need to index. There are meaningful challenges here.  Want security and verifiability? Read block by block linearly using API calls on ethereum nodes. StreamingFast solved these problems with Firehose powered substreams. Using a streaming first approach and the flat files structure from Firehose we have rich protobuff models, stream cursors, and the ability to use parallelization to improve speeds without sacrificing verifiability.  We now have a more efficient and scalable approach to indexing data. Furthermore, substreams provide a way to ensure data verifiability and security, in fact you could rebuild the entire node from the flat files. Everything is cachebale and hashable. Instead of using the traditional handlers in assembly developers can use Rust to write substreams modules. These can run in parallel boosting performance.

Key Takeaway:  Get up to 100x speed improvements with Firehose and Substreams on Ethereum clients. It is fast, responsive, and modular, making The Graph the fastest and most efficient way to get data from blockchains. Initial implementation is built and has been tested for over a year. On March 16 StreamingFast announced the substreams reached general availability! On April 27 at core dev call 20 StreamingFast showed off the new UI tool which is available to use now.

More resources can be found below