Overview of the ZK Big Data Stack

The Lagrange Labs ZK Big Data stack is a propriety proof construction optimized for generating large scale batch storage proofs, concurrently with arbitrary dynamic distributed computation. The ZK Big Data stack is extensible to any distributed computational frameworks, ranging from MapReduce to RDD to distributed SQL.

With the Lagrange Labs ZK Big Data stack, a proof can be generated from a single block header, proving both an array of historical storage slot states at arbitrary depths and the result of a distributed computation executed on those state.

For example, consider an application that wishes to consume an outlier resistant 1 week TWAP of a DEX on Ethereum for ETH / USDC pricing on a DEX. A developer must first specify the memory slots and block range (~50,400) that they are interested in including in the data set. Next the developer can write distributed computations that can execute across the data set to derive a standard deviation, factor outliers and finally compute a final averaged price. Once the proof is generated, it will prove both the storage inclusion and the aggregated computation as valid with respect to a given block header.

ZK Big Data proofs are modular and can also be generated from any block header, even originating from existing cross-chain bridges and messaging protocols. ZK Big Data proofs can also be combined with Lagrange State Committee proofs to create expressive and non-interactive n-to-1 chain relationships, involving both historical and multi-chain data.

The First ZKBD Product: ZK MapReduce

ZK MapReduce is the first product in Lagrange's ZK Big Data stack. ZKMR enables DApps to generated programmable storage proofs, at an unprecedented scale, with programming interfaces familiar to Web2 data engineers.

The first version of ZKMR, relies on highly optimized built-in functions, similar to RDD programming in Apache Spark. These functions are optimized to generate circuits that are both expressive and performant across a large amounts of concurrent on-chain states.

//Sample ZKMR Built-In Functions
ethereum.createDataFrame({...})
‍
dataframe.select(...)  

dataframe.map(...)    

dataframe.filter(...)

dataframe.concat(...)

dataframe.reduce(...)

dataframe.union(...)

dataframe.generateProof({...})    

lagrange.submitProof({...})