Table of contents
- Executive overview
- Detailed overview
- Products susceptible to blockchain reorgs
- Metrics for protocols and auditors
- Product detailed reorg issues explained
- On-chain random based systems
- Off-ramp systems
- Protocols that require users to create something and then interact with it in a two step manner
- Cross-chain mechanisms
- Systems where an on-chain action has a gradual effect depending on its proximity to a specific block or timestamp
- Blockchain validation software
- Conclusion
- References
Executive overview
There exists a blockchain event referred to as blockchain reorganization or reorgs. When this event happens, transactions can be discarded or reordered within a block and this can span across multiple blocks
Depending on the Web3 product type being built, reorgs can be a natural issue that needs to be taken into consideration. See Products susceptible to blockchain reorgs for more information
Depending on the blockchain, these events are more or less frequent. In some extreme cases a reorg has affected as much as 5 minutes of history. See Metrics for protocols and auditors for more information
A coordinated malicious attack can also cause a blockchain reorg. Although improbable it has happened in the past leading to a double spending attack
For auditors, there are examples provided of issues for each product type
Detailed overview
In Web3 we often hear the idiom the blockchain is immutable but there are cases when it's not, or to be more precise it becomes practically immutable (reach finality) only after some time has passed since the block, which contains users' transactions, was minted. This is due to events called blockchain reorganizations, or reorgs for short.
Paradigm has one of the best articles on reorgs, how and why it happens. I highly recommend you give it a read as we will only briefly touch on why it fundamentally happens. From the mentioned article, we have the following good block reorg definition:
A reorg is an event where a block that was part of the canonical chain becomes no longer part of the canonical chain because a competing block beat it out.
Reorgs happen because of the way blockchain validation client softer works. The algorithm that determines which is the canonical chain (fork choice rule), under certain circumstances, can conclude that the current chain history is not optimal and choose a different chain from those provided by competing miners (fork chains), to become the canonical chain.
When a block reorg happens, as an external observer of the chain you see that:
several blocks have been discarded and replaced with others
this does not imply that all transactions in the original blocks were discarded, a large number will probably still be included in the new blocks, possibly even all but not in the same order
Reorgs appear more often than people realize. Using on-chain explorers such as polygonscan.com/blocks_forked, for Polygon reorgs, you can see they appear often:
The first reorg in the image indicates a ReorgDepth
of 11, meaning 11 blocks were discarded (approx. 22 seconds of history was changed).
Products susceptible to blockchain reorgs
Protocols or systems that are naturally vulnerable to blockchains reorgs are:
Products vulnerable to reorgs | Examples and details |
On-chain random based systems | Examples: - betting protocols - on-chain casinos - random reward systems |
Off-ramp crypto systems | Examples: - centralized exchanges (CEX) - crypto-to-fiat providers |
Protocols that require users to create something and then interact with it in a two step manner | - yes, this is vague - the majority of problematic systems here are those that have user deploy a contract then interact with it Example behavior to look for: - deploying a vault in one transaction and depositing funds to it in another - user creates a proposal then votes on it in an on-chain voting system |
Cross-chain mechanisms | - yes, this is also vague Examples: - cross-chain communication protocols - token bridges - cross-chain gas acquisition mechanisms |
Systems where an on-chain action has a gradual effect depending on its proximity to a specific block or timestamp | Examples: - on-chain Dutch auctions |
Blockchain validation software | Example: - node validation clients |
Metrics for protocols and auditors
When designing or auditing a protocol that may be vulnerable to reorg issues there are some data points that must be taken into account which are particular to each chain. The following table serves as a reference point.
Blockchain | Reorgs/day | Max depth/day | Largest known reorg | Average block time (approx. in seconds) |
Ethereum | 10-25 | 1 | 1 | 12.08 |
Cronos | 10+ | 5-7 | 7 | 5.6 |
Polygon PoS | 2-10 | 5-15 | 157 | 2.2 |
BNB Chain | 0-2 | 1 | 5 | 3 |
Avalanche C-Chain | reorgs not possible | reorgs not possible | reorgs not possible | 2 |
Fantom | reorgs not possible | reorgs not possible | reorgs not possible | 1.41 |
Arbitrum One | 0 | 0 | 0 | 0.25 |
Optimism | 0 | 0 | 0 | 2 |
Base | 0 | 0 | 0 | 2 |
zkSync Era | 0 | 0 | 0 | 2 |
Notes and observations
the above data has been collected either from 3rd party aggregators or from sites displaying the raw data directly. They are also taken after any major changes in the network, e.g. for Ethereum after PoS and for BNB Chain after their latest major scalability changes that drastically reduced reorg rates
When deciding how much blocks to wait before confirming a transaction off-chain in sensitive systems (such as crypto to fiat off-ramps) it is best to wait a number of blocks more then highest know reorg depth on that chain that happened historically
- For example, deposits on a CEX from Polygon can wait even 200 blocks before being regarded as final
L2s theoretically can experience reorgs but it has never happened in practice (at least not know to or found out by the author of this article).
In the References section you can find links from where the above numbers were collected
Product detailed reorg issues explained
For each of the aforementioned type of products or system susceptible to blockchain reorg issues that were mentioned, a detailed description follows.
On-chain random based systems
On-chain protocols that have a random component are naturally affected by blockchain reorgs. These type of protocols can be betting, casino types, games with a random reward system, practically anything that calculates randomness using an oracle.
Example of situations that can appear:
a player wins a good reward in random draw but due to a blockchain reorg he loses it
a gambler loses an on-chain dice game but due to a blockchain reorg he now actually wins
In practice, Chainlink's VRF (Verifiable Random Function) is often used to calculate random number. This API accepts a block confirmation time, which, to quote the documentation represents:
how many blocks the VRF service waits before writing a fulfillment to the chain to make potential rewrite attacks unprofitable in the context of your application and its value-at-risk.
Examples of this type of vulnerability seen in audits:
07/10/2021
Miners Can Re-Roll the VRF Output to Game the Protocol01/06/2023
Polygon chain reorgs will often change game results20/11/2023
Polygon chain reorgs will change mystery box tiers which can be gamed by validators
To mitigate this issue, it is enough to choose an appropriate block confirmation time. See Metrics for protocols and auditors for recommended numbers.
Off-ramp systems
To off-ramp crypto, refers to the process of exchanging cryptocurrencies for fiat currency. Systems doing crypto off-ramping include:
centralized exchanges (CEX)
crypto-to-fiat providers
Crypto deposits into these systems are confirmed after a specific number of blocks have passed since the transaction in order to mitigate any issues possibly caused by blockchain reorgs.
This article gives some specific block confirmations requirements found on know CEXs. Example Coinbase requires 35 block confirmations for ETH deposits while Binance requires only 12.
As an observation, CEXs tend to severely over-wait for confirmations. Example, a blockchain reorg on Ethereum of more then 2 blocks has not happened in years.
Off-ramp systems, particularly CEXs, are an attractive target for 51% attacks. In the past, there were several cases where attackers gained 51% of a network validation resources (hashing rate) and proceed to initiate withdrawals on several exchanges then reorg the chain in order to re-initiate the withdrawals on other exchanges.
Most notable 51% attacks are mentioned in this article and involve the Verge, Ethereum Classic, Bitcoin Gold, Feathercoin and Vertcoin tokens.
Mitigating this form of attack is not straight forward. One component of the solution is to overly increase the required confirmation count. Although in those cases this is not enough, as the network takeover can last an undefined amount of time. An on-chain monitoring solution coupled with withdrawals/deposits pausing need to implemented.
Protocols that require users to create something and then interact with it in a two step manner
(yes, the title is too big)
This type of reorg issues are the most frequently seen in this vulnerability category. They appear when a protocol requires users to first create or initiate a component and then, in a different transaction to interact with it.
Another important and key precondition is that the deployed contracts use the Solidity new keyword for contract creation (or the CREATE opcode). Both of these rely on the calling contract nonce to determine the address of the newly-deployed contract and do not take into consideration any custom input.
Example, a protocol uses a factory-like pattern to deploy contracts, whos functions are permissionless. Say the factory logic deploys a contract via one call while giving special permissions to the deployer. After the deploy, the user send a deposit to the newly deployed contract.
An example attack scenario in the above case:
alice
creates a vault via the factory contract (transaction A)alice
then deposits into the vault tokens (transaction B)a block reorg happens and transaction A is discarded (transaction B still exists)
normally transaction B now would revert if executed
bob
deploys himself the vault that initiallyalice
did (via frontrunning) and because of the underlying issue, it is created on the same address as the initial vaultalice
's deposit reaches nowbob
's vault, and he can withdraw it
Another interesting example seen ITAW (In The Audit Wilderness) is:
alice
creates a proposal; proposal ID is determined by a counter increment only (transaction A)alice
votes on her proposal (transaction B)a block reorg happens and transaction A is discarded (transaction B still exists)
normally transaction B now would revert if executed
bob
creates himself a different proposal (via frontrunning) which receives the same ID asalice
's due to faulty implementationalice
's vote now reaches a malicious proposal
Examples of this type of vulnerability seen in audits:
12/04/2023
Re-org attack in factory23/05/2023
Attacker could abuse victim's vote to pass their own proposal30/05/2023
Manycreate
methods are suspicious of the reorg attack07/07/2023
An attacker can front-rundeployVault
to deploy at the same address17/07/2023
Identifying publications using its ID makes the protocol vulnerable to blockchain re-orgs
To mitigate the issue use the create2 opcode and have the salt contain elements distinctive of the caller (e.g. hash on the inputs plus the msg.sender
).
Cross-chain mechanisms
Similar to how CEXs or crypto off-ramp solutions need to consider block reorganizations so does any cross-chain mechanism that needs an off-chain component to transfer data from one chain to another.
Issues can appear if one of the destination or source chain reorgs which may create situations where a bridging was removed on the source chain but on the destination chain it was created. Leading to double spending.
Examples:
cross-chain communication protocols
token bridges
cross-chain gas acquisition mechanisms (e.g. gasbot)
Examples of this type of vulnerability seen in audits:
Mitigation is again straightforward and involves waiting a specific amount of blocks before considering a transaction as done (finalized). See Metrics for protocols and auditors for exact numbers
Systems where an on-chain action has a gradual effect depending on its proximity to a specific block or timestamp
(yes, this is another title that is too big)
There are protocols for which the result of an on-chain action depend gradually on their proximity to a specific block number or timestamp. There are, by their nature vulnerable to reorgs since transactions can be shuffled within a reorg leading to different outcome then expected.
The predominant (only) example see in this case is that of a Dutch auction:
alice
initiates a payment atauctionStartBlock + 3
needing to payprice/3
a block reorg happens and
alice
's transaction is now put atauctionStartBlock + 2
the transaction goes through and
alice
ends up payingprice/2
Examples of this type of vulnerability seen in audits:
25/03/2023
useLoan doesn't allow liqudator to specifiy maximum price11/12/2023
No check for sequencer uptime can lead to dutch auctions failing or executing at bad prices
The issue is mitigated per project value logic. In the Dutch auction example, a maximum payable amount should also be passed as a parameter when initiating the buy.
Blockchain validation software
There are over 1000 different blockchains in existence, each with its own validation logic and client software. While there are some common validation software being used, in the rare cases a blockchain development team decides to implement their custom validation logic, it is imperative to take blockchain reorgs into consideration.
Consider an example where a validation node always validates the oldest block. But because of reorgs, that block is often discarded and depending on the validation logic, the node may be slashed and lose funds.
Examples of this type of vulnerability seen in audits:
01/09/2019
Malicious clients can use forks or reorgs to convict honest nodes01/09/2019
in3-server - should enforce safe settings for minBlockHeight12/09/2023
Risk of double-spend attacks due to use of single-node Clique consensus without finality API19/12/2023
Block depth used does not offer guarantees against reorgs under edge cases
Conclusion
Blockchain reorganization is an event that has always existed and will probably exist for years to come. Reorgs impact specific application types and as such need to be taken into consideration otherwise severe financial loss may happen, as it has in the past.
As a manager, if the project your team is building is the type that can suffer when reorgs happen, share this link with your team and ask them to check. Resolving reorg issues should not have its own KPI, it is enough to include it in the (hopefully already existing) security related ones.
As an auditor you can either focus on the type of protocols that are susceptible to reorg issues or focus on the event itself. It is enough to ask the question:
how would a block reorg affect this protocol?
and to follow up from there to identify if such issues exist. The current article shows issues regarding specific products but a pattern can be deduced and applied to similar protocols.
References
Ethereum
Cronos
the only external source (without having a node) for block reorgs is https://cronos.org/explorer/reorgs which does not provide data in a user-friendly manner. The metrics in the article were extrapolated from monitoring the page sporadically during the day (I kid you not)
Polygon PoS
BNB Chain
Avalanche C-Chain
Fantom
Arbitrum One
Optimism
Base
ZkSync Era