Hello, Harmony Community: I want to give you an update on the recent mainnet stability issue and our plan to resolve it. We’ve seen a significant increase of txns on mainnet since a week ago, with close to 100 txns every block. And now based on various evidence on the txn content and sending pattern, most of the increased txn are very likely spams that’s part of a targeted attack taking advantage of our negligible limit on minimum txn gas price (1/1000000000000000000 ONE). The increased txn volume added burden on our rpc endpoint which at some time affected normal user’s usage. However, our mainnet network and consensus functioned well with the increased txn volume.
Starting from around 8AM PST June 9th, we saw significant increase (~50x-100x) on p2p network traffic in our mainnet, which later confirmed as targeted p2p network spam attack. The attack slowed down our block time and eventually caused network downtime of around 2 hours later on June 9th. We were able to recover the network after adding various mitigations and fixes on components including block sync, crosslinks, p2p message validation and block size limit. Currently the spamming attack is still on-going and we are planning a few more solutions to the attack. First, we are working on a p2p level rate limiting mechanism to stop the spam from the source. Second, we are updating our minimum txn gas price requirement to 1/1000000000 ONE which is the same as Ethereum’s 1 Gwei requirement. With the updated min gas price, our network will be stronger against txn spam attacks since it will be more costly to do so.
What we need from our community? For validators, please update your node’s binary in time when we publish a new release with the additional fixes soon in next 1 or 2 days. For users and developers using harmony tools and sdks (including exchange partners), please update your txn gas price to above 1/1000000000 ONE (For eth-compatible tools, no action required since it’s set by default). And if your txns aren’t accepted in time, try increase your gas price to outbid the spammer’s txns.
I would like to thank our community for the continuous support and understanding, and we are working non-stop to make our network better and stronger against attacks. For any questions and issue reports, please go to our discord channel https://harmony.one/mainnet-nodes and tag @harmony-team for help. Thanks.