Hello, Harmony Community: I want to give you an update on the recent mainnet stability issue and our plan to resolve it. We’ve seen a significant increase of txns on mainnet since a week ago, with close to 100 txns every block. And now based on various evidence on the txn content and sending pattern, most of the increased txn are very likely spams that’s part of a targeted attack taking advantage of our negligible limit on minimum txn gas price (1/1000000000000000000 ONE). The increased txn volume added burden on our rpc endpoint which at some time affected normal user’s usage. However, our mainnet network and consensus functioned well with the increased txn volume.
Starting from around 8AM PST June 9th, we saw significant increase (~50x-100x) on p2p network traffic in our mainnet, which later confirmed as targeted p2p network spam attack. The attack slowed down our block time and eventually caused network downtime of around 2 hours later on June 9th. We were able to recover the network after adding various mitigations and fixes on components including block sync, crosslinks, p2p message validation and block size limit. Currently the spamming attack is still on-going and we are planning a few more solutions to the attack. First, we are working on a p2p level rate limiting mechanism to stop the spam from the source. Second, we are updating our minimum txn gas price requirement to 1/1000000000 ONE which is the same as Ethereum’s 1 Gwei requirement. With the updated min gas price, our network will be stronger against txn spam attacks since it will be more costly to do so.
What we need from our community? For validators, please update your node’s binary in time when we publish a new release with the additional fixes soon in next 1 or 2 days. For users and developers using harmony tools and sdks (including exchange partners), please update your txn gas price to above 1/1000000000 ONE (For eth-compatible tools, no action required since it’s set by default). And if your txns aren’t accepted in time, try increase your gas price to outbid the spammer’s txns.
I would like to thank our community for the continuous support and understanding, and we are working non-stop to make our network better and stronger against attacks. For any questions and issue reports, please go to our discord channel https://harmony.one/mainnet-nodes and tag @harmony-team for help. Thanks.
I just did a transaction and it charged me network fee of “0.000000000000372905” ONE. I think we need to drop several zero’s to make it a meaningful cost for the attackers. This is ok for now but i have seen all sorts of attacks when genuine project launches happen and the current fees actually arent really costing anything at all for such attacks.
Yes, exactly, soon we will add the minimum gas price requirement of 1/1000000000 ONE on txns. Though it’s still very cheap, but definitely more reasonable than the current situation without lower limit.
the tx that i did (on lootswap) costed me 1/100 billionth ONE and it had 0.000000001 gas. does that mean even with the change, an attacker could do 100 billion spam txs per ONE? sorry i don’t understand the pricing, trying to understand & comprehend the proposal
The minimum gas requirement for a txn is 21000 unit of gas and with a min gas price per unit of 0.000000001 ONE, one txn could cost at minimum 0.000021 ONE. So with 1 ONE, you can send around 47620 basic txns.
If you look at iota I believe they automatically increase the gas fee exponentially over time when they get a suspected spam attack. Plus they can also automatically reconfigure the IP addresses under attack? Might be worth studying this and learning from it?
Hi I am educating myself on the finer aspects of the harmony architecture. Does the harmony architecture support linear infinite scaling to be able to operate as a global DLT? If so then it should be able to scale up and cope with spam attacks?
Could you clarify if there is a limit to the number of shards that can be deployed and whether additional shards can be invoked on-the-fly based on demand peaks based on a pre-deterministic allocation of wallet addresses to shards?
I am aware of the Radix DLT which deterministically pre-shards every address to its own shard based on the address itself. This means that by definition every transaction is a cross-shard transaction and the vast majority of transactions i.e. simple transfers involving only 2 addresses can be executed in parallel without full confirmation processing. It’s only the more complex cross-chain or cross-shards atomic composable transactions which involves several addresses that requires to be “stitched” together and orchestrated as required by the consensus layer. They also use an address space of 2**256 to provide massive scalability for global use as processing proliferates to includes zkp based unique address per transaction and device based internet of things usage.
Probably additional food for thought to consider to position harmony as viable global player?
So why is this being done? Can Harmony not handle the transaction count? I’m surprised how a team that boasted 2000 txn/sec/shard is having issues at just a fraction of its capabilities.
Can Harmony actually handle what it claims, or we just putting a band-aid and ignoring the real issue here?
