Summary of the Horizon Bridge Incident

An individual, group or groups of perpetrators began transferring Harmony’s Horizon Bridge’s assets on the Ethereum chain, starting on Jun-23-2022 11:06:46 AM +UTC, 14 bridged assets, including USDC, ETH and USDT on Ethereum, and also BNB on Binance Smart Chain, into a previously unrecognized account 0x0d04…d000. The perpetrator(s) compromised at least two out of four private keys of the bridge validators and gained control of the bridged assets, to then begin funneling the assets into a combination of new wallets (known as wallet hopping) and eventually into a mixer called Tornado Cash. Forensics teams are actively monitoring the activities of these wallets and the mixer.

Harmony had offered a $1M bounty to return the remaining stolen amount. The bounty was soon after increased to $10M but Harmony has not received a legitimate response to date. The global hunt continues with investigations passed on to the Federal Bureau of Investigation (FBI) with the cooperation of partners, including multinational cryptocurrency exchanges. We believe this was a coordinated attack targeting our internal infrastructure. There is no evidence to date that the bridge smart contracts, or the blockchain protocol, were compromised.

Decrypting the bridge keys would have required several operations from within a secure set of servers to generate keys on-the-fly performed on servers with privileged access (authorized roles). We believe the attacker(s) 1) employed a phishing schemes to trick at least one software developer to install malicious software on their laptop, that 2) enabled the attacker to either read chat threads to understand how to operate the bridge, and/or gain access to non-public bridge infrastructure code, plus 3) gaining backdoor access to one or more servers, to perform the hack. The perpetrator(s) were successfully able to do all three.

An attack inducing installation of trojan-horse software occurred as recently as June 17th. We are still investigating what this software is capable of. Meanwhile, we discovered a vulnerability on June 18th related to software packages included with our internal subgraph service. This leads to exposing all server addresses within a private cloud environment. This may have been potentially used to probe internal server addresses. The engineering team was in the process of addressing this vulnerability before the bridge hack on June 23rd. Post-hack, we discovered evidence that there were server logs with date and timestamps, hinting that the perpetrator began reviewing the Horizon Bridge implementations as early as June 2nd. This, in itself, is insufficient to compromise the bridge. The combination of all of the above demonstrates an orchestrated attack to perform the hack.

The Horizon Bridge has since been halted but not until approximately 64,000 wallets, with approximately 50,000 of them being bridged wallet owners, and the rest spread across the entire DeFi ecosystem, were affected. To regain trust from our community, Harmony also paused the unaffected Trustless Bitcoin bridge, reinforced it with an expanded key set, and swapped the multisig signers to be managed by a new set of keys. We have since released the Bitcoin bridge funds and have the owners reclaim their funds, effectively rendering the bridge offline, in order to secure user funds on this bridge. We plan to reconvene bridge activities at a later date.

Harmony has now set up a Security Operations team to lock down Cloud infrastructure to minimize any further compromise by reducing the attack surface. Access to any of the cloud services, including bridge nodes, were decoupled in an effort to preserve forensic evidence while maintaining minimal blockchain node operations for the Harmony blockchain. A focused Security Operations team is reviewing the practices around Cloud operations to review and rollout security best practices, such as the use of Single Sign-On with Multi Factor Authentication (MFA), use of VPN, clear separation of sensitive and insensitive data, full traceability of access, in all identifiable entry points, in order to minimize any potential attack vectors in the future. Some of these new security practices have been put in place to enable team members to continue their limited set of operations while the process continues over the next calendar quarter aimed to enforce measures to prevent any similar incidents from happening in the future.

We have engaged Chainalysis and Anchain for forensic analysis, are working with the FBI, and are tracking the hackers with the assistance of partners, including exchanges worldwide. As mentioned earlier, funds were funneled into the mixer, Tornado Cash. At the time of this writing, there are no movements to or from high-confidence withdrawal candidates from Tornado Cash.

8 Likes

We appreciate the extremely untimely response to the incident and hope to next see which OpSec personnel are taking the fall for this breach, as well as the formation of new SOPs for the chain in regards to responding to an emergency situation, INCLUDING proper communication channels. Would have loved to see many of these best security practices in places prior to handling over $100M in assets!

10 Likes

#HarmonyONE building back stonger :muscle::blue_heart:

3 Likes

This has been the first instance of clear, precise communication that was much needed for months. I have to commend @Jacksteroo and team for distillating the underlying circumstances around the hack & the team’s incident response.

Looking forward to hearing more about the restitution process related to the bridge in the upcoming days. Well done, and this is a good start. :saluting_face::grinning:

4 Likes

Thank you for the detailed summary.

If my interpretation from reading the summary is accurate, Harmony, the engaged forensic teams and FBI are still tracking the assets in Tornado Cash. This is excellent!

Also, if available, please provide a name list of the Security Operations Team. Thank you.

1 Like

I was worried when I saw this update. I thought you guys were going to do something crazy and generate coins out of nowhere and give them to the people that lost their money which could have been leveraged just so the people could take their 100 mil back out of the chain because they lost faith in the security… Thank God you guys aren’t doing that.

4 Likes

Steps in the right direction. Just keep building. We’re still early crypto wise , we can learn from mistakes. As long as the core tech teams is in place here, Harmony will be ok…Shards and finality will win in the end… keep hitting the tech. That’s what attracted me 2 years ago… the tech.

6 Likes

I know that crypto is the wild west and this stuff happens, but I think that people like me who have never used the bridge are effected. Thust me I understood how it works, but it doesn’t make it right. Having a stablecoin go to almost zero is absurd, essentially when I went to stable coins to stop the bleeding. Harmony should be accountable and put the money back and if they dont have it, bring someone who does. If not, the chain is dead and all the good protocols will leave if they already didnt start the process.

1 Like

I think it unwise to disclose names of a Security Team that can then be targeted.

2 Likes

At least, the person in-charge should be named? Or, the team should be invisible at all?

1 Like

Have the team isolate the hack and is the protocol safe to use ?

1 Like

Hi Comunity! I actually sorry this situation! My situation is that I have USDC in My wallet in the Harmony red! But when I going to exchange the USDC to ONE to get out the funds of the Harmony red, the parity is different of the real parity! On any DEX ONE is $ 0.22 but the real price now is 0.022! Could someone tell me why? What I can do ?

2 Likes

There is a single solution to this incident and it involves the use of the treasury to peg 1:1 every asset on the bridge, this should be executed over a extended period of time and considering the corresponding liquidity of each asset so each asset that gets functional under the bridge is backed 1:1, doing so will slowly recover the utility of the chain as of now it is a total mess without DeFi. Recovering the peg of each bridged asset could actually help increase ONE locked on protocols reducing its availability and actually increasing its price to recover by buying additional unpeged assets.

Unfortunately that is the problem. The assets were stolen from the ETH side of the bridge, which made the Harmony assets lose their value “depeg”. Your 1USDC is now only worth a fraction of its value until the on-chain value can be re-established.

The protocol always was safe to use, but all eth/bsc assets are depegged so that wont change in the near future. :frowning:

One of the consequences of the exploit is AAVE locking all users funds. If you want to know about this and curious whether Harmony Team is aware or not, please upvote my topic:

It is bizarre to me that 4 private keys controls entire ETH bridge. Blockchain all about Decentralization.
How come Trojan was able to compromised sensitive information on local computer.

Wondering if Harmony Devs using FSTP for data transferring.

1 Like

May the same method that’s been used for month over a job offer and a attached PDF.

About vulnerability one important thing is always to not make IPs public. I remember the Ddos attack back in January and with the medium article there was a Printscreen from grafana. But instead of using an alias name for the server the IP was visible, same happened several times on other social media.

Also I would not open up the BTC bridge till you fix the oracle and price feeds. There was a 15% difference, so you buy ONE on Binance, send to network buy 1BTC bridge out and send to Binance you was able to make magic internet money or arbitrage.

A suggestion for the Multi-sig, I saw there is a option to have up to 50 signer. Why not using some of the permanent and long term active validator and decentralise the keys and increase the required signer??

If you read the reimbursement proposal, it lists AAVE as one of the affected dapps.

1 Like

There are not enough funds to do this. They state this in the reimbursement proposal.

1 Like