Definition: “RPC” ~ Remote Procedural Call … let’s get this out of the way. It’s origin goes way back to 1981. Blockchain data is difficult to access. Hence the RPC service layer is added to nearly all blockchains out there.
The Harmony blockchain is accessible by an RPC service via the URL api.harmony.one (commonly referred to as an “endpoint”). This endpoint is generously hosted for free by the Harmony Foundation to access data stored in the Harmony blockchain. A (gas) fee is charged in the native Harmony ONE token when users decide to add or make a change to Harmony’s blockchain data, otherwise, reading from the blockchain dataset, via the RPC, is free.
Highlights
The Harmony’s RPC services hosting the api.harmony.one endpoint has drastically evolved over the last 12 months, and will continue so for another quarter. Most of the Harmony community using MetaMask to connect to api.harmony.one felt the biggest pain points when there were really high demand for DeFi activity in the Harmony ecosystem at the peak in December 2021. Things have quieted down at the time of this writing, which gives us an opportunity to look back into the past year at what Harmony did to scale up, the issues we faced, the evolution that’s taking place and where it is now.
All in the name of servicing the vibrant blockchain ecosystem of users accessing the Harmony’s blockchain dataset.
When Things Were Quiet
To give the audience the sense of where the Harmony hosted the initial RPC services at the start, users were actually dialing into a single location in the US West Coast region since the start of Harmony’s Mainnet in June 2018 up to the end of August 2021. The latency for users across the ponds to reach Harmony’s blockchain was over 200 milliseconds (in my opinion, anything over 20 milliseconds is unacceptable)
Up until August 2021: One (1) data center hosting RPC services for api.harmony.one
To meet the needs of future customers, we decided to begin expanding into multiple geographic locations around the world, starting with US East Coast, then to Europe and then Asia. We wouldn’t have otherwise be able to provide a world-class service for our ecosystem builders’ eventual customers during DeFi summer.
Partnership Kickoffs
Courtesy of “In The Heights”
We sought out partners in Q3 2021 with infrastructure partners (who were also super busy!) and found those able and willing to work with us. We were excited to begin our first integration partnership with Pocket Networks, followed by a rapid fire series of partnerships with Chainstack, QuickNode and Ankr.
We couldn’t wait for the partnership integrations to complete, where some required a few months (others at half a year!) of backlogged engineering time before completion. Harmony’s ecosystem success depended on a reliable service, and the expansion began in September 2021 and completed in multiple phases. We struggled to keep up with inbound traffic that kept increasing in double digit percentages week-over-week!
Getting Close to Our Customers
Courtesy of “In The Heights”
We initially grew our service availabilities by using one cloud provider into 4 regions around the globe by then end of November 2021. Through geo latency-based routing technology, we were able to direct users to the closest data center. This improved their experience, and they wanted more!
By December 2021: Expansion into 4 geographic regions
Users kept complaining about brownout situations, especially during their peak hours in their region, or when there were events which kicked off unbeknownst to us in the DeFi ecosystem. We have the capability to detect a region’s outage and slowly drain traffic out to the next closest region. But we didn’t have the architecture to auto scale based on traffic. Each data region started failing almost on a daily basis to keep up with the traffic and started falling behind, until we dropped in more nodes behind a region’s Load Balancer, which could take up to a day.
Also, Load Balancers weren’t quick enough to detect each server’s brown out (falling behind by over 5 blocks) as our settings were tuned to only let a Harmony blockchain node attached to Load Balancers to be taken offline after 30-seconds of reporting a brownout. This is a classic scaling issue in blockchain RPC services.
This is a classic scaling issue in blockchain RPC services.
High Availability with Multi Cloud
We then followed up in December 2021 to double the capacity by using another cloud provider at a more cost effective rate. We were able to reach a steady state in February 2021.
Two months later, beyond adding more servers, we then added an nginx reverse proxy server in front of each Harmony node to relieve its duties as a web service (e.g. queuing) but it only improved by its ability to service RPC traffic slightly. This also enables us to aggregate traffic analytics to gather data to help understand our traffic patterns. Instead of relying on time-series data, we now had the ability to look at every single request and report on the RPC traffic pattern with incredible fidelity.
That’s when we realized we had 600 Million RPC requests per day. Holy sh*t.
Over 600 Million requests per Day
Courtesy of “In The Heights”
At the peak of DeFi summer, we were able to estimate over 1 Billion requests per day, but we didn’t quite have the confidence to make that call due to the lack of a highly reliable aggregated set of datapoints.
With the launch of Pocket Networks in Q4 2021 followed by the expansion of Harmony’s RPC infrastructure, we kept this up at a steady state while cost of hosting increased by incrementally adding more servers to a point where it was unsustainable. When we configured the reverse proxy service to aggregate data into an ELK stack, we instantly saw it for reals. In April 2022, we confirmed and recorded servicing over 600 Million RPC calls per day.
Averaging 12.5M RPC calls every 30 mins, that’s 25M every hour, or 600M RPC calls per day
A month later, I realized that the 600M RPC calls per day weren’t the only traffic accessing Harmony blockchain data. While attending Pocket Network’s InfraCon, they reported that the Pocket Network’s validator community running Harmony nodes witnessed another 200M RPC per calls per day! That’s a total of at least 800M RPC calls per day access Harmony’s blockchain data. This means, in a blink of an eye, over 3,000 RPC calls would’ve flashed right by us.
This means, in a blink of an eye, over 3,000 RPC calls would’ve flashed right by us.
Fighting Spam Traffic
We witnessed cases of an entire data center region (a load balancer over a dozen of servers behind it) falling behind trying to sync the blockchain’s quick 2-second block times, while servicing RPC traffic at the same time. The architecture was archaic, as these were the same nodes. The nginx reverse proxy server enabled us to implement additional rate limiting to block out spam traffic.
There were so much spam traffic that even the Harmony blockchain protocol had to be upgraded (see v4.3.2 and v4.3.3) to mitigate unfriendly spam traffic and MEV-like attacks. We also increased gas fees twice, once to 30 gwei in January 2022 and another to 100 gwei in July 2022 (see GitHub changes).
Time to be Cost Effective … hence Elastic
Courtesy of “In The Heights”
We knew this isn’t sustainable and set off on a path to launch an initiative to research on what plausible options there are. Ankr’s Erigon implementation on Binance Smart Chain seemed promising, but Harmony’s architecture had too many changes under the hood to allow an easy switch.
The team finally set off on a path to build out Elastic RPC.
Elastic RPC is named after the inspiration of Amazon Web Services’ Elastic Cloud Compute (or EC2). This new architecture is a “write into one, read out from many” concept, similar to how a typical database horizontal scaling solution looks like. We were able to achieve building a highly efficient and indexed datastore able to service all types of EVM-compatible RPC requests.
Elastic RPC (future) to be scale up based on demand
The process behind building out Elastic RPC has its growing pains. While we were shrinking the amount of servers hosted behind load balancers across the world from over 100 nodes down to under 50, we were also relying on a new blockchain database, which is highly efficient (indexed), to primarily service the Harmony RPC service. We then split the traffic between Elastic RPC and the regular Harmony validator nodes, syncing blockchain data while servicing RPC traffic
We started off in April 2022 and launched the new service in June 2022, in one undisclosed geographic location. The services encountered edge cases where the indexing service initially being battle tested saw issues with running a healthy database, while servicing live traffic, causing further angst to Harmony ecosystem users. This was soon solved but not before we faced angry Harmonauts demanding for a more reliable RPC service.
As of the time of this writing, we are still in the midst of synchronizing a second undisclosed data center location to host Elastic RPC services. In time, we will be able to launch worldwide. We also plan to open source this architecture to enable any RPC provider to improve upon their RPC service offerings, and possibly even integrating this architecture into their RPC services, within any EVM-compatible chain.
Hang In There
We’ve been building and the quiet time from the crypto lull is helping us focus. Harmony’s RPC services will be more cost effective with Elastic RPC. In addition, initial measurements witnessed an improvement in response times and service reliability. We are also looking to transition out these services to both the decentralized Pocket Networks community, and centralized RPC service providers such as Infura, Chainstack, Ankr and QuickNode.
We’re All Gonna Make It … hashtag.
Splash photos and imagery courtesy of “In The Heights” and Google Maps
