Harmony's RPC Network Evolution -- How Harmony ecosystem expansion drove the evolution of RPC services into a scaleable elastic service

Definition: “RPC” ~ Remote Procedural Call … let’s get this out of the way. It’s origin goes way back to 1981. Blockchain data is difficult to access. Hence the RPC service layer is added to nearly all blockchains out there.

The Harmony blockchain is accessible by an RPC service via the URL api.harmony.one (commonly referred to as an “endpoint”). This endpoint is generously hosted for free by the Harmony Foundation to access data stored in the Harmony blockchain. A (gas) fee is charged in the native Harmony ONE token when users decide to add or make a change to Harmony’s blockchain data, otherwise, reading from the blockchain dataset, via the RPC, is free.


Highlights

The Harmony’s RPC services hosting the api.harmony.one endpoint has drastically evolved over the last 12 months, and will continue so for another quarter. Most of the Harmony community using MetaMask to connect to api.harmony.one felt the biggest pain points when there were really high demand for DeFi activity in the Harmony ecosystem at the peak in December 2021. Things have quieted down at the time of this writing, which gives us an opportunity to look back into the past year at what Harmony did to scale up, the issues we faced, the evolution that’s taking place and where it is now.

All in the name of servicing the vibrant blockchain ecosystem of users accessing the Harmony’s blockchain dataset.


When Things Were Quiet

To give the audience the sense of where the Harmony hosted the initial RPC services at the start, users were actually dialing into a single location in the US West Coast region since the start of Harmony’s Mainnet in June 2018 up to the end of August 2021. The latency for users across the ponds to reach Harmony’s blockchain was over 200 milliseconds (in my opinion, anything over 20 milliseconds is unacceptable)

Up until August 2021: One (1) data center hosting RPC services for api.harmony.one

To meet the needs of future customers, we decided to begin expanding into multiple geographic locations around the world, starting with US East Coast, then to Europe and then Asia. We wouldn’t have otherwise be able to provide a world-class service for our ecosystem builders’ eventual customers during DeFi summer.


Partnership Kickoffs

Courtesy of “In The Heights”

We sought out partners in Q3 2021 with infrastructure partners (who were also super busy!) and found those able and willing to work with us. We were excited to begin our first integration partnership with Pocket Networks, followed by a rapid fire series of partnerships with Chainstack, QuickNode and Ankr.

We couldn’t wait for the partnership integrations to complete, where some required a few months (others at half a year!) of backlogged engineering time before completion. Harmony’s ecosystem success depended on a reliable service, and the expansion began in September 2021 and completed in multiple phases. We struggled to keep up with inbound traffic that kept increasing in double digit percentages week-over-week!


Getting Close to Our Customers

Courtesy of “In The Heights”

We initially grew our service availabilities by using one cloud provider into 4 regions around the globe by then end of November 2021. Through geo latency-based routing technology, we were able to direct users to the closest data center. This improved their experience, and they wanted more!

By December 2021: Expansion into 4 geographic regions

Users kept complaining about brownout situations, especially during their peak hours in their region, or when there were events which kicked off unbeknownst to us in the DeFi ecosystem. We have the capability to detect a region’s outage and slowly drain traffic out to the next closest region. But we didn’t have the architecture to auto scale based on traffic. Each data region started failing almost on a daily basis to keep up with the traffic and started falling behind, until we dropped in more nodes behind a region’s Load Balancer, which could take up to a day.

Also, Load Balancers weren’t quick enough to detect each server’s brown out (falling behind by over 5 blocks) as our settings were tuned to only let a Harmony blockchain node attached to Load Balancers to be taken offline after 30-seconds of reporting a brownout. This is a classic scaling issue in blockchain RPC services.

:bulb: This is a classic scaling issue in blockchain RPC services.


High Availability with Multi Cloud

We then followed up in December 2021 to double the capacity by using another cloud provider at a more cost effective rate. We were able to reach a steady state in February 2021.

Two months later, beyond adding more servers, we then added an nginx reverse proxy server in front of each Harmony node to relieve its duties as a web service (e.g. queuing) but it only improved by its ability to service RPC traffic slightly. This also enables us to aggregate traffic analytics to gather data to help understand our traffic patterns. Instead of relying on time-series data, we now had the ability to look at every single request and report on the RPC traffic pattern with incredible fidelity.

That’s when we realized we had 600 Million RPC requests per day. Holy sh*t.


Over 600 Million requests per Day

Courtesy of “In The Heights”

At the peak of DeFi summer, we were able to estimate over 1 Billion requests per day, but we didn’t quite have the confidence to make that call due to the lack of a highly reliable aggregated set of datapoints.

With the launch of Pocket Networks in Q4 2021 followed by the expansion of Harmony’s RPC infrastructure, we kept this up at a steady state while cost of hosting increased by incrementally adding more servers to a point where it was unsustainable. When we configured the reverse proxy service to aggregate data into an ELK stack, we instantly saw it for reals. In April 2022, we confirmed and recorded servicing over 600 Million RPC calls per day.

Averaging 12.5M RPC calls every 30 mins, that’s 25M every hour, or 600M RPC calls per day

A month later, I realized that the 600M RPC calls per day weren’t the only traffic accessing Harmony blockchain data. While attending Pocket Network’s InfraCon, they reported that the Pocket Network’s validator community running Harmony nodes witnessed another 200M RPC per calls per day! That’s a total of at least 800M RPC calls per day access Harmony’s blockchain data. This means, in a blink of an eye, over 3,000 RPC calls would’ve flashed right by us.

This means, in a blink of an eye, over 3,000 RPC calls would’ve flashed right by us.


Fighting Spam Traffic

We witnessed cases of an entire data center region (a load balancer over a dozen of servers behind it) falling behind trying to sync the blockchain’s quick 2-second block times, while servicing RPC traffic at the same time. The architecture was archaic, as these were the same nodes. The nginx reverse proxy server enabled us to implement additional rate limiting to block out spam traffic.

There were so much spam traffic that even the Harmony blockchain protocol had to be upgraded (see v4.3.2 and v4.3.3) to mitigate unfriendly spam traffic and MEV-like attacks. We also increased gas fees twice, once to 30 gwei in January 2022 and another to 100 gwei in July 2022 (see GitHub changes).


Time to be Cost Effective … hence Elastic

Courtesy of “In The Heights”

We knew this isn’t sustainable and set off on a path to launch an initiative to research on what plausible options there are. Ankr’s Erigon implementation on Binance Smart Chain seemed promising, but Harmony’s architecture had too many changes under the hood to allow an easy switch.

The team finally set off on a path to build out Elastic RPC.

Elastic RPC is named after the inspiration of Amazon Web Services’ Elastic Cloud Compute (or EC2). This new architecture is a “write into one, read out from many” concept, similar to how a typical database horizontal scaling solution looks like. We were able to achieve building a highly efficient and indexed datastore able to service all types of EVM-compatible RPC requests.

Elastic RPC (future) to be scale up based on demand

The process behind building out Elastic RPC has its growing pains. While we were shrinking the amount of servers hosted behind load balancers across the world from over 100 nodes down to under 50, we were also relying on a new blockchain database, which is highly efficient (indexed), to primarily service the Harmony RPC service. We then split the traffic between Elastic RPC and the regular Harmony validator nodes, syncing blockchain data while servicing RPC traffic

We started off in April 2022 and launched the new service in June 2022, in one undisclosed geographic location. The services encountered edge cases where the indexing service initially being battle tested saw issues with running a healthy database, while servicing live traffic, causing further angst to Harmony ecosystem users. This was soon solved but not before we faced angry Harmonauts demanding for a more reliable RPC service.

As of the time of this writing, we are still in the midst of synchronizing a second undisclosed data center location to host Elastic RPC services. In time, we will be able to launch worldwide. We also plan to open source this architecture to enable any RPC provider to improve upon their RPC service offerings, and possibly even integrating this architecture into their RPC services, within any EVM-compatible chain.


Hang In There

We’ve been building and the quiet time from the crypto lull is helping us focus. Harmony’s RPC services will be more cost effective with Elastic RPC. In addition, initial measurements witnessed an improvement in response times and service reliability. We are also looking to transition out these services to both the decentralized Pocket Networks community, and centralized RPC service providers such as Infura, Chainstack, Ankr and QuickNode.

We’re All Gonna Make It … hashtag.


Splash photos and imagery courtesy of “In The Heights” and Google Maps

16 Likes

I can’t say how happy I am to see some quality marketing and communication from the team on a Sunday! Thank you for the great post, I hope this continues! Also, I hope that you enjoyed “In the Heights”, I’m a huge Lin fan myself!

8 Likes

Thank you Sir for sharing this :pray:

2 Likes

Wow, this is the amazing content that we want to see🙏

3 Likes

Brilliant Jack, you’re the MVP!

3 Likes

Education with images & memes is perfectly down my aisle. One Secure-resharding, cross-shard messaging later - and we could be a scalable, functional chain back in business before NEAR, next year.

The Competition

Fast-Stream Sync is simply a design improvement goal. Getting Zk-related products into action (Identity, authentication, rollups) should be the next goal as competition will be tough from permissioned privacy, modular & succinct blockchains. Polygon Avail, Mina, Celestia, FRA are in the field.

A Sea of Change

Some services will complement the chain, while others, will others will disrupt the ecosystem. I highly advise not to ignore the business development roadmap and revisit those ideas very quickly (w.r.t retaining & letting go of ecosystem partners as well).

Feedback : Agenda-based community roundtables could be a good start.

  • Getting close to Cosmos was a good decision. Cross-chain liquidity will always be a contentious topic when wrapped assets are considered.

  • Using open-source licenses for features could offer robustness, but technical drift is a bigger possibility as well - and without bizdev growth to complement it. There is no enough code refactoring in the world to compete with novel solutions.

  • Big enough war chest helps - but dilution would make this arduous, but there’s no separate R&D fund in the tokenomics either that could be offered as a token-swap or direct financial capital provided to “more” research institutions & labs. Next option is strategic hires & active head-hunting.

  • The marginal utility & the narrative is shifting even in VC circles on what’s promising new tech (or even the next ROI gainer).

The unique value proposition to inject capital, enable R&D expansion, strategic partnerships & responsible scaling - requires a better narrative thesis to be presented & codified into operation.

Any active business development plan needs to be looked from the ground-up, with appropriate key operational goals, staff hiring & with data senstivity to active trends. Initiatives were not well-run previously, or people were spread too thin.

Please feel free to look around & ask for opinions.
Still, I believe that’s this an extremely positive step of clear & concise communication - that people know about the decent things.

Handling peak RPC traffic, sophisticated arbitrage bots*, smart contracts with block gas limit upto 90M.

* - DFK traffic isn’t arbitrage bots.I hope
some of those gas change increases are temporary in nature.

To close on a funny note, the community does not want to pilot the chain, just requires the team be receptive to critical feedback with urgency. Effective transparency, not radical.

WhatsApp Image 2022-08-08 at 7.04.49 AM

3 Likes

wen we reduce gas fee ?

Thank you so much and please keep building, Harmony core team :pray: :sparkling_heart:

2 Likes

Excellent content, thanks for sharing!

4 Likes

GREAT POST! We crave more of this!

2 Likes

I love this kind of content. Sharing the process instead of just the results is key; it makes the community/investors feel like they are also participating in this, as opposed to only being spectators of Harmony’s wins and failures.

1 Like

Beautifully structured, thanks a lot for sharing this!

Please create public RPC that can query more than 10K logs like Fantom/FTM, Polygon. So we as a developer can focus to build useful app for community. It cost more than $250 -$500/month using digital ocean if we setup our own node. Chainstack/ANKR/POKT they cannot query more than 10K logs. :unamused: