Date: Monday, January 10th 2022
This is a continuation of a previous thread.
- See link below for previous updates.
https://www.reddit.com/r/harmony_one/comments/ruspz6/state_of_the_network_jan_2nd_2022/
Note: This thread will be updated on a near daily basis as work continues to improve performance for our community and ecosystem partners. See replies below for updates.
We feel it’s important to keep the community informed with communication and transparency, while accepting constructive feedback and listening to your concerns. The on-going RPC issues have our attention, and we will not be treating this lightly. Improvements to the infrastructure are underway and with more to come. Some of the upcoming work includes:
a. Arbitrage Bot Transactions & Gas Price
(Status: Planning)
Nodes are receiving an urgent update to increase recommended gas price from 10 gwei to 30 gwei and minimum gas price from 1 gwei to 30 gwei. The fee increase will combat the arbitrage bots spamming the network with transactions.
When: We will soon begin focusing and planning on increasing the minimum gas. This has been low priority while stabilizing RPC functionality was prioritized above all else.
b. RPC Change #2 | Rate Limiting
(Status: Monitoring)
We understand the community has been experiencing an issue where an error was displayed:
“wait(n=1) would exceed context deadline”
To resolve this, rate limits will be relaxed so that these RPC calls are not throttled to the point of retry exhaustion. This will have been completed across all RPC nodes as of 11PM PST on Jan 10th.
c. RPC Change #3 | Sync Status Checks
(Status: Monitoring)
Our load balancers query RPC nodes to determine their health. When a node is out-of-sync, and to avoid impact, the load balancers are expected to stop sending queries in their direction. However, we’re finding queries are still sent to out-of-sync nodes despite their failed health check.
Our latest fix seems to be working well in this area. The sync status appears to be returning true data from the nodes, most importantly from those that fall behind in block count. This means our load balancers are able to proactively avoid problem RPC nodes. All nodes have this fix applied.
d. Pocket Network | Routing to an External RPC Provider
(Status: In Progress)
We have ramped up our load distribution to Pocket Network from 1% to 2%. This will be a gradual increase to 5% on Thursday as we monitor performance and user experience. If all goes well, we will increase the amount of traffic direct to Pocket significantly, monitor, and repeat.
Expected Timeline:
Tuesday [Jan 11]
Get the binaries updated throughout the network, ensuring all fixes we’ve applied this past week is replicated to Pocket Network RPC nodes.
Wednesday [Jan 12]
Raise redirect of traffic to 3% early in the day.
Thursday [Jan 13]
Assess and then shift towards 5% for the weekend.
Monday [Jan 17]
Assess community feedback, plan next steps.
e. Increase of RPC Node Footprint
We have increased our RPC node footprint by adding 20 servers to the infrastructure over the past week. This increase in server count will help handle the increase in traffic we are observing on the network.
Harmony Foundation 
+++++++++++++++++++++++++++++++++++++++++++
Potential Workaround
If you’re impacted, please follow the steps documented below and configure your Metamask to use the Pocket network. The team at Pocket has made great advancements over the past few weeks with Harmony RPC nodes and appears to be handling traffic very well.
- Click on the Networks drop-down menu, then press Custom RPC
- Under the Network Name field, type Harmony Mainnet 0 POKT Portal
- Within the New RPC URL field, copy and paste this endpoint URL https://harmony-0-rpc.gateway.pokt.network
- Put the decimal 1666600000 in the ChainID field
- Write ONE as the Symbol
- Add https://explorer.harmony.one as the Block Explorer URL
- Don’t forget to save
+++++++++++++++++++++++++++++++++++++++++++
