State of The Network | Jan 2022 | Thread-2

Date: Monday, January 10th 2022



This is a continuation of a previous thread.

  • See link below for previous updates.


Note: This thread will be updated on a near daily basis as work continues to improve performance for our community and ecosystem partners. See replies below for updates.

We feel it’s important to keep the community informed with communication and transparency, while accepting constructive feedback and listening to your concerns. The on-going RPC issues have our attention, and we will not be treating this lightly. Improvements to the infrastructure are underway and with more to come. Some of the upcoming work includes:

a. Arbitrage Bot Transactions & Gas Price

(Status: Planning)

Nodes are receiving an urgent update to increase recommended gas price from 10 gwei to 30 gwei and minimum gas price from 1 gwei to 30 gwei. The fee increase will combat the arbitrage bots spamming the network with transactions.

When: We will soon begin focusing and planning on increasing the minimum gas. This has been low priority while stabilizing RPC functionality was prioritized above all else.

b. RPC Change #2 | Rate Limiting

(Status: Monitoring)

Applied fix: add rate limit on PendingTransactions, Call, and GetDelegationsByValidator rpc by rlan35 · Pull Request #3971 · harmony-one/harmony · GitHub

We understand the community has been experiencing an issue where an error was displayed:

“wait(n=1) would exceed context deadline”

To resolve this, rate limits will be relaxed so that these RPC calls are not throttled to the point of retry exhaustion. This will have been completed across all RPC nodes as of 11PM PST on Jan 10th.

c. RPC Change #3 | Sync Status Checks

(Status: Monitoring)

Our load balancers query RPC nodes to determine their health. When a node is out-of-sync, and to avoid impact, the load balancers are expected to stop sending queries in their direction. However, we’re finding queries are still sent to out-of-sync nodes despite their failed health check.

Our latest fix seems to be working well in this area. The sync status appears to be returning true data from the nodes, most importantly from those that fall behind in block count. This means our load balancers are able to proactively avoid problem RPC nodes. All nodes have this fix applied.

d. Pocket Network | Routing to an External RPC Provider

(Status: In Progress)

We have ramped up our load distribution to Pocket Network from 1% to 2%. This will be a gradual increase to 5% on Thursday as we monitor performance and user experience. If all goes well, we will increase the amount of traffic direct to Pocket significantly, monitor, and repeat.

Expected Timeline:

Tuesday [Jan 11]
Get the binaries updated throughout the network, ensuring all fixes we’ve applied this past week is replicated to Pocket Network RPC nodes.

Wednesday [Jan 12]
Raise redirect of traffic to 3% early in the day.

Thursday [Jan 13]
Assess and then shift towards 5% for the weekend.

Monday [Jan 17]
Assess community feedback, plan next steps.

e. Increase of RPC Node Footprint

We have increased our RPC node footprint by adding 20 servers to the infrastructure over the past week. This increase in server count will help handle the increase in traffic we are observing on the network.

Harmony Foundation :blue_heart:

+++++++++++++++++++++++++++++++++++++++++++

Potential Workaround

If you’re impacted, please follow the steps documented below and configure your Metamask to use the Pocket network. The team at Pocket has made great advancements over the past few weeks with Harmony RPC nodes and appears to be handling traffic very well.

  • Click on the Networks drop-down menu, then press Custom RPC
  • Under the Network Name field, type Harmony Mainnet 0 POKT Portal
  • Within the New RPC URL field, copy and paste this endpoint URL https://harmony-0-rpc.gateway.pokt.network
  • Put the decimal 1666600000 in the ChainID field
  • Write ONE as the Symbol
  • Add https://explorer.harmony.one as the Block Explorer URL
  • Don’t forget to save

+++++++++++++++++++++++++++++++++++++++++++

1 Like

Sync-status and sync speed improvement related PRs:

#3976 - additional fix for node-sync status
#3973 - release one rlock and change default value at getMaxPeerHeight
#3971 - Rate-limits on multiples RPC : PendingTransactions, Call, and GetDelegationsByValidator
#3975 - Added metrics for dns server for sync service
#3974 - extend the expiration for sync status cache for non-validator nodes
#3970 - remove two unnecessary use of global mutex in StateSync
#3927 #3968 - Rate-limits on multiples RPC : StakingNetworkInfo, SuperCommittees, CurrentUtilityMetrics
#3918 - refactor and make sync status check interval smaller

Other updates:

#3947 - replace go-ethereum/ethereum/rpc with harmony-one/harmony/eth/rpc
#3939 - Add revert mechanism for UpdateValidatorWrapper - this will be used in future PR to enable staking transaction with metamask
#3949 - fix DecodeSigBitmap
#3958 - Do not process txns if node is out of sync to help with a faster sync
#3953 - fix block decode bug resulting to node panicking, issue #3955
#3951 - new RPC : GetHeaderByNumberRLPHex and GetProof

Thank you for the list, @sophoah!

Date: Tuesday, January 11th 2022

a. Arbitrage Bot Transactions & Gas Price

(Status: Scheduled)

UPDATE: We are aiming to increase minimum gas fees from 1 gwei to 30 gwei at 8PM PST on Thursday, January 13th / 4AM UTC of Friday, January 14th. Proactive notifications will be distributed to as many ecosystem partners as possible as we come closer to the scheduled date and time.

b. RPC Change #2 | Rate Limiting

(Status: Fine Tuning)

Applied fix : add rate limit on PendingTransactions, Call, and GetDelegationsByValidator rpc by rlan35 · Pull Request #3971 · harmony-one/harmony · GitHub

We are still observing excessively high amounts of calls on doEVMcall.

UPDATE: Rate limits will be tightened on this RPC as we continue to fine tune this configuration and optimize performance of our RPC nodes.

c. RPC Change #3 | Sync Status Checks

(Status: Success/Closed)

UPDATE: We have monitored the performance of sync status checks for a number of days without observing issues. This is now considered closed but we will continue to keep an eye on the sync status check.

d. Pocket Network | Routing to an External RPC Provider

(Status: In Progress)

UPDATE: We are sending 2% of all RPC traffic to Pocket while observing performance. This will be increased to 3% tomorrow, Wednesday, January 12th. See timeline below.

Expected Timeline:

Tuesday [Jan 11]
Get the binaries updated throughout the network, ensuring all fixes we’ve applied this past week is replicated to Pocket Network RPC nodes.

Wednesday [Jan 12]
Raise redirect of traffic to 3% early in the day.

Thursday [Jan 13]
Assess and then shift towards 5% for the weekend.

Monday [Jan 17]
Assess community feedback, plan next steps.

e. Increase of RPC Node Footprint

(Status: Success/Closed)

Update: We have increased our RPC node footprint by adding 20 servers to the infrastructure over the past week. This increase in server count will help handle the increase in traffic we are observing on the network.

f. Multi-Sig Wallets:

(Status: Investigating)

Update: The team is investigating issues impacting our multi-sig wallets and hope to have the problem resolved soon. More to follow on this as progress is made.

g. Bridging Issues:

(Status: Applied Fix / Monitoring)

UPDATE: We have identified an issue impacting the bridge and have taken corrective action by dedicating two archival nodes exclusively to bridging operations.

3 Likes

Jan 12th - Transaction Issues

Update: The network suffered a block propagation issue today due to heavy spam traffic. We’ll have a post-mortem announcement to share with the community once the issue is resolved.

The solution is applied but will need a couple more hours to fully recover.

Next Steps: Being this was caused by heavy spam traffic, our upcoming PR to increase minimum gas should prevent this moving forward and greatly improve overall performance. The gas increase will not completely remove the spam traffic, but will reduce it significantly which will be more than sufficient for an increase in overall performance.

PR for minimum gas increase to 30 gwei:

2 Likes

Date: Thursday, January 13th 2022

a. Arbitrage Bot Transactions & Gas Price

(Status: Scheduled | Today)

Please note that we are proceeding with the minimum gas increase from 1 gwei to 30 gwei today at approximately 8PM PST / 4AM UTC. See the pull request below for information on the change.

» Link: bump min gas price to 30 gwei by rlan35 · Pull Request #3960 · harmony-one/harmony · GitHub

This has been merged to our testnet a few hours ago.

To our ecosystem partners, I have reached out directly to some of you in chat. Feel free to run some tests against our testnet to confirm the minimum gas increase will not have an adverse impact on your services. If your app is pulling the minimum gas from Harmony, then you should have no issues and no changes should be needed.

TLDR: Min gas will increase today at approx. 8PM PST / 4AM UTC which will reduce the number of arb bot transactions on our network.

Our network has experienced a set of issues which required an all-hands-on-deck approach to troubleshooting and issue resolution. Today, the primary components the infrastructure are working well with some residual hiccups due to recent changes, all of which we intend to continue optimizing over the coming days.

From this, we will learn and become stronger as a chain, community, and ecosystem.

We are working on a full postmortem to share with the community and our ecosystem partners and hope to have that ready in the next 48 hours. Thank you to everyone as we worked diligently to restore service.

3 Likes

Thank you, the other Harmony team and validators :pray:

2 Likes

Looking forward to that read.

1 Like

Any update about the gas price plzz :roll_eyes:

1 Like