Harmony testnets (Devnet + Testnet)

Name of Project

Harmony testnets
This proposal is to formalize the grant for running 2 testnets for Harmony Foundation.

Application type

Grant

Proposal overview

Scope

  1. Devnet setup and operation (1 shard with only 1 or 2 internal node, 1 dns node, 1 explorer, 1 endpoint, faucet, explorer dashboard and documentation for the dev to refers to)

  2. Testnet operation (assuming this is the same as now) (4 shards, 8 dns node, 4 explorer, 8 external; validators, 1 explorer dashboard, faucet)

  3. Maintenance of both testnets, upgrades, monitoring

Miscellaneous

Any additional requests or work outside of this agreement is billed separately or the agreement is amended to include the extra requests.
Example: new features testing

Team

https://pops.one

Time

This agreement is valid for 1 year with possibility of extension if there are no changes to the scope.
Start: 01-01-2022
End: 31-12-2022

Proposal ask

(15K monthly)
180.000$ in ONE tokens for 1 year, in equal monthly installment.

Justification

  • Server costs (higher end so the testnets are stable)
  • Periodic upgrades and monitoring
  • Maintenance and troubleshooting if testnet goes down
  • Dev and testnet user support

Metrics for success

Testnet + Devnet Network reliability and uptime

External links

https://pops.one
https://faucet.pops.one/
https://explorer.pops.one/

8 Likes

Hello everyone, just wanted to add a few words to the proposal.

We have been with Harmony since the first testnet and the mainnet 0 (FN nodes). Since then we have tested and supported Harmony network for a few years and gained a lot of experience running nodes and understanding how it all works and help debug. We have also ran an iteration of Harmony testnet for a long time already so we do have the capabilities and experience to run these two testnets and supprt them with timely responses, monitoring, devops and debugging in case there are any issues. We also plan to make them even more stable by supplying more resilient servers to make the experience for devs, validators and testnet users much smoother.

5 Likes

You guys are the OG in Harmony! Glad to see you’ve submitted a proposal to Harmony’s $300M Ecosystem Fund. Let’s chat about this soon.

We are in the process of reviewing your proposal. We would also love :blue_heart: to have the Harmony community participate to ask questions and provide feedbacks.

3 Likes

It sounds like you are one of the best to do it so thanks for the proposal. :rocket:

I am new to Harmony One, so pardon if question is too basic.
What are the specific functions of each of the testnets? What would you use the second one for specifically? Why do you think its important to have two of them running.

Hey, thanks for the questions!

Basically we have discussed with the team about these testnets a few months ago already. Having 2 was basically their choice and we also think its reasonable.

So why 2? The team will use one to push in the latest code which can mean a lot of bugs, downtime and instability on it, so it wouldnt be suitable for developers to test their builds on it, or testnet validators to learn how to manage their nodes. This is where the second testnet comes in which will be much more stable with the same code as mainnet, or maybe just slightly newer. This is to ensure everyone who needs a stable chain to test out stuff can do so without the hassle of instability and bugs :slight_smile:

5 Likes

Thank you, that totally makes sense!

Great proposal. Looking forward to a solid testnet!

1 Like

Payments to be sent to faucet.pops.one on testnet? :grin:

3 Likes

This sounds great!
Further decentralizing operations and expanding them sounds like a great idea!

Thank you for leading the initiative!

1 Like

Upon reviewing the current Testnet setup, adding the scope of a Devnet may not be the best use of our resources at this time. Doubling down on Testnet would be a higher priority:

  1. Validator critical mass – there’s not enough Validators today running enough BLS keys to replicate the Mainnet behavior. It will be more ideal to see more validators, say 40-50 (currently at 10), running more keys on every shard, say 20-25 each (currently between 3 to 6 per shard).
  2. Node Uptime – 3 out of 11 validators have uptimes of less than 95%. Besides increasing the amount of validators, having a higher overall uptime amongst all validators will be desirable, with 95% uptime as the threshold (36 hours outage, or missed blocks, per month).
  3. Faucet uptime – a highly available faucet will facilitate an unstoppable development cycle with 99.9% availability (equiv. to 43 mins outage per month). Harmony is growing fast, and funding resources to mitigate faucet outages make sense. We witnessed several faucet outages that lasts for hours, if not longer, in the last couple of months.

As for the node types:

Can we elaborate on what hardware is behind these shards, and where they’re located? We’d like to propose having the Testnets be placed in at least two regions around the globe, 8 to 16 timezones away (pruned and DNS nodes).

Hey Jack,

let me respond :slight_smile:

With regards to having two testnets, its to ensure one is stable for developers to test (and validators if they want) and the other one to test new upgrades and for validators. Current testnet is still a bit unstable due to the upgrades that happen, and this is also why validators will have a hard time staying at high uptimes. This is also the reason why there was a discussion for 2 testnets the last time we spoke with @leo , but ofc this can be changed if you want just one. The question is do you want it a stable one then, or not (meaning way less upgrades before theyre internally tested by the team).

With regards to validator metrics, it is a testnet so most dont have an incentive to run it there and go straght to mainnet. We have frequently suggested an academy where validators that would perform well on testnet, would get Harmony Foundation delegation. And this is how we can keep them on testnet, otherwise they will just shut them down at some point. So basically in order to get delegation on mainnet, they have to run and maintain a testnet validator also. I am not sure if this is feasible since i am unsure if the foundation can spare that many tokens (eg like Solana does it).

If you want we can then discuss about changing the grant, but it needs to have realistic goals based on what you want from the testnet validators, as like i said, they dont have any incentive right now to run one, and it does cost them money.

Faucet is mostly online but we decided not to allow large funding so it does not drain fast - so or validator, we fund and help them separately of that.

For current testnet we run double nodes, located in Germany and in USA. Machines are 6-18 cores, 16-64GB RAM and disk space as needed, all based on function (explorer nodes, leaders, etc).