HIP-24: Harmony testnets (Devnet + Testnet)

Funding Proposals Grants
1

Name of Project

Harmony testnets
This proposal is to formalize the grant for running 2 testnets for Harmony Foundation.

Application type

Grant

Proposal overview

Scope

  1. Devnet setup and operation (1 shard with only 1 or 2 internal node, 1 dns node, 1 explorer, 1 endpoint, faucet, explorer dashboard and documentation for the dev to refers to)

  2. Testnet operation (assuming this is the same as now) (4 shards, 8 dns node, 4 explorer, 8 external; validators, 1 explorer dashboard, faucet)

  3. Maintenance of both testnets, upgrades, monitoring

Miscellaneous

Any additional requests or work outside of this agreement is billed separately or the agreement is amended to include the extra requests.
Example: new features testing

Team

Time

This agreement is valid for 1 year with possibility of extension if there are no changes to the scope.
Start: 01-01-2022
End: 31-12-2022

Proposal ask

(15K monthly)
180.000$ in ONE tokens for 1 year, in equal monthly installment.

Justification

  • Server costs (higher end so the testnets are stable)
  • Periodic upgrades and monitoring
  • Maintenance and troubleshooting if testnet goes down
  • Dev and testnet user support

Metrics for success

Testnet + Devnet Network reliability and uptime

External links

13 Likes
2

Hello everyone, just wanted to add a few words to the proposal.

We have been with Harmony since the first testnet and the mainnet 0 (FN nodes). Since then we have tested and supported Harmony network for a few years and gained a lot of experience running nodes and understanding how it all works and help debug. We have also ran an iteration of Harmony testnet for a long time already so we do have the capabilities and experience to run these two testnets and supprt them with timely responses, monitoring, devops and debugging in case there are any issues. We also plan to make them even more stable by supplying more resilient servers to make the experience for devs, validators and testnet users much smoother.

7 Likes
3

You guys are the OG in Harmony! Glad to see you’ve submitted a proposal to Harmony’s $300M Ecosystem Fund. Let’s chat about this soon.

We are in the process of reviewing your proposal. We would also love :blue_heart: to have the Harmony community participate to ask questions and provide feedbacks.

6 Likes
4

It sounds like you are one of the best to do it so thanks for the proposal. :rocket:

I am new to Harmony One, so pardon if question is too basic.
What are the specific functions of each of the testnets? What would you use the second one for specifically? Why do you think its important to have two of them running.

1 Like
5

Hey, thanks for the questions!

Basically we have discussed with the team about these testnets a few months ago already. Having 2 was basically their choice and we also think its reasonable.

So why 2? The team will use one to push in the latest code which can mean a lot of bugs, downtime and instability on it, so it wouldnt be suitable for developers to test their builds on it, or testnet validators to learn how to manage their nodes. This is where the second testnet comes in which will be much more stable with the same code as mainnet, or maybe just slightly newer. This is to ensure everyone who needs a stable chain to test out stuff can do so without the hassle of instability and bugs :slight_smile:

7 Likes
6

Thank you, that totally makes sense!

1 Like
7

Great proposal. Looking forward to a solid testnet!

3 Likes
8

Payments to be sent to faucet.pops.one on testnet? :grin:

4 Likes
9

This sounds great!
Further decentralizing operations and expanding them sounds like a great idea!

Thank you for leading the initiative!

2 Likes
10

Upon reviewing the current Testnet setup, adding the scope of a Devnet may not be the best use of our resources at this time. Doubling down on Testnet would be a higher priority:

  1. Validator critical mass – there’s not enough Validators today running enough BLS keys to replicate the Mainnet behavior. It will be more ideal to see more validators, say 40-50 (currently at 10), running more keys on every shard, say 20-25 each (currently between 3 to 6 per shard).
  2. Node Uptime – 3 out of 11 validators have uptimes of less than 95%. Besides increasing the amount of validators, having a higher overall uptime amongst all validators will be desirable, with 95% uptime as the threshold (36 hours outage, or missed blocks, per month).
  3. Faucet uptime – a highly available faucet will facilitate an unstoppable development cycle with 99.9% availability (equiv. to 43 mins outage per month). Harmony is growing fast, and funding resources to mitigate faucet outages make sense. We witnessed several faucet outages that lasts for hours, if not longer, in the last couple of months.

As for the node types:

Can we elaborate on what hardware is behind these shards, and where they’re located? We’d like to propose having the Testnets be placed in at least two regions around the globe, 8 to 16 timezones away (pruned and DNS nodes).

1 Like
11

Hey Jack,

let me respond :slight_smile:

With regards to having two testnets, its to ensure one is stable for developers to test (and validators if they want) and the other one to test new upgrades and for validators. Current testnet is still a bit unstable due to the upgrades that happen, and this is also why validators will have a hard time staying at high uptimes. This is also the reason why there was a discussion for 2 testnets the last time we spoke with @leo , but ofc this can be changed if you want just one. The question is do you want it a stable one then, or not (meaning way less upgrades before theyre internally tested by the team).

With regards to validator metrics, it is a testnet so most dont have an incentive to run it there and go straght to mainnet. We have frequently suggested an academy where validators that would perform well on testnet, would get Harmony Foundation delegation. And this is how we can keep them on testnet, otherwise they will just shut them down at some point. So basically in order to get delegation on mainnet, they have to run and maintain a testnet validator also. I am not sure if this is feasible since i am unsure if the foundation can spare that many tokens (eg like Solana does it).

If you want we can then discuss about changing the grant, but it needs to have realistic goals based on what you want from the testnet validators, as like i said, they dont have any incentive right now to run one, and it does cost them money.

Faucet is mostly online but we decided not to allow large funding so it does not drain fast - so or validator, we fund and help them separately of that.

For current testnet we run double nodes, located in Germany and in USA. Machines are 6-18 cores, 16-64GB RAM and disk space as needed, all based on function (explorer nodes, leaders, etc).

1 Like
12

To maintain a stable devnet for all the dApp developers is a mandatory requirement. Our current testnet is mainly used by developers for the nightly update to check the regression of the codebase. A stable devnet on par with the mainnet is critical for dapp development and testing. Many blockchains have more than one testnet anyway.

Agree with another point by @Jacksteroo that our current testnet setup may be expanded to have better network coverage. Noted we are doing major protocol upgrade work next year and we need both a bigger testnet and stable devnet. I would support this grant proposal.

Some metrics have to be defined though about the uptime, downtime SLA, accessibility, educational material, supporting channel about the devnet. I think those should be added to the scope of this proposal though. Also to expand the current testnet needs to be included as well.

Thanks,

1 Like
13

Hi @leo , thanks for replying!

To update the original proposal, lets discuss these things here first (or if we see its too slow we can move onto a DM group with us and you and @Jacksteroo ).

  1. TESTNET
  • this is the “unstable” network, supposed to be used for testing purposes for new codebase and nightly upgrades
  • testnet currently runs 4 shards, nodes are located in US and EU (Germany) already, is that satisfactory? (8 external nodes, 4 explorer nodes, 8 dns nodes)
  • do we have any historical metrics for our current testnet? Due to nightly upgrades I am not sure at what uptime we should aim here

DEVNET

  • 2 shards is enough? And stable codebase same or similar to mainnet (then added 4 external nodes, 2 explorer nodes and 4 dns nodes)
  • nodes in EU and USA
  • uptime of 99%, recovery within 4-8h in case of downtime

OTHER

  • we will support both and have almost 24/7 coverage on telegram and/or discord
  • network set ups, monitoring, upgrades, maintenance
  • we will use a lot of existent educational material and provide with new one too (also based on new changes like cross shard smart contracts, resharding etc)

When we finalize discussion i will update the proposal with the changes :slight_smile:

4 Likes
14

So we have discussed with the team via discord channel and here is the final take:

PANGAEA (stable testnet - the one currently active)

  • setup and operation
  • 4 shards, 8 dns node, 4 explorer, 8 external validators
  • Faucet, explorer
  • Nodes in EU and US
  • We add more BLS keys and external validators to take more seats (~360)
  • Config changes to resemble mainnet (block rewards, epoch times, etc)
  • 99% uptime and recovery within up to 4h from incident

PANNOTIA (unstable testnet - have to create it)

  • setup and operation
  • 2 shards, 4 external nodes, 2 explorer nodes, 4 dns nodes
  • Faucet, explorer
  • Nodes in EU and US
  • 90% uptime, recovery within 24h from incident
  • Nightly upgrades

The updated proposal ask is $240k for 1 year, in equal monthly installments ($20k/month) in ONE tokens. The higher costs are due to additional man power and 24/7 coverage!

6 Likes
15

I think it’s worth the cost. Fully support.

4 Likes
16

I support this and think it’s a great idea. The cost seems justified to me.

3 Likes
17

I fully support this proposal, mONEy well spent.

2 Likes
18

Looking towards the Validator DAO to seek approval from the community through a vote. Harmony Foundation will fund the initiative based on the voting results.

7 Likes
19

Has my backing. Room for growth is a great thing :100::facepunch::blue_heart:

3 Likes
20

Posting on behalf of the VDAO

Pops two testnet proposal Terms for Funding

Miscellaneous work

Additional work outside of this agreement must be requested and approved by the VDAO for funding. This includes requests from the VDAO, Harmony, and POPS team. Any work done without prior authorization WILL NOT BE PAID.

Terms for funding

$18k USD will be paid by the VDAO in $ONE tokens monthly. We will withhold 10% of the monthly payment and pay it back at the end of the 1 Year term. The final payment will be 42k (18k + 24k). The total funding amount will be 240k. If there is a request to the termination of this agreement via POPS or from the VDAO, the withhold funds of 10% will be paid out on the following day of the last date of service. This is including the final 2 months of service. (Termination of service is detailed in section “Request for Termination of this Agreement”)
(DATES:TBD)

Funding will be determined by the completion of the agreement per month.

Failure to uphold this agreement will result in a deduction of funding from the initial start of failure. Each day is equivalent to $666.66 in a month of 30 days and $645.16 in a month of 31 days. (30 days) $666.66/24 hrs = $27.77 per hour. (31 days) 645.16/24 hrs = $26.88 per hour. If the failure of upholding the agreement extends for the totality of the month, the result will be no payment for that month.

PANGAEA (stable testnet - the one currently active) was proposed a 99% uptime and recovery within up to 4hr from incident. VDAO will expect a 5 hour recovery. After the 5 hour mark we will begin deducting hourly per the initial point of failure per the rates noted above by the qualifying month.

PANNOTIA (unstable testnet - have to create it) was proposed a 90% uptime, & recovery within 24hr from incident. VDAO will expect a 25 hour recovery. After the 25 hour mark we will begin deducting hourly per the initial point of failure per the rates noted above by the qualifying month.

Exceptions for these terms can be reviewed by the VDAO following a report by POPs and will be put up to an internal vote in the VDAO to continue to approve the funding for those hours missed.

Request for Termination of this Agreement

At any given time any party can request a termination of this agreement with the following conditions.

For POPS:
A 2 month notice of termination of this agreement to allow the VDAO to take support of both testnets.

For VDAO:
A breach in uptime on either testnet from the original agreement. This also will be a 2 months notice take over both testnets. This condition does need to be approved by the validators and not the VDAO. The following qualify.
Stable testnet: Avg below 94% for the month
Unstable testnet: Avg below 84% for the month

These terms and conditions are to be debated before finalization.

3 Likes