Grant: Node Pilot (1-click deployment and validation)

Web3 Bounties
1

Node Pilot <> Harmony Integration

Node Pilot Harmony Proposal
Node Pilot Harmony Proposal1000Ă—1000 135 KB

Summary

Decentralized Authority would like to apply for a Harmony token grant to:

  1. Add Harmony Validation to Node Launcher, a new, open-source, TypeScript, Docker deployment engine, making Harmony nodes and validator nodes deployable from any application with a few lines of JS code.
  2. Support full and archival nodes on both mainnet and testnets.
  3. Integrate Harmony JavaScript SDK into Node Launcher, giving developers the ability to create/update Harmony nodes/validators and access the full Harmony API from a single JavaScript source.
  4. Add Harmony one-click validation to Node Pilot, a user-friendly, GUI-driven node deployment application built on the Node Launcher engine.
  5. Add Harmony validator importer so existing validators can utilize Node Pilot with simplicity.
  6. Maintain and update Harmony nodes with all integrations in accordance to latest stable releases.
  7. Operate Harmony Validators to provide continual mainnet testing and build a long-term partnership.

Background

Shane and Ryan Burgett started developing Node Pilot in August 2020 after being unable to find a comprehensive node deploy/management solution. Shane was a full-time business development contributor to Pocket Network since January 2019. Ryan is a full-stack crypto application developer with years of experience in the areas of wallets, interoperability platforms, mining, and node deployment.

Motivation/Rationale

To keep blockchain networks both secure and censorship resistant, crypto projects need to have a strong node community that is decentralized and running the most up-to-date core software. However the technical skills required to run crypto nodes severely limit who can participate, and the lack of node toolings leaves most networks running out-of-date software. Decentralized Authority is driving a new solution that addresses node deployment on both a technical and user-experience level to democratize node running while adhering to industry best practices for each blockchain.

This was first done by building a new node deployment engine called Node Launcher that is approachable to all devs. Node Launcher is a TypeScript library that enables any application to deploy nodes and validators via Docker with only a few lines of Javascript code. For the first time, developers can programmatically build full stack blockchain applications using only JavaScript. No more CLI, bash scripts, or wrestling with docker config files. Blockchains that are supported in Node Launcher have a powerful node/validator tool that provides automatic version updating to ensure seamless network upgrades.

With Node Launcher addressing the technical side of node deployments, Node Pilot was created to be free-software that enables users to deploy, manage, and monetize crypto nodes and validators with a few clicks. Seamless Docker integration through Node Launcher brings enterprise-level security and scalability while providing users a simple, self-hosted GUI interface. No third party OS-level dependencies and automatic TLS certifications give users the ability to access their nodes with privacy and security. Node Pilot currently supports 10+ blockchain nodes and is bringing 1-click validation capabilities to non-technical users.

There are some limited node management platforms out there like DappNode and AVADO, which are great for simple node deployment, but are locked to a dedicated OS or proprietary hardware and lack the features required for someone to be an effective validator operator on most networks. Validators that are not run properly or run outdated core software become a burden on their networks. Decentralized Authority is designing flexible tools from the ground up to be just as easy as these other platforms while providing important features that make for good validator operators.

When users don’t have a user-friendly, self-hosting solution for participating in their favorite blockchains, their only option is to turn to hosting services where they must trust their keys to 3rd parties. When blockchains don’t have a diverse set of nodes supporting the network, the ecosystems become vulnerable to centralization, security risks, and performance issues. Many of these services are great, but crypto should be an ecosystem with options, and a solution like Node Pilot gives everyone the freedom to be an independent validator operator by overcoming the technical barriers that exist today.

Objective(s)

  1. Give the Harmony community an approachable, fully self-hosted option for deploying/managing Harmony nodes/validations, while maintaining industry best practices and security.

  2. Ensure Harmony node runners can always stay up-to-date with 1-click updates.

  3. Open the door to multiple levels of monetization through seamless Pocket integration.

    • Harmony Validators can “double dip” in the native ONE rewards and Pocket rewards.

  4. Enable any applications to deploy and manage Harmony nodes with only JavaScript.

    • This would give the Harmony ecosystem an approachable developer tool to create node deployment applications.
    • Support all Harmony node types, including full and archival on both mainnet and testnets.

  5. Provide support and high quality resources to demystify participating in the Harmony node community.

  6. Provide the Harmony community with a node management platform that can scale to future Harmony features.

  7. Enable Decentralized Authority to actively join in the Validator DAO and conduct mainnet validator testing.

    • This will allow Node Pilot updates to be tested internally before being pushed to the larger community. It is crucial to ensure updates work on mainnet validators using software from Decentralized Authority.

  8. Reduce pressure on the Harmony Foundation to take on more in-house, node development projects.

    • Decentralized Authority would be a highly motivated partner, with alline incentives, to deliver, maintain, and grow this platform.

Deliverable(s)

  • Add harmony node suite to Node Launcher
    • Full nodes
    • Validator nodes
    • Mainnet and Testnet support
  • Integrate Harmony JavaScript SDK into Node Launcher
  • Add Harmony to Node Pilot v1
    • Full Harmony node suite
    • 1-click validation
    • Import existing validators
    • 1-click updating
    • CLI Command Entry support via GUI
    • Available for Node Pilot v1 launch
  • User resources and documentation for Node Launcher and Node Pilot
    • Written documentation
    • Step-by-step guides
    • Video tutorials for Harmony Validators
  • VMs/images that enable 1-click deployments (date TBD after v1 launch)
    • AWS
    • Digital Ocean
    • VMware

Budget

$30,000 grant to build initial integrations and documentation

  • Node Launcher
  • Harmony JavaScript SDK <> Node Launcher integration
  • Node Pilot
  • QA testing and support
  • User Resources

$21,500 to operate Harmony ONE Validators for multiple shards

  • Paid out at the competition of the integration for testing before the public launch
  • 18 month lock-up
  • Provides continual mainnet validation testing for both Node Launcher and Node Pilot
  • Establish a long-term, cross-ecosystem partnership for support and maintenance.

Goals Beyond This Proposal

As Harmony grows to more features and services, Node Launcher / Node Pilot is designed to scale in tandem. Future features could include:

  • Future service deployments
    • Example: Trustless bridge deployments or other services that may accompany Harmony DAOs.
  • Node Pilot <> Harmony DAO
  • Node PIlot Mobile application (iOS & Android)
  • Real-time notifications
  • Node Pilot Fleets (enterprise-ready API platform for multi-instance orchestrations)

Contributor(s)

Shane Burgett - Pocket Network bizdev contributor with a background in business development, marketing, design, and media production.

Ryan Burgett - Full-stack Web3 application developer with 4+ years of experience in the areas of wallets, blockchain interoperability platforms, mining, and node deployment.

8 Likes
2

Hello @shane this is a really great proposal. I would have a few questions.

  1. CLI Command Entry support via GUI how extensive is this ? would a technical validator allowed to issue any commands for troubleshooting ?
  2. with 1) I assume there will be no SSH-able access to the node directly (ie connecting to the docker instance directly?)
  3. the document never mentioned about how and where those docker are being built :
    3a) are all the docker hosts be disperse in multiple region ?
    3b) how would you allocated the resource (memory/disk/cpu)
    3c) the shard 0 disk is incrementing a lot, and one of the primary concern of the node runner would be the disk upgrade and management, how would that be handled ?
    3d) there will be for sure during high network load, increased of memory/cpu usage, how would that be handled ?
  4. How would you manage the shard 0 (and others 1 2 3) DB snapshot and restoration during a new node build for instance (both for pruned and archival node)?
  5. I don’t see any monitoring and alerting dashboard as part of the proposal which would be something that validator also struggling to build (ie grafana/prometheus setup …) and requires some level of technicality, is that something you could also add ? you can revised the proposed budget.

VDAO, feel free to ask any questions here.

3 Likes
3

1 & 2- Yes, you can send commands directly to your node via the GUI. No SSH is required. Any node calls should be able to be sent here. We will ensure that this works seamlessly with Harmony nodes.

image
image994Ă—521 28.1 KB

3A- We are 100% a self-hosting solution, so Node Pilot can be run on any Debian based machine or VM (though we recommend Ubuntu) anywhere in the world. Probably about half of Node Pilot users opt to run their own bare-metal servers, while many will install NP on VPS instances running Ubuntu Server. We do not provide our own hosting at this time, as our main goal is to first enable true self-hosting.

3B- When you setup a node, you get to determine the amount of resources that container is allowed: Deploy Chain Nodes - Node Pilot

3C- Node Pilot is 100% transplantable from one server to another. All of NP’s data is stored in a single folder off the home directory called /.node-pilot. That folder has your entire instance and can be moved to another server by just copying. If someone need to upgrade their storage, they can install an new drive, install Ubuntu, copy the /.node-pilot folder over, run the install script, and it will boot up perfectly and deploy your validator.

3D- By utilizing Docker, we are able to take advantage of Docker’s well-designed resource management. We always recommend that node runners provision their Validators for the most resources possible so that if it needs to scale it can. For v1 we will have the ability for user to even edit node resources via the GUI if they need to give specific nodes more resources.

4- We are actually building snapshot downloading abilities into Node Launcher. That development is already underway. Users can either add an outside source, or an internal source (another node they run) to bootstrap a new node or re-sync. We do plan to offer snapshot downloading for Decentralized Authority in the future, but for now it will work with other outside sources, including the one Harmony offers in the docs. For this integration, we would ensure that your snapshot work with NP.

5- Because NP is Docker driven, we are able to automate the deployment of tools like Grafana and Prometheus. While we currently offer extensive hardware monitoring through automatically deploying and configuring Netdata, we can look into node/network monitoring as well as part of this proposal. I will add an amendment to the proposal to spec out those features once I talk with our tech lead :+1:

Thanks for the questions @sophoah!! Drop any more you may have :slightly_smiling_face:

2 Likes
4

@sophoah regarding 5, it would be great to hear about what monitoring metrics are important for the validator community. What metric would be important for you?

I’m tempted to do a post in the Staking - Harmony Community Forum channel to get some feedback on what validator community would be interested in.

With that info, I would be interested in seeing how Node Pilot can address those needs.

3 Likes
5

Hope all is well @shane. Couple things when you find some time:

1-- Can you please update the proposal to reflect USD instead of ONE?

2-- I was curious if you could explain how Node Pilot will handle BLS key manipulation. Will key manipulation via GUI be supported or would users send commands to their node instead? Will the interface include information on what keys are in use by each node?

3-- I’ll reach out to the Validator DAO to ask if they can chime in as well!

Thanks Shane!

1 Like
6

Thanks for the questions @dpagan-harmony.

1– Done!

2– I may not be following what “BLS key manipulation” means, but if you point to me what a manipulation command is, I can give you a more specific answer as to if we can add it to the GUI. But users would be able to do commands via Command Entry (shown in a comment above) via the GUI without the need for SSH.

We would indeed add all the important node information directly in the GUI under “Validator Info” as shown here for a POKT validator. We are open to any and all feedback on what information should be displayed for Harmony validators.

image
image2199Ă—811 85.2 KB

3– Thank-you!! More feedback on what Validators need/want would be amazing :+1:

1 Like
7

@shane amazing proposal, will be good to see how the community as a whole flush out the details here.

Soph’s point on BLS keys are more relevant from a validator perspective since a single node can potentially host up to 10 BLS keys (or more), and when massive delegations occur during an epoch, BLS keys will need to be added on-chain, and reflected on the node (with a restart). The BLS key changes need to complete on-chain and on the node(s), before the next epoch starts.

For non-validating node runners, this shouldn’t be an issue.

See this article for more info

2 Likes
8

not me but @dpagan-harmony :slight_smile: from my point of view, this solution will be to manage the node (not the validator) but if there is why not.

1 Like
9

At first the basics one cpu/memory/disk so they are being alerted if any issue resource wise.

but I would really love to see what the VDAO have in mind for that.

1 Like
10

Thanks for the detail answer.

So now another question, node runner will surely ask. What is the budget like for them ? resource scaling usually means more $$$

could you elaborate on that.

1 Like
11

Ah, thanks for the clarification @Jacksteroo! Yes, it looks like all of this should be able to be handled in a GUI-driven fashion. Just like we tailored POKT’s user experience to account for their custom processes, we would do the same level of integration for Harmony’s BLS keys.

I do have a question regarding the quote above: Are the BLS key changes a manual process that node runners have to do periodically, or are the changes done in an automatic fashion?

12

Correct me if I’m wrong, but I believe you are referring to my comment about resource scaling :point_down:

NP doesn’t help with provisioning or resource scaling of instances on cloud services. Cloud resource scaling will have to be set up and provisioned before running NP on top of Ubuntu. NP can help with resource delegation within the Ubuntu instance. You can provision the amount of vCPUs and RAM that a single container (or node) is allowed to access. This way if say a one node starts getting too greedy, it will have to stay within the resources it was permitted and won’t be able to take resources away from another node. NP is basically giving a GUI to basic Docker resource provisioning.

In terms of how expensive would it be for node runners? Well, that is up to them. Harmony Validators aren’t resource-intensive, according to the recommended hardware, but if they are wanting to run a Harmony RPC node with POKT as well, then they will have to make sure they have the resources which will cost more but could produce more rewards. Those choices on cost will have to be made before installing NP.

Hope that answers your question :slightly_smiling_face:

1 Like
13

2 questions.

Does setup the node to use root as the working directory or does it setup a separate working directory and a sudo user?

For the validator wallet that needs to be created for each validator, will the software have access to the wallet private key or mnemonic and what is stored on your servers?

2 Likes
14

The presence of the BLS key on a node, vs. the on-chain BLS key addition are two separate but matching activities. On-chain BLS key additions (or removals) can be done via CLI, from any machine with hmy binary installed, and the validator key, using --add-bls-key or --remove-bls-key params as seen here

1 Like
15

Great questions :+1:

NP is meant to be run as a regular user (not root) on the host machine. The Node Pilot folder with all data is stored in the user’s home directory under ~/.node-pilot.

Node Pilot stores all the node’s standard config data in ~/.node-pilot/config (folder structure subject to change with v1). Since Node Pilot is deploying the standard Docker container for Harmony, the folder structure will be the same as all other Harmony deployments.

Passphrases are stored in plain text, enabling automations within deployment/validation/management processes, and making all data fully transparent to the user. While we have plans for data encryption options in the future, to start with our focus was giving users the ability to easily understand and modify configs. If you check out our Discord, users are always modifying config files to create custom setups (including side-loading external Harmony nodes :joy:). Our premise right now is your data is safe as long as your server is safe.

This does mean your data could be compromised if someone gets access to your server. At the beginning of our docs, the first thing we do is warn users about not giving untrusted applications or untrusted 3rd parties access to your server. To make Node Pilot transparent, modifiable, and portable, we opted to trust the OS for security. As mentioned, native Node Pilot data security features are planned for the future.

In terms of data privacy, Node Pilot doesn’t send any out information and all data, including user data (like email address used for TLS) and node data, is stored locally. Since all data is stored locally it’s up to the user to ensure their server is safe from untrusted applications or 3rd parties. Node Pilot is a true self-hosting solution where we have zero data, which also separates this from custodial services because we can’t provide any kind of recovery. It’s your own responsibility to ensure you have your data backed up. We do have a “Backup” button in the GUI to enable users to download their data for safe backup storage (which is highly recommended). Even that feature is just between your computer browser and your server with no 3rd parties.

Happy to answer any other questions :slightly_smiling_face:

1 Like
16

For setting up:

We have easy-node which can setup a node very quickly, is open source and heavily supported and documented.

Harmony provided a 2k bounty for it. Myself and others have also contributed somewhat (not nearly as much as @easynode though)

This already takes into consideration key management and security as well as setup.

For key management
We have @RoboValidator amazing autobidder GitHub - Robovalidator/harmony_autobidder: Python service which autobids and manages BLS keys for validators on the harmony blockchain

Again opensource and supported and also widely used throughout the community.

For monitoring and resource management, I have created automatic HDD resizing and alerts to both email and @FortuneValidator amazing telegram Bot with plans to extend this to CPU resources, CLI updates and support other providers upon request.

So far I have Digital Ocean and Linnode but it is no problem to implement any other provider…

I should rename the repo :stuck_out_tongue:

All of the above a open source and maintained by existing time-served validators and are already fully operational. There are also plans to integrate the 3 together to form 1 Validator Node setup that has all the functionality of the above and more.

[Disclaimer - I HATE GUIs but LOVE Docker]
I personally do not see the advantages of a docker container and gui. Considering what we already have, this is an extra layer of complexity that is unnecessary for the average validator and it is not currently developed.

If you have a large server and run several validators for several chains, it may be beneficial in that context.

I certainly do not doubt the skills and expertise of the proposers. They appear highly qualified, professional and skilled and I congratulate them on a nice product and company.

I hope someone can offer some more insights to this proposal.

2 Likes
17

I think having an easier way to deploy a Validator would attract more people that do not understand or have computer engineering train.
Just one simple question.
Can any validator operator leave your system anytime they want?
How do you manage a Validator that leaves without sound? I mean. If a Validator isn’t taking care anymore of it, how do you manage to communicate or take care of the delegators? Does this validator has a red flag, underperforming KPI?

Thank you and good luck.
My vote is a yes. Because of the number that this project expose in a marketing manner would attract.
Best

1 Like
18

From a business and outreach perspective, this could be a good thing for both companies but I think there is more discussion to be have about the details, especially surrounding transparency and security.

2 Likes
19

Isn’t something similar ANKR is providing already.
How many of Validator node operators are using their solution, not so many.

Also, as @Maffaz mentioned already there is easy-node solution as well.

Personally I, as IT person, would like to have more control over my Validator node Setup and do not outsource it to some 3d party, that’s why I was using Harmony One official docs in building Validator node. Also, it gave me more exposure how Validator node is operating.

I agree, that such solution could lower down entry point to become Validator, but if this model will be sustainable? When some kind of technical issues appears, the only party, who will be able to provide help to such Non technical Validator node operator would be that 3d party itself, that is providing it’s own solution.

It is bigger risk for network stability, IMHO…

2 Likes
20

Some validators lost their ONE after creating with ANKR.

This was mainly due to ANKR being the sole owner of the Private Keys / Seed Phrases. I don’t know the full story but ANKR no longer supports Harmony.

1 Like