Rclone Database Shard 0 vs Established Node Shard 0 sizes

Web3 Developers
1

Hello,

This week we’ve had many nodes fill up on disk space.

I’ve been running a lot of tests loading nodes and noticed over the last few weeks an rclone of shard 0 now takes up 167GB of storage.

On an old node I have been running for 2+ months, shard 0 is only at 130GB. On a server I have from a month ago it’s at 155GB.

Here’s a few questions pertaining to storage that some validators have had:

Is there anything that can be done to reduce the storage needed for a fresh rclone?
Should we all plan on much more storage space being the norm for a validator going forward?
Will anything be done to reduce the size of the database a validator needs to hold?

Thank you,

Patrick @ Easy Node
easynode.one

13 Likes
2

@leo @rongjian :point_up_2: Can you say something about this issue?

3 Likes
3

I would also like to know as well as several other validators I have spoken with.

2 Likes
4
  1. the rclone db may not be the cleanest db. @leo maybe we can refresh the rclone db with smaller version like the one with 130GB?

2/3. In the short term, yes, the db will grow as the txn volume keep at a high level. But eventually, we will implement pruned version of the db to cap the storage needed for validators.

6 Likes
5

@slugo_slugom_crypto we’ll circle back with an estimate completion by the end of the week. Hang in there.

2 Likes
6

I’ve witnessed something similar with my nodes, here’s a few things we’ll be working on Patrick @slugo_slugom_crypto

  • To deep dive into the differences in the DB and investigate the root cause
  • To create a fresh copy from block 0 (this will take a week)
  • (bonus) To have the rclone db snapshot occur more frequently (e.g. 3x per day)

This will take us 2-3 weeks to resolve. The current copy should still work, we suspect that the pruning isn’t working as consistently as it should.

3 Likes
7

ive synced 170gb so far. on day two of syncing. will i still be able to validate even if i dont have the full db?

8

it depends … non shard 0 you could validate but if your node is s0 it won’t validate until you are fully synced

9

@slugo_slugom_crypto @harmonious … thanks to @sophoah for restoring a fresh copy, we now have a cleaner Shard 0 db. Closing the loop here based on confirmations on Discord

2 Likes
10

Imgur

Hello - just a heads up that we’re up to 229GB on shard 0 on a fresh rclone again. Any chance this can be reduced again?

7 Likes
11

A week ago shard 0 was 218GB, that’s 5% increase in just a week!

4 Likes
12

Mine is currently 275GB, I’m constantly in fears that it would soon take up the remaining 15% space remaining and I have to upgrade more disk space and garner extra cost of servers.

Something needs to be done as running a validator on Harmony is getting costly every week and many validators especially new ones like us are still on 0% fees so as to attract delegators.

This needs to be capped and fast too.

5 Likes
13

+1 on worries about this.

If there’s anything I can do, I’d like to help

3 Likes
14

I hope something can be done. New validators will find it hard to begin or keep up. My first node is barely two months old and I had to whip a new one with a mounted volume because it got full.

2 Likes
15

My db0 is at around 253 GB, so I’ll have to increase my volume again tomorrow. I’ve already increased it twice since I set up my node (+1 month ago). The cost of running a validator is getting higher and that’s particularly hard on unelected validators that don’t generate rewards yet.

Short term this is OK, but if the db size keeps on getting bigger and bigger month after month, then it’ll reach a point where some validators won’t be able to continue running their nodes. Remember that we have validators from all over the world and +$100 a month is already a lot of money for people that aren’t from the so called “first world countries”.

Well, I’m sure you’re already working on this, guys, so please let us know when you have some news.

Question: Will there be a prune on db0 on the upcoming hard fork?

4 Likes
16

Shard 0 for me is 230GB and Shard 3 is 13GB. It would be nice to keep things under 250GB and at this rate it is concerning for costs.

3 Likes
17

I’m sure many are in the same boat as me. I’m an unelected validator with the 160gb+250gb mounted volume. I’ve recently had to bump that up to 300gb since the shard0 db was reaching the 250gb mark.

This roughly increases costs by 30%, to run an unelected validator, trying to support the network. pls halp.

4 Likes
18

Just checked today, Shard 0 is at 237 GB. I sincerely hope we find a solution too, it’s getting prohibitive to start up or run a node.

19

RJ - any idea how close we are to being able to do this? The DB0 size is getting so big the entry level barrier for new validators is becoming an issue. We are going to lose a lot of validators who simply can’t afford to provide the service. Looking forward to hearing your thoughts!

3 Likes
20

Thanks for reiterating the issue. What would be the upper limit for storage where it’s not profitable for the validators in concern? I would like to get a better sense about the urgency and prioritize engineering tasks accordingly. As you know, we have many other initiatives going on and the eng resource is limited.

2 Likes