# Weekly OnCall Rotation - Aug 10-17, 2021

**URL:** <https://talk.harmony.one/t/weekly-oncall-rotation-aug-10-17-2021/2400>\
**Category:** Team Ops\
**Created:** [September 13, 2021, 6:15pm UTC](https://talk.harmony.one/t/weekly-oncall-rotation-aug-10-17-2021/2400 "2021-09-13T18:15:46Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![leo](https://yyz1.discourse-cdn.com/flex035/user_avatar/talk.harmony.one/leo/32/21_2.png) [@leo](https://talk.harmony.one/u/leo)\
**Post date:** [September 13, 2021, 6:15pm UTC](https://talk.harmony.one/t/weekly-oncall-rotation-aug-10-17-2021/2400/1 "2021-09-13T18:15:46Z")

</div>

# Weekly OnCall Rotation

Oncall: Leo, Soph({lc,soph}@harmony.one)

Duration: 8/10 12:00pm - 8/17 12:00pm

# Summary

- Lots of low incidents that are auto-resolved. Out of sync issue on mainnet that are usually temporary
- S0 Leader disk space issue

# Details

- 8/10

- 8/11

- 8/12

- 8/13

- 8/14

- 8/15

- 8/16

- 8/17

# Operation Improvement

- Apart from “Node Disk Space Alert”, no other needed action. How can they be fine tuned to avoid too many notifications?

- Average UnHealthyHostCount GreaterThanOrEqualToThreshold 1.0 is super annoying

- 
# Node CPU Usage Rate Alert can be fine tuned?
- Watchdog auto-resolve “beacon out-of-sync” pager now. 22 tickets were automatically resolved.

# Resources

Runbook: [GitBook](https://app.gitbook.com/@harmony-one/s/onboarding-wiki/devops-run-book/harmony-mainnet-devops-runbook)

Uptime Status: [https://status.harmony.one/](https://status.harmony.one/)

Grafana Server: [http://grafana.harmony.one:3000/d/DoL\_wq7Zk/harmony-nodes-monitoring?orgId=1&refresh=5s](http://grafana.harmony.one:3000/d/DoL_wq7Zk/harmony-nodes-monitoring?orgId=1&refresh=5s)

Node out-of-sync: [https://harmony.one/status](https://harmony.one/status)

Pager duty: [https://harmonyone.pagerduty.com/](https://harmonyone.pagerduty.com/)
