Update BP NetworkGraph and Scorer persist frequency
#2226
Conversation
| @@ -371,8 +372,7 @@ macro_rules! define_run_body { | |||
|
|
|||
| have_pruned = true; | |||
| } | |||
| let prune_timer = if have_pruned { NETWORK_PRUNE_TIMER } else { FIRST_NETWORK_PRUNE_TIMER }; | |||
| last_prune_call = $get_timer(prune_timer); | |||
| last_prune_call = $get_timer(NETWORK_PRUNE_TIMER); | |||
| } | |||
|
|
|||
| if $timer_elapsed(&mut last_scorer_persist_call, SCORER_PERSIST_TIMER) { | |||
There was a problem hiding this comment.
Wondering is it still necessary to keep this regular 30s persist if we're persisting on every update to the scorer?
There was a problem hiding this comment.
I'm not too familiar with the scorer, but I don't think it does anything in the background to warrant persisting on a timer, as it mostly relies on datapoints received from events (cc @TheBlueMatt).
There was a problem hiding this comment.
It doesn't, indeed, though I also don't think there's any harm in persisting it once an hour with the graph, so would prefer to just leave it.
There was a problem hiding this comment.
The timer for the scorer was set to persist every 30s, so I just changed it to an hour to be similar with the graph.
Codecov ReportPatch coverage:
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. Additional details and impacted files@@ Coverage Diff @@
## main #2226 +/- ##
==========================================
+ Coverage 91.56% 92.63% +1.06%
==========================================
Files 104 105 +1
Lines 51749 70952 +19203
Branches 51749 70952 +19203
==========================================
+ Hits 47386 65725 +18339
- Misses 4363 5227 +864
☔ View full report in Codecov by Sentry. |
| @@ -352,8 +347,7 @@ macro_rules! define_run_body { | |||
| // falling back to our usual hourly prunes. This avoids short-lived clients never | |||
| // pruning their network graph. We run once 60 seconds after startup before | |||
| // continuing our normal cadence. | |||
| let prune_timer = if have_pruned { NETWORK_PRUNE_TIMER } else { FIRST_NETWORK_PRUNE_TIMER }; | |||
| if $timer_elapsed(&mut last_prune_call, prune_timer) { | |||
| if !have_pruned || $timer_elapsed(&mut last_prune_call, NETWORK_PRUNE_TIMER) { | |||
There was a problem hiding this comment.
I'm not sure if we want to prune immediately when syncing via P2P. Previously, we'd wait 60 seconds, which gave us some time to sync with peers any updates that happened while offline. If we happened to be offline for a while, we may end up pruning a large portion of our graph before we sync with any peers, just to add them back once syncing completes. Maybe we should also expose a notion of "initial sync" when syncing via P2P that checks whether we've done a sufficient amount of syncs?
There was a problem hiding this comment.
Yea, indeed, probably we should not.
There was a problem hiding this comment.
I couldn't find a great way to expose when an "initial sync" would be considered over for P2P, since IIUC we basically do an "unofficial" sync of just getting either a full dump (no-std) or latest 2 weeks (std) through the gossip timestamp filter for our first 5 peers we connect to? For now I just left the 60s initial prune timer for P2P but allowed pruning before this once RGS initial sync has completed.
| @@ -371,8 +372,7 @@ macro_rules! define_run_body { | |||
|
|
|||
| have_pruned = true; | |||
| } | |||
| let prune_timer = if have_pruned { NETWORK_PRUNE_TIMER } else { FIRST_NETWORK_PRUNE_TIMER }; | |||
| last_prune_call = $get_timer(prune_timer); | |||
| last_prune_call = $get_timer(NETWORK_PRUNE_TIMER); | |||
| } | |||
|
|
|||
| if $timer_elapsed(&mut last_scorer_persist_call, SCORER_PERSIST_TIMER) { | |||
There was a problem hiding this comment.
I'm not too familiar with the scorer, but I don't think it does anything in the background to warrant persisting on a timer, as it mostly relies on datapoints received from events (cc @TheBlueMatt).
Previously we would wait 60 seconds after startup, however for RGS we prune/persist after its initial sync since 60 seconds is likely too long.
Now that we persist the scorer upon events, we extend timer persistence from 30 seconds to 1 hour, similar to network graph persistence.
alecchendev
force-pushed
the
2023-04-persist-network-graph-on-rgs
branch
from
8ac39c9 to
2afbdf5
Compare
Closes #2117.
For
NetworkGraph, we normally wait an initial 60 seconds for an initial prune, but since that's likely too long for RGS, we allow pruning before the 60s timer when the RGS initial sync completes. For P2P it still waits the 60s, otherwise both prune/persist hourly as it currently does.For
Scorer, instead of persisting every 30 seconds, this persists each time the scorer is updated from an event, and once every hour similar to the network graph.