Store and forward - Conditional store/forward rates

Store and forward, I usually set to about 100 records for the forward write size. This works great for my continually data logging which is about 15 queries per second.

However if I get a backlog in my local cache for some reason (took down the database server), of say, 500k records (my local cache is set to 1 million records), I would like a different update rate to clear the backlog faster. At the moment I manually change the write size to about 10k so that I get far more efficient querying (about 500 queries/second) then I set it back when the backlog is cleared.

I am proposing two new parameters.
The first parameter is the “record count at which the forwarding write size is changed”.
The second parameter is the new forwarding write size parameter to use when the condition above is met.


Why not just leave it at the large size all of the time, and just set the “forward time” to a small number, like 100ms?

The forward size and forward time work together to trigger the forward: if either condition is hit, it forwards. The forward size, though, also affects how data is loaded from the cache, and therefore can have an impact on catching up, as you’ve noticed. However, as long as the “time” portion is small enough to cause data to be forwarded constantly in the normal state, I don’t see a problem with with “size” portion be as large as you’d ever want it to be.