Store and Forward only Storing

Ignition
1

I have two tag servers, both of them have JDBC connectors to the same database.

One of them has store rate and forward rate roughly equal and tag history works on that gateway.

However on the other gateway, tag history stopped working today and store rate is there but forward rate is N/A meaning data is only going to disk but never makes it to DB. I noticed this today when historical values which were previously working stopped.

We run the same version and same settings on both machines so I'm not immediately sure why this is happening.

Input from anyone who has experienced similar is much appreciated.

EDIT: I can still query tag history from DB on the trouble server so the issue is only in the store direction, retrieval seems like no problem.

Thanks,

Nick

2

This morning the other tag server who's tag history was working, has stopped and shows the following, the same as the other tag server:

image
image889×312 13.9 KB

One thing to note is this seems to only affect tag history and alarm history. We store things to that DB directly (not via Store and Forward) and those data streams are still working (we can store and retrieve fine).

Thanks,

Nick

3

We seem to have recovered by doing the following:

  1. For the target store and forward, disable disk storage
  2. On the root machine, delete the corresponding folder in: Ignition\data\datacache
  3. Re-enable disk storage on the target store and forward

While we do see the expected behavior now from history tags, I am skeptical because we do not know why this happened.

What would be the best way to determine what event(s) caused the cache to fill up and behave abnormally?

Thanks,

Nick

4

Provide your logs to support (from when the problem started).

5

@pturmel I think we have solved the problem based on current behavior.

Basically what was holding up the S&F was long alarm event inserts. I am not certain at this moment why those were taking so long, but what we did is setup 2 different JDBC connections to the same database forcing the alarms to take their own S&F route. This way if alarms have some issue, only alarms are affected and tag history still makes it in without blockage.

During the whole time we had things that go to DB that were non S&F that continued to work just fine.

Currently monitoring and would call it resolved after a few days.

Nick

1 Like
6

I have monitored this all day today and have marked the above comment as the solution. That is: setting up a separate JDBC connection for the alarms so that alarms and tag history flow through different store and forward paths.

Nick