An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception

earnhartj84 · March 25, 2024, 1:47pm

I'm getting this error.

Is there a way to stop this error

Also, when I go into the gateway, click on the store's status, and forward, it takes 5 minutes for the status to appear.

When I restart the gateway, it takes 10 minutes to start up.
It should be a lot faster than that.

Here is a copy of the log file.
032524wrapper.log (3.8 MB)

pturmel · March 25, 2024, 1:58pm

Please fix your post to wrap that traceback with a "Preformatted Text" marker.

earnhartj84 · March 25, 2024, 2:27pm

fixed. thanks

pturmel · March 25, 2024, 2:41pm

Your log is full of these:

INFO   | jvm 1    | 2024/03/25 08:03:48 | E [t.e.a.history                 ] [13:03:48]: Error storing tag history data 
INFO   | jvm 1    | 2024/03/25 08:03:48 | java.lang.Exception: Unable to add data to memory buffer, buffer full.
INFO   | jvm 1    | 2024/03/25 08:03:48 | 	at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.insertInBuffer(MemoryStore.java:185)

You database isn't keeping up with the historian load. Fix your database.

earnhartj84 · March 25, 2024, 2:51pm

I created a Support ticket.
I hope they can guide me toward what went wrong in my database.
Since the database hasn't changed.

Thanks.

pturmel · March 25, 2024, 2:56pm

Have you changed how many tags you are historizing? Are you running the database on the same hardware as other things (like the gateway itself), and not managing the conflicting resources?

Your logs also show database connection breakage. Is your overloaded MySQL rebooting itself?

earnhartj84 · March 25, 2024, 3:05pm

I have changed the number of tags I am historizing I've added maybe 20 more tags being recorded to TC0
The SQL database is on the same machine as the gateway.
I have small clients using Ignition, and there may be 3k total tags.
I set up 4 DB connections, 1 for each test cell. TC0, TC1 TC2, TC3.

Each of these stores the data in a separate drive (D:) and does not use the C drive to store and forward this data.

Conflicting resources? Like RAM and CPU usage?

pturmel · March 25, 2024, 3:24pm

Yes, those. Sounds like a very small system.

By separate databases, do you mean multiple schemas in one single database process? Or multiple independent database processes?

You almost certainly will need support to look over your shoulder.

earnhartj84 · March 25, 2024, 3:28pm

Yes,
multiple Schemas in 1 single database.

the CPU and Ram usage don't seem to be too high.
C: is for processing
D drive is for store and forward. data.
E is for backups ,

pturmel · March 25, 2024, 3:32pm

Concur.

There's plenty of RAM for any ordinary system.

You will probably have to examine the database's own log files to see why it has been choking. (Your wrapper log clearly shows your DB choking and your S&F memory cache running out of space.)

pturmel · March 25, 2024, 3:33pm

Hmmm. That read speed on your C: drive suggests something bad is happening.

earnhartj84 · March 25, 2024, 3:46pm

When I stop the ignition gateway, the C drive goes 0-1 %.
It's just another symptom of an underlying issue.

looking into SQL Log files.

earnhartj84 · March 25, 2024, 6:04pm

looking more into how to resolve this issue. When I check my plugins for MYSQL, I don't see any. Since I'm not using any, there might be built-in plugins I'm not unaware of.

did find this warning in the SQL log file.
09:55:31 [INF][ WBContext]: WbContext::init
09:55:31 [INF][ WBA]: Looking for extension modules for WBA...
09:55:31 [INF][ WBA]: 0 extension modules found
09:55:31 [WRN][ grt]: Duplicate plugin name wb.tools.cmdlineClient There is more than one plugin with the name wb.tools.cmdlineClient (in PyWbUtils and PyWbUtils).
09:55:31 [WRN][ grt]: C:\Users\earnh\AppData\Roaming\MySQL\Workbench\connections.xml:26: link '{0B8C220D-1180-47EE-B9FC-60C62C3339D5}' key=owner could not be resolved
09:55:31 [WRN][ grt]: Duplicate plugin name wb.tools.cmdlineClient There is more than one plugin with the name wb.tools.cmdlineClient (in PyWbUtils and PyWbUtils).
09:55:31 [INF][ WBContext]: System info:
MySQL Workbench Community (GPL) for Windows version 8.0.30 CE build 2054668 (64 bit)

earnhartj84 · March 25, 2024, 6:30pm

I thought SSD drives have a "good" read and write speed under 1000-500MB/s
But you're the expert, so I'll take your word for it. there is definitely something going on with my S&F. still diffing through log files. and waiting to hear back with the support ticket.

pturmel · March 25, 2024, 6:37pm

Actual performance varies greatly with the kinds of SATA ports involved and, particularly, the size of specific transfer requests.

But 130MB/s reflects a rather intense load for an OS disk. (I personally find it annoyingly difficult to troubleshoot Windows-based applications, so I won't be much more help.)

earnhartj84 · March 27, 2024, 2:21pm

need to disable the Disk Cache settings first. for each database connection Then, rename the folder under the name of the database in the C:\Program Files\Inductive Automation\Ignition\data\datacache directories. Next, we will enable the disk cache again and set it back to the default number values.