Core Historian Shows RUNNING Status but Stops Writing Data

Ignition
1

Hi everyone, I wanted to share an issue I ran into while working with Ignition and the Core Historian, and also ask if anyone has seen something similar or has any recommendations.

On a Gateway running Ignition 8.3.1, the Core Historian provider showed a RUNNING status with no visible errors. Tags could be configured and historized without any issues, but when trying to view the data in PowerChart, no historical values were shown and the message “No Trend Data Source available (Bad_NotFound)” appeared.

I tried several things without success, including disabling and re-enabling the Historical Provider, creating new tags and historizing them, restarting the Gateway, and deleting and recreating the Historical Provider. None of these actions changed the behavior.

While reviewing the Core Historian database, I noticed there were around 16 million records and that the last stored value was from 2025-12-17 , which made it look like the historian had stopped writing new data. The provider was configured with one year of retention per partition, so I started to suspect that some internal file limit might have been reached or that one of the historian files had become locked or corrupted.

Before taking more invasive actions, I believe a good practice in this situation would be to export the existing historical data directly from the Core Historian database (for example from the core_historian.datapoints table), either completely or filtered by a date range, in order to preserve the data.

The solution I applied (clarifying that this was done only in a test environment) was to stop the Ignition service, move the Core Historian files to the backup folder, and then start the service again. After restarting, Ignition automatically recreated the Core Historian files and the historian started working again, with data correctly visible in PowerChart.

My question to the community is whether anyone else has encountered this Core Historian behavior when dealing with large data volumes, and if there are any official recommendations or best practices to avoid reaching this state (for example retention settings, partitioning, or known limits of the Core Historian).

Any feedback or similar experiences would be greatly appreciated. Thanks!

2

Having similar issues here with the core historian.

I’m able to insert into historian using system.historian.storeDataPoints() and it returns ‘Good’ but I’m unable to query using neither the tag history binding nor the system.historian.queryRawPoints()

Funny thing is, if I setup a new historian then it starts working and it keeps working for a while then at some point it starts falling apart and I’m no longer able to retrieve anything from it.

I can continue to insert into historian with system.historian.storeDataPoints() but I just can’t retrieve the newly inserted points anymore but I can still see the points that were stored while the historian was still working.

I really wish there was a way to interrogate the questDB underneath the hood but it looks like the ways to do this are very limited from what I could gather from the documentation.

3

Thanks for the detailed reports, @cristian.garcia and @k.jin.

What you're both experiencing is likely related to QuestDB table suspension. The core datapoints table can become suspended when corrupt transactions accumulate in the WAL (Write-Ahead Log) files. When this happens, new data continues to be accepted and written to the WAL, but queries against the table fail because the table itself is in a suspended state waiting for those corrupt transactions to be discarded.

The root cause ties back to how the Core Historian currently handles durability. It uses QuestDB's default nosync commit mode, which relies on OS-level durability—the OS decides when to flush dirty pages to disk. While this provides better write throughput, it means that during an unexpected shutdown or power loss, the OS may not have flushed all pending writes. When QuestDB recovers, it attempts to replay the WAL to restore consistency, but if incomplete or corrupt transactions are present, the table can become suspended.

The alternative is sync commit mode, which invokes fsync() on each commit. This forces the data to be physically written to disk before the commit returns, ensuring data survives OS crashes or power loss. The tradeoff is reduced write throughput since every commit waits for disk I/O to complete.

For more details on QuestDB's durability model, see: QuestDB Storage Engine | QuestDB

The good news is that we have changes coming in 8.3.4 that will expose QuestDB's configuration files, allowing you to customize nearly all QuestDB settings including the commit mode. The server.conf and log.conf files are QuestDB-specific configuration files (more info here: Configuration | QuestDB) and will be auto-created with defaults at:

{InstallDirectory}/data/var/com.inductiveautomation.historian/corehistorian/conf/

Once this is available, you'll be able to modify the commit mode in server.conf. For more technical background on this issue, see: Automatic recovery for suspended (WAL) tables on start-up · Issue #4829 · questdb/questdb · GitHub

We're also exploring whether to change the default durability mode to sync rather than the current nosync, which would help prevent this issue from occurring in the first place. This is still being evaluated since it does come with performance implications, so no guarantees there—but regardless, the configuration file will give you the ability to set whichever mode fits your environment.

This is currently being vetted by our QA team and should be available in a nightly build soon. I'll update this thread when it's released.

1 Like
4

Eeeewwww!

This puts a lot of the performance claims against other technologies in a new light. Like, apples and oranges. :frowning:

Drop all the consistency mechanisms from other DBs and see how fast they can go, too.

3 Likes
5

Very true, but throughput benchmarks are almost always run under best-case scenarios with controlled environments, consistent hardware, and often with durability relaxed or disabled entirely.

For context, here's how some other databases handle durability by default:

Relational databases (generally default to durable):

  • PostgreSQL/TimescaleDB – synchronous_commit=on, fsync on each commit
  • SQL Server – flushes transaction log on commit by default
  • MySQL/InnoDB – innodb_flush_log_at_trx_commit=1, fsync on each commit
  • SQLite – synchronous=FULL by default

Time-series databases (more of a mixed bag):

  • InfluxDB – wal-fsync-delay=0s by default, meaning fsync on every write
  • ClickHouse – fsync_after_insert=0 by default (disabled), opt-in for durability
  • TDengine – wal_level=1 by default (WAL but no fsync), need wal_level=2 for fsync

So QuestDB isn't entirely alone. ClickHouse and TDengine also default to the faster-but-riskier approach, but InfluxDB and the relational databases do default to sync.

That said, yeah, I get it. It's why we're looking at flipping the default to sync and letting users opt into nosync if they want the throughput and are okay with the tradeoff. QuestDB's also got an open issue digging into this: Optimise QuestDB performance when running in durable sync mode · Issue #4976 · questdb/questdb · GitHub