New Historian - QuestDB

Ignition 8.3 Early Access
21

Change took place just a few weeks ago (July 19). As I understand, TimescaleDB is now "TigerData".

1 Like
22

Looks like the company name changed, but the software (PostgreSQL extension) is still called TimescaleDB.

1 Like
23

This sort of thing is common in order to distinguish an open source project from the company that sponsors its development (or most of it) and offers commercial support. FWIW, PostgreSQL itself is sponsored by EnterpriseDB, where you would go for pure PostgreSQL support.

1 Like
24

Hi @Kyle_Chase1 - are you able to provide some guidance on these questions?

26
  1. We only support the embedded QuestDB historian right now. We will support the external QuestDB historian at some point in the future.
  2. By default, the embedded historian will use 25% of available system memory. You can adjust this limit, specifying either a specific memory amount or by percent.
2 Likes
27

Is store & forward a planned future capability? The current user manual section explicitly states that QuestDB doesn’t support it.

28

Why do you need store and forward if you are writing to the local, embedded historian?

I do not feel there is a need for S&F when writing locally.

29

I agree with Kyle. I’d love to know a use case here, but if your system state is such that you can’t write to the embedded historian, your chances of being able to write to a S&F file on disk are pretty low.

30

Goes back to an earlier question that went unanswered. IAs recommendations have always been to have the database on a seperate box. Many of those reasons haven’t changed. So, I'm guessing that many are wondering if there will be a remote questDB box option, or will it forevermore be married to the Ignition Gateway box?

31

:smiley:

Disclaimer: Not my area of the product, but (while QuestDB is enormously powerful) it's important to think of the 'Power Historian' as it will exist in 8.3.0 as a deliberately limited implementation.

It's not meant to be all the things for all the people [1] - it's specifically meant to be "here's a turn key historian you don't have to think about" - specifically for folks who don't want to manage a SQL DB on an additional server, or don't want to bother their IT department with permissions to add tables/a schema, or are just used to XYZ other software that proudly proclaims how easy their historian is to set up.

  1. yet ↩︎

1 Like
32

The Internal Historian - QuestDB (IHQ) is an early step to building a more robust historian solution. The IHQ is able to provide fast, localized access to your historical data, and will be part of our ongoing historian development. Access and moving data between instances in a distributed, heterogenous architecture is on the long term road map, and the new IHQ provides a key building block in that architecture.

33

Hopefully there will be tools provided to ease those transitions with existing historians, whether merging or bridging of data and databases. With a large installed base of MSSQL/MySQL historians, customers are going to want a transparent migration path that utilizes the new high speed questDB coupled with continued access to their existing data.

34

To chime in here on this question specifically:

One architecture that I’m thinking won’t be uncommon for folks with medium sized systems would be one with the Historian separate from a main server, for a lot of the reasons you’re describing. This provides the benefits of QuestDB, doesn’t require a SQL database, and takes advantage of store & forward and the Gateway Network’s ability to stream historical data.

image
image211×112 6.06 KB

In that image, the Ignition on the left would be a standard Gateway with full visualization and device connections, and would have a Gateway Network connection to a dedicated Ignition Historian server / VM / container (pictured on the right), which would be using the Internal Historian - QuestDB. You could have several Ignition Gateways all share the same Ignition Historian server.

I think this architecture will increase in popularity after we ship with redundancy support for the Internal Historian - QuestDB, which we’re aiming to have out within a few months from initial 8.3.0 release. The initial release won’t have redundancy support, so standard IT tools for file backups are the main method of resiliency until we do release that support.

We’re doing a lot of work on our Historian Guide for 8.3, so we’ll try to package information like this and other related things in a nice way for the whole community.

4 Likes
35

So you’re saying that your grand plan is to charge for entire gateway licenses for what can already be done with Postgres and a single gateway for free?

36

No, if Postgres is working for folks, absolutely keep using that. No reason not to.

QuestDB will have higher ingest rates and likely better data compression, but probably 90% of folks using Ignition and the current SQL historian have systems that are working well. There’s no reason to change from that architecture if it’s meeting the current needs.

37

Nice. I can see the advantages of a setup like this. Assuming that a remote gateway/questdB module would be a reduced license cost than the full gateway on the left.

38

Yep, it would be much lower cost. Just the platform and the historian package.

39

Not sure if there are plans to show comparisons between all the common databases and QuestDB at ICC this year, but I’d love to see a slide/benchmark of QuestDB vs SQLite, MSSQL, MySQL/MariaDB, and Postgres (both with and without TimescaleDB w/ and w/o compression enabled) for ingestion, storage, query speed, etc. on the same hardware. If it’s ready for ICC, it would be even better to show with the store/forwarding to a separate server, but even on the same server to compare apples to apples would be great.

Also, I know ZFS compression was discussed, so as a separate metric, it could be shown with and without it since there’s probably a lot of people who are using Windows and wouldn’t be able to take advantage of ZFS compression. Where as with TimescaleDB, compression is also cross platform compatible. (I’m not familiar enough with QuestDB to know if it has built-in compression like TimescaleDB).

3 Likes
40

Wouldn’t this architecture require S&F or what happens if the connection between the two gateways ist down or we have to shutdown the database gateway for maintenance reasons?

41

Yes this architecture requires store and forward, but at initial release with only the local historian capability, store and forward isn't needed.