In general yes,. We are testing if Ignition is in the first step, a good integration layer between lines and the site historian/db, with the requirement to not loose data between the machine and the site historian e.g in case of network issues.
The lines have edges, but we can also upgrade to full gateways. And the idea is to have central gateways as the link to the site historian
We do have constraints like the options to put data into the site historian (Only OPC UA , CSV and in the near future MQTT) or not having full network redundancy. But the data must eventually end in the site historian.
Can you name the brand of that site historian?
(So I can tell anyone considering it how terrible it is, to avoid in any new projects, and to plan on replacement in any existing projects.)
Unsure if this requirement is based on a network security model (Purdue, CPwE, etc.) or based on physical proximity. In either of these cases, I would leave Edge gateways at the machines and place a central gateway 'as close as possible' (same layer & location) to the site historian. Let Edge S&F do its job to buffer data and ensure that it all gets sent to the central gateway (& its Historian). From there, you'll be able to use the full gateway to query data from all machines, retrieve the data and format it whatever way you want. In the future, when this process is streamlined, you'll be able to connect that central Ignition server directly to the site historian, possibly without touching the configuration on your Edge gateways at all.
If we assume that importing a CSV is the best method to get data out of Ignition and into the site historian, what does that process look like? As in, if you had a CSV file sitting on an accessible filesystem somewhere:
- What EXACTLY does that file look like? What does the site historian REQUIRE? What are the exact column names, column data types, give example of a couple of rows of data, etc.
- What is the process by which that file gets imported to the site historian? Manually? By some file/folder listening service? By streaming of 'CSV style data directly to a socket connection' on the site historian?
1 Like
- Yes, the edge gateways are part of the machine and the central gateway is per site very close to the historian.
- We can already S&F the data to the central gateway (and its historian) in different ways. That works fine in our test env.
Regarding CSV:
- The file is configurable but the standard is to have a row like so:
Timestamp | Tag 1 | Tag 2 | Tag 3 | Tag 4
Everything else will be configured in the central site historian
- There are services whatching specific folders, get the csv and store the data eventually in the db The internals are not known but it does some checks and is then using SQL to insert the data.
We tried to just script the export from the historian to a csv file. Ignition already provides the required functions. Unfortionatly we want to have the raw data which comes event based. So, reading just the raw data takes a long time and provide a not-expected csv result. But this is also described in the documentation. The other option is to provide an interval or the return Size. But this is not know. But, I guess we make this working with some integrator support or directly query the Ignition historian (read replica). Lets see, not sure what the best option is 
This is not compatible with Ignition's data collection. Different tags will have different timestamps, totally independent of each other. If you try to force data from Ignition's historians into a common-timestamp format, you will lose information. If you query from Ignition's historian in "wide" format, it will either use aggregation to collapse the samples by requested intervals into single rows, or will duplicate rows with as many timestamps as actually occurred.
To faithfully ingest data from an Ignition historian, you need to query in "tall" format, with no aggregation or interpolation (aka "as stored"). Every row will be one tag value with name and timestamp.
Timestamp - What format? Unix (integer)? String?
Tag1/2/3.. - As a tag ID (numeric)? Full tag path as string? With provider or without?
All configurable. Time in UTC, string format which is configurable. Also the tag names are configurable, but the tag string name would make sense.
Thats what we experienced in a first naive try.
Thank you for the hint with the tall format!
Not sure if this is the best solution, but we can use full gateways and switch from ignition historian to sql scripts
In your situation, I would recommend using the legacy site historian only for legacy applications, with best effort to push data to it, and use Ignition as the "source of truth" going for all new applications. Extract old data from the current site historian to backfill Ignition's history.
1 Like