Importing tag history from previous system into Ignition

So a customer is inquiring about importing in historical data from their old system into ignition. Im not talking about a few thousand lines of data, I would guess millions of rows of data that would need to be added to the Ignition taghistory tables.

I guess it would be possible to get this data into a database(its csv right now), then thin it out by pulling 1 sample per datapoint per day, then work from there.

Would it be possible to write a script of something that can look at this data and write it into the taghistory tables?

Yes, it would be possible. If you already have a CSV file it would be pretty easy. The only problem is that it might take a while to complete if you have millions of rows. If you have a small sample we can help you write the script.

Hi,

If the data is in CSV, you could definitely use some sort of scripting to bring it in. I can work with you to explain the table structures and how I would go about it, or perhaps if you send me a sample of the CSV, I can help with the script.

Before that though, just a quick question: if it were possible to interface with the current system, would it still make sense/be preferable to import it? That is, imagine that you could access that history through the same mechanism that you accessed tag history currently: you could browse for the tags from a “historical provider”, and use them on charts, tables, scripting, etc. mixed in with SQLTags History. I’m just curious, because we’ll shortly be able to write these types of modules.

Regards,

Yeah I am worried about doing the import thru ignition. I have done some with a few thousand rows and it takes quite a while(20-30 minutes) using the csv parsing script.

I guess my first question if we use the import option thru scripting, what kind of performance hit would we take by writing this data into the tag history tables while the system is live? would we run into any deadlocks or anything?

Colby, we dont really have that option in this situation, since a 3rd party was hosting the data. I dont think it is a bad idea though. I may be looking at something similar pretty soon to access wonderware tag data. kepware makes a wonderware to opc driver that i may use to access tag data from a wonderware system without actually pulling the plug on it.

Hi,

I’m sure we can get it to go quite a bit quicker than that. I would guess at least 100 values per second, up to 1000. However, I personally would probably do it outside of Ignition, directly with python… but with the transaction functions available now, it could probably also be done in the client.

As for table locking, I don’t think there will be an issue. I would recommend importing all of the data into it’s own “partition” (table) anyhow, so it won’t touch the tables that are currently in use. Do the tags in this system correspond directly to tags in the new system? That is, is it expected that you’ll be able to query “tag a” and get some data stored by sqltags history and some data imported from this system? I don’t really think it’s a problem either way, but it would affect how I suggest laying out the tables.

Regards,

yes, the data will correspond to tags currently in the system, although their may need to be some massaging of the data to do the correlation between tagnames in our system and tagnames in the previous system. Yes, we would like to be able to have the data show up in taghistory.

Ill post a sample of the history later on today.