# Local data caching for historical data logging **URL:** \ **Category:** Archived Topics\ **Created:** [April 2, 2006, 1:30am UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48 "2006-04-02T01:30:34Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![nathan](https://avatars.discourse-cdn.com/v4/letter/n/c5a1d2/32.png) [@nathan](https://forum.inductiveautomation.com/u/nathan)\ **Post date:** [April 2, 2006, 1:30am UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/1 "2006-04-02T01:30:34Z")
This a feature request announcement. We’re working on a feature that will allow FactorySQL to locally cache logged data when connection to the SQL database is lost. Upon reconnection, FactorySQL will write all the missed records into the database. ---
**Author:** ![admin1](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@admin1](https://forum.inductiveautomation.com/u/admin1)\ **Post date:** [October 25, 2006, 6:54am UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/2 "2006-10-25T06:54:57Z")
This feature is now implemented. ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [January 30, 2009, 9:46pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/3 "2009-01-30T21:46:45Z")
Can this feature be modified so that when the database connection is re-established, the cached values are sent to the database before any new values? In other words, once FactorySQL starts caching, new values should continue to be cached until the cache is empty. I have written a procedure that analyzes the database to determine the sequence of events that occurred in the process. When FactorySQL is “backfilling” after losing the database connection it intermingles current values with historical values and the analysis procedure doesn’t work correctly. ---
**Author:** ![Carl.Gould](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/carl.gould/32/7262_2.png) [@Carl.Gould](https://forum.inductiveautomation.com/u/Carl.Gould)\ **Post date:** [January 30, 2009, 10:26pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/4 "2009-01-30T22:26:43Z")
The timestamp should still be accurate, can your analysis use that to impose order on the records? ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [January 30, 2009, 10:35pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/5 "2009-01-30T22:35:48Z")
Carl’s right, but I would just add that you should almost always use an ORDER BY clause when selecting from the database. Without it, you’re simply assuming the database is going to return them in some meaningful order- which isn’t always true. MySQL seems be be nice and return them ordered on the primary key, but other databases don’t (and this could change with future versions of mysql, since nothing says it has to do this). So, making sure the t\_stamp column is indexed and then ordering on it is the best way to go in this case. I just wanted to point that out because it’s not always apparent, and it’s a big backwards compatibility/portability issue. Regards, ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [January 30, 2009, 10:37pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/6 "2009-01-30T22:37:06Z")
The timestamp is accurate and I am using an ORDER BY clause. But if my procedure analyzes data during a time that FactorySQL is in the process of “backfilling” there will be gaps in the historical record. The gaps won’t be filled in until the “backfilling” is complete. My procedure has no way of knowing when the “backfilling” processs is going on or when it is complete. If FactorySQL inserted the cached values in chronological order before inserting the new values, there would be no gaps in the historical record at any time. ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [January 30, 2009, 10:54pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/7 "2009-01-30T22:54:32Z")
Instead of changing the way that that data is “backfilled”, if you could provide a “backfill” indicator of some sort in the database then my procedure could be written so that it doesn’t try to analyze the data while the “backfill” process is active. ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [January 30, 2009, 11:16pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/8 "2009-01-30T23:16:11Z")
Ah, yes, that makes sense. We’ll have to look into what we can do, though a setting like you described that would cause it to load the data cache first doesn’t seem like it would be too difficult. Regards, ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [March 1, 2009, 2:23pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/9 "2009-03-01T14:23:20Z")
What is the status of this feature request? ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [July 1, 2009, 5:42pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/10 "2009-07-01T17:42:07Z")
Are you planning to implement some functionality to address this issue? If so, what is the schedule for that implementation? ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [July 1, 2009, 6:57pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/11 "2009-07-01T18:57:27Z")
Hi, Indeed, the next (major) version of the software will include a revamped caching system, which is now actually more of a store and forward system. All data will pass through the system, and new data will only be written once cached data has been exhausted. We’re currently targeting a release in early 4th quarter (though beta/test access will be available before that). Regards, ---
**Author:** ![larryjones](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/larryjones/32/4102_2.png) [@larryjones](https://forum.inductiveautomation.com/u/larryjones)\ **Post date:** [September 17, 2009, 3:36pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/12 "2009-09-17T15:36:01Z")
Is the new version still on schedule to be released “early 4th quarter” and is this feature still included? When exactly is “early 4th quarter”? Thank you ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [September 17, 2009, 5:07pm UTC](https://forum.inductiveautomation.com/t/local-data-caching-for-historical-data-logging/48/13 "2009-09-17T17:07:10Z")
Hi, The schedule is currently being described as “4th quarter”. We haven’t committed to a firm release date, but we are still confident that it will be in that time frame. We are taking a fairly non-committal approach on this because given the scope of the update, we really want to be sure that it’s ready to release. In regards to this feature- YES, it is there. The data caching system has been re-written and made into a store and forward system. Regards,