On my searching the manual and forums I’ve not seen not seen any question or answer about why the disk cache is defined in terms of records and not disk size eg. 1 gig. I assume that the record size might be variable so the 25000 max records might be a varied amount of disk space. So how does one determine the actual disk space required by the record setting?
Also in experimenting with VM’s it seems that the disk cache just keeps growing in size unless you restart the machine which seems to clear the file. So is there any garbage collection that reclaims the disk space without having to restart the machine?
I think it’s measured that way mostly because it’s easier for the system to book keep. It’s probably not the most intuitive, but then on the other hand, people usually aren’t too concerned about disk space (as long as it doesn’t grow too big!). There probably should be some sort of facility for limiting the disk size, as well.
In regards to reclaiming space: this requires an expensive compacting operation, so it is only performed on system shutdown. Assuming the size limits work, reclaiming space while running probably isn’t a good idea, because it will likely just need to grow back to the same size again- and growing the file causes operations to take longer.
What type of environment are you trying to run under? Sounds like you have fairly limited resources… is it an embedded type system? I’m just curious, because for some time now we’ve been looking at modifying the disk cache system to work better for these types of environments (primarily by switching to sqlite, or perhaps going about it a different way). Feedback on stuff like this helps us prioritize.
“people usually aren’t too concerned about disk space (as long as it doesn’t grow too big!).”
That’s what I’m concerned about. Using a remote system doing data collection over a somewhat unreliable link it seems that this file could grow big enough to fill up a hard drive. Not all systems will have massive disk space on a remote system. This would create an extra maintenance headache to have to restart program to recover disk space.
Seems something like the linux log file rotation could work here. Write to a file as required and when link comes up and store and forward catches up close file and open new file for next time link is lost. Maintain a limit of X number of files then start deleting files over the X limit.
Yeah, I figured you were working on an embedded system. We can definitely look at ways to optimize the cache for this. I’m afraid it might be a lot of work to switch over to a file based system instead of a db based one, but if nothing else we could probably have it compact the files after big transfers. Anyhow, we’re open to all options, though it’s going to be a few months before we get around to this (but just so you know, we have other customers who need this improved as well).
I’m also curious to know how number of records translates into disk space.
In a test environment, I recently had a database hard drive fill up and the maximum 50,000 records got quarantined with error “The table ‘sqlt_data_XXX’ is full”.
This is 3 second data so it reached the max within 4 days. I want to know if I can safely double or triple the 50,000 so that at least a week of data is cached on the gateway. How much disk space would 150,000 records take up? Is there a way to check the disk space of the current 50,000 record cache?
Hope you all don’t mind me piggy-backing my question on this, it’s also about store & forward.
I also had a database fill up and then the cache filled up too (200k+ records) After removing a lot of columns from the transaction groups’ tables, the cache is taking an excessively long time (stopwatch estimate is ~10.4 days) to empty out, meanwhile new data isn’t accessible.
Looking at the console, every record is erroring due to the missing columns; that’s probably what’s taking so long so I’m wondering if there’s a way to clear the cache or at least tell store & forward to ignore errors; reenabling the cache or shrinking the size has no effect. Currently I’ve disabled it so new data is getting stored.
Version is 7.6.6
Data that cannot be stored generally goes into quarantine. You can delete any quarantined data from the gateway web page. Go to Configure / Databases / Store and Forward and click on the Quarantine Control tab. You should then be able to click ‘Delete’ next to any quarantine store you want to empty. Remember this throws away all the data - you will have no way to recover it afterwards.
If the data is already in the cache and the inserts are failing because of changes to the database, either reinstate the columns in the database and delete them again once all the data has been processed, or restart the gateway.