Hello! Our database is nearing 16TB (in about 3-4 months), and we need to pull some data out into cold storage. We must retain the ability to query and possibly even trend this data in the future. I'm assuming we could set up a cold storage-only gateway for this purpose.
Does anyone have any pointers or suggestions on how we can achieve this?
There are a number of tools in various databases that have native partitioning support that can seamlessly query different kinds of inner tables. And then you can use a number of techniques to transform old partitions into read-only, highly compressed tables. (I've played with, but not deployed, the PostgreSQL column-store archive table type for just this purpose. Lots of interesting capabilities mixed into the Foreign Data Wrappers technology.)
But whatever brand of DB you are using, investigate its native partitioning capabilities.
(Time-series data that has been re-organized into grouped data under the covers can often be losslessly compressed 10x or more.)
It isn't an accident that Amazon's Redshift, Google's AlloyDB, and Timescale are all built on top of PostgreSQL technology. Microsoft SQL Server is feel-good technology for IT dinosaurs, IMNSHO.