Hey,
I am investigating the Translation Manager in Perspective. I am trying to find a way to synchronize projects with each other and the import functionality caught my attention.
When it comes to large files (70.000+) the import time seems to increase per new key.
I tested approaches like the Translation Manager import function, TranslationPackageDiff (with different chunk sizes) or the system.util.modifyTranslation function.
When I used the TranslationPackageDiff function, I was able to test the runtime and I discovered that the import time per 10.000 keys increases the more keys are stored in the Translation Manager. I assume that each key is checked for singularity(?).
Is there a more performant way to import language packages?
Here is some data to show what I mean 
| Time |
Difference [s] |
Key Count |
| 13:34:50 |
00:00:36 |
10000 |
| 13:35:26 |
00:00:47 |
20000 |
| 13:36:13 |
00:01:14 |
30000 |
| 13:37:27 |
00:01:35 |
40000 |
| 13:39:02 |
00:02:15 |
50000 |
| 13:41:17 |
00:02:40 |
60000 |
| 13:43:57 |
00:03:08 |
70000 |
| 13:47:05 |
00:03:40 |
80000 |
| 13:50:45 |
00:04:09 |
90000 |
| 13:54:54 |
00:04:39 |
100000 |
The import function uses the TranslationPackageDiff and every 10.000 keys I run the applyDiff function.
Kind regards
Janis
What version of Ignition are you using?
Also, all the reads and writes to the database - which seems to be SQLite? - seems to disturb the entire system. Not performance wise as RAM and CPU utility does not noticeably increase.
I am also curious because the export of the same amount of files happens within a second. So the issue does not seem to be the amount of data that must be processed.
Note: This picture is taken from another device with redundance enabled. Seems like this makes more issues than a non redundant gateway as both GW try to access the SQLite db.
Right. So, this is among the reasons 8.3 moves away from a single "internal database" backed by SQLite in favor of a bunch of configuration files on disk. Unfortunately, that means there's not really much you can do about performance/these problems while you're on 8.1.
In 8.3, translations are modeled on disk as a series of files containing JSON, e.g.
en_US.json:
{
"someKey": "some translations in en_US"
}
and so on for other locales, e.g. de_DE.json, based on their BCP-47 language tag.
The big gain is in the way translation data is stored on disk and read into memory - the actual way data is dealt with once in memory is essentially the same as 8.1. So I would expect an upgrade to get you some performance more or less for free, plus improved VCS compatibility.
And drop the entire category of "SingleConnectionDatasource" errors.
In 2027.2, our next major version, things improve another step - performance throughout the system should be a lot better, and you'll be able to directly import and export translations in XLSX or that new JSON format, plus some new scripting functions to work with translations in bulk.
Thank you for your answer. This helps me a lot!
