Hello, I have a script to open a log file, search for a string, and if found insert the string into a database.
There is a log file generated per day of the month for each test cell. There are 16 test cells. Each log file holds around 650K results but I am not interested in a lot of them. I am currently experiencing a bottle neck however whenever I exceed 4 search strings or 10 test cells which results in ~38K db inserts.
If I search 2 strings in 3 cells it takes 15 seconds and generates 4.6k records.
3 in 3 takes 22 secs with 9.2k results.
4 in 3 takes 30s with 13k results
4 in 8 takes 60s with 28k results
For any of these looking at performance in task manager the 8 threads are happily ticking over peaking at maybe 15%.
Once I exceed 38K results the 8 threads go to 95% and I have to kill the j2plauncher process, no error messages are displayed.
Anyway, here’s the code as it stands, many thanks to althepal for getting me started on this;
cells = range(153,162) day = "01" string_1 = "NOREAD" string_2 = "cu to rotate to slide" string_3 = "center unit to rotate to robot" string_4 = "closing gripper 2" strings = [string_1, string_2, string_3, string_4] for cell in cells: filepath = "\\\selcfts" + str(cell) + "\\SEL\\Logs\\" + day + "_Handling.log" for string in strings: file = open(filepath) for line in file: if line.find(string) != -1: system.db.runPrepUpdate("INSERT INTO logdata (time, string, cell, date) VALUES (?,?,?,?)", [line[:12], string, cell, "01/02/2016"]) file.close()
There is a problem with 152’s network path, I can’t remote browse to it. Tried taking it off and back on the domain which fixed it for 30 mins but then reverted to non-browseable. The end goal is cells in range 150-165 on this site. I need to add some form of error handling, because if I do range(150,166) it fails on 152 and therefore doesn’t do any more. There is a chance that when this script runs at midnight every day that one of the cells could be powered down. Not sure how to go about this?
I have increase the max ram in the .conf file to 8GB. Are there any other reasons why I can’t exceed 38k entries? The sql server is MS SQL MS 2012.