Shutil copy taking too much time

General Discussion
1

Hi,
I am using the following code to export a CSV file. But it is not copying sometimes and running too long without exit and halt the whole tag change script even i cancelled the running query. It will back to normal only gateway restarts


def CreateRecordCSV(job, serial):
	import os.path
	import datetime
	import shutil
	localPath = 'R:/OracleIgnition/'
	remotePath = 'Z:/'
	timestamp = ''
	fileName = ''
	d = datetime.datetime.today()
	
	_params = {}
	_params['day'] = d.strftime('%d')
	_params['month'] = d.strftime('%b').upper()
	_params['year'] = d.strftime('%Y')
	_params['hour'] = d.strftime('%H')
	_params['minute'] = d.strftime('%M')
	_params['second'] = d.strftime('%S')	
	_params['mili'] = d.strftime('%f')[:3]
	_params['mon'] = d.strftime('%m')
	
	timestamp = _params['day'] + '-' + _params['month'] + '-' + _params['year'] + ' ' + _params['hour'] + ':' + _params['minute'] + ':' + _params['second']
	fileName = 'ignition_' + _params['year'] + _params['mon'] + _params['day'] + _params['hour'] + _params['minute'] + _params['second'] + _params['mili'] 
	
	import csv  
	data = [job, serial, timestamp]
	completeLocal = os.path.join(localPath, fileName+".csv")    
	with open(completeLocal, 'w+') as f:
		writer = csv.writer(f)
		writer.writerow(data)
		
#	Copy to Oracle server
	completeRemote = os.path.join(remotePath, fileName+".csv")	
	shutil.copy(completeLocal, completeRemote)

Could you please someone help me

2

Some general thoughts that might be helpful:

  1. You mention using a tag change script. If this is a tag event script you certainly shouldn't be initiating long-running actions (like file writes). A better place is in a gateway event tag change script.
  2. If there's more than one way to accomplish a task, you should pick the tools in this order.
    1. system.* functions. (Documentation)
    2. Java libraries
    3. Python libraries
    • In particular, I would [strongly] recommend performing your date logic something like this:
      now = system.date.now()
timestamp = system.date.format(now, 'dd-MM-yyyy HH:mm:ss')
fileNameDate = system.date.format(now, 'yyyyMMddHHmmssSSS')
fileName = r'ignition_{}.csv'.format(fileNameDate)
      and your file writes using the system.file.* functions.
      • The Jython implementation of the Python standard libraries is sometimes buggy. I wouldn't be surprised if shutil or os.path are running slowly and causing you problems where one of the system.file.* functions would perform much better.
2 Likes
3

If either of these are network drives, latency is going to be insanely variable. shutil.copy has no timeout parameter that I can see, so you're easily prone to ~infinite loops if the network drive hiccups or something else happens, even aside from the often buggy Jython standard library @zacht mentioned.

This is also a crazy complicated script to append a single line of text to a file.

1 Like