Looking for a way to import csv values without changing certain characters like "°"

Ignition
,
1

I've tried changing a few different things but nothing's working. I changed the format that the Excel VBA is using when saving the CSV file but then that added quote marks in the CSV which I couldn't get ignition to deal with.

On the ignition side I've tried changing the encoding to a few different things but the best I can do is change the character I end up with from "�" to "�"

I'm sure I'm missing something simple but I don't know why it's so hard to just give me the character that's actually in the CSV file?

			try:
				# This is the direct implementation of your suggestion:
				# Use io.open with 'utf-8-sig' to handle the BOM
				f = io.open(filePath, 'r', encoding='utf-8-sig')
				REST.Utilities.Tag_Functions_logger.info("Opened CSV file with utf-8-sig encoding: {}".format(filePath))
			except Exception as e:
				REST.Utilities.Tag_Functions_logger.error("Error opening file with utf-8-sig: {}".format(e))
				REST.Utilities.Tag_Functions_logger.error("Traceback: {}".format(traceback.format_exc()))
				REST.Utilities.Tag_Functions_logger.error("Attempting fallbacks (cp1252, latin-1) if utf-8-sig fails.")
				
				# --- Fallback if io.open with utf-8-sig explicitly fails ---
				# This block is what my previous solution focused on more
				opened_successfully_fallback = False
				for encoding_attempt in ['cp1252', 'latin-1']: # Removed utf-8 as utf-8-sig failed already
					try:
						f = codecs.open(filePath, 'r', encoding=encoding_attempt) # codecs.open can be more reliable for fallbacks in Jython 2
						f.read(1024) # Force a read to test encoding
						f.seek(0)
						REST.Utilities.Tag_Functions_logger.info("Opened CSV file with {} encoding: {}".format(encoding_attempt, filePath))
						opened_successfully_fallback = True
						break
					except UnicodeDecodeError as e_fallback:
						REST.Utilities.Tag_Functions_logger.error("Failed to decode CSV with {}. Error: {}. Trying next encoding.".format(encoding_attempt, e_fallback))
					except Exception as e_fallback_gen:
						REST.Utilities.Tag_Functions_logger.error("An unexpected error during fallback with {}: {}".format(encoding_attempt, e_fallback_gen))
						REST.Utilities.Tag_Functions_logger.error("Traceback: {}".format(traceback.format_exc()))
				
				if not opened_successfully_fallback:
					REST.Utilities.Tag_Functions_logger.error("Could not open CSV file with any of the attempted encodings. Returning.")
					return
2

The file is probably encoded in 8859-1, not utf8.

I'd be using java libraries, not jython, when dealing with encodings.

Start with StandardCharsets.

1 Like
3

Notepad++ says it's ANSI and I tried changing it to UTF-8. That is when it added quotes around each of the values and I was having trouble finding a way to deal with that. CSVs and Excel always seem so squirrely, like when you open a CSV and Excel immediately tries to reformat some of your values and so on.

Thanks for the info and I'll see if I can make sense of standard character sets. :slight_smile:

4

Consider reading the source excel file with the Apache POI libraries instead of passing through CSV. CSV is a lossy format.

1 Like