Script transform apply to whole table, not just visible part

Hello!

I´ve got a script transform to aply to a table so show me the text on UTF-8 format...

def transform(self, value, quality, timestamp):

 
    if not isinstance(value, list):
        return value  

    transformed_data = []
    
    for row in value:
        new_row = {}
        for key, val in row.items():
            # Check if the value is of type bytes and decode it
            if isinstance(val, bytes):
                try:
                    val = val.decode('utf-8')
                except UnicodeDecodeError:
                    val = str(val)  # Fallback if decoding fails
            new_row[key] = val
        transformed_data.append(new_row)
    
    return transformed_data

I just realize, it just show me formated the text loaded on the current table page, If I filter and load othr part of the table it doesn´t come formated.

Any solution for this issue you can think of?

Cheers!

Can you post a screenshot of what you're seeing that is different than expected?

1 Like

You can pass an errors parameter to decode, to tell it what to do if it can't decode some bytes.
In your case, I'd say errors='ignore' or maybe errors='replace'.

Put something like that in your lib (If you're using it for display purposes in a table, it's likely you'll use it in other places):

def decode_bytes(value, errors='ignore'):
    """Decode bytes to UTF-8 string with configurable error handling.
    
    Args:
        value: The value to decode (bytes or other type)  
        errors (str): How to handle decoding errors ('ignore', 'replace', 'strict')
        
    Returns:
        str: Decoded string if input was bytes, otherwise returns input unchanged
        
    Example:
        >>> decode_bytes(b'Hello World')
        'Hello World'
        >>> decode_bytes(b'Bad \xff bytes', errors='replace')  
        'Bad � bytes'  
        >>> decode_bytes('Already a string')
        'Already a string'
    """
    if isinstance(value, bytes):
        return value.decode('utf-8', errors=errors)
    else:
        return value

Then your transform can be just this:

def transform(self, value, quality, timestamp):
	if not isinstance(value, list):
		return value
	return [
		{
			k: some_lib.decode_bytes(v)
			for k, v in val.items()
		} for val in value
	]

I like making transform as simple as possible.

Now, your actual issue...
Does it still happen if you uncheck virtualized ?

2 Likes

Thanks @pascal.fragnoud

the thing is as follow

I have a data binding to a table with the last script transform like this to put all text on UTF-8 format:
image

image

def transform(self, value, quality, timestamp):
	
	
	if value is None:
	    return value  # No hacer nada si no hay datos
	
	try:
	    py_data = system.dataset.toPyDataSet(value)
	    headers = py_data.getColumnNames()
	    output_data = []
	
	    for row in py_data:
	        new_row = []
	        for col in headers:
	            val = row[col]
	            if isinstance(val, str):
	                try:
	                    val = val.encode('latin1').decode('utf-8')
	                except:
	                    pass
	            new_row.append(val)
	        output_data.append(new_row)
	
	    return system.dataset.toDataSet(headers, output_data)
	
	except Exception as e:
	    system.perspective.print("Transform error: " + str(e))
	    return value

Then when I open the page I see the table in good format:

but when I navigate to other part of the table it doesn´t show UTF-8 format:

This happen even when I aply the format to the datasource bia script transform

Also, why do you have multiple script transforms chained ?
You should probably be doing all your scripting in the same transform.
If it's too big or too complicated, use library functions.

2 Likes

Yes, I just tried with the virtualized unchecked and doesn´t work neither

I use multiple scripts sometimes when I have to modify something I did before, for me it makes the data transform more clear.

Maybe if I can cancel the pagination it could work.

If you move your initial data binding and transform somewhere else, like view.custom.data, then bind your table props.data to it, would there still be the same issue?

Yes, I tried to aply the script transform to a external custom.data property and have the same issue...a bit extrange

Is it formatted properly in the custom prop ?

Yes, in the custom prop everithing is formated properly
image

image

There is just the invalid key warning...

image

I think is related to use UTF-8 characters on named query with json return format

but without json return format I can´t aply the Expression Structure Binding to fister with checkboxes