Unable to Close FileChannel, opened with NIO

Ignition
,
1

I’m running a script where I want to get byte channel from file. Everything works fine, “apparently”. because when ever I try to delete the file I’m getting this

image

Of course I’m closing the channel

path = Paths.get("C:\\file.xlsx")
byteChannel = Files.newByteChannel(path, StandardOpenOption.READ)
byteBuffer = byteChannel.map(FileChannel.MapMode.READ_ONLY, 0, byteChannel.size())
byteChannel.close()

I’m able to delete the file only after closing the designer.

Does Ignition fully supports nio package?
Any Java Pros Advices are appreciated!

2

Try MapMode.PRIVATE instead?

3

Got a little hope that your advice might work, but

java.nio.channels.NonWritableChannelException: java.nio.channels.NonWritableChannelException
4

Also tried to close the channel from io.RandomAccessFile. Same behavior.
When doing the mapping with MapMode.PRIVATE. Same exception.

5

Do you have to use a memory mapped file for some reason?

These work in a OS-dependent manner and the memory/file isn’t released until the buffer is GC’d. Your channel close is meaningless because you still have a buffer pointing to memory the file has been mapped to.

1 Like
6

I was afraid to hear that.

The reason is because is the most efficient method to compare objects.

byteBuffer1.equals(byteBuffer2)

The idea is to determine if content of files are the same, beyond the comparison of file names. Trying to collect bytes has became challenging for xlsx big files, about 100Mb.

I was using readAllBytes() but from time to time I was getting OutOfMemory Exceptions.

Any advice you can provide?

7

Only briefly sanity checked:

    private static boolean compare(String f1, String f2) throws IOException {
        byte[] bs1 = new byte[1024];
        byte[] bs2 = new byte[1024];

        try (FileInputStream fis1 = new FileInputStream(f1);
             FileInputStream fis2 = new FileInputStream(f2)) {

            while (true) {
                int r1 = fis1.read(bs1);
                int r2 = fis2.read(bs2);
                if (r1 != r2) {
                    return false;
                }
                if (r1 == -1) {
                    return true;
                }
                if (!Arrays.equals(bs1, 0, r1, bs2, 0, r2)) {
                    return false;
                }
            }
        }
    }

You’d want to maybe play with the buffer size.

1 Like
8

Let me translate it to Jython and give it a test.

I also tried FileInputStream() but the reason why I stopped using it is because nio.Files isReabled() is saying False. Even after closing the stream.

But, also, I’ll retry.

Any chance to notify the GC to trigger manually?

Thanks for the knowledge!

9

Not reliably and you’d have to trigger it in a separate script invocation after your first had gone out of scope.

Using memory mapped files just so you can do a comparison is over kill.

10

Right, I was delighted by the speed of mapped files using this approach.

Just an additional question, How much time would take the GC to act over these files. About 20 mins according to this graph maybe? or that is not even related?

image

11

Maybe, but there could be smaller collections happening in between the big ones that don’t register on the graph. You’d have to turn on GC logging to see when it’s actually happening. And even then it would vary depending on what the gateway is up to.

You can request a GC be done by calling System.gc().

I’m curious what the performance difference between the regular code with varying buffer sizes and the memory mapped files is.

edit: comparing two equal 105MB files (worst case is when they are equal) with varying buffer sizes:

true
bufferSize=1024 took 1015ms
true
bufferSize=2048 took 456ms
true
bufferSize=4096 took 261ms
true
bufferSize=8192 took 140ms
true
bufferSize=16384 took 83ms
true
bufferSize=32768 took 54ms
true
bufferSize=65536 took 38ms
true
bufferSize=131072 took 31ms
true
bufferSize=262144 took 30ms
true
bufferSize=524288 took 36ms
true
bufferSize=1048576 took 85ms
2 Likes
12

Shouldn't be an statement fo r r2 != -1 as well, like below?

Good, the script works but I'm having way worst performance compared to your results (buffer = 262144). Way worst, like 30 seconds hehe.
Increasing the buffer to 2^23 did not make a huge difference.

Running from script console, designer with 4GB memory

13

Kevin was almost certainly running from a local SSD, as his time implies a drive sequential rate of >315MB/sec.

(The extra comparison isn’t needed because r1 and r2 are known to be equal at that point.)

(Hmm. Kevin, did you flush your drive cache between runs?)

1 Like
14

No, didn't do anything like that. I did a comparison against memory mapped files as well right after:

memory mapped file comparison took 49ms

I am of course using a SSD.

And this was done on Windows, which is pretty shit compared to Linux or macOS when it comes to I/O performance, in my experience.

2 Likes
15

Oh ok, you’re right.

Am doing this also in a SSD, but Windows. Can result vary that much based on OS?

Are you getting non satisfactorily results maybe?
Assuming you also did testing.

16

If you are running on any java interpreter different of Ignition and on any OS different from Windows.
Those seems to be biggest variables for me.

May I ask, Do you have dates cells, bools, long and formulas on your excel files?
Your PC specs maybe…

17

I ran it using JDK 11 on Windows 10. Not through Jython. I didn’t use an Excel file, the contents of the shouldn’t matter.

image

1 Like
18

Mine is fast in Jython code too… maybe you did something slow in your translation? Post the code you ended up with?

19

mmm, this are my specs for now

image
image726×22 1.47 KB

8 RAM

That is the code, do you see something maybe I don’t

then = system.date.now()
buffer = bytearray(2**23)
path1 = Paths.get("C:\\fileA.xlsx")
path2 = Paths.get("C:\\fileB.xlsx")
fileInputStream1 = FileInputStream(path1.toFile())
fileInputStream2 = FileInputStream(path2.toFile())
same = False
while True:
	read1 = fileInputStream1.read(buffer)
	read2 = fileInputStream2.read(buffer)
	if read1 != read2:
		break
	if read1 == -1:
		same = True
		break
	if not Arrays.equals(buffer, 0, read1, buffer, 0, read2):
		break
		
fileInputStream1.close()
fileInputStream2.close()
now = system.date.now()
print same
print system.date.millisBetween(then,now),'millis'
20

Use jarray instead of bytearray:

buffer = jarray.zeros(262144, "b")

and then fix the fundamental flaw in your code where you read the files into the same buffer.

1 Like