[FB12678-Investigating] Issues with upgrade

Ignition Early Access
21

What method did you use for installation (zip or GUI installer)?
Could we get a copy of the install_log.txt file from your Ignition directory on the Gateway?
The fact that there are only three modules present on your Gateway is a heavy indication that installation steps were not followed correctly, and so that's why we're double-checking your installation steps.

You claimed that you did a 7.9 install, then restore, then upgrade to 8.0,

but you also later claim you uninstalled the gateway,

Did you use the uninstaller, or did you delete files manually?

Could you please send a screenshot of the contents of your /lib/core/common/ directory from the Gateway? (this is due to you mentioning "deserialization")

Essentially, we need a crystal-clear breakdown of the steps you took to get in this position because somewhere along the line something happened to your environment which doesn't line up with the information you've provided so far.

Please clarify each step you took to get in this situation, and please be exact - we need to know:

  1. if you installed via .zip installer or the .run installer
  2. if you uninstalled by deleting files or via the uninstaller we provide
  3. when you restored your .gwbk
  4. when your Windows machine was able to successfully perform creating/modifying content (was that before or after you uninstalled and then re-installed?)
  5. Does your Windows machine still work with the environment?
  6. After performing the removal Garth suggested, is you MacBook now able to create/modify content?
22

I did delete my Mac’s ~/.ignition, reinstalled the Designer Launcher, and the problem remains. I don’t think it is locking or loading the views, because I can use a Windows PC to open and load the same View from it while the Mac pinwheels through its timeout. It seems clear that the issue is actually with the Mac, not with the Gateway.

The three modules are on purpose- we are not a typical SCADA and don’t use more than these anyway, except for Web Browser in Vision. I installed the 3 only in order to keep the test as clean as possible.

At this point, I would keep our 7.9 project out of the mix- I will worry about that when the first problem is solved. I am only trying to be able to edit Perspective on those basically blank projects.

I am sure I have confused you because I was basically trying different things in order to feed you relevant information. To answer your numbered questions:

  1. I used the RUN installer
  2. I uninstalled using the uninstaller via terminal
  3. At this time I can’t edit the gateway you have accessed; it was created fresh and I did not attempt to import my 7.9 project or gateway. What you see is just a clean install.
  4. I admit to not scientifically documenting that, it did not occur to me to sequentially try the Mac then the PC. At this time, with the clean install and no attempt to install my old content, I can edit with a W10 machine but cannot with my Mac.
  5. Yes
  6. No

Below is a screenshot of the directory you mentioned.

50%20PM
50%20PM2234×1702 422 KB

23

Nest step:

  1. What’s your MacBook OS version?
  2. On your MacBook, we’re going to run a complicated set of commands:
    2a. Navigate to ~/.ignition/clientlauncher-data
    2b. Run the following command: cat designerlauncher.log
    2c. In the resulting output, scroll up until you find a line that starts with “Starting Java with”, and copy everything between “nohup " and " &” to your clipboard (note the spaces - we don’t want them).
    2d. Paste the contents of your clipboard to the terminal window as your next command and output the result to a file by including " > DConsole.txt" at the end of the command (again, note the spaces - this time we want them).
    2e. Please send us the resulting DConsole.txt file.
  3. One further question for now: When you try to open a Designer on your MacBook, are you able to actually open a project, or does the Designer never actually open?
24

Mojave 10.14.2 (18C54)

The chunk of the output you asked for is this:

/Applications/Designer Launcher.app/Contents/Resources/jre/bin/java

-classpath /Users/boblee/.ignition/clientlauncher-data/launchclient.jar -Djavaws.sr.gateway.addr.0=http://go.fusion.cloud:8088 -Djavaws.sr.launchts=1549483798166 -Djavaws.sr.main=com.inductiveautomation.ignition.designer.DesignerStartupHook -Djavaws.sr.platform.edition= -Xms64M -Xmx1024M -Djavaws.sr.memory.init=64M -Djavaws.sr.memory.max=1024M -Xdock:name=Ignition-nuclear-1 -Xdock:icon="/Users/boblee/.ignition/clientlauncher-data/DesignerIcon.icns" -Dsun.java2d.d3d=false -Dsun.java2d.noddraw=false -Djavaws.sr.screen=0 -Djavaws.sr.scope=D com.inductiveautomation.ignition.client.launch.BootstrapSwing

When I execute this with or without the output to the file, I get a bash error "-bash: /Applications/Designer: No such file or directory". Do I need to put quotes around the highlighted part to get over that embedded space?

The Designer opens, and it does show in the tree the Perspective resources. It will, for instance, show a new View created by another Designer (on W10 machine). However, it fails when trying to open any of these, and when trying to save.

26

I ran it with the quotes and it started the Designer. Here is the output file.

DConsole.txt (9.7 KB)

27

Did the Designer this opened encounter those failures, or are you just informing me of past behavior? I ask because the provided file gives no indications of any issue - nor does it appear you actually opened any project. That’s probably my fault for not specifying, but we really need to do this and then trigger the issue.

Could you please delete the existing DConsole.txt file, then repeat the process and this time when the Designer opens, attempt to trigger the error you’re encountering. Once you’ve encountered it, then please shut down the Designer and the Designer Launcher, and then please send the new DConsole.txt file.

28

This Designer also fails to open or save Perspective resources. I did in fact open a project too.

I will do it again.

29

I opened the Designer using the script writing to the text file attached.
In Designer I opened the " QAVerifyOne" project you created. It opened fine.
I double clicked on the View “CreatedOnW10” which I had created from my W10 machine. FYI, Designer is not running on that W10 machine.
It went to pinwheel for the usual few minutes, gave same error as I have been seeing.
I clicked the close button (top left in Mac) It asked if I wanted to Save. I pressed Save and Exit. A Dialog came up saying that it wanted to save modifications to Session Props. On this dialog I hit save, and the (failed) View seemed to close, and I was back to looking at the Designer with the pages screen showing. Again I hit close, again was given the option to save and exit. I again chose Save and Exit. I again got the dialog saying it was saving changes to Session Props. It briefly showed a dialog that said it was Updating Project, then closed. Attached is the file. FYI I was in my home dir in the Terminal when I ran this, I doubt that matters but I figured I would point it out.

DConsole.txt (9.7 KB)

Would Designer activity actually get logged by the Designer starter? It looks the same as the last dump.

30

Look at the “Running:” line, second to last in your log. That’s the command that gets run by the launcher to start the designer itself. Extract the args from that list and run it directly in your terminal.

31

@NorthPoint Can we try a ‘nuclear’ start from scratch?

  1. Delete your ~/.ignition/cache/ directory entirely.
  2. Shut down and restart your development machine.
  3. Once it’s back up, verify there are no chromium or related processes running (check ps in terminal)
  4. Try to launch the designer via the Designer Launcher again, and copy the logs out of the output console within the designer.

Based on the last log you sent Cody, there appears to be an issue with JxBrowser/Chromium sessions getting created and not cleared up, but we’re not able to replicate it in house. Restarting your machine should resolve it, although obviously that may remain to be seen.

32

I think we tried that before, regardless I did it again, with the same result.

33

@NorthPoint: could you send one of us the resulting file?

34

Sorry guys, I am distracted, as usual. I will try to slow down and read your requests more carefully, I really appreciate your attention to this. This has been happening literally since V8 beta was released. Frankly, I thought “wow, V8 has a lot of issues”, and dropped it for a long time. Returning now, I still have the issue but clearly now it is specifically me, or my laptop. I am disappointed at the loss of development and testing time, I could have been working on it all along on a different PC…

I repeated the test, because I had already closed Designer.

Attached is a file containing the output from the Designer console. Visually the error looks the same as it always has- attempt to open a View, pinwheel of death for a few minutes, error popups.

output from Designer Console.txt (8.1 KB)

35

We’re going to try something different while we wait on others…

  1. Open your Designer Launcher and click the vertical ellipsis for your Gateway, then select “Manage”.
  2. In the JVM Arguments area, supply the following argument : -Djxbrowser.chromium.dir=Chromium2
  3. Save your changes.
  4. Launch the designer

If the same issue occurs please do the following:

  1. Navigate to ~/.ignition/clientlauncher-data in your terminal
  2. Run the following command: cat designerlauncher.log
  3. In the resulting output, scroll up from the bottom until you find the first line that starts with “Starting Java with”, and copy everything between “nohup " and " &” to your clipboard (note the spaces - we don’t want them).
  4. Paste the contents of your clipboard to the terminal window as your next command and output the result to a file by including " > Chromium2Console.txt" at the end of the command (again, note the spaces - this time we want them).
  5. Open a Project.
  6. Attempt to open a View.
  7. Close the Designer
  8. Please send us the resulting Chromium2Console.txt file
36

Sorry I somehow missed this yesterday. Below is the resulting file. Just to be clear, I have to add quotes around the path and java command in the console to get it to ignore the space.Chromium2Console.txt (9.7 KB)

37

It looks like you cleared out your cache again because the file you posted contains logging around downloading the required runtime and then shortly after that it launches a JVM with the downloaded runtime, at which point the logging moves to the new JVM and we stop seeing what’s happening with your environment.

Sometimes we want that cache cleared and sometimes we don’t - right now we’re trying to see how your environment behaves with 11.0.1 and chromium2. For us to see that, the 11.0.1 runtime must already have been downloaded and cached.

Your environment should now have the 11.0.1 JRE downloaded and cached. Please delete the Chromium2Console.txt file and run through the prescribed steps again. Do NOT clear out the .ignition folder. When you have completed the steps, please send us the Chromium2Console.txt file.

38

I am really confused- I haven’t cleared the cache. Also, when I run the cat designerlauncher.log, I get a file that is nearly 3000 lines long. The last "starting java with " is from yesterday afternoon. Is that what you want? In between that “starting java with” and the end of the file are thousands of lines of errors starting with "IOException thrown when attempting to receive a socket message:

java.net.SocketTimeoutException: Receive timed out".

I just did the following:

a) opened designer launcher from the GUI, opened a project in designer, tried to open a View, let it fail, then closed the view and the designer.

b) This time at the bottom of the socket errors there was a “starting java with”
c) ran this to file as requested, opened a project and attempted a view open, which failed as usual.
d) resulting file attached

Chromium2Console.txt (37.8 KB)

39

We’re still examining the issue, but the developer best-suited to getting to the bottom of this issue is out today. We are becoming increasingly confident that this issue does indeed stem from your personal machine as more time goes by without anyone else in the Beta having encountered the issue and the fact that none of us in-house have been able to replicate it.

Continue using your Windows machine until we can investigate further and if we haven’t gotten back to you by Tuesday of next week please reach out to me again.

40

Hi Northpoint,

I’ve added some flags to perspective in an effort to try and identify what you are seeing. I’d like to do three things to start.

  1. enable deeper logging
  2. use in-memory cache
  3. increase designer memory

Follow the instructions below, doing each step in order, and trying to launch the designer (and collect logs) in between each attempt. By the end, you’ll want all 3 changes enabled/set when you launch.

Instructions

First, we’re going to enable some direct logging to disk from the chromium engine by setting a new flag we’ve added that will stream output from the internal browser engine directly to log files on the machine. The hope here is that we can collect logging information earlier in the startup process. Also, in setting this flag, we set all browser loggers to log all messages.

To enable this mode, we simply need to add a flag to our designer launcher in the “Manage” section of the launcher config

image

Once there, in the JVM Arguments field, add the line -Dperspective.browser.logToDisk=true and without making additional changes, save the changes, and try launching the designer.

On startup, a successfully launched designer will now have a log message that looks like 08:40:37.202 [Designer-Startup] INFO Perspective.Designer - JxBrowser engine logging to file system at /Users/someuser/.ignition/cache/some.host.com_29980/C1/perspective/log/jxbrowser

If you navigate to that folder, you’ll find 3 binary log files named:

  • jxbrowser-browser.log
  • jxbrowser-ipc.log
  • jxbrowser-process.log

If you don’t get far enough in startup to view the designer logs, simply run a sudo find / -name "jxbrowser-browser.log" and you should find the appropriate cache directory.

These files are viewable in the standard mac log viewer, go ahead and send them as well as any additional info/log files to us.

With that step done (and assuming no change), shutdown the designer and we’ll add another flag:

-Dperspective.browser.cache.useMemory=true

This flag puts the internal pages into an ‘incognito’ mode, which keeps all cacheing in memory rather than writing to disk. In the designer, this has some minor memory and performance implications, but the tradeoff is reduced disk IO (and the potential read/write/permission errors that can originate from file IO).

After adding this flag, go ahead and save and try launching the designer again without any further changes.

If you see the same behavior, grab the jxbrowser logs again, as well as designer logs and send them our way.

After that step, let’s increase the memory of the designer by changing the Max Heap value in the designer launcher management screen. Set it to somewhere between 2048M and 4096M and try launching again, just to rule out memory issues.

Again, collect the log files and any additional info you are able to provide and send it over.

Let’s see where that gets us, and we’ll find some next steps depending on the outcome.

-Perry

1 Like
41

Summary instead of separate emails:

  1. Added new JVM arg to log to disk. Designer launched, no change in behavior. Files attachedjxbrowser-browser.log (1.4 KB)
    jxbrowser-ipc.log (2.0 MB)
    . The third file (process) was zero bytes.

  2. Added the JVM arg to use memory, same result. Files attached with “mem” prefix. Again the process file was zero bytes, and again it also had left open a lock file (jxbrowser-ipc.log.lck), also zero bytes.
    mem jxbrowser-browser.log (2.2 KB)
    mem jxbrowser-ipc.log (1.9 MB)

  3. Changed max heap to 2048M, tried again, same result. Files attached with “third” prefix.
    third jxbrowser-browser.log (2.2 KB)
    third jxbrowser-ipc.log (2.0 MB)