Popen with ping doesn't work in a Docker container

This is related to Using 3rd python library: pyping
I can't get ping to work with popen as of Ignition 8.1. Kind of...

In the script tool on 8.1 or 8.1.5 (pulled from Docker), I have:

from subprocess import Popen, PIPE
entry = Addresses[1]
args = ["ping", "-n","1", "-w","300", "192.168.1.122"] 
print(str(args))
p = Popen(args, stdout=PIPE, stderr=PIPE, stdin=PIPE)

output = p.stdout.read()
#print("output = " + output)
err = p.stderr.read()
#print("err = " + err)
Strout = output.decode('UTF-8')
print Strout

This works just fine. However, if I place a button on a Perspective view, and have this code:

logger = system.util.getLogger(WhosHomeLogger)|
from subprocess import Popen, PIPE|
entry = Addresses[0]|
args = [ping, -n,1, -w,300, "192.168.1.122" ] 
args = [ls,gwcmd.sh]|
args = [ping,192.168.1.122]|

logger.info(str(args))|
# p = Popen(args  , stdout=PIPE) # , stderr=PIPE, stdin=PIPE)|
p = Popen(ping -n 1 www.google.com,stdout = PIPE)|
output,err = p.communicate()|
logger.info(pingoutput =  + output)|

and I get the following on the logger:

Error running action 'component.onActionPerformed' on View@C/root/Button: Traceback (most recent call last): File "function:runAction", line 20, in runAction File "/usr/local/share/ignition/user-lib/pylib/subprocess.py", line 859, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/share/ignition/user-lib/pylib/subprocess.py", line 1369, in _execute_child raise OSError(errno.ENOENT, os.strerror(errno.ENOENT)) OSError: [Errno 2] No such file or directory

This works just fine, however, in both script tool and on a button, in 8.0.12 (native, not in Docker)

I don't see any differences between the two subprocess.py. I'm sure the fact that it works in the scripting tool and not in Perspective is a significant clue, but not one that does anything for me.

After doing a little more testing, I ran this under 8.0.17 and 8.0 under Docker (using kcollins’ images) and got the same thing. So it seems like it is a container vs non-container issue.

Works fine in script tool, but not in Perspective.

You can see in the code I also tried

args = [ls,gwcmd.sh]

and that worked, at least on the image I was using last night. I can’t remember what iamge that was. But I tried the same thing just now on Docker image 8.0, and ls gwcmd.sh gave same No such file or directory error

You are violating the spirit of containers by using popen and friends. Containers usually don’t have any other executables in their images, deliberately, to keep tools away from any hackers that compromise the main process.

And for ping, you don’t need to execute anything outside Ignition. Use .isReachable() on a java.net.InetAddress instead.

2 Likes

Also, watch out for how you are using strings and lists/arrays. You have two VERY different scripts in your original post. I'm not surprised the second one didn't work for many reasons.

The above line is not equivalent to

p = Popen(["ping", "-n", "1", "www.google.com"], stdout=PIPE)

Same goes for this:

Lastly, probably not a big worry for you, but if you do use ping, be careful as the arguments are not the same for every OS (i.e. - unix-based OS systems use -c instead of -n, etc). @pturmel's .isReachable() is a lot cleaner here.

@WillMT10 thanks for the catch. I know better, but apparently still screwed it up, and then published it! When I correct the string and put it into an array I still have the problem.

@pturmel thanks for the input, in fact when I open a bash shell for the Ignition instances ping doesn’t exist, that is probably the problem! So I tried to use .isReachable()

Looking back in threads, I found this so I implemented it:

def ping(address,timeout=2000):
	from java.net import InetAddress
	ip = InetAddress.getByName(address)
	return ip.isReachable(timeout)

When I run this in a container, it always comes back false. When run non-container, it works correctly.

I’m running following to start the container:

docker run -p 8092:8088 --name my-ignition -e GATEWAY_ADMIN_PASSWORD=password -e IGNITION_EDITION=full -d kcollins/ignition:8.0.17

Thanks for the help everyone!

Interesting. @kcollins1, do your containers run Ignition as root? If not, that could explain the java.net failure.

1 Like

Currently, kcollins/ignition does not run as root… inductiveautomation/ignition does at the moment, but will not in the future (strategy still TBD, though I’m leaning towards entrypoint as root and de-escalate to standard user for gateway launch)…

@ken.maze, I’ll take a quick look at this for ya…

EDIT: I read through the rest of the thread and I’ll mention that you’d have to install inetutils-ping package in the kcollins/ignition image to be able to use the ping program…

2 Likes

You learn something new everyday it seems… I didn’t realize that ICMP on Linux required extra privileges. Looks like /bin/ping is usually suid root, which explains why you can ping www.google.com as a normal user. But the Ignition process will not be able to use the raw socket form (see this thread for a good explanation: linux - Which Privilege Should be Obtained for ICMP Request at Java? - Stack Overflow) and thus likely return false (because port 7/tcp is not likely open on the target host).

Another point for @pturmel:smiley:

2 Likes

Consider running as root in the container so ports 80 and 443 are automatically allowed.

It would be worth experimenting to see if CAP_NET_RAW tricks java into using ICMP as non-root.

Then, with CAP_NET_BIND_SERVICE, all might work without root.

That’s a good call. I was envisioning a routing problem with docker and the target having overlapping subnets, I hadn’t even considered the possibility that ICMP wasn’t available at all by default. I’ll have to keep that in mind for Docker deployment in general.

I remember reading somewhere that you can’t do ICMP without root. This is for my home maker project that I am switching from node-red to Ignition since I’m using Ignition at work, and since there is so much mu UI support. So I guess for now I just won’t run it in a container until if/when this gets worked out. Thanks for all the input on this!

BTW I tried the ping function mentioned above in the script tool, and that works, just like the popen works. Is that jython running as root?

I changed the title of this to more accurately reflect what it is addressing.

:astonished: ... Let's not start being extreme here.. :laughing:

I looked into this a little and was hopeful that the Docker run args of --cap-add=NET_RAW would be enough to provide the right privileges. I also tried --sysctl "net.ipv4.ping_group_range=999 999" with no success.. It appears that the java executable itself needs to be flagged for this capability.

This is a situation where you can build your own Docker image. Thankfully you can inherit from other images and only add the things you want. The Dockerfile below will enable the InetAddress.isResolvable() calls to use the raw socket ICMP option:

ARG IGNITION_IMAGE_VERSION=8.1.6
FROM kcollins/ignition:${IGNITION_IMAGE_VERSION}

# Ideally the retrieval of libcap2-bin would be done in the first stage of a multi-stage
# build so that we don't have those tools hanging around in the final image.
USER root
RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        libcap2-bin && \
    rm -rf /var/lib/apt/lists/*

# ... we'd then just flag the `java` executable with the capability in that first stage,
# and then only have to add the ldconfig steps for libjli.so to the final stage.
RUN setcap cap_net_raw+ep ${IGNITION_INSTALL_LOCATION}/lib/runtime/jre/bin/java && \
    echo ${IGNITION_INSTALL_LOCATION}/lib/runtime/jre/lib/jli > /etc/ld.so.conf.d/java-ignition.conf && \
    ldconfig

USER ignition

Here is a quick demo of how to build this image and use it yourself:

asciicast

2 Likes

Wow, thanks for this, I will take a look at it and let you know how it goes.

OK, I have tried this a couple of times. I have had problems with the build.
I get:

pi@raspberrypi:/usr/local/dockerTrial $ sudo docker build -t kcollins/ignition:8.1.6-netraw --build-arg IGNITION_IMAGE_VERSION=8.1.5 .
Sending build context to Docker daemon  3.584kB
Step 1/6 : ARG IGNITION_IMAGE_VERSION=8.1.5
Step 2/6 : FROM kcollins/ignition:${IGNITION_IMAGE_VERSION}
 ---> 6e57c1e763ad
Step 3/6 : USER root
 ---> Using cache
 ---> f6dc452156aa
Step 4/6 : RUN apt-get update &&    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends         libcap2-bin &&     rm -rf /var/lib/apt/lists/*
 ---> Running in 2a4ff2285a7a
Get:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [114 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease [101 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [114 kB]
Err:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease
  At least one invalid signature was encountered.
Err:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease
  At least one invalid signature was encountered.
Err:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease
  At least one invalid signature was encountered.
Err:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease
  At least one invalid signature was encountered.
Reading package lists...
W: GPG error: http://ports.ubuntu.com/ubuntu-ports focal InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports focal InRelease' is not signed.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease' is not signed.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease' is not signed.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports focal-security InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports focal-security InRelease' is not signed.
The command '/bin/sh -c apt-get update &&    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends

This is apparently a known ubuntu bug, that hopefully i will get the time to address someday. Most of the suggested solutions were kernel rebuilds, and I’m not nearly linux-adept enough for that.

So, I took out the apt-get update just to try the rest of it. I got

pi@raspberrypi:/usr/local/dockerTrial $ sudo docker build -t kcollins/ignition:8.1.6-netraw --build-arg IGNITION_IMAGE_VERSION=8.1.5 .
Sending build context to Docker daemon  3.584kB
Step 1/6 : ARG IGNITION_IMAGE_VERSION=8.1.5
Step 2/6 : FROM kcollins/ignition:${IGNITION_IMAGE_VERSION}
 ---> 6e57c1e763ad
Step 3/6 : USER root
 ---> Using cache
 ---> f6dc452156aa
Step 4/6 : RUN   DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends         libcap2-bin &&     rm -rf /var/lib/apt/lists/*
 ---> Running in df68b2bdbb41
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package libcap2-bin
The command '/bin/sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends         libcap2-bin &&     rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Do I need to copy the libcap2-bin package somewhere?

Thanks

Without the apt-get update you’ll not have package indices and therefore follow-up apt-get install calls will fail… must be an issue with the ARM Ubuntu distro—the guide above was worked up on Intel… I’ll try to have a go at it on one of my Pi’s. I’m assuming you’re running 64-bit ARM?

Based on some additional research (this thread included), I think you're facing the issue with libseccomp2 on your host Pi system. Take a look at this forum thread for more insights: Docker Container Does Not Start - #4 by kcollins1

First thing I'd do is a sudo apt-get update && sudo apt-get upgrade on your Pi and try the build again. If this works, we should probably add that guidance to the other thread (since users would no longer need to go manually update that library and instead be able to have the system take care of the update). I'm running Ubuntu 21.04 64-bit on my Pi system (discussed briefly here) which is already running libseccomp2 v2.5.1... I can't say whether that is necessarily a good option if you're making use of RaspPi GPIO, I haven't done any hardware projects under Ubuntu on RaspPi yet...

Yeah, it’s the libsecomp2 issue. Running Raspbian release 10, 32 bit. I may try moving to the Ubuntu release if I get some time, but I don’t have enough time to mess with this at this point.

Right now I’m having node-red do my pings, and set up and API and having Ignition check it. I’m trying to move from node-red into Ignition, so I’m still using the old node-red instance to do the pinging and it will remain even though I pull the rest of the functionality into Ignition. Sometime in there I will try Ubuntu 21.04 64-bit. Thanks for all the help, and I will update when I update to Ubuntu (or recompile Raspbian, which is unlikely)

Ken