[FEATURE-1953] How to capture customer IP in audit log when behind AWS ELB (load balancer)?

We run an Ignition 8.1 instance in the AWS cloud, with a load balancer (ELB) in front of it to ease management.

I just noticed that the gateway’s audit log is only capturing the internal IP of the load balancer on user login events. We’re using Perspective and the internal IDP, so they are showing as “Web Auth Status Change” actions. I’m not seeing the user’s external IP being captured anywhere else in the audit log.

Is it captured somewhere else? If not, what is the recommended way to capture it in the audit log?

Hmm. I don’t think if there’s anything you can do about this right now.

It might be something we can add support for, though. Assuming they set the X-Forwarded-For header correctly we might be able to look for that on incoming requests and add a new session prop or use it to change the value of the existing one that identifies the client hostname/ip.

This is something we need. As it is, we have no way of knowing where logins are coming from. I suspect this is also breaking security zones, but we thankfully don’t use them.

I would argue this doesn’t “break” security zones so much as it’s an incompatible model for use with Security Zones and you should know better when designing a system. X-Forwarded-For will not be suitable for use with Security Zones because a client could set any value they want in that header.

edit: and the same caveat would exist for this header when it comes to auditing if we added support for it - you are now trusting a header value that can be arbitrarily set rather than the hostname or IP as seen by the gateway, which is more difficult to spoof.

I admit to being only vaguely aware of X-Forwarded-For, but I sort of assumed that a load balancer would refuse to pass through an X-Forwarded-For setting that was provided externally, and would instead override it with a new value based on the IP it saw.

Do forwarders just chain them together, so you could find multiple stacked up in your HTTP requests? If that’s the case, then our network architecture enforces that no client can reach the gateway directly and we know the most recent one is trustworthy since our load balancer added it. In this situation, I would want to log all of the forwarded headers but rely on just that one our system added.

If that’s not the case, then it’s almost useless.

I’m thinking mostly of a scenario where clients may hit the gateway without going through the load balancer as well - maybe that’s not realistic. I doubt that load balancers will arbitrarily forward theses headers with client-provided IPs.

Another thing you may need to consider since you’re running Ignition in the cloud is that every user who logs in from the same facility/building/whatever may very well be coming from the same external IP address and even with support for X-Forwarded-For you’re just going to have some external IP address rather than the load balancer IP address.

I agree there is definitely a risk of abuse if clients could reach the gateway directly OR through a load balancer, in that the direct clients could fake a X-Forwarded-For. In my case, it would be sufficient if the AUDIT log included both the direct and forwarded IPs, and it was up to me to figure out what to do with that information.

If IA implements any further behavior based on X-Forwarded-For, then you probably also need to guide admins away from a network architecture that allows both forwarded and direct gateway connections, or at least only allow direct connections from only the most trusted clients (i.e. administrator tunneling/VPN).

And seeing all users from the same facility showing the same IP is understood and expected.

The latest 8.1.9 early access build includes a new capability to opt-into making Ignition reverse-proxy-aware. The user manual still need to catch up, but for now, you can refer to Jetty’s ForwardedRequestCustomizer’s javadoc page for which headers Ignition respects. For additional detail, see: Nightly 8.1 Changelogs - 2021 - #128 by sreis