# Using & authenticating system.net.httpGet() on a gateway **URL:** \ **Category:** Ignition\ **Created:** [July 11, 2018, 8:01pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799 "2018-07-11T20:01:49Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [July 11, 2018, 8:01pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/1 "2018-07-11T20:01:49Z")
Hi guys, I was thinking I may want to access a project’s own gateway web page to get some info that may not be accessible otherwise (from the client). I’m trying to use system.net.httpGet and supply a url that needs authentication to get to, plus a username and password that should have access to it, but the response I get is just the source for the sign in page. Is there something else I need to do here to login to the gateway web page? ---
**Author:** ![phillip.bow](https://avatars.discourse-cdn.com/v4/letter/p/ce73a5/32.png) [@phillip.bow](https://forum.inductiveautomation.com/u/phillip.bow)\ **Post date:** [July 11, 2018, 8:49pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/2 "2018-07-11T20:49:42Z")
You would likely need to jump through some hoops using python’s httplib. When you first hit the site you need to look at the response headers and pull out your assigned session id. Then you’d need to make a POST with your auth credentials, and that session ID to the endpoint that processes logins. Then you could make a request to get the page you want to scrape as long as you still include that session ID and as long as that session remains valid. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [July 11, 2018, 8:59pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/3 "2018-07-11T20:59:53Z")
This is something that we have eventual plans to improve - exactly how is up in the air, but the rough idea was some kind of API token table (in the gateway configure page) where you can add/remove random tokens, then use those as authentication against endpoints. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [July 11, 2018, 9:56pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/4 "2018-07-11T21:56:11Z")
> [@paul-griffith](#): > > exactly how is up in the air Consider having a version/overload of httpGet() that only goes to the gateway, and automatically inherits the client session (cookie). Or an isolated child session, identified by the main session's cookie. Gateway endpoints already enforce various role requirements, so it would leverage existing features. This would be especially helpful for generic webdev endpoints. ---
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [July 13, 2018, 8:20pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/5 "2018-07-13T20:20:43Z")
I think I’ve got the jist of what you’re saying, I get the cookie part after trying once, then use that cookie trying to post my username/password, then use get again, and I got another ‘found’ … but no response with any data. Here is my test code - I’ve cleaned it up a bit but its pretty rough right now. It uses some parameters for the sensitive bits, which have been left out of what I’m posting here. This is also created in the script console, so assumes variables will be carried over (like cookies). ```auto import httplib, urllib conn = httplib.HTTPConnection(url, port) params = urllib.urlencode({'id13_hf_0': '', 'username': username, 'password': password, 'button': "Sign In"}) if cookie is not None: cookie = cookie.split(";")[0] else: cookie = None #print cookie headers = { 'Cookie': cookie, 'Content-Type': "application/x-www-form-urlencoded", 'Connection':" keep-alive", 'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" } #print headers conn.request("POST","/main/web/signin?2-1.IFormSubmitListener-sign~in~form",params,headers) res = conn.getresponse() print res.status, res.reason resRead = res.read() hdrs = res.getheaders() cookieNew = res.getheader('set-cookie') cookie = cookieNew if cookieNew is not None else cookie print cookie print "\n".join(["%s: %s" % r for r in hdrs]) print "----" headers = { 'Cookie': cookie, 'Content-Type': "application/x-www-form-urlencoded", 'Connection':" keep-alive", 'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 'Accept-Encoding': "gzip-deflate", 'Accept-Language': "en-US, en;q=0.9", 'DNT': 1, 'Host': "%s:&s" % (url, port), 'Upgrade-Insecure-Requests': 1, 'User-Agent': "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36" } params = urllib.urlencode({'3': ''}) conn.request("GET", "/main/web/status/?3", params, headers) res = conn.getresponse() print res.status, res.reason resRead = res.read() hdrs = res.getheaders() print "\n".join(["%s: %s" % r for r in hdrs]) print resRead conn.close() ``` ---
**Author:** ![phillip.bow](https://avatars.discourse-cdn.com/v4/letter/p/ce73a5/32.png) [@phillip.bow](https://forum.inductiveautomation.com/u/phillip.bow)\ **Post date:** [July 13, 2018, 9:48pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/6 "2018-07-13T21:48:18Z")
You’re on the right track, but there could be some extra trickery going on behind the scenes that make this unviable(and obviously unsupported). The 302 - response indicates a redirect to another page to the browser. I tested your code and the submission seems to work but I never actually get access - the 302 redirects back to the sign in page as if the login failed. If I hard-code an already logged in session ID from my browser I can get the pages without any extra hoops which makes me think there’s some magic to posting through the login form that is still missing. ---
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [July 17, 2018, 9:35pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/7 "2018-07-17T21:35:39Z")
Thanks for that, I’ve continued to try other methods as much as I can with no luck. Anyone else have an idea for this? Any IA people around and able to shed some light on what we’re missing here? Edit: New problem, sorta. I think I was able to get in by passing the username & parameters POST’d to the form action url in the signin source thats provided on the signin page. But now, the source I get for something like the log viewer page just doesn’t match what I get when I look via the browser. Its missing… well, the good stuff. Seems like some script takes care of this later in the process in a normal browser, I’m not sure if we can easily replicate it and get it. ---
**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\ **Post date:** [July 18, 2018, 3:37am UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/8 "2018-07-18T03:37:31Z")
> [@mrogers](#): > > But now, the source I get for something like the log viewer page just doesn't match what I get when I look via the browser. Its missing... well, the good stuff. Seems like some script takes care of this later in the process in a normal browser, I'm not sure if we can easily replicate it and get it. That's because it's all generated dynamically by React. You'll have to monitor the calls a browser makes to the gateway to figure out how to retrieve what you want. If that's even possible. Consider adding the React tools to your browser. ---
**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\ **Post date:** [July 18, 2018, 2:39pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/9 "2018-07-18T14:39:08Z")
> [@mrogers](#): > > But now, the source I get for something like the log viewer page just doesn't match what I get when I look via the browser. If you're specifically looking for logs information, then you can use the internal endpoint (rather than trying to parse the frontend/React output): `http:///main/data/status/logs?page=0&pageSize=100&minLevel=ALL&queryText=` Should be fairly self explanatory. This will return a JSON object that you can then freely parse/manipulate. All of the status pages (to my knowledge, anyways) work through some internal endpoint that returns plain JSON - so using the browser's network tracing tools is your best bet to get programmaticly useful data out of these pages. ---
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [July 18, 2018, 3:14pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/10 "2018-07-18T15:14:46Z")
Fantastic, even better than what I was hoping for. As far as I can tell so far, all the ones I’m looking for have the json retrieval and it works great through scripting. Thanks! ---
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [October 22, 2018, 11:03pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/11 "2018-10-22T23:03:10Z")
I’m revisiting this project and have another problem - using system.net.httpGet() from the gateway seems to have a timeout error. I have a UDT with a boolean memory tag, when that tag goes from low to high it runs a shared script that will ideally use httpGet to login to a given gateway URL and get the JSON information as discussed above. In particular, I’m trying to access the gateway the script is on, so nothing should be too odd. ```auto def signin(urlP): url = "http://%s:8088" % urlP webURL = "/main/web" username = 'admin' password = 'password' signinTest = shared.HTTP.checkSignin(url) retStr = [] if signinTest == "": retStr.append("signed in") else: retStr.append( "not signed in: %s" % signinTest ) retStr.append( "trying signin now" ) signinResponse = system.net.httpPost(url + webURL + signinTest, {'username': username, 'password': password}) retStr.append('Signin Response:\n%s' % signinResponse) newTest = shared.HTTP.checkSignin(url) if newTest == "": retStr.append( "signin successful") else: retStr.append( "signin not successful") return retStr def checkSignin(url, ext=None): import re ext = ext if ext is not None else "/main/web/status/" firstGet = system.net.httpGet(url = url + ext) patternPre = ".*?action=\"\.(.*?)\"" pattern = re.compile(patternPre, re.DOTALL) match = pattern.match(firstGet) if match: subText = match.group(1) return str(subText) else: return "" ``` This isn’t the most optimized code, but on the ‘firstGet’ assignment in checkSignin I have a timeout error. The same code I use to call signin() works just fine when in the script console, so I think something about the gateway scope makes it different, but I’m not sure why. ---
**Author:** ![ethomason](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ethomason/32/43338_2.png) [@ethomason](https://forum.inductiveautomation.com/u/ethomason)\ **Post date:** [October 23, 2018, 2:13pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/12 "2018-10-23T14:13:29Z")
It’s likely an infinite loop because you don’t have a user session. You need to create a cookie first. ---
**Author:** ![mrogers](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mrogers/32/4110_2.png) [@mrogers](https://forum.inductiveautomation.com/u/mrogers)\ **Post date:** [October 23, 2018, 3:27pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/13 "2018-10-23T15:27:52Z")
There aren’t really any opportunities there for an infinite loop - it goes into checkSignin when assigning signinTest, and then has the error there. And, after all this I set another memory tag to the retStr returned value. I just wrapped that assignment causing trouble in a try/except so the retStr finally returned - no duplicated entries at all. As for the user session - thats what this is trying to test. checkSignin tries to access a page you need to login to get to, and if that acts like you’re already logged in it will return true else false, THEN it’ll actually login. So it should be fine if there is no user session just yet, that will be made later. Edit: Actually I’m not sure if I’ve changed something or just misread the error before but now its saying ‘Server redirected too many times (20)’. So, an infinite loop after all, just not one caused by the script directly. Edit 2: Got it working! For now at least. Used httplib to get the cookie (as far as I can tell the [system.net](http://system.net) functions don’t give the headers at all). Passed that in as a headerValue in the httpGet in checkSignin and its working well. The getCookie code: ```auto def getCookie(url): import re, httplib conn = httplib.HTTPConnection(url.replace("http://", "")) conn.request("GET", "/main/web/home") r1 = conn.getresponse() cookieStr = [list(t)[1] for t in r1.getheaders() if list(t)[0] == 'set-cookie'] conn.close() cookies = cookieStr[0] if len(cookieStr)>0 else 'none' patternPre = "JSESSIONID=(?P.*?);" pattern = re.compile(patternPre) match = pattern.match(cookies) return str(match.group('cookie')) ``` ---
**Author:** ![ethomason](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ethomason/32/43338_2.png) [@ethomason](https://forum.inductiveautomation.com/u/ethomason)\ **Post date:** [August 24, 2019, 2:00am UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/14 "2019-08-24T02:00:19Z")
Just a heads up, this won’t work for 8. In getCookie, changes the GET url to ```auto "/web/signin" ``` ---
**Author:** ![cristihcd](https://avatars.discourse-cdn.com/v4/letter/c/7bcc69/32.png) [@cristihcd](https://forum.inductiveautomation.com/u/cristihcd)\ **Post date:** [August 25, 2019, 2:42pm UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/15 "2019-08-25T14:42:35Z")
Nice! Do you have a working final example? I tried to make it work but missed the checkSignin (shared.HTTP.checkSignin(url)) function. I am trying to get the RAM/uptime of clients from [this post](http://forum.inductiveautomation.com/t/get-client-ram-consumption-and-more/17277/3) and don’t know how to get the json for `/main/data/vision/status/sessions`. Thanks ---
**Author:** ![ethomason](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ethomason/32/43338_2.png) [@ethomason](https://forum.inductiveautomation.com/u/ethomason)\ **Post date:** [August 27, 2019, 4:00am UTC](https://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/16 "2019-08-27T04:00:26Z")
checkSignin is a function that can be found at the bottom of Michael’s snippet [here](http://forum.inductiveautomation.com/t/using-authenticating-system-net-httpget-on-a-gateway/18799/11?u=ethomason)