Web Browser component login to website

I need to set up a script or something to allow the web browser component to automatically login to SharePoint webpage.

I have had a project deployed for over a year now that was able to access SharePoint documents; it automatically logged in based on the windows user. But something changed this weekend and it no longer works.

I have been struggling to find a way for a script to interact with the web page in order to login. Does anyone know how to accomplish this?

I have been reading through the jxbrowser documentation and it looks like it has the capability to do this. But all of the examples are in java, I cannot figure out how to use these functions in a python script.

If you post the script you are trying to convert, one of us can walk you through how to convert it to jython. It's usually just just a matter of importing and creating instances of the objects you need rather than simply declaring them.

1 Like

I need to enter a username/password into text fields:

And click a button:

The web browser component has a .getBrowser() scripting function so you should be able to get a reference to the DOM and thus documentElements.

You can use dir() to find what properties and methods are are your disposal. I don't have the Web Browser Component, so I'm going completely off of the manual.

from the JxBrowser docs:
image

This is not something I've ever messed with before, but it looks like you can use the code as it is written convert the code in the documentation to javascript and apply it in this way:

webComponent = event.source.parent.getComponent('Web Browser')#replace with relative path to your component
webbrowser = webComponent.browser
frame = webbrowser.focusedFrame()
if frame.isPresent():
	frame.get().executeJavaScript("const element = document.getElementById('firstname'); if (element !== null) {element.value = 'John';}")

Edit: Updated per feedback below

I'm pretty sure what you've written inside the 'executeJavascript' call there is more Java code. But the general principle is about right.

I don't have a web page to test this on, but probing the component with print dir(frame.get()) I see two methods that look useful: execute and executeJavaScript. Would you guess that execute is actually the correct one?

Full Console Output
['__class__', '__copy__', '__deepcopy__', '__delattr__', '__doc__', '__ensure_finalizer__', '__eq__', 
'__format__', '__getattribute__', '__hash__', '__init__', '__ne__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__subclasshook__', '__unicode__', 'browser', 
'children', 'class', 'close', 'closed', 'createJsProxyFunction', 'createJsProxyObject', 'document', 
'domContext', 'equals', 'execute', 'executeJavaScript', 'find', 'findImpl', 'getClass', 'hasSelection', 
'hashCode', 'html', 'id', 'inspect', 'isClosed', 'isCommandEnabled', 'isMain', 'json', 'loadData', 'loadHtml', 
'loadUrl', 'localStorage', 'main', 'name', 'notify', 'notifyAll', 'of', 'onEvent', 'parent', 'print', 'renderProcess', 
'selectionAsHtml', 'selectionAsText', 'sessionStorage', 'toString', 'unregister', 'wait']

I mean that this bit is still Java code. The point of the 'executeJavaScript' function is that you provide JS code (as a string) that will be run directly on the webpage.

1 Like

Understood. I've corrected the above untested answer.

1 Like

With this I was able to fill in the text field.

When I attempt to fire the click event on the button, microsoft says it is an invalid username/email. But if I click on the button manually, it accepts it.

filling the password box doesn't seem to work either.
(edit: It seems it will not fill in any of the text boxes after I click on the page. Until reloading the page)

if event.propertyName == "testbool":
	def fill_form():
		from time import sleep
		
		print "Starting login attempt"
		
		webComponent = event.source      #replace with relative path to your component
		webbrowser = webComponent.browser
		frame = webbrowser.focusedFrame()
		
		if frame.isPresent():
			print "frame present"
			# fill email
			frame.get().executeJavaScript("const element  = document.getElementById('i0116');       if (element  !== 'null') {element.value = 'user@email.com';}")
			sleep(1)
			
			# press "next" button
			frame.get().executeJavaScript("const element2 = document.getElementById('idSIButton9'); if (element2 !== 'null') {element2.click();}")
#			sleep(5)
#			
#			# enter password
#			frame.get().executeJavaScript("const element3  = document.getElementById('i0118');       if (element  !== 'null') {element3.value = 'password';}")
#			sleep(1)
#			
#			# click "sign in" button
#			frame.get().executeJavaScript("const element2 = document.getElementById('idSIButton9'); if (element2 !== 'null') {element2.click();}")
	
	
	if "authorize?client" in event.source.currentUrl:
		system.util.invokeAsynchronous(fill_form)

image

I'm glad we have been able to make some progress on the problem. Unfortunately, I don't have an adequate test environment for this usage case, so I doubt I will be of much further assistance. My guess is that the problem is with my implementation of javascript, so perhaps somebody who works with this language on a regular basis, such as @victordcq, can offer you further guidance from here.

ive done stuff similar like this before.

ill come back here tommorow

1 Like

eh we did this quite different...

Anyways the problems with accessing the browser in vision is that its super hard to know when the browser finishes with something.
So this sleep thing is really not consistent.

the only kinda consistent thing i was able to do was access the localstorage( nd i assume cookies will be the same)
since those work before the page is fully loaded.
so like that you could technically log in even before the page is loaded and store the log in in one of those (you'll ahve to know what the site uses for this tho, and how to do that for microsoft)

idk what ms uses, but you can probably even get a token through python/java code

from com.teamdev.jxbrowser.navigation.event import NavigationFinished
from com.teamdev.jxbrowser.event import  Event, Observer

class obs(Observer):	
	def __init__(self,fn):   		
   		self.on=fn 
def login(ev):
  key= ?
  token= ?
  # if you need it... ev.frame().executeJavascript("")
  storage= ev.frame().localStorage()
  storage.putItem(key,token)
browser = system.gui.getParentWindow(event).getComponentForPath('Root Container.Web Browser').browser
browser.navigation().on(NavigationFinished,obs(lambda ev: login(ev)))

note this is for doing stuff before the page is loaded, you want be able to acces any buttons

if its just a minor thing tho and it has worked before, i suggest trying to increase the sleep between actions and maybe check if the element id's are still the same

In the past, It would just login on it's own, without any scripts.

That might be related to a Windows security hole that got plugged, cutting off certain auto-login functions. I don't recall the details (a couple months or so ago?).

Yesterday, I implemented a java.awt robot to simulate a user input to login. It is working, but I can foresee a few issues.

  1. If a user interacts with the mouse/touchscreen during the login process, it might be interrupted and not login properly.

  2. I have no way to check if an element is visible, since the username and password are on different elements that become visible at different times, It might be possible for the password to be typed in the username field or typed in before the page is ready to receive it.

  3. This will also only work on the expected screen size. Which is not a problem for me, as all stations running this project use the same screen.

Property change on web browser component

if event.propertyName == "current_URL":
	
	from time import sleep
	from java.awt import Robot
	from java.awt.event import InputEvent
	from java.awt.event import KeyEvent
	import subprocess
	
	def login():
		username = "username"
		password = "password"
		
		def copy_paste(txt):
			cmd='echo ' + txt.strip()+'|clip'
			subprocess.check_call(cmd, shell=True)
			
			robot.keyPress(KeyEvent.VK_CONTROL)
			robot.keyPress(KeyEvent.VK_V)
			robot.keyRelease(KeyEvent.VK_CONTROL)
			robot.keyRelease(KeyEvent.VK_V)
			
			robot.keyPress(KeyEvent.VK_ENTER)
			robot.keyRelease(KeyEvent.VK_ENTER)
		
		# make sure login screen is still present
		if "authorize?client" in event.source.currentUrl:
			
			# Make component over web-component invisible so clicks can go through to web-component
			event.source.parent.getComponent('Swipe Scroll Area').visible = False
			
			robot = Robot()
			
			# Get coordinates of web-component position in ignition window
			# add mouse coordinates within web-page to click on textbox
			comp_coords = misc.get_abs_comp_coords(event)
			new_x = comp_coords[0] + 500
			new_y = comp_coords[1] + 570
			
			# move to textbox and click
			robot.mouseMove(new_x, new_y)
			robot.mousePress(InputEvent.BUTTON1_MASK)
			robot.keyRelease(InputEvent.BUTTON1_MASK)
			
			# copy username to clipboard and paste, then press enter
			copy_paste(username)
			
			# copy password to clipboard and paste, then press enter
			sleep(2)
			copy_paste(password)
			
			#accept "stay logged in" prompt
			sleep(2)
			robot.keyPress(KeyEvent.VK_ENTER)
			robot.keyRelease(KeyEvent.VK_ENTER)
			
#			event.source.parent.getComponent('Swipe Scroll Area').visible = True
			
			
		
	
	def login_async():
		system.util.invokeAsynchronous(login)
	
	# if login page is present, invoke async login function later 
	if "authorize?client" in event.source.currentUrl:
		event.source.parent.getComponent('Swipe Scroll Area').visible = False
		system.util.invokeLater(login_async, 5000)
	
	# if no login page is present, make component over web-component visible to allow swipe scrolling
	elif "login.microsoftonline.com" not in event.source.currentUrl:
		def clicks_for_scroll():
			robot = Robot()
			robot.mouseMove(430, 1000)
			
			robot.mousePress(InputEvent.BUTTON1_MASK)
			robot.keyRelease(InputEvent.BUTTON1_MASK)
			
			robot.mouseMove(430, 1265)
			robot.mousePress(InputEvent.BUTTON1_MASK)
			robot.keyRelease(InputEvent.BUTTON1_MASK)
		
		event.source.parent.getComponent('Swipe Scroll Area').visible = True
		
		#needed to insure swipe scrolling works.
		system.util.invokeLater(clicks_for_scroll, 1000)

"misc" script

# gets coordinates of a component from an event relative to the ignition window
def get_abs_comp_coords(event):
	
	window = system.gui.getParentWindow(event)
	
	root_container = window.getRootContainer()
	
	container = event.source
	
	
	x = window.getX()
	y = window.getY()
	
	while container != root_container:
		
		x += container.x
		y += container.y
		
		container = container.parent
	
	
	return [x, y]
1 Like

Idk what has changed.

But usually authentication for iframes (which is what the browser component basicly is) is done through temporary tokens. Be it stored through localstorage/cookies (with jwt which i used but not for ms). Or by creating a unique and temporary link.

Im pretty sure microsoft should have something like that. but its probably not something trivial to do

meh, seems a bunch of work

1 Like

I commend you for getting out of the box, and while I agree with Victor that robot doesn't seem like the ultimate approach, this does seem like a decent work around. With the exception of the occasional hiccups you've mentioned, this should buy you all the time you need to work out a proper behind the scenes handshake.