Have Robot type any arbitrary message via keyboard in vision?

Ignition
1

I’m writing end to end tests and am using java.awt.Robot to simulate a user. I can’t figure out how to have my robot type in any message I want it to. From the documentation I have to feed it either an KeyEvent.VK_Something, or just an integer, but how can I do that on on the fly for any string of characters? I figure I’ll need to iterate through the message

def type_on_keyboard(message):
	"""
	message: str, the key strokes you want simulated
	"""
    key_presses = [convert(x) for x in message]
    for key_to_press in key_presses:	
        robot.keyPress(key_to_press)
    	robot.keyRelease(key_to_press)

But I don’t know what the convert function should look like. Do I have to parse it and convert each character to VK_A, VK_B, etc or is there a better way?

2

https://docs.oracle.com/javase/7/docs/api/constant-values.html#java.awt.event.KeyEvent.VK_A

VK_A through VK_Z are the same as their ascii equivalent.

3

Do you need to actually type for the purposes of your test, or could you put the ‘message’ into the clipboard buffer and simulate a paste?

4

This (getExtendedKeyCodeForChar) may be useful as well: KeyEvent (Java SE 9 & JDK 9 )

1 Like
5

I don’t think actual typing is required if you now how to do this with a clipboard copy and paste?

@Kevin.Herron So I could use the getExtendedKeyCodeForChar on each char in my message and that would be the key to be pressed and released?

6

I don’t know, I’m just looking at the docs, try it and see.

7

Paste method would be something like:

from java.awt.datatransfer import StringSelection
from com.inductiveautomation.ignition.client.util import ClipboardUtil

def toClipboard(message):
	ClipboardUtil.doCopyTransferableToClipboard(StringSelection(message))

And then simulate a Ctrl + V keypress.

1 Like
8

Keycode method might be something like this:

from java.awt.event import KeyEvent

message = u"hello, world"
codes = [KeyEvent.getExtendedKeyCodeForChar(ord(c)) for c in message]

for code in codes:
	print code
1 Like
9

Oh I never seen the ord function before. That was the missing piece of the puzzle for me. Thanks.

10

ord may or may not behave weirdly if you ever pass unicode in, I don’t know whether Jython sticks to what Java would do or what Python would do, and character encoding is a total mess anyways…so here’s an alternate way to get the same info that’s less likely to have weird edge cases, I think:

from java.awt.event import KeyEvent
from java.lang import String

message = String(u"hello, world")
codes = [KeyEvent.getExtendedKeyCodeForChar(k) for k in message.chars().iterator()]

for code in codes:
	print code
1 Like
11

Awesome this works nicely. If I can bother you about one last thing regarding this, I am trying to put a tiny pause between pressing the keys to make it look like a real user. I currently have this

def small_delay():
	def foo():
		pass
	system.util.invokeLater(foo, 500)

def type_on_keyboard(message):
	"""
	message: str, the key strokes you want simulated
	"""
	str_message = String(message)
	key_presses = [KeyEvent.getExtendedKeyCodeForChar(k) for k in str_message.chars().iterator()]
	for key_to_press in key_presses:
		rob.keyPress(key_to_press)
		small_delay()
		rob.keyRelease(key_to_press)

But it’s still instantaneous, and I realize that my small_delay won’t work the way I have it, but I don’t see how I can do this without time.sleep, which I know is to be avoided. Any ideas?

Edit: Think I got it like this in case anyone else needs this:

def type_on_keyboard(message):
	"""
	message: str, the key strokes you want simulated
	"""
	str_message = String(message)
	key_presses = [KeyEvent.getExtendedKeyCodeForChar(k) for k in str_message.chars().iterator()]
	for num, key_to_press in enumerate(key_presses):
		delay = (num+1) * 200
		def typeKey(key=key_to_press):
			rob.keyPress(key)
			rob.keyRelease(key)
		system.util.invokeLater(typeKey, delay)
12

invokeLater calls will already be serial/in order, you should just be able to put them on the queue:

def type_on_keyboard(message):
	"""
	message: str, the key strokes you want simulated
	"""
	str_message = String(message)
	
	def typeKey(key):
		rob.keyPress(key)
		rob.keyRelease(key)

	for key_to_press in (KeyEvent.getExtendedKeyCodeForChar(k) for k in str_message.chars().iterator()):
		system.util.invokeLater(lambda: typeKey(key_to_press))
1 Like
13

I do like how clean it is. After getting the typing to work, I realized all my robot instructions had to be utilizing system.util.invokeLater otherwise while I was still logging on, the mouse would start moving due to the next function call. I chose the convention that each function returns the ms needed for the next call of system.util.invokeLater, and have a constant DELAY_FACTOR to easily shorten or lengthen time between actions. My setup is like this -

DELAY_FACTOR = 200

WINDOW_SELECT_PIXELS = {
	"Privileged Access": (1300, 1050),
	"Overview": (500, 1050),
	"Print Screen": (1900,1050),
	"Login": (1400,40)
}

def goToWindow(window, initial_delay=0):
	"""
	Goes to a specific window via bottom navigation.
	Args:
		window: str, which window do you want to go to?
	Returns:
		None, moves mouse and clicks button, or throws KeyError if bad window name
	"""
	x, y = WINDOW_SELECT_PIXELS[window]
	def leftClickForWindow(x=x, y=y):
		leftClick(x, y)
	system.util.invokeLater(leftClickForWindow, initial_delay)
	return initial_delay+DELAY_FACTOR

def pressTab(initial_delay=0):
	def pressVKTab():
		rob.keyPress(KeyEvent.VK_TAB)
		rob.keyRelease(KeyEvent.VK_TAB)
	system.util.invokeLater(pressVKTab, initial_delay)
	return initial_delay+DELAY_FACTOR

def pressEnter(initial_delay=0):
	def pressVKEnter():
		rob.keyPress(KeyEvent.VK_ENTER)
		rob.keyRelease(KeyEvent.VK_ENTER)
	system.util.invokeLater(pressVKEnter, initial_delay)
	return initial_delay+DELAY_FACTOR

def type_on_keyboard(message, initial_delay=0):
	"""
	message: str, the key strokes you want simulated
	initial_delay: int, - for the first thing you type in, but probably otherwise.  needed for chaining
	"""
	str_message = String(message)
	key_presses = [KeyEvent.getExtendedKeyCodeForChar(k) for k in str_message.chars().iterator()]
	maxDelay = None
	for num, key_to_press in enumerate(key_presses):
		delay = ((num+1) * DELAY_FACTOR) + initial_delay
		maxDelay = delay
		def typeKey(key=key_to_press):
			rob.keyPress(key)
			rob.keyRelease(key)
		system.util.invokeLater(typeKey, delay)
	return maxDelay + DELAY_FACTOR

def moveMouse(x, y, initial_delay=0):
	"""
	Call leftClick at a specified pixel.
	"""
	def moveMouseDelay(x=x, y=y):
		rob.mouseMove(x, y)
	system.util.invokeLater(moveMouseDelay, initial_delay)
	return initial_delay+DELAY_FACTOR

def loginAsMe(initial_delay=0):
	wait_0 = goToWindow("Login", initial_delay=initial_delay)
	wait_1 = type_on_keyboard("secretusername") 
	wait_2 = pressTab(initial_delay=wait_1) 
	wait_3 = type_on_keyboard("secretpassword", initial_delay=wait_2)
	wait_4 = pressEnter(initial_delay=wait_3)
	return wait_4

This works for me. It looks like a realistic user at 200 ms delay. I wish my loginAsMe function looked nicer, as I’m always getting the result of the previous function and feeding it to the next initial_delay, but it’s readable enough and works. Just posting this here for comments/concerns/posterity, as I haven’t seen too many posts on trying to write end to end tests with Robot.

1 Like
14

I wonder if something like this would work? You’ll have to invoke loginAsMe from an async thread - EventQueue will throw if you try to call invokeAndWait from the EDT.

from java.awt import EventQueue

DELAY_FACTOR = 200

WINDOW_SELECT_PIXELS = {
	"Privileged Access": (1300, 1050),
	"Overview": (500, 1050),
	"Print Screen": (1900,1050),
	"Login": (1400,40)
}

def __invokeBlocking(fn, *args):
	EventQueue.invokeAndWait(lambda: fn(*args))

def __pressKey(keyCode):
	rob.keyPress(keyCode)
	rob.keyRelease(keyCode)

def goToWindow(window, initial_delay=0):
	"""
	Goes to a specific window via bottom navigation.
	Args:
		window: str, which window do you want to go to?
	Returns:
		None, moves mouse and clicks button, or throws KeyError if bad window name
	"""
	x, y = WINDOW_SELECT_PIXELS[window]
	__invokeBlocking(leftClick, x, y)

def pressTab():
	__invokeBlocking(__pressKey, KeyEvent.VK_TAB)

def pressEnter():
	__invokeBlocking(__pressKey, KeyEvent.VK_ENTER)

def type_on_keyboard(message, initial_delay=0):
	"""
	message: str, the key strokes you want simulated
	initial_delay: int, - for the first thing you type in, but probably otherwise.  needed for chaining
	"""
	str_message = String(message)
	for keyCode in (KeyEvent.getExtendedKeyCodeForChar(k) for k in str_message.chars().iterator()):
		__invokeBlocking(__pressKey, keyCode)
	
def moveMouse(x, y):
	"""
	Call leftClick at a specified pixel.
	"""
	__invokeBlocking(rob.mouseMove, x, y)

def loginAsMe():
	goToWindow("Login")
	type_on_keyboard("secretusername") 
	pressTab() 
	type_on_keyboard("secretpassword")
	pressEnter()
1 Like
15

That looks much cleaner, I will try it tomorrow. One weird thing is I can’t seem to get it this to work for undrescores unfortunately. Right now this keeps throwing me an error that it’s an invalid key code, even when I use the KeyEvent.VK_UNDERSCORE. I thought I maybe needed a caveat for special characters and made a dictionary map from numbers to KeyEvent.VK_UNDERSCORE, but that’s the same exact key code the getCharCode is giving in the first place. Here’s a script you can run in the script console and replicate the problem. Not sure what is going wrong.

import java.awt.Robot
from java.awt.event import InputEvent, KeyEvent
from java.lang import String
import java.lang.IllegalArgumentException

ROBOT_DELAY_MS = 100
DELAY_FACTOR = 200

SPECIAL_CHARACTERS = {
	523: KeyEvent.VK_UNDERSCORE 
}

rob = java.awt.Robot()
rob.delay(ROBOT_DELAY_MS)

def getKeyCode(char):
	return KeyEvent.getExtendedKeyCodeForChar(char)

def type_on_keyboard(message, initial_delay=0):
	"""
	message: str, the key strokes you want simulated
	initial_delay: int, - for the first thing you type in, but probably otherwise.  needed for chaining
	"""
	str_message = String(message)
	print "working on %s"%(str(str_message))
	key_presses = [getKeyCode(k) for k in str_message.chars().iterator()]
	maxDelay = None
	for num, key_to_press in enumerate(key_presses):
		delay = ((num+1) * DELAY_FACTOR) + initial_delay
		maxDelay = delay
		def typeKey(key=key_to_press):
			print "about to press %s"%(str(key))
			try:
				rob.keyPress(key)
				rob.keyRelease(key)
			except java.lang.IllegalArgumentException, e:
				# Not try/excepting keyerror here as if we don't have it, we probably want to alert the dev so they add it
				special_key = SPECIAL_CHARACTERS[key]
				print "special_key:" + str(special_key)
				rob.keyPress(special_key)
				rob.keyRelease(special_key)
		system.util.invokeLater(typeKey, delay)
	return maxDelay + DELAY_FACTOR

type_on_keyboard("some_cool")
16

Extended characters that don’t have an actual mapping on the current keyboard layout will throw that error. You’ll have to simulate it via a Shift + - ‘stack’ of key presses:

1 Like