I finally solved the issue. At first I thought the problem was that triggering the authentication challenge as scripted above was causing the page to reload, which led to a loss of the values I wanted to store. However, I tried to alleviate this by writing the values to memory tag prior to the authentication challenge, and upon success of the challenge the message handler on the root container would retrieve the values from the memory tags and would process them as needed. However this did not work. The values would write to the memory tags prior to the auth challenge, but the auth challenge failed to execute the message handler upon success.
I found that the issue has something to do with the auth challenge reloading the page maybe disconnecting the session for a second and terminating the auth challenge prematurely, hence the websocket error mentioned above. However, I cannot be certain that this is the root cause.
However I did find that having the popup open as an embedded frame did not reload the session and worked at completing the sequence of events I desired. See code and picture below.
system.perspective.authenticationChallenge(timeout = 5, framing = "Embedded", payload = {'pageID':self.page.props.pageId})
