How to restrict the mouse operation within popup window

The behavior you are describing is generically called a modal window. (By generically, I mean across all GUI programming platforms, not solely Ignition.) Unfortunately, Ignition's architecture makes this rather difficult to implement. This thread has the most comprehensive options for you: