When you are editing the user source in project properties in vision, the role suggestion popup and validation icons do not update. You have to change the project’s user source, save the project, then the role list and validation icons are updated to the new user source.
Also, it looks like you have to close the project and reopen it for the roles to update for component security
This is a known bug with the required client roles field. I have added this forum thread to the internal bug ticket we have open for it. For now, the workaround is exactly what you have described.