The downside is that the Gateway will make LDAP requests to the AD server every time the list of users or roles is required from the AD server, instead of hitting the cache. Depending on how often your users require the list of users or roles and depending on how large the directory is, this could be a huge performance burden.
There are many settings you can take advantage of in the user source profile to tune performance to your liking. You could increase the cache invalidation timeout depending on your tolerance for stale cache on the Gateway side since the last AD query. You could refine your user search and list base if you only need a subset of the directory (and not the entire 11K users). If you only want to show the list of users that you have created on the internal DB side of the hybrid source, you could disable the List Users from Active Directory setting, which would make the system show the list from the internal DB side instead of querying the AD server for the list of roles and users.