Ok, found some further info.
It looks like Google Noto separates the fonts by language. The fonts are primarily needed for web development: by splitting it per language they limit the download size, and CSS can accept multiple fallback fonts anyway. So you can get the full range of characters (see Guidelines for Using Noto – Google Noto Fonts).
The really strange part is how Noto is now split in different languages, but neither Noto Symbols nor Noto Symbols 2 supports a majority of arrows (though they do have different support).
In 7.9 however, Lucida was apparently shipped and used as fallback font, but this isn’t shipped anymore in new JRE’s. (see Font issues with Symbols - #3 by fred.berryman). Strangely enough, I still see Lucida Sans Unicode in my list, which has a pretty good support for symbols, but not as good as in 7.9.
But now I’m wondering if Java has a feature similar to CSS where you can define multiple fallbacks. And how to even configure fonts for the designer.