So I finally used media query to handle mobile stuff. But rather than do it on the individual items, I used it to set rem at the html level and it seems to be working pretty well that way. A 4x multiplier got things sized pretty decently (including those darn handles on docks).
/* Set rem */
/* Mobile first - gets a 4x multiplier to the large screen size */
html {
font-size: calc((1vw / 19.2) * var(--base-font-size) * 4);
}
@media (min-width: 640px) {
html {
font-size: calc((1vw / 19.2) * var(--base-font-size));
}
}