No, the designer does not expose any of the SVG elements if the SVG is displayed using the Icon or Image component. You only have direct access to the elements if you embed the SVG.
As a different approach, you would need to be very strategic about it, but since the ICON component eventually is just rendered in the browser as an SVG (this is not true of the image component), then you should be able to target the specific elements by the ID using the CSS Stylesheet.
Something like:
rect#rect3723 {
color: rgb(250,250,251);
}
Would be a lot of work, but would eliminate the need for embedded views just to "simulate" an icon component.
The real performance degradation with embedded views is when they are nested, however, I have also hear that having many embedded SVG's will degrade performance. I would think that so long as the SVG's are relatively simple you wouldn't see much of a hit up to hundreds of SVG's on a single view.