Factory Map using Perspective Map Component and Custom SVG

I have seen a few people do something like this but have not figured it all out. I am looking to use a custom SVG of my factory map inside the map component. I want to use the pan/zoom features of the map component, in addition to adding markers/other data points for the user to click on.

When I add my SVG as a tile layer, it tile repeats the SVG over and over. If I add it as an image layer I can get close to what I want, but I have to leave the openstreetmap tile layer behind the svg image layer and restrict the bounds.

Anyone have some guidance on this type of application?