Development work in production

We currently do development work in production. Recently our gateway server logged a (Restarting gateway scripts…) message. It seems this happened directly at the same time a timer script / Gateway script should have run, but the script never ran and some important data was never collected. I’ve tried repeatedly, over a span of months, to explain to management that we need a development environment, for our Ignition/mes system and our database but the conversation seems to never get any traction. The priority is more like, keep chugging, we still have features we need added etc. Our gateway server also logs so much (sometimes multiple per second or at most with a few seconds between logs) that it’s impossible to know what all are. They literally scroll off the screen trying to read them (unless we disable the feed) . I think my question is, how can I explain this or push this idea of (needing a development environment) in a more effective way?

1 Like

IMO this shouldn’t even need explanation. Having a development environment lets you make changes and test them without breaking anything. Depending on your industry, making untested changes to production can result in various degrees of catastrophic failure. Google dev > staging > production, there is no shortage of explanations as to why it is a good thing to do.

For my case, working in mostly corrections, an untested change could result in letting inmates out of jail, which would earn me a firing, and a court appearance.

2 Likes

I would try to frame it in the one way sure to get people to listen - the dollars it will cost them.

You already had one instance where important data wasn’t collected and sent out, probably cost them some money. If you can convey that a development system would cost them less money and more stability than what their current practice - it might actually get some traction.

I sympathize with you though. There are definitely some people who simply will not care. When you ask for a development system and mention it will cost more than 0 dollars all of a sudden it is falling on deaf ears. I recently had this happen, despite a issue where a testing system was pointing to a production database due to human error. Was not enough to convince anyone of anything.

Best I could suggest is if you’re able to, at least have a local development environment on your laptop inside an VM for the company so you can test out your changes first, though I know this is not always possible.

Edit: If you can get a couple of your coworkers (or operators at the plant/comany you’re working at who have had their jobs made harder due to hot fixes) on your side, it might help as well.

1 Like

It’s unfortunate that your bosses think that way. But the key to getting them to open their eyes is to talk to them about what matters to them - money. You need to show them that a proper development system will save them money in the long term. This could be by documenting the lost hours debugging the production system that could have been avoided with testing on dev, or the time rebuilding your prod system after a screw up. And potentially positioning the dev system as a fallback for your prod system, thus eliminating risk of downtime.

So rather than saying you need a dev environment, you need to show why you need a dev environment, and how not having one is hurting your company.

Here are some questions that other people have asked about similar concepts:

Should I propose a big change as a newcomer?

How do I approach my new boss in my new job about major changes in my working environment?

How to deal with a boss who has lost touch with modern software development?

1 Like

At least you can always let yourself out of jail though if they sent you there :wink:

5 Likes

Heh, that might be more true than you know. I may or may not have the actual get out of jail keys (;

4 Likes