It sounds like you're aware of your tendency to think about everything that can go wrong.
Experienced Cautious Developers often learn to follow the mantra YAGNI, you ain't gonna need it, when they try to return to a lean, agile and productive workflow after getting too choked up in the weeds of failure-mode-analysis-gone-amok.
However, if you are indeed writing something in a domain where that level of care is no less than what Professionalism demands, then you should realize that your "velocity", your "productivity", in net terms, is measurable by how much good (or harm) you are doing to your company, your customers, and the software suite, or product family you are building or maintaining.
Remember to:
Include total cost of maintenance, total cost of ownership, and total cost of deploying and maintaining solutions when you consider changes in your approach. Going faster and making more mistakes may or may not make things better.
If you work in a good company, you can probably discuss this in your own team, and with your own supervisor, without it being a Career Limiting Move. If you can't, now is a good time to find that out, and find a new job.