Yeah, I used to program a lot in high school and loved making little games and whatnot, when I could create what I wanted to. I thought this is great and I would love to do this the rest of my life. Then I went to college for CS in 2000, and when I got into all the advanced calculus and whatnot, and some of the lower level language stuff, and realized I would have to do this crap forever, I just couldn't keep up and lost all interest and decided it was not what I wanted to do the rest of my life, and ended up dropping out.
I have always wanted to program and create games and whatnot, but I know I could never handle sitting there coding programs that I'm not passionate about for 8-12 hours a day. But the funny thing is, thanks to the emergence of easy access to publishing (Google Play, Steam, Amazon, iTunes Appstore, etc), and actual realistic income levels for indie developers, I have been self learning Java and intend to start programming some apps and games and hopefully make a supplemental income (or if I'm lucky, full income) on some app stores, and eventually move into C# which is very similar to Java, or C++. By being able to make what I want, and being able to be passionate about it, it makes a world of difference. I don't have to get into super complicated math and whatnot if I don't want to, and I'm not trying to program any super complicated things. I can program to my ability level.
For me personally, its miserable to go into super advanced math to make stuff for other people's designs, and much more enjoyable to program on my own terms. I would STRONGLY recommend before quitting your job, and enrolling and whatnot, either picking up a book and self-learning a useable but not super complicated language like Java or C# (I'd recommend 'Teach Yourself Java in 24 Hours', and 'Head First Java') or take a couple programming classes at a local junior college, and see if you enjoy it or hate it, and on what terms you enjoy it (doing it for yourself vs doing it under specific guidelines for an employer). I also work in the medical field (IT), and honestly I would much rather be a respiratory therapist the rest of my life, than program databases and business applications the rest of my life, but would choose indie development over either of those in a heartbeat, and you don't need a degree for that.