I jumped on the bandwagon a few years ago and have never looked back.
Today it looks like the best bang for the buck on a new table is the Audio Technica LP120:
It's based off of the legendary Technics sl-1200mk2 (which is what I use) which means it's rock solid but not beaten to death by a dj. Plus it can play 78rpm and reverse which mine can't (jealous!). Plus it's only about $200 now and that's about the best you can do on a new table without going into heavy audiophile territory. (Just look at some of the threads on the Steve Hoffman Music Forums...goodness...)
I always knew about analog warmth of vinyl, but what really got me was mastering and the loudness war. To be perfectly honest, I had lost all interest in music during high school. I couldn't understand why, but I just didn't enjoy listening to music anymore. It wasn't always the music being something I disliked-there was just never a time where I wanted to listen to anything.
And then in college I discovered the world of vinyl needledropping to digital formats. Oh. My. Dear. God. Music came back to me. I'm not usually one to make these kinds of statements, but I felt like a little kid who just couldn't wait to hear new things. It really did change my life. (3 long useless but fantastic years in the college AM station too!)
With a good quality rig and transfer chain, you can easily best most any CD simply because of the mastering. But I'd stick with FLAC and lossless formats because bouncing down to mp3 just seems like a waste. (I swore off all lossy formats years ago.) Now most people go to higher resolution flac (24/96 and up) and it can really bring out a difference.
It really all depends on the mastering of the music and how the record was cut. It can easily trump the CD...or not if it was not done with fidelity in mind.
If you're on a budget the best option is still to scan locally and through craigslist for a decent TT like an old Technics. These can be found in various models all usually for around $25-$50. Then grab an old stereo reciever with a phono input for around $25.
And now we get to one of the more finicky parts:
I must stress this: You need a good cartridge and needle. This is key to vinyl and most will ignore this step entirely. Be sure and find one you like and set it up properly in order to have proper playback and not damage your records.
Best cartridges to start with are the Shure m97xe (Straightforward sound, what I'm currently using and can be had for around $50), and the Audio Technica AT95e (good straightforward sound with a natural warmth $40-50 in most places).
You will also need a preamp if there isn't one built into your reciever. Don't use a USB one on a new TT because they really aren't that great.
Get a anti-static record brush. Especially if you can't afford to clean records now.
Last thing I can think of right now is pressing quality. When I started collecting, I wound up grabbing all kinds of things I wanted and not really thinking about pressings. This way I wound up with quite a few inferior record club pressings. Generally you want to avoid these and later re-pressings and always try to go for the original issue. Also, certain mastering engineers are revered for their particular cuttings of certain albums, and these will sound miles ahead of the other versions.(Most notably UK pressings of British albums, and Robert Ludwig mastered albums, like his cut of Led Zeppelin II.)
Check out the above mentioned SH forums for all kinds of more audio info. I joined over there two years ago and quickly became an addict.
And remember, vinyl is a constant building process. There's always something you can change or upgrade, but it's all about your personal preference.