I guess it's different for everyone, but I knew someone who drank WAAAY too much coffee, like 10 cups a day. I guess it's an extreme case but the point is, one day she didn't have time for coffee and went the whole day without it. She ended up passing out and hallucinating...
So yeah, forget what I said about being a casual user, you would need to be a coffee maniac to have really bad withdrawals. But my point was just that coffee is a drug that can be abused just like coccaine or heroin... it's just not as strong (and has different effects).