I had a cellphone once. I got it as part of a marketing study I participated in. The company was studying whether people with WAP-enabled cell phones would get any value out of having text-message ads sent to them. (Short answer: no, or as I put it in one of the feedback surveys I filled out, "This is the stupidest technology ever.") Anyways, they paid for it, so I got to play around with one for free and develop some opinions.
The big thing is, I determined that a cell phone is the kind of thing that I would be really picky about because it's a device you use all the time, and if there are little details that bug you, constant exposure will soon develop them into gigantic annoyances. So that's the first barrier: it would be a lot of effort to find a mobile phone that I would be content with.
Another barrier is the cost. In comparison to a landline (which I have to keep because we have DSL), which just costs $N per month, a mobile phone has all kinds of complicated pricing options for in-network versus out-of-network, roaming charges, long distance (or not), time of day, and so on. The problem here is that, again, it's a huge amount of effort to find a plan that is a good match for your usage patterns (which, for me, are currently unknown). Worse yet, this is the kind of thing where if I get it wrong, I will feel enormously ripped-off. I know that a lot of this stuff is getting better, depending on which provider you go with, but there's still too many possibilities for things like taking the phone on a trip, not realizing you'll be stiffed with obnoxious out-of-area connection charges and being disgruntled for weeks after you get your phone bill.
Finally, cell phones don't quite behave properly. They're getting close, but there's still some pieces missing. Garrison Keillor did this amusing bit on Prarie Home Companion a few years ago in which he observes that 90% of all cell-phone conversations are interchangeable: they basically consist of "I am here, I am on my way there, and I'll be there in half an hour." And it's true; many of the instances where I would want a mobile phone it would just be so that I can find out where someone else is in transit and what their ETA is (or vice-versa). And for that purpose, it just seems like an awful lot of extra rigamarole to have to call the person, get them to answer, talk to them, and extract the data. Especially if they're driving. Why shouldn't I just be able to ping their phone and ask "where are you?" If my friend's GPS-enabled cell phone shows that they're on highway 36 headed north at about 65 mph, I know that
they'll be here in a couple minutes. I don't have to bother the person to find out what I need to know. (Sure, you need a clever whitelist mechanism that grants people permission to ping your phone, but this is a relatively simple UI/social engineering problem.)
Similarly, phone-is-ringing-you-must-answer-it-NOW is an outmoded interaction model. Phones should support both synchronous and asynchronous interaction. Sometimes, I just want to leave a message. Why can't I call someone's voicemail? And if I have an incoming call, there should be some way for me to quickly and unobtrusively identify the caller and either a) answer the phone now, b) bounce them directly to voicemail, or c) have my phone answer with a message that says "Hi! I will talk to you, gimme just a minute to wrap up whatever it is I'm doing (or hang up and I'll call you right back)." Because that makes the phone less interrupty -- it becomes more like a person who wants to talk to you wandering over and sort of hovering in the background for a minute until you're available, which is a social interaction that you can be polite about.
The bottom line is that I'm stingy, and I can't quite bring myself to shell out the money for an expensive and very personal gadget that doesn't quite do what I want it to in so many little ways, when I have a cheap alternative that works 95+% of the time. (Oh, and let's not forget that sound quality is still not fantastic.)