There are many differences between a cheap machine and the more expensive machines. The price *does* make a difference, although it is a personal decision whether that difference justifies the price. Of course a BMW runs smoother than a Hyundai, but some people can't afford a BMW, and some people who can are just happy enough with a Hyundai.
Some of the major differences ... first, you need to realize what espresso is. It's supposed to be the essence of coffee, purified and concentrated. You're trying to take a mound of beans, and extract from it exactly all the good essence of coffee that's in it ... any less and you lose some of the flavor ... any more and you start getting the bad stuff. It's a pretty narrow window, really ...
So, to start, the cheap machines are essentially boilers connected to the portafilter (which holds the coffee). Water boils, forcing steam through the pipes and through the coffee. Since the good stuff in coffee is best extracted at a lower temperature than steam, these machines end up not only scorching the coffee (sometimes creating a nasty burnt smell/taste), but extracting mostly the bitter tastes. In addition, the amount of water that ends up in contact with the grinds is inconsistent, and the amount of time that it is in contact with the grinds is inconsistent.
More expensive machines basically control those variables better. First and foremost, they do not work by forcing steam thru the coffee. They typically heat water up in a boiler, to the proper temperature just below that of steam ... then, some mechanism (e.g., pump or piston) delivers that water to the coffee at exactly the right pressure (I think it's something like 9 psi). That's oversimplifying ... there's actually a complex interaction between how fine you grind the coffee, how hard you pack (tamp) the coffee, how much water you run through the machine, etc.
Anyway, as you can see, it starts getting pretty complicated. In addition, while it's easy to say "water at the proper temperature", it's actually a veritable engineering feat to maintain water temperature, especially considering the water is not static ... it's being pumped thru pipes, and more is being added to the boiler constantly.
These are some of the basic variables that differentiate machines. As you can see, the dept store espresso machine is pretty primitive comparatively. If you can't see the difference, consider this ... properly made espresso (which you won't find at most of even the "best" cafes/restaurants) shouldn't be very bitter at all.
That said, it's still a personal choice. My father likes the harsh bitterness of coffee. I still can drink a latte at Starbucks -- the milk and sugar mask the bitter taste -- but along with most of the good tastes, yielding a sweetish drink with a bitter undertone, nothing more. Once in awhile, however, I drop by Gimme Coffee in Brooklyn, and they serve me a drink that tastes yummy without any sugar at all ... and that's why I no longer own a $30 espresso machine.
Some suggestions for machines ... decide first of all if you're in it for the journey or the destination. If you want good espresso with minimal work, look at some of the automatics. If you can't afford that and are interested in spending some time playing with things, look into a Rancilio Silvia. If even that's too expensive, you could get away with a Starbucks Barista ... but that's really the absolute minimum, and you are likely to outgrow it quickly. If you're happy with drinking just milk drinks with sugar, and don't want to spend the money, you could be happy enough with a $30 steam machine.
Hope that helps. You gotta just remember, espresso is the king of coffee ... it's an attempt to pull out the best parts of coffee as efficiently as possible. It's a perfect balance of time, pressure, heat, water, and coffee.
Oh, and remember ... the first most important step is to get a good burr grinder. That's probably the single best first step to take, since coffee beans start losing flavor quickly as soon as they're broken up. Within minutes, a good part of the flavor is gone. Within a day, all the good stuff is gone, leaving you with a kind of dull "coffee" smell. If you don't believe, me, go to a gourmet store and grind some coffee ... smell it right away, then smell it again the next day. Better yet, order some fresh-roasted coffee from some online roasters, like Intelligentsia, Vivace, Gimme, Terroir, Stumptown, or Sweet Maria's. They ship you coffee roasted within a few days, instead of months like most grocery stores. The smell will knock your boots off, I promise.
j