They are the simplest machines and usually have the fewest features. The boiler inside heats up the water and it uses the pressure of the steam to force the water through your coffee grounds. It's a pretty simple process, which is why the machines are inexpensive. But steam alone will only reach a pressure level of 3 bars (the standard unit for measuring pressure). That's not very strong, and won't be enough to create the foamy crema layer on your espresso. To get enough steam, these machines will have to heat the water to at least 200 degrees, which will create an extremely hot cup of espresso. There is some danger in the kitchen at these temperatures, and the high heat can effect the overall flavor too.
Pump Powered Espresso Pump powered espresso machines are more expensive because they are more complicated equipment. They use an electric pump to pressurize the water rather than just heat. With this added power, a pump machine can get between 8 and 15 bars of pressure instead of just 3. You get a bolder espresso and a good layer of crema. The pump also means you don't have to heat the water to such high heat. It's safer to use, and means a less scalding shot of espresso.
On the other hand, the lower temperature is not sufficient for foaming or steaming milk. The cheaper pump machines may not have any milk foaming attachments because of this, but the more advanced ones usually have a second thermostat inside that will allow for hotter water if you want to steam your milk. When you do steam milk, you will usually have to wait again while the water heats up to steaming temperature. High-quality pump powered espresso machines sometimes have 2 water reservoirs and 2 boilers to accommodate both purposes without any lag time.
Other Types of Espresso Machines Though choosing between pump or steam espresso machines may be the main options, there are other ways to make espresso. One of the simplest ways is with a stove-top moka pot. Bialetti is the biggest manufacturer of these little espresso kettles, that look like metal coffee pots with a narrow waist in the middle. Water goes in the bottom, and packed coffee grounds are added to a little funnel between the two sections. When the water is heated, it steams and bubbles up through the middle to produce espresso in the top have of the pot. It's no where near as strong as even a steam powered machine, but its a simple (and very cheap) alternative to an electric machine.
Gaggia also makes a piston-style espresso machine with a long handle that you manually pull to force the water through the coffee grounds under pressure. You can get up to 8 bars this way, just as much as an electric pump machine. So even if pump or steam powered espresso machines aren't for you, you do have other options.