Hiccups are the result of a diaphragm spasm. The diaphragm is a muscle that separates your abdomen from your chest and helps you inhale and exhale. When it spasms and suddenly contracts, your vocal cords then shut tight and trap air. This combination of factors causes the hiccup sound you're familiar with.
No one is completely sure why adults get hiccups, so it makes sense that the cause of hiccups in babies is also a bit murky. However, there are some reasonable hypotheses. One has to do with the size of babies. In adults, eating too much food is a known trigger for hiccups. Since babies are smaller than adults, it's relatively easy to fill their stomachs quickly. Another reason some babies get hiccups is that their lower esophageal sphincter isn't fully developed yet. This ring of muscle is what opens and closes to let food into your baby's esophagus. When your baby's lower esophageal sphincter isn't all the way developed, it might not be able to close completely. This allows food to leave the esophagus the way it came in, causing gastroesophogeal reflux and hiccups, in turn. As the baby grows, he'll finish developing and this problem will go away.
As with adults, babies can get hiccups if they swallow too much air. It's more likely that a baby will take in more air than adults because of all the time they spends crying and drinking out of bottles with air bubbles. To reduce the instance of swallowed-air hiccups, you should burp your baby often. Also, if you bottle feed instead of breastfeed, check the hole on the bottle's nipple to make sure it doesn't let out too much liquid at a time. If it flows too fast, your baby will take in air trying to keep up.