You've got your best man, but the guys who are "next in line" are just as important! Unless you have serious family issues, always include your brothers and your fiancée's brothers -- friends come and go, but family is forever.
Other criteria? Pick men who are active participants in your life as a couple, and choose those who know you well and who you can still see yourself being friends with later in life. Think about childhood friends that you're still in close contact with.
Also, keep in mind that being in a wedding can be expensive, and some of your friends might not be able to foot the bill. If that's the case, would you be willing to pick up the expenses for them?
If you're having a hard time choosing your guys, start out by making a list of potential candidates and go from there. With enough thought and planning, you'll be confident that you have your best buds standing next to you on your wedding day!