Numbers Do Matter
While the actual number of groomsmen you have isn't important, you should consider how many bridesmaids your fiancée will have. Your wedding party doesn't need to be perfectly symmetrical, but try to keep the ratio as even as possible since you'll need enough guys to walk the gals down the aisle. If you end up with fewer groomsmen, think how happy the guys will be to escort two ladies!
Keep in mind that your groomsmen will also double as ushers. A good rule of thumb is to have one usher for every 50 guests. Because of this, bigger weddings call for bigger wedding parties, and vice versa. The key is to keep things proportional.