Men has worn bags since they needed to carry things along.
However, it is know that in the 14th to 17th century, before pockets was a common thing, men was wearing small bags held up by belts called a girdle pouch. These were usually hung around the waste or a leg.
Later the when the pockets came around, the girdle pouches was less needed, but was instead used as a form of "perfume" to overcome BO. The bags was called "swete bagges" and was filled with nice smelling things.
More history to follow...