The Brute is the earliest Barbarian unit. Although it is weak to stronger units, such as Spearmen and Swordsmen, a fortified Brute can stand toe-to-toe with most units up to the Pikeman, especially if the Brute is in hill, forest, or jungle terrain. If you lack superior units, overrunning Brutes with a melee unit backed by an archery unit, or simply multiple melee units at once will do the job. Once you have units such as Longswordsmen and Musketmen, you will easily be able to destroy Brutes with one attack.
If you play as the Germans, you can convert Brutes to your side by capturing Barbarian encampments early on in the game. Additionally, in the Brave New World expansion, the reformation belief Heathen Conversion allows Missionaries to convert Barbarians to your side.
Barbarian is a term used by civilized people to describe those who have not yet embraced civilization. Such people are generally seen as inferior and warlike. The term was coined by the Greeks, to whom it meant "anybody who wasn't Greek." In Europe, the term was applied to the various raiders who rode out of the East and pillaged everything they could get their hands on, even when these raiders came from a perfectly fine civilization of their own. In the New World, the term was applied to the aboriginal inhabitants (even though it was the Europeans who were doing the actual pillaging). In short, the question of who the barbarians are depends a lot on which end of the spear one happens to be on at any given moment.