To clear an alien nest, in base Beyond Earth simply walk onto the tile containing the nest. If an alien unit is present on said tile, you will need to defeat the unit or wait until it moves. Nests yield a bounty of Energy and, with the Scavenging Virtue, Science. In the Rising Tide expansion, players must manually select the option, destroy Alien Nest, in order to destroy one. Destroying an Alien Nest will result in angering nearby Aliens.
You should note, however, that a Xenomass, Fixarite or Floatstone resource covered by Miasma will often spawn a new nest. This does not seem to happen in every circumstance, and simply adding Miasma to an existing Xenomass plot will not spawn a new nest.
Alternatively, leaving a nest 60 turns within your territory whilst not attacking the aliens will allow them to calm down and consider you non-threatening, and so not attacking your colonists or military units when they pass by. Eventually, after several turns of living within your borders, the aliens will regard your people as their own, eventually giving you a sort of independent free militia force. This will ONLY apply to the nests within the borders, foreign hives will respond accordingly. Additionally, the rate at which you gain positive "influence" with aliens seems to be at least partially determined by how many nests you have within your borders. If you get a really lucky colony with access to 3 nests and you don't attack the aliens you can sometimes get them to be allies after just a dozen turns or so.
Be advised that aliens are NOT barbarians, so they are highly feral and not tactical, and you can lure them into ambushes. After antagonizing the aliens for some time, nests can spawn Siege Worms.
In the Rising Tide expansion pack, Alien nests can also be found in the ocean. As with the changes in the land gameplay in BERT, nests are not automatically destroyed when a unit travels over them.
Another change in rising tide is that the player must manually pillage nests. They do not go away by just moving a unit on top of them.