By default buildout used pypi.org has its index - the location it downloads the eggs from. The server can sometimes be very slow. So, basically two steps to help speeding things up:
Alternative 1
Go to http://www.pypi-mirrors.org/, and choose a fast mirror. At the time of writing this is e.g. http://g.pypi.python.org/. Set the index section in the buildout accordingly.
[buildout]
index = http://g.pypi.python.org/simple
Alternative 2
Make things even faster (and potentially usuable offline) by using an eggproxy. Follow the instructions on https://pypi.python.org/pypi/collective.eggproxy, and set a fast mirror as the index (see step1).
Sometimes I get timeout issues with using the eggproxy though - even though I set a high timeout using "-t 120" when running buildout. In the end waiting for eggproxy to finish downloading and restarting buildout helps.