User-agent: * (or)  a specific crawler(BadBot)
Disallow: /   (or)  a  specific directory alone