#Default robots.txt file (which asks all bots to crawl slowly but still index everything). User-agent: * Crawl-delay: 1 #Disallow not found links #Disallow: /portfolio/portfolio-3-cols/2nd-category #Disallow: /portfolio/portfolio-4-cols #Disallow: /portfolio/portfolio-3-cols