# Some files (on domain or on S3) we don't want crawled by the FreeFind spider # Per spec, no-spider must be the top of the path User-agent: * Disallow: /no-spider/