User-agent: * Disallow: /articles/ Disallow: /pdf/ Disallow: /url/ Disallow: /admin.php