adsense728x90


facebook microsoft google naver robots.txt




User-agent: baiduspiderDisallow: /ajax/Disallow: /album.phpDisallow: /checkpoint/Disallow: /contact_importer/Disallow: /feeds/Disallow: /file_download.phpDisallow: /hashtag/Disallow: /l.phpDisallow: /p.phpDisallow: /photo.phpDisallow: /photos.phpDisallow: /sharer/Disallow: /topic/User-agent: Bingbot...
User-agent: Googlebot
...

각 bot 별로 allow해 두되, 일부 directory만 비공개.




bot 종류는 상관없이 일부 directory만 비공개.
User-agent: *Disallow: /searchAllow: /search/aboutDisallow: /sdchDisallow: /groupsDisallow: /catalogsAllow: /catalogs/about



User-agent: msnbot-media Disallow: /Allow: /shopping/$Allow: /shopping$Allow: /th?User-agent: TwitterbotDisallow: User-agent: *Disallow: /account/Disallow: /bfp/search


naver는
User-agent: *
Disallow: /





덧글

댓글 입력 영역