다양한 bot 의 세계 > 기술자료

• 목록

검색엔진에서 방문하는 다양한 bot 을 정리해봤습니다.  robots.txt 에 활용합니다.

1~0
1Noonbot
360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
360Spider-Image
360Spider-Video

A
AdsBot-Google
Mozilla/5.0 (compatible; AdsBot-Naver/1.0; +http://searchad.naver.com)
AhrefsBot/5.0
AhrefsBot/5.1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

B
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
BaiduSpider
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Baiduspider-image
Baiduspider-news
Baiduspider-video
Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://ecommerce.bigdatacorp.com.br/faq.aspx)
BLEXBot/1.0
BusinessBot: Nathan@lead-caddy.com
Bytespider

C
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
Contacts-Crawler/0.2 (+http://www.scrapinghub.com)
cowbot

D
Mozilla/5.0 (compatible; Dataprovider.com;)
Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
daumos
DialogSearch.com Bot 1.4;http://dialogsearch.com/webmasters
Digincore bot
Digincore crawler bot. See https://www.digincore.com/crawler.html for rules and instructions.
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot)
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
DuckDuckGo-Favicons-Bot/1.0

E
empas
Exabot/3.0

F
facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

G
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot-Image
Mozilla/5.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Googlebot-news

H
360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

I
ia_archiver
IDBot/1.1

J


K
kisaBot v1.0
KisaBotD/2.0

L
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)
linkdexbot/2.2

M
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible, Linux x86_64; Mail.RU_Bot/Fast/2.0; +//go.mail.ru/help/robots)
Mozilla/5.0 (compatible, Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots)
Mozilla/5.0 (compatible, Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)
meanpathbot/1.0
Mediapartners-Google
MediavBot/1.0 (+http://www.mediav.com/)
Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
mfibot/1.1 (http://www.mfisoft.ru/analyst/; <admin@mfisoft.ru>; en-RU)
MJ12bot/v1.4.4
MJ12bot/v1.4.5
MJ12bot/v1.4.6
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

N
Naverbot
NeumobBot/0.2.16 (+http://www.neumob.com/bot)

O
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)

P
Mozilla/5.0 (compatible; PageAnalyzer/1.5;)
PagesInventory (robot +http://www.pagesinventory.com)
Perviibot/1.0
Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm)
psbot
PrivacyAwareBot/1.1

Q
 
R
RankvalBot/1.0
roboto
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)
RSNBOT CONTACT US 031-924-2025

S
SafeDNSBot (https://www.safedns.com/searchbot)
semanticbot (info@semanticaudience.com)
SemrushBot/0.98~bl
SemrushBot/0.99~bl
SemrushBot/1~bl
SemrushBot/1.1~bl
Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)
Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)
Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
Slurp
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
spbot/5.0
spbot/5.0.2
spbot/5.0.3
SurveyBot/2.3 (DomainTools)

T
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)
TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html)

U
Unnecessarybot
Uptimebot/0.1.73
Uptimebot/0.2.42
Uptimebot/0.2.43
Uptimebot/1.0
UXCrawlerBot

V
Virusdie crawler/3.0

W
WBSearchBot/1.1
WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1
wisebot
WoTBoT
Wotbox/2.01 (+http://www.wotbox.com/bot/)

X
XoviBot/2.0

Y
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
yahoo-blog
Yahoo-MMCrawler
Yahoo Pipes 1.0
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)
Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
YisouSpider

Z
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)


참고사이트
http://www.robotstxt.org/db.html
http://www.botreports.com