- 浏览: 1243235 次
- 性别:
- 来自: 深圳
文章分类
- 全部博客 (608)
- 数据结构 (2)
- AJAX (3)
- 设计模式 (3)
- java (117)
- js (14)
- css (9)
- jsp (10)
- 杂文 (49)
- htmlparser (6)
- 数据库 (29)
- 算法 (14)
- 数据挖掘 (11)
- 电脑杂症 (12)
- 网络爬虫 (7)
- 应用服务器 (9)
- PHP (2)
- C# (14)
- 测试 (3)
- WEB高性能开发 (3)
- swt (1)
- 搜索引擎 (16)
- HttpClient (4)
- Lite (1)
- EXT (1)
- python (1)
- lucene (4)
- sphinx (9)
- Xapian (0)
- linux (44)
- 问题归类 (1)
- Android (6)
- ubuntu (7)
- SEO (18)
- 数学 (0)
- 农业资讯 (12)
- 游戏 (3)
- nginx (1)
- TeamViewer (1)
- swing (1)
- Web前 端 (1)
- 主页 (0)
- 阿萨德发首发身份 (0)
- 软件设计师 (0)
- hibernate (5)
- spring3.0 (5)
- elastic (1)
- SSH (3)
- ff (0)
- oracle 10g (9)
- 神经网络 (1)
- struts2.0 (2)
- maven (1)
- nexus (1)
- 辅助工具 (3)
- Shiro (1)
- 联通项目 (0)
- 2014年专业选择 (0)
- freemarker (1)
- struts1.2 (8)
- adfasdfasfasf (0)
- TortoiseSVN (1)
- jstl (1)
- jquery (1)
- eclipse plugin (0)
- 游戏外挂 (1)
- 推广 (0)
- 按键精灵 (1)
- ibatis3.0 (1)
最新评论
-
水野哲也:
不不不, 这个您真错了!其实是你引用的那个jsp和本身的jsp ...
解析关于jsp页面指令冲突问题contentType="text/html;charset=UTF-8" -
caobo_cb:
import xx.cn.weibo.Util;
[ java版]新浪微博之ruquest_token篇 -
caobo_cb:
你好 Util包没有
[ java版]新浪微博之ruquest_token篇 -
小桔子:
你好!我遇到个问题 max_allowed_packet值总是 ...
mysql查询占用内存,优化的技巧 -
donghustone:
谢谢大神!
用JSmooth制作java jar文件的可执行exe文件教程(图文)
GOOGLE
---------------------------------------------------------------------
66.249.70.212 - - [11/Jan/2009:00:03:35 -0700] "GET www.vidun.com/user-f2fc990265c712c49d51a18a32b39f0c.html?umid=f2fc990265c712c49d51a18a32b39f0c HTTP/1.1" 200 8148 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.212 - - [11/Jan/2009:03:27:23 -0700] "GET www.youxigao.com/images/pink/demo.gif HTTP/1.1" 200 2367 "-" "Googlebot-Image/1.0"
Referer: ""
UserAgent: "Googlebot-Image/1.0"
209.85.238.7 - - [11/Jan/2009:00:02:58 -0700] "GET www.youxigao.com/rss/c/1009 HTTP/1.1" 404 37 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"
Referer: ""
UserAgent: "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"
百度
---------------------------------------------------------------------
60.28.22.38 - - [11/Jan/2009:01:28:09 -0700] "GET www.vidun.com/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "http://www.vidun.com/" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
Referer: ""
UserAgent: "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
YAHOO
---------------------------------------------------------------------
202.160.180.81 - - [11/Jan/2009:00:02:44 -0700] "GET www.vidun.com/ HTTP/1.0" 200 14250 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)"
67.195.37.167 - - [11/Jan/2009:00:23:00 -0700] "GET www.vidun.com/postmsg-tech-2-120.html?type=tech&id=2&tid=120 HTTP/1.0" 200 12609 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
有道
---------------------------------------------------------------------
2008-03-04 09:54:12 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 61.135.219.7 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) - 200 0 0
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
61.135.249.120 - - [11/Jan/2009:09:44:46 -0700] "GET vidun.com/robots.txt HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
SOSO
---------------------------------------------------------------------
58.61.164.207 - - [11/Jan/2009:03:13:53 -0700] "GET www.vidun.com/robots.txt HTTP/1.1" 404 - "http://www.vidun.com/robots.txt" "Sosospider+(+http://help.soso.com/webspider.htm)"
Referer: ""
UserAgent: "Sosospider+(+http://help.soso.com/webspider.htm)"
2008-03-04 10:48:28 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 124.115.4.218 Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm) http://www.tatabao.com/ 200 0 0
Referer: ""
UserAgent: "Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)"
Sogou
---------------------------------------------------------------------
219.234.81.41 - - [11/Jan/2009:03:26:49 -0700] "GET www.vidun.com/ HTTP/1.0" 200 14250 "-" "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"
Referer: ""
UserAgent: "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"
220.181.61.217 - - [11/Jan/2009:13:10:57 -0700] "GET www.youxigao.com/play/3615?id=3615 HTTP/1.1" 302 5 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.19.74 - - [11/Jan/2009:06:20:37 -0700] "GET www.vidun.com/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "-" "Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.19.78 - - [11/Jan/2009:10:55:18 -0700] "GET www.vidun.com/robots.txt HTTP/1.1" 404 - "http://pic.sogou.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
Referer: "http://pic.sogou.com/"
UserAgent: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
219.234.81.27 - - [11/Jan/2009:23:53:41 -0700] "GET www.vidun.com/ HTTP/1.1" 200 14271 "-" "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"
Referer: ""
UserAgent: "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"
2008-03-04 17:23:59 W3SVC226223753 222.33.192.54 HEAD /index.php - 80 - 220.181.19.107 Sogou+head+spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) - 200 0 0
---------------------------------------------------------------------
66.249.70.212 - - [11/Jan/2009:00:03:35 -0700] "GET www.vidun.com/user-f2fc990265c712c49d51a18a32b39f0c.html?umid=f2fc990265c712c49d51a18a32b39f0c HTTP/1.1" 200 8148 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.212 - - [11/Jan/2009:03:27:23 -0700] "GET www.youxigao.com/images/pink/demo.gif HTTP/1.1" 200 2367 "-" "Googlebot-Image/1.0"
Referer: ""
UserAgent: "Googlebot-Image/1.0"
209.85.238.7 - - [11/Jan/2009:00:02:58 -0700] "GET www.youxigao.com/rss/c/1009 HTTP/1.1" 404 37 "-" "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"
Referer: ""
UserAgent: "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 10 subscribers; feed-id=8474979256887526569)"
百度
---------------------------------------------------------------------
60.28.22.38 - - [11/Jan/2009:01:28:09 -0700] "GET www.vidun.com/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "http://www.vidun.com/" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
Referer: ""
UserAgent: "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
YAHOO
---------------------------------------------------------------------
202.160.180.81 - - [11/Jan/2009:00:02:44 -0700] "GET www.vidun.com/ HTTP/1.0" 200 14250 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)"
67.195.37.167 - - [11/Jan/2009:00:23:00 -0700] "GET www.vidun.com/postmsg-tech-2-120.html?type=tech&id=2&tid=120 HTTP/1.0" 200 12609 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
有道
---------------------------------------------------------------------
2008-03-04 09:54:12 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 61.135.219.7 Mozilla/5.0+(compatible;+YodaoBot/1.0;+http://www.yodao.com/help/webmaster/spider/;+) - 200 0 0
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
61.135.249.120 - - [11/Jan/2009:09:44:46 -0700] "GET vidun.com/robots.txt HTTP/1.1" 404 - "-" "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
Referer: ""
UserAgent: "Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )"
SOSO
---------------------------------------------------------------------
58.61.164.207 - - [11/Jan/2009:03:13:53 -0700] "GET www.vidun.com/robots.txt HTTP/1.1" 404 - "http://www.vidun.com/robots.txt" "Sosospider+(+http://help.soso.com/webspider.htm)"
Referer: ""
UserAgent: "Sosospider+(+http://help.soso.com/webspider.htm)"
2008-03-04 10:48:28 W3SVC226223753 222.33.192.54 GET /index.php - 80 - 124.115.4.218 Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm) http://www.tatabao.com/ 200 0 0
Referer: ""
UserAgent: "Sosoimagespider+(+http://help.soso.com/soso-image-spider.htm)"
Sogou
---------------------------------------------------------------------
219.234.81.41 - - [11/Jan/2009:03:26:49 -0700] "GET www.vidun.com/ HTTP/1.0" 200 14250 "-" "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"
Referer: ""
UserAgent: "Sogou Web Sprider(compatible; Mozilla 4.0; MSIE 6.0; Windows NT 5.1; SV1; Avant Browser; InfoPath.1; .NET CLR 2.0.50727; .NET CLR1.1.4322)"
220.181.61.217 - - [11/Jan/2009:13:10:57 -0700] "GET www.youxigao.com/play/3615?id=3615 HTTP/1.1" 302 5 "-" "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.19.74 - - [11/Jan/2009:06:20:37 -0700] "GET www.vidun.com/vwsoft-vwantileechs-download.html?pr=vwantileechs&vi=download HTTP/1.1" 200 27406 "-" "Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
Referer: ""
UserAgent: "Sogou Orion spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)"
220.181.19.78 - - [11/Jan/2009:10:55:18 -0700] "GET www.vidun.com/robots.txt HTTP/1.1" 404 - "http://pic.sogou.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
Referer: "http://pic.sogou.com/"
UserAgent: "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
219.234.81.27 - - [11/Jan/2009:23:53:41 -0700] "GET www.vidun.com/ HTTP/1.1" 200 14271 "-" "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"
Referer: ""
UserAgent: "Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)"
2008-03-04 17:23:59 W3SVC226223753 222.33.192.54 HEAD /index.php - 80 - 220.181.19.107 Sogou+head+spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) - 200 0 0
发表评论
-
小卒:关键词竞争度分析方法
2012-04-11 01:01 851关键词分析在SEO中的地 ... -
404页面该怎么做?
2012-04-11 00:51 857404页面的目的是:告诉浏览者其所请求的页面不存在或链接错误, ... -
濮阳SEO:RSS也可以增加外链
2012-04-09 23:18 1331第一种是比较简单的直接提交到RSS目录站;主要是搜藏,书签等R ... -
如何分析关键字优化的难度?了解你的关键字竞争力
2012-04-09 09:06 870如何分析关键字优化的 ... -
浅谈如何建设高质量外链
2012-04-07 11:13 928做外链有许多渠道,人 ... -
教你如何分析百度蛛蛛Baiduspider日志
2011-03-02 13:47 1184今天将昨日的网站日志进行下分析,很多人都不知道网站日志该如何分 ... -
seo-关键字优化:关键字(一)
2011-02-23 17:24 1570seo-关键字优化:关键字 ... -
提升网站在GOOGLE中排名的方法
2011-02-19 10:31 955此文将介绍如何让你的网站成为google搜索中的焦点,如果你是 ... -
百度和谷歌更新日期和时间
2011-02-18 13:28 1046作为一名网站站长,来 ... -
网站收录减少怎么办
2011-02-17 22:49 901网站收录的突然减少往 ... -
什么是反向链接?
2011-02-17 14:15 755举个简单的例子,看上面这行内容中, “农联网”这三个字的链接指 ... -
robots 中 Sitemap 的 XML 格式 和 用法
2011-02-14 15:35 6169此文档介绍适用于 Sitemap 协议的 XML 架构。 S ... -
百度只收录一页的解决方法
2011-02-13 00:18 1156百度只收录一页,我想 ... -
百度只收录首页怎么办
2011-02-13 00:15 904对搜索引擎来说,最讨厌的就是JS,框架,漫天广告,弹窗。 所 ... -
让百度迅速收录的方法
2011-02-12 11:35 8241.在百度提交网址 但现 ... -
如何让搜索引擎更快的收录一个新站
2011-01-24 09:56 965目前,虚拟网络世界中,搜索引擎可以说占有很重要的地位。网民查找 ... -
新手怎样利用博客推广
2010-04-01 08:02 768第一招,撒网捕鱼!百度空间,新浪,雅虎,网易,阿里巴巴,和讯这 ...
相关推荐
以下javascript脚本代码可以实现判断访问是否来自搜索引擎。 代码如下: [removed] if(window.name != 'ad_app6'){ var r = document.referrer; r = r.toLowerCase(); //转为小写 var aSites = new Array('google.'...
fake_useragent json缓存 爬虫 fake 消息头
各种useragent,也是分析ua收集来的 涵盖各种手机机型的ua了,差不多1w多个
包含大量UserAgent数据信息,满足相关行业人员需要,相关数据为2015年至今,比较新,可放心使用
所以通过UserAgent判断请求的发起者是否是搜索引擎爬虫(蜘蛛)的方式是不靠谱的,更靠谱的方法是通过请求者的ip对应的host主机名是否是搜索引擎自己家的host的方式来判断。 要获得ip的host,在windows下可以通过...
C#代码获取WebBrowser中的 cookie和userAgent 参数,方法和请求类
谷歌浏览器的一款实用插件,用于修改UserAgent请求头,内置了各种客户端的默认请求头,例如Linux、windows、android、IOS等,使用方法也很简单,选中目标UA头,然后直接激活就可以了,插件表面显示为ON时,就表示...
手机最新UserAgent
解决UserAgent库最大请求次数
手机useragent.pdf 数据库资源
Laravel开发-useragent Laravel 4的用户代理包
复制代码 代码如下: function write_naps_bot(){ $useragent=get_naps_bot(); // echoExit($useragent); if ($useragent==”false”) return FALSE ; date_default_timezone_set(“Asia/Shanghai”); $date=date(“Y...
移动端useragent,18w条,包含platform,颜色深度信息
fake_useragent 本地文件,版本为 0.1.11。FakeUserAgentError('Maximum amount of retries reached') 彻底解决办法中,fake_useragent.json 文件需要从 herokuapp 下载。为无法打开 herokuapp 网站的朋友提供下载。
某网站pc端浏览最新useragent#动态分,如果感觉下载分数太高了可以私信我
改资源的注释已经很详细了,但是还是建议配合我的博客一起学习:博客地址:https://blog.csdn.net/qq_23832313/article/details/82775316
一年以来收集到的UserAgent数据,总数45184条,已做字符去重
fake_useragent_0.1.11.json
那么下面分享一款用php写的获取各搜索蜘蛛爬行记录代码 支持搜索引擎如下 可以记录Baidu,Google,Bing,Yahoo,Soso,Sogou,Yodao爬行网站的记录! php代码如下 复制代码 代码如下: <?php function get_naps_bot() { $...