Hadoop运行mapreduce实例时,抛出错误 All datanodes xxx.xxx.xxx.xxx:xxx are bad. Aborting… - 加俊 - ITeye博客

`

iammonster

浏览: 1778822 次
性别:
来自: 北京

最近访客更多访客>>

yuanyuan7891

sagadan

JianCaesar

led19880901

博主相关

博客

微博

相册

收藏

留言

关于我

文章分类

社区版块

存档分类

最新评论

奔跑的小牛：例子都打不开
如何使用JVisualVM进行性能分析
蜗牛coder：好东西[color=blue][/color]
Lucene学习：全文检索的基本原理
lovesunweina：不在haoop中是在linux系统中，映射IP的时候，不能使用 ...
java.io.IOException: Incomplete HDFS URI, no host
evening_xxxy：挺好的, 谢谢分享
如何利用 JConsole观察分析Java程序的运行，进行排错调优
di1984HIT：学习了~~~
ant使用ssh和linux交互如：上传文件

Hadoop运行mapreduce实例时,抛出错误 All datanodes xxx.xxx.xxx.xxx:xxx are bad. Aborting…

博客分类：

HBase

Hadoop Mapreduce Java Apache Linux

阅读更多

Hadoop运行mapreduce实例时,抛出错误 All datanodes xxx.xxx.xxx.xxx:xxx are bad. Aborting…

java.io.IOException: All datanodes xxx.xxx.xxx.xxx:xxx are bad. Aborting…

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2158)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)

java.io.IOException: Could not get block locations. Aborting…

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2143)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1735)

at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1889)

经查明，问题原因是linux机器打开了过多的文件导致。

用命令ulimit -n可以发现linux默认的文件打开数目为1024

修改/ect/security/limit.conf，增加hadoop soft 65535
再重新运行程序（最好所有的datanode都修改），问题解决

非常感谢淘宝团队：http://rdc.taobao.com/blog/dw/archives/245

分享到：

在windows平台下Eclipse调试Hadoop/Nutch | JAVA基础对象克隆实现java.lang.Clonea ...

2010-04-29 14:26
浏览 6370
评论(0)
分类:企业架构
查看更多

评论

发表评论

您还没有登录,请您登录后再发表评论

相关推荐

hadoop-mapreduce-client-core-2.5.1-API文档-中文版.zip: 赠送jar包：hadoop-mapreduce-client-core-2.5.1.jar；赠送原API文档：hadoop-mapreduce-client-core-2.5.1-javadoc.jar；赠送源代码：hadoop-mapreduce-client-core-2.5.1-sources.jar；赠送Maven依赖信息文件：...

hadoop-mapreduce-client-jobclient-2.6.5-API文档-中文版.zip: 赠送jar包：hadoop-mapreduce-client-jobclient-2.6.5.jar；赠送原API文档：hadoop-mapreduce-client-jobclient-2.6.5-javadoc.jar；赠送源代码：hadoop-mapreduce-client-jobclient-2.6.5-sources.jar；赠送...

hadoop-mapreduce-client-common-2.6.5-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-common-2.6.5.jar；赠送原API文档：hadoop-mapreduce-client-common-2.6.5-javadoc.jar；赠送源代码：hadoop-mapreduce-client-common-2.6.5-sources.jar；赠送Maven依赖信息...

hadoop-mapreduce-client-jobclient-2.6.5-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-jobclient-2.6.5.jar；赠送原API文档：hadoop-mapreduce-client-jobclient-2.6.5-javadoc.jar；赠送源代码：hadoop-mapreduce-client-jobclient-2.6.5-sources.jar；赠送...

hadoop-mapreduce-client-app-2.6.5-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-app-2.6.5.jar；赠送原API文档：hadoop-mapreduce-client-app-2.6.5-javadoc.jar；赠送源代码：hadoop-mapreduce-client-app-2.6.5-sources.jar；赠送Maven依赖信息文件：...

hadoop-mapreduce-client-app-2.6.5-API文档-中文版.zip: 赠送jar包：hadoop-mapreduce-client-app-2.6.5.jar；赠送原API文档：hadoop-mapreduce-client-app-2.6.5-javadoc.jar；赠送源代码：hadoop-mapreduce-client-app-2.6.5-sources.jar；赠送Maven依赖信息文件：...

hadoop-mapreduce-client-core-2.7.3-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-core-2.7.3.jar；赠送原API文档：hadoop-mapreduce-client-core-2.7.3-javadoc.jar；赠送源代码：hadoop-mapreduce-client-core-2.7.3-sources.jar；赠送Maven依赖信息文件：...

hadoop-mapreduce-client-app-2.7.3-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-app-2.7.3.jar；赠送原API文档：hadoop-mapreduce-client-app-2.7.3-javadoc.jar；赠送源代码：hadoop-mapreduce-client-app-2.7.3-sources.jar；赠送Maven依赖信息文件：...

hadoop-mapreduce-examples-2.7.1.jar: hadoop-mapreduce-examples-2.7.1.jar

hadoop-mapreduce-client-core-2.6.5-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-core-2.6.5.jar 赠送原API文档：hadoop-mapreduce-client-core-2.6.5-javadoc.jar 赠送源代码：hadoop-mapreduce-client-core-2.6.5-sources.jar 包含翻译后的API文档：...

Ubuntu系统上Hadoop与MapReduce 运行实例 .docx: Ubuntu系统上Hadoop与MapReduce 运行实例

Hadoop源代码分析（包org.apache.hadoop.mapreduce）: 包org.apache.hadoop.mapreduce的Hadoop源代码分析

hadoop-mapreduce-client-common-2.7.3-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-common-2.7.3.jar；赠送原API文档：hadoop-mapreduce-client-common-2.7.3-javadoc.jar；赠送源代码：hadoop-mapreduce-client-common-2.7.3-sources.jar；赠送Maven依赖信息...

Hadoop mapreduce实现wordcount: Hadoop 用mapreduce实现Wordcount实例，绝对能用

hadoop-mapreduce-client-jobclient-2.5.1-API文档-中文版.zip: 赠送jar包：hadoop-mapreduce-client-jobclient-2.5.1.jar；赠送原API文档：hadoop-mapreduce-client-jobclient-2.5.1-javadoc.jar；赠送源代码：hadoop-mapreduce-client-jobclient-2.5.1-sources.jar；赠送...

hadoop-mapreduce-client-jobclient-2.7.3-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-jobclient-2.7.3.jar；赠送原API文档：hadoop-mapreduce-client-jobclient-2.7.3-javadoc.jar；赠送源代码：hadoop-mapreduce-client-jobclient-2.7.3-sources.jar；赠送...

hadoop-mapreduce-client-jobclient-2.5.1-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-jobclient-2.5.1.jar；赠送原API文档：hadoop-mapreduce-client-jobclient-2.5.1-javadoc.jar；赠送源代码：hadoop-mapreduce-client-jobclient-2.5.1-sources.jar；赠送...

hadoop-mapreduce-examples-2.6.5.jar: hadoop-mapreduce-examples-2.6.5.jar 官方案例源码

hadoop-mapreduce-client-shuffle-2.5.1-API文档-中文版.zip: 赠送jar包：hadoop-mapreduce-client-shuffle-2.5.1.jar；赠送原API文档：hadoop-mapreduce-client-shuffle-2.5.1-javadoc.jar；赠送源代码：hadoop-mapreduce-client-shuffle-2.5.1-sources.jar；赠送Maven依赖...

hadoop-mapreduce-client-shuffle-2.5.1-API文档-中英对照版.zip: 赠送jar包：hadoop-mapreduce-client-shuffle-2.5.1.jar；赠送原API文档：hadoop-mapreduce-client-shuffle-2.5.1-javadoc.jar；赠送源代码：hadoop-mapreduce-client-shuffle-2.5.1-sources.jar；赠送Maven依赖...

Global site tag (gtag.js) - Google Analytics