- 浏览: 107354 次
- 性别:
- 来自: 北京
文章分类
最新评论
-
zenoh:
是啊,我也纳闷,apache hadoop2.2.0的已经修复 ...
安装CDH5 hadoop2.2.0遇到的配置问题 -
Molisa:
你好,我有两个问题请教一下:1.为了密码安全,你有没有试过使用 ...
使用oozie调度sqoop从oracle导入hbase
hadoop版本:hadoop-2.2.0-cdh5.0.0-beta-1
今天在安装hadoop后,启动了namenode,在执行hadoop fs -put /tmp/test.dat /test命令后,报了下面的一个错:
上面的意思大概是说只有0个datanode,也就是说没有连接到namenode的datanode,但是我到DN节点上jps,datanode进程是存在的;我又用hadoop dfsadmin -report命令查看,发现确实是0个DN......
这是后只有查看DN的日志:
发现确实是DN一直连不上NN, 怀疑是防火墙的问题,因此把所有节点上的防火墙都关闭了,
上述错误消失,hdfs正常。
今天在安装hadoop后,启动了namenode,在执行hadoop fs -put /tmp/test.dat /test命令后,报了下面的一个错:
13/11/05 23:40:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/11/05 23:40:37 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test/test.dat._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2478) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514) put: File /test/test.dat._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. 13/11/05 23:40:37 ERROR hdfs.DFSClient: Failed to close file /test/test.dat._COPYING_ org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test/test.dat._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2478) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:330) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1226) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1078) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:514)
上面的意思大概是说只有0个datanode,也就是说没有连接到namenode的datanode,但是我到DN节点上jps,datanode进程是存在的;我又用hadoop dfsadmin -report命令查看,发现确实是0个DN......
这是后只有查看DN的日志:
edSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:04,222 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:04,223 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: master1/10.95.3.100:8020 2013-11-06 16:42:10,226 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:11,228 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:12,230 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:13,232 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:14,234 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:15,238 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:16,241 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:17,243 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2013-11-06 16:42:18,245 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master1/10.95.3.100:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
发现确实是DN一直连不上NN, 怀疑是防火墙的问题,因此把所有节点上的防火墙都关闭了,
上述错误消失,hdfs正常。
发表评论
-
-snn.saturn.local:8485 failed on connection exception: java.net.ConnectException
2014-05-07 15:12 02014-01-26 00:46:35,631 INFO ... -
Hadoop 归档 和HIVE 如何使用har 归档 文件
2014-05-01 23:47 1025Hadoop archive 唯一的优势可能就是将众多的小 ... -
Hadoop的JVM重用
2014-05-01 20:21 1068Hadoop中有个参数是mapred.job.reuse. ... -
Hadoop 优化总结(一)
2014-04-27 12:09 15541. 使用自定义Writable 自带的Text很好用,但 ... -
hadoop调优
2014-04-26 12:01 0硬件级别提高磁盘IO的 ... -
启动hadoop 报错:A fatal error has been detected by the Java Runtime
2014-04-25 10:10 0今天在重启hadoop集群的时候,一个节点报了下面的一个错,由 ... -
java.io.IOException: Cannot obtain block length for LocatedBlock{BP-
2014-04-25 10:07 0今天的job在执行的时候,出现了下述的异常: java. ... -
hadoop报的一个job提交的错
2014-04-24 10:56 0Diagnostics: Application a ... -
hadoop hdfs HA 启动不了ZKFC
2014-04-23 11:47 0测试环境master1 系统重做了,原来的所有数据都丢失,后来 ... -
Hadoop平台优化综述(一)
2014-04-19 18:48 01. 概述 随着企业 ... -
大数据处理分析的六大最好工具
2014-04-19 11:59 0Hadoop Hadoop 是一个能够对大量数据进行分布式处 ... -
Shuffle$ShuffleError
2014-04-06 10:10 0Error: org.apache.hadoop.mapr ... -
hadoop2.2.0 CDH 源码编译
2014-03-27 13:29 01 下载 源码 wget chd5 2 安装软件 yum ... -
Map/Reduce Task JVM 堆大小设置优化
2014-03-13 15:15 1865前一阵子发现用户提交的hive query和hadoop jo ... -
hadoop参数配置优化
2014-03-13 15:12 683hadoop.tmp.dir 默认值: /tmp 说明: 尽 ... -
Hadoop为什么处理小数据量时效果不好?
2014-03-11 16:50 1127Every file, directory and bloc ... -
Hadoop : MapReduce中的Shuffle和Sort分析
2014-03-03 14:06 860MapReduce 是现今一个非常流行的分布式计算框架,它被 ... -
关于Hadoop的shuffle
2014-03-03 11:16 857我们知道每个reduce task输入的key都是按照k ... -
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError:
2014-03-01 23:17 4214hadoop执行JOB在reduce阶段 ... -
hadoop Caused by: java.io.IOException: Filesystem closed
2014-02-25 13:46 6480今天在执行hive的时候报了下面的错: 2014-02 ...
相关推荐
CDH5 Hadoop发行版离线安装手册,亲自实验的经验总结
hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装
Hadoop安装部署手册,使用CDH在线离线部署大数据集群环境
CDH5 Hadoop集群完全离线安装说明
一开始使用的是原生的hadoop2.6.0编译的hadoop.dll,但是一直遇到这个问题Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String...
Centos 7安装配置Hadoop生态圈CDH5版本 其中包括hadoop hbase hive spark 等组件安装
CDH_hadoop安装汇总整理,基于CDH平台使用hadoop,本文亲自整理,没有坑。
CDH5安装手册:Apache Hadoop的服务的部署比较繁琐,需要手工编辑配置文件、下载依赖包等。Cloudera Manager以GUI的方式的管理CDH集群,提供向导式的安装步骤。由于需要对Hive,HBase,Impala,Spark进行功能测试,就...
Centos6.7 + CDH5.4.5 HADOOP 集群环境离线安装
使用CDH搭建hadoop集群,yarn集群,内附详细搭建流程,并记录了里面踩过的一些坑。
CDH搭建hadoop环境文档指南,本文档是搭建基于CDH的 hadoop开发环境的文档,详细描述了如何搭建CDH的环境,同时还有其他环境的搭建,如jstorm,ElasticSearch等
cdh_hadoop5。8 kerberos认证,快捷不繁琐
cdh离线安装文档,安装hadoop yarn 等标准资源。配置cdh
而搭建CDH_Hadoop不是一件容易的事情,步骤很多,配置繁琐,这也是大数据门槛高的一个体现,很多新手都卡在这一关。经过实践和参考相关资料,个人总结整理成文档,为自己为他人,希望都有帮助。
Ubuntu安装CDH集群 CDH CM NTP HADOOP Spark 等插件集成
之前一直在网上找社区版hadoop的配置,发现CDH安装直接界面搞定,自动生成配置。供检查学习。
(CDH5.13.1)hadoop集群搭建手册,搭建流程都有包括,强烈推荐
CDH (Cloudera's Distribution, including ...Cloudera Manager则是为了便于在集群中进行Hadoop等大数据处理相关的服务安装和监控管理的组件,对集群中主机、Hadoop、Hive、Spark等服务的安装配置管理做了极大简化。
CDH版本Hadoop 伪分布式环境搭建 服务规划 步骤 第一步:上传压缩包并解压 cd /export/softwares/ tar -zxvf hadoop-2.6.0-cdh5.14.0.tar.gz -C ../servers/ 第二步:查看Hadoop支持的压缩方式以及本地库 cd /...
本文档主要以使用Cloudera Manager5.9.3 自动搭建Hbase集群为例,介绍如何使用Cloudera Manager自动搭建Hadoop相关集群,简化为多台服务器搭建Hadoop集群工作,提高工作效率。至于如何使用Cloudera Manager来管理...