`
shutiao2008
  • 浏览: 209647 次
  • 性别: Icon_minigender_1
  • 来自: 杭州
社区版块
存档分类
最新评论

hadoop put异常

HadoopJavaApacheSUNAccess 
阅读更多

09/01/19 17:32:41 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 WARN fs.FileSystem: "10.5.57.81:9000" is a deprecated filesystem name. Use "hdfs://10.5.57.81:9000/" instead.
09/01/19 17:32:42 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:42 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 4
09/01/19 17:32:43 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 3
09/01/19 17:32:44 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:44 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 2
09/01/19 17:32:45 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:45 WARN dfs.DFSClient: NotReplicatedYetException sleeping /user/hadoop/input/1.test retries left 1
09/01/19 17:32:48 WARN dfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hadoop/input/1.test could only be replicated to 0 nodes, instead of 1
 at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1120)
 at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
 at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)

 at org.apache.hadoop.ipc.Client.call(Client.java:715)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
 at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2448)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2331)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1920)

09/01/19 17:32:48 WARN dfs.DFSClient: Error Recovery for block null bad datanode[0]
put: Could not get block locations. Aborting...
Exception closing file /user/hadoop/input/1.test
java.io.IOException: Could not get block locations. Aborting...
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2151)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1743)
 at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1897)

 

Hadoop DFSClient警告NotReplicatedYetException信息

有时，当你申请到一个HOD集群后马上尝试上传文件到HDFS时，DFSClient会警告NotReplicatedYetException。通常会有一个这样的信息 -

WARN hdfs.DFSClient: NotReplicatedYetException sleeping <filename> retries left 3
08/01/25 16:31:40 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:java.io.IOException: File <filename> could only be replicated to 0 nodes, instead of 1

当你向一个DataNodes正在和NameNode联络的集群上传文件的时候，这种现象就会发生。在上传新文件到HDFS之前多等待一段时间就可以解决这个问题，因为这使得足够多的DataNode启动并且联络上了NameNode。

分享到：
| 改 hadoop ssh 端口
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    Hadoop常见异常

    Hadoop常见异常，以及hadoop配置，等资料

    hadoop2.7.3 hadoop.dll

    在windows环境下开发hadoop时，需要配置HADOOP_HOME环境变量，变量值D:\hadoop-common-2.7.3-bin-master，并在Path追加%HADOOP_HOME%\bin，有可能出现如下错误： org.apache.hadoop.io.nativeio.NativeIO$Windows....

    云计算Hadoop平台的异常数据检测算法研究.pdf

    云计算Hadoop平台的异常数据检测算法研究.pdf

    《Hadoop大数据开发实战》教学教案—01初识Hadoop.pdf

    《Hadoop大数据开发实战》教学教案—01初识Hadoop.pdf《Hadoop大数据开发实战》教学教案—01初识Hadoop.pdf《Hadoop大数据开发实战》教学教案—01初识Hadoop.pdf《Hadoop大数据开发实战》教学教案—01初识Hadoop.pdf...

    hadoop安装以及常见异常处理.docx

    Hadoop安装及常见异常处理，记录了在Hadoop安装中可能出现的几类常见异常及其解决方案

    Hadoop下载 hadoop-2.9.2.tar.gz

    Hadoop 是一个处理、存储和分析海量的分布式、非结构化数据的开源框架。最初由 Yahoo 的工程师 Doug Cutting 和 Mike Cafarella Hadoop 是一个处理、存储和分析海量的分布式、非结构化数据的开源框架。最初由 Yahoo...

    Hadoop下载 hadoop-3.3.3.tar.gz

    Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不...

    Hadoop权威指南 中文版

    本书从hadoop的缘起开始，由浅入深，结合理论和实践，全方位地介绍hado叩这一高性能处理海量数据集的理想工具。全书共14章，3个附录，涉及的主题包括：haddoop简介：mapreduce简介：hadoop分布式文件系统；hadoop的i...

    hadoop_tutorial hadoop入门经典

    hadoop_tutorial hadoop入门经典 Hadoop 是一个能够对大量数据进行分布式处理的软件框架。Hadoop 是可靠的，因为它假设计算元素和存储会失败，因此它维护多个工作数据副本，确保能够针对失败的节点重新分布处理。...

    hadoop2.7.3 Winutils.exe hadoop.dll

    hadoop2.7.3 Winutils.exe hadoop.dll

    hadoop的dll文件 hadoop.zip

    hadoop的dll文件 hadoop.zip

    Hadoop集群pdf文档

    Hadoop 集群配置详解 Hadoop_Hadoop集群（第1期）_CentOS安装配置 Hadoop_Hadoop集群（第2期）_机器信息分布表 Hadoop_Hadoop集群（第4期）_SecureCRT使用 Hadoop_Hadoop集群（第5期）_Hadoop安装配置 Hadoop_Hadoop...

    hadoop最新版本3.1.1全量jar包

    hadoop-annotations-3.1.1.jar hadoop-common-3.1.1.jar hadoop-mapreduce-client-core-3.1.1.jar hadoop-yarn-api-3.1.1.jar hadoop-auth-3.1.1.jar hadoop-hdfs-3.1.1.jar hadoop-mapreduce-client-hs-3.1.1.jar ...

    hadoop配置资源 ，hadoop-3.0.0，hadoop.dll，winutils

    调用保存文件的算子，需要配置Hadoop依赖 将文件夹中的 hadoop-3.0.0 解压到电脑任意位置 在Python代码中使用os模块配置：os.environ[‘HADOOP_HOME’] = ‘HADOOP解压文件夹路径’ winutils.exe，并放入Hadoop解压...

    Hadoop大数据资料集锦

    Hadoop大数据资料集锦Hadoop大数据资料集锦Hadoop大数据资料集锦Hadoop大数据资料集锦

    hadoop-3.3.4 版本（最新版）

    Apache Hadoop （hadoop-3.3.4.tar.gz）项目为可靠、可扩展的分布式计算开发开源软件。官网下载速度非常缓慢，因此将hadoop-3.3.4 版本放在这里，欢迎大家来下载使用！ Hadoop 架构是一个开源的、基于 Java 的编程...

    hadoop-3.1.3安装包

    Hadoop实现了一个分布式文件系统(Hadoop Distributed File System)，简称HDFS。HDFS有高容错性的特点，并且设计用来部署在低廉的(low-cost)硬件上;而且它提供高吞吐量(high throughput)来访问应用程序的数据，适合...

    Hadoop多版本 hadoop.dll和winutils.exe 下载

    支持如下版本的Hadoop hadoop-2.6.0 hadoop-2.6.3 hadoop-2.6.4 hadoop-2.7.1 hadoop-2.8.1 hadoop-2.8.3 hadoop-3.0.0

    hadoop的hadoop.dll和winutils.exe

    hadoop hadoop的hadoop.dll和winutils.exe 解决方法， 把winutils.exe加入你的hadoop-x.x.x/bin下 Could not locate executable null\bin\winutils.exe in the Hadoop binaries

    Hadoop3.1.3.rar

    hadoop3.1.3安装包

Magicbox
Global site tag (gtag.js) - Google Analytics