`
wsql
  • 浏览: 12034252 次
  • 性别: Icon_minigender_1
  • 来自: 深圳
文章分类
社区版块
存档分类
最新评论

安装hadoop错误一则:/tmp/hadoop-grid/mapred/system/jobtracker.info could only be replicated to 0 nodes

 
阅读更多

现象:

--节点进程显示都是正常的呀
[grid@gc logs]$ /usr/java/jdk1.6.0_18/bin/jps
4434 JobTracker
4346 SecondaryNameNode
4194 NameNode
8291 Jps

[grid@rac1 conf]$ /usr/java/jdk1.6.0_18/bin/jps
32423 Jps
29224 DataNode
29348 TaskTracker

[grid@rac2 logs]$ /usr/java/jdk1.6.0_18/bin/jps
26358 DataNode
26457 TaskTracker
1210 Jps

--master节点的jobtracker日志
[grid@gc logs]$ tail -100f hadoop-grid-jobtracker-gc.localdomain.log
2012-11-23 15:49:26,973 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
2012-11-23 15:49:26,973 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/tmp/hadoop-grid/mapred/system/jobtracker.info" - Aborting...
2012-11-23 15:49:26,974 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://gc.localdomain:9000/tmp/hadoop-grid/mapred/system/jobtracker.info failed!
2012-11-23 15:49:26,975 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
2012-11-23 15:49:26,977 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /tmp/hadoop-grid/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

at org.apache.hadoop.ipc.Client.call(Client.java:740)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2937)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2819)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)

--slave 1节点日志
[grid@rac1 logs]$ more hadoop-grid-datanode-rac1.localdomain.log
2012-11-23 15:33:28,815 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.101:9000. Already tried 7 time(s).
2012-11-23 15:33:29,817 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.101:9000. Already tried 8 time(s).
2012-11-23 15:33:30,818 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.101:9000. Already tried 9 time(s).
2012-11-23 15:33:30,819 INFO org.apache.hadoop.ipc.RPC: Server at /192.168.2.101:9000 not available yet, Zzzzz...
2012-11-23 15:33:32,820 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.101:9000. Already tried 0 time(s).
2012-11-23 15:33:33,821 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.101:9000. Already tried 1 time(s).
--slave 2节点日志
[grid@rac2 logs]$ more hadoop-grid-datanode-rac2.localdomain.log
2012-11-23 15:34:19,661 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 9 time(s).
2012-11-23 15:34:19,663 INFO org.apache.hadoop.ipc.RPC: Server at /192.168.2.102:9001 not available yet, Zzzzz...
2012-11-23 15:34:21,665 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 0 time(s).
2012-11-23 15:34:22,666 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 1 time(s).
2012-11-23 15:34:23,667 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 2 time(s).
2012-11-23 15:34:24,674 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 3 time(s).
2012-11-23 15:34:25,683 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 4 time(s).
2012-11-23 15:34:26,685 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 5 time(s).
2012-11-23 15:34:27,690 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 6 time(s).
2012-11-23 15:34:28,703 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: /192.168.2.102:9001. Already tried 7 time(s).
jobtracker监控界面:

namenode集群监控界面:

原因:

是因为两个slave节点的core-site.xml和mapred-site.xml文件配置的问题
之前对安装说明解决有误,一直以为在ore-site.xml和mapred-site.xml配置文件中IP或机器名应为各自机器的IP或机器名。
所以我之前把两个slave节点的core-site.xml和mapred-site.xml文件配置成了自己机器的IP。
应该都改成master机器的IP或机器名,如下:
--1 master 及 2 slave nodes 的 core-site.xml 和 mapred-site.xml 配置
[grid@gc conf]$ cat core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://gc:9000</value>
</property>

<property>
<name>hadoop.tmp.dir</name>
<value>/home/grid/hadoop/tmp</value>
</property>

</configuration>

[grid@gc conf]$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
<name>mapred.job.tracker</name>
<value>gc:9001</value>
</property>

</configuration>

然后重启hadoop正常

问题帖子:http://f.dataguru.cn/thread-32858-1-1.html

总结:

在安装配置hadoop时,如果出现问题,可以主要按下面的步骤检查:

1、节点的映射,master,slave1;slave2。。。;
2、ssh互信,master—》slave1;master—》slave2;
3、各个服务器上的hadoop的配置文件确认正确,安装目录一致,tmp,data目录也要一致;
4、防火墙要关闭:/etc/init.d/iptables stop
分享到:
评论

相关推荐

    hadoop常见问题及解决办法

    同时,可以修改conf/hadoop-env.sh文件,增加export HADOOP_HEAPSIZE=4000参数。 问题4:无法访问DataNode 问题描述:无法访问DataNode,且无法结束错误。 解决办法:该问题是由于dfs.name.dir路径已经被使用过所...

    hadoop-core-0.20.2-cdh3u3.jar

    export CLASSPATH="$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:${HADOOP_HOME}/lib/commons-logging-1.0.4.jar:${HADOOP_HOME}/lib/guava-r09-jarjar.jar:${HADOOP_HOME}/hadoop-core-0.20.2-cdh3u3.jar:/usr/...

    hadoop-lzo-master

    1.安装 Hadoop-gpl-compression 1.1 wget http://hadoop-gpl-compression.apache-extras.org.codespot.com/files/hadoop-gpl-compression-0.1.0-rc0.tar.gz 1.2 mv hadoop-gpl-compression-0.1.0/lib/native/Linux-...

    centos7系统中的hadoop3.2.0的配置文件.rar

    hadoop 3.2.0在(win10+cento7+vmware)环境的配置文件,包括/etc/profile和hadoop安装目录下etc/hadoop文件下的文件:hadoop-env.sh,mapred-env.sh,yarn-env.sh,core-site.xml,hdfs-site.xml,mapred-site.xml,yarn-...

    hadoop安装与配置 Hadoop的安装与配置可以分成几个主要步骤: 1. 安装Java 2. 下载Hadoop 3. 配

    hadoop安装与配置 hadoop安装与配置 Hadoop的安装与配置可以分成几个主要步骤: 1. 安装Java 2. 下载Hadoop 3. 配置Hadoop 4. 格式化Hadoop文件系统 5. 启动Hadoop 以下是基于Linux系统的简化安装与配置步骤: 1. ...

    hive-exec-*.jar包

    Missing Hive Execution Jar: /hive/hive1.2.1/lib/hive-exec-*.jar

    flink-shaded-hadoop-3-uber-3.1.1.7.1.1.0-565-9.0.jar

    Flink-1.11.2与Hadoop3集成JAR包,放到flink安装包的lib目录下,可以避免Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.这个报错,实现...

    hadoop-2.4.1安装软件包以及教程jdk.zip

    &lt;value&gt;/home/hadoop/hadoop-2.4.1/tmp&lt;/value&gt; &lt;/property&gt; 第三个:hdfs-site.xml hdfs-default.xml (3) &lt;!-- 指定HDFS副本的数量 --&gt; &lt;name&gt;dfs.replication&lt;/name&gt; &lt;value&gt;1&lt;/value&gt; &lt;/...

    hadoop-3.0.0.tar.gz

    Hadoop实现了一个分布式文件系统(Hadoop Distributed File System),简称HDFS。HDFS有高容错性的特点,并且设计用来部署在低廉的(low-cost)硬件上;而且它提供高吞吐量(high throughput)来访问应用程序的数据...

    hadoop伪分布式安装.pdf

    ### Hadoop伪分布式安装知识点详解 #### 一、Hadoop伪分布式概述 Hadoop是一种能够处理海量数据的大规模分布式计算框架。它通过将任务分解到多个计算机节点上并行处理来提高数据处理效率。Hadoop支持多种运行模式,...

    hadoop-lzo-0.4.20.jar

    hadoop-lzo-0.4.20 centOS6.5 64位编译出来的 拷贝jar包到hadoop和hbase中 cp /opt/hadoopgpl/lib/hadoop-lzo-0.4.20-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/ cp /opt/hadoopgpl/lib/hadoop-lzo-0.4.20-...

    win安装部署hadoop3.0.0并启动上传文件到HDFS(博文相关代码)

    - 打开`hadoop/etc/hadoop`目录下的`hadoop-env.sh`(在Windows上可能是`hadoop-env.cmd`),设置`JAVA_HOME`指向你的JDK安装路径。 - 修改`core-site.xml`,配置HDFS的默认FS(默认文件系统)和通信缓冲区大小,...

    hadoop-eclipse-plugin-2.10.0.jar

    Eclipse集成Hadoop2.10.0的插件,使用`ant`对hadoop的jar包进行打包并适应Eclipse加载,所以参数里有hadoop和eclipse的目录. 必须注意对于不同的hadoop版本,` HADDOP_INSTALL_PATH/share/hadoop/common/lib`下的jar包...

    利用ansible 自动 安装Hadoop 集群

    Hadoop作为大数据处理的核心组件,包括HDFS(Hadoop Distributed File System)和MapReduce等模块,构建一个Hadoop集群通常涉及多台服务器的配置、软件安装、服务启动和集群配置。通过Ansible,我们可以简化这个过程...

    hadoop2.6-common-bin.zip

    标题 "hadoop2.6-common-bin.zip" 指示这是一个包含Hadoop 2.6版本通用二进制文件的压缩包。这个压缩包主要针对Windows用户,旨在解决在该操作系统上运行Hadoop时可能遇到的"Could not locate executable"错误。这个...

    zookeeper+hadoop+hbase+hive(集成hbase)安装部署教程(超详细).docx

    jdk1.8.0_131、apache-zookeeper-3.8.0、hadoop-3.3.2、hbase-2.4.12 mysql5.7.38、mysql jdbc驱动mysql-connector-java-8.0.8-dmr-bin.jar、 apache-hive-3.1.3 2.本文软件均安装在自建的目录/export/server/下 ...

    【CDH部署】全网最简单CDH6.2.0环境部署

    下载链接 ...主机规划 sparkproject1 manager 8G 50GB sparkproject2 agent 4G 50GB sparkproject3 agent 4G 50GB 1. 设置IP, hostname, hosts, ssh免密登录 可参考Hadoop中linux基本配置...【Hadoop部署】Hadoop环境部署1

    Hadoop安装步骤及软件下载地址

    根据提供的文件信息,我们可以归纳出以下关于Hadoop安装步骤及相关软件下载的知识点: ### Hadoop基础知识 1. **Hadoop简介**:Hadoop是一个能够对大量数据进行分布式处理的软件框架,它允许用户轻松地在由...

Global site tag (gtag.js) - Google Analytics