目前集群上某台机器卡住导致出现大量的Map端任务FAIL,当定位到具体的机器上时,无法ssh或进去后terminal中无响应,退出的相关信息如下:
[hadoop@xxx ~]$ Received disconnect from xxx: Timeout, your session not responding.
AttemptID:attempt_1413206225298_24177_m_000001_0 Timed out after 1200 secsContainer killed by the ApplicationMaster. Container killed on request. Exit code is 143
The number of milliseconds before a task will be terminated if it neither reads an input, writes an output, nor updates its status string. A value of 0 disables the timeout.
Map.Entry<TaskAttemptId, ReportTime> entry = iterator.next(); boolean taskTimedOut = (taskTimeOut > 0) && (currentTime > (entry.getValue().getLastProgress() + taskTimeOut)); if(taskTimedOut) { // task is lost, remove from the list and raise lost event iterator.remove(); eventHandler.handle(new TaskAttemptDiagnosticsUpdateEvent(entry .getKey(), "AttemptID:" + entry.getKey().toString() + " Timed out after " + taskTimeOut / 1000 + " secs")); eventHandler.handle(new TaskAttemptEvent(entry.getKey(), TaskAttemptEventType.TA_TIMED_OUT)); }
public void progressing(TaskAttemptId attemptID) { //only put for the registered attempts //TODO throw an exception if the task isn't registered. ReportTime time = runningAttempts.get(attemptID); if(time != null) { time.setLastProgress(clock.getTime()); } }
Report progress
If your task reports no progress for 10 minutes (see the mapred.task.timeout
property) then it will be killed by Hadoop. Most tasks don’t encounter this situation since they report progress implicitly by reading input and writing output. However, some jobs which don’t process records in this way may fall foul of this behavior and have their tasks killed. Simulations are a good example, since they do a lot of CPU-intensive processing in each map and typically only write the result at the end of the computation. They should be written in such a way as to report progress on a regular basis (more frequently than every 10 minutes). This may be achieved in a number of ways:
- Call
setStatus()
onReporter
to set a human-readable description of
the task’s progress - Call
incrCounter()
onReporter
to increment a user counter - Call
progress()
onReporter
to tell Hadoop that your task is still there (and making progress)
但是,事情还没完,集群中会不定时地有任务卡死在某个点上导致任务无法继续下去:
"main" prio=10 tid=0x000000000293f000 nid=0x1e06 runnable [0x0000000041b20000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000006e243c3f0> (a sun.nio.ch.Util$2) - locked <0x00000006e243c3e0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006e243c1a0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
/now wait for socket to be ready. int count = 0; try { count = selector.select(channel, ops, timeout); } catch (IOException e) { //unexpected IOException. closed = true; throw e; } if (count == 0) { throw new SocketTimeoutException(timeoutExceptionString(channel, timeout, ops)); }
Error: java.net.SocketTimeoutException: 70000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=xxx remote=/xxx] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1490) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:962) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:930) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
while (true) { long start = (timeout == 0) ? 0 : Time.now(); key = channel.register(info.selector, ops); ret = info.selector.select(timeout); if (ret != 0) { return ret; } /* Sometimes select() returns 0 much before timeout for * unknown reasons. So select again if required. */ if (timeout > 0) { timeout -= Time.now() - start; if (timeout <= 0) { return 0; } } if (Thread.currentThread().isInterrupted()) { throw new InterruptedIOException("Interruped while waiting for " + "IO on channel " + channel + ". " + timeout + " millis timeout left."); } }
java.nio.channels.Selector public abstract int select(long timeout) throws java.io.IOException Selects a set of keys whose corresponding channels are ready for I/O operations. This method performs a blocking selection operation. It returns only after at least one channel is selected, this selector's wakeup method is invoked, the current thread is interrupted, or the given timeout period expires, whichever comes first.
- 至少一个已经注册的Channel被选择,返回的就是被选择的Channel数量;
- Selector被中断;
- 给定的超时时间已到;
但是,这也没完,难道超时了不会重试?到底会重试几次?
经过继续分析,发现往下的堆栈中的DFSInputStream调用了readBuffer方法,可以看到retryCurrentNode在第一次失败后,将IOException捕获,会进行必要的重试操作,如果还是发生超时,并且找不到就将其加入黑名单作为失败的DataNode(可能下次不会进行重试?),并转移到另外的DataNode上(执行seekToNewSource方法),经过几次后才会将IOException真正抛出。
try { return reader.doRead(blockReader, off, len, readStatistics); } catch ( ChecksumException ce ) { DFSClient.LOG.warn("Found Checksum error for " + getCurrentBlock() + " from " + currentNode + " at " + ce.getPos()); ioe = ce; retryCurrentNode = false; // we want to remember which block replicas we have tried addIntoCorruptedBlockMap(getCurrentBlock(), currentNode, corruptedBlockMap); } catch ( IOException e ) { if (!retryCurrentNode) { DFSClient.LOG.warn("Exception while reading from " + getCurrentBlock() + " of " + src + " from " + currentNode, e); } ioe = e; } boolean sourceFound = false; if (retryCurrentNode) { /* possibly retry the same node so that transient errors don't * result in application level failures (e.g. Datanode could have * closed the connection because the client is idle for too long). */ sourceFound = seekToBlockSource(pos); } else { addToDeadNodes(currentNode); sourceFound = seekToNewSource(pos); } if (!sourceFound) { throw ioe; } retryCurrentNode = false; }
相关推荐
Hadoop Map Reduce教程,介绍hadoop map/reduce框架的各个方面
hadoop中map/reduce自学资料合集
hadoop的map reduce 学习手册,很实用
Hadoop Map-Reduce教程,hadoop,mapreduce
hadoop map reduce 的中文简易教程,能轻松帮助普通用户不需了解太多hadoop底层知识就能实现分布式编程,很好的入门教程。
人脸识别,车辆识别,一人一档,一车一档 hadoop map reduce hbase
Hadoop Map Reduce 教程.doc
hadoop开发文档
Hadoop学习总结之三:Map-Reduce入门
hadoop调优参数列表.hadoop调优参数列表.hadoop调优参数列表.
讲述了Windows平台的Hadoop安装... 最后,以最简单的求和为例,剖析Hadoop的Map/Reduce工作机制,对于初学Hadoop及Map/Reduce的读者有很大的帮助。相信通过最简单的求和为例,读者可步入Hadoop的Map/Reduce开发者行列。
hadoop调优参数列表
NULL 博文链接:https://sgq0085.iteye.com/blog/1879442
Hadoop的MapTask类源代码分析
hadoop作业调优参数整理及原理,并且针对部分的原理和视图详细说明
4.1.3 Hadoop配置参数格式;4.1.3 Hadoop配置参数格式;4.1.4 获得Hadoop集群全部配置信息;4.2 在Master节点上安装Hadoop;(1)解压缩hadoop-2.7.1.tar.gz安装包到/usr目录下 [root@master ~]# tar zxvf hadoop-2.7.1....
hadoop map reduce mapreduce
Hadoop参数调优,各种参数修改优化,还是需要大量实验才能得到最合适的设置
hadoop_map_reduce Hadoop Map reduce 示例
在windows环境下开发hadoop时,需要配置HADOOP_HOME环境变量,变量值D:\hadoop-common-2.7.3-bin-master,并在Path追加%HADOOP_HOME%\bin,有可能出现如下错误: org.apache.hadoop.io.nativeio.NativeIO$Windows....