论坛首页 → 综合技术论坛 →

hadoop map任务在reduce部分执行完成后重新执行

浏览 1994 次
锁定老帖子 主题：hadoop map任务在reduce部分执行完成后重新执行
精华帖 (0) :: 良好帖 (0) :: 新手帖 (0) :: 隐藏帖 (0)
作者 正文
  • brandNewUser
  • 等级: 初级会员
  • 性别:
  • 文章: 2
  • 积分: 40
  • 来自: 北京
   发表时间：2014-10-09   最后修改：2014-10-09
虽然任务最终成功了，但是期间发生的问题觉得还是很诡异。

在console上输出的内容如下，map任务都已经100%，开始执行reduce任务，并且reduce任务也执行了80%成功，但此后两个map任务重新启动，严重拖慢整个流程。

引用

14/10/08 21:41:04 INFO mapreduce.Job:  map 37% reduce 10%
14/10/08 21:41:05 INFO mapreduce.Job:  map 38% reduce 11%
14/10/08 21:41:07 INFO mapreduce.Job:  map 39% reduce 12%
14/10/08 21:41:08 INFO mapreduce.Job:  map 40% reduce 12%
14/10/08 21:41:09 INFO mapreduce.Job:  map 41% reduce 12%
14/10/08 21:41:10 INFO mapreduce.Job:  map 42% reduce 12%
14/10/08 21:41:12 INFO mapreduce.Job:  map 44% reduce 12%
14/10/08 21:41:14 INFO mapreduce.Job:  map 45% reduce 12%
14/10/08 21:41:16 INFO mapreduce.Job:  map 46% reduce 13%
14/10/08 21:41:18 INFO mapreduce.Job:  map 48% reduce 13%
14/10/08 21:41:21 INFO mapreduce.Job:  map 49% reduce 13%
14/10/08 21:41:22 INFO mapreduce.Job:  map 50% reduce 13%
14/10/08 21:41:25 INFO mapreduce.Job:  map 51% reduce 14%
14/10/08 21:41:29 INFO mapreduce.Job:  map 52% reduce 15%
14/10/08 21:41:34 INFO mapreduce.Job:  map 52% reduce 16%
14/10/08 21:41:38 INFO mapreduce.Job:  map 53% reduce 17%
14/10/08 21:41:53 INFO mapreduce.Job:  map 53% reduce 18%
14/10/08 21:42:09 INFO mapreduce.Job:  map 54% reduce 18%
14/10/08 21:42:18 INFO mapreduce.Job:  map 55% reduce 18%
14/10/08 21:42:37 INFO mapreduce.Job:  map 56% reduce 18%
14/10/08 21:42:49 INFO mapreduce.Job:  map 56% reduce 19%
14/10/08 21:42:53 INFO mapreduce.Job:  map 57% reduce 19%
14/10/08 21:42:57 INFO mapreduce.Job:  map 58% reduce 19%
14/10/08 21:42:59 INFO mapreduce.Job:  map 59% reduce 19%
14/10/08 21:43:00 INFO mapreduce.Job:  map 60% reduce 19%
14/10/08 21:43:05 INFO mapreduce.Job:  map 61% reduce 19%
14/10/08 21:43:06 INFO mapreduce.Job:  map 62% reduce 19%
14/10/08 21:43:09 INFO mapreduce.Job:  map 62% reduce 20%
14/10/08 21:43:13 INFO mapreduce.Job:  map 63% reduce 20%
14/10/08 21:43:16 INFO mapreduce.Job:  map 63% reduce 21%
14/10/08 21:43:19 INFO mapreduce.Job:  map 64% reduce 21%
14/10/08 21:43:36 INFO mapreduce.Job:  map 65% reduce 21%
14/10/08 21:43:42 INFO mapreduce.Job:  map 66% reduce 21%
14/10/08 21:43:44 INFO mapreduce.Job:  map 67% reduce 22%
14/10/08 21:44:00 INFO mapreduce.Job:  map 68% reduce 22%
14/10/08 21:44:09 INFO mapreduce.Job:  map 70% reduce 22%
14/10/08 21:44:12 INFO mapreduce.Job:  map 70% reduce 23%
14/10/08 21:44:18 INFO mapreduce.Job:  map 71% reduce 23%
14/10/08 21:44:24 INFO mapreduce.Job:  map 72% reduce 23%
14/10/08 21:44:28 INFO mapreduce.Job:  map 72% reduce 24%
14/10/08 21:44:35 INFO mapreduce.Job:  map 73% reduce 24%
14/10/08 21:44:44 INFO mapreduce.Job:  map 74% reduce 24%
14/10/08 21:44:50 INFO mapreduce.Job:  map 75% reduce 24%
14/10/08 21:44:51 INFO mapreduce.Job:  map 76% reduce 24%
14/10/08 21:44:54 INFO mapreduce.Job:  map 77% reduce 25%
14/10/08 21:45:00 INFO mapreduce.Job:  map 78% reduce 25%
14/10/08 21:45:07 INFO mapreduce.Job:  map 78% reduce 26%
14/10/08 21:45:15 INFO mapreduce.Job:  map 79% reduce 26%
14/10/08 21:45:17 INFO mapreduce.Job:  map 80% reduce 26%
14/10/08 21:45:20 INFO mapreduce.Job:  map 81% reduce 26%
14/10/08 21:45:24 INFO mapreduce.Job:  map 82% reduce 26%
14/10/08 21:45:27 INFO mapreduce.Job:  map 83% reduce 26%
14/10/08 21:45:29 INFO mapreduce.Job:  map 83% reduce 27%
14/10/08 21:45:34 INFO mapreduce.Job:  map 85% reduce 27%
14/10/08 21:45:37 INFO mapreduce.Job:  map 86% reduce 27%
14/10/08 21:45:42 INFO mapreduce.Job:  map 86% reduce 28%
14/10/08 21:45:43 INFO mapreduce.Job:  map 88% reduce 28%
14/10/08 21:45:45 INFO mapreduce.Job:  map 89% reduce 28%
14/10/08 21:45:47 INFO mapreduce.Job:  map 89% reduce 29%
14/10/08 21:45:56 INFO mapreduce.Job:  map 89% reduce 30%
14/10/08 21:45:58 INFO mapreduce.Job:  map 90% reduce 30%
14/10/08 21:46:00 INFO mapreduce.Job:  map 91% reduce 30%
14/10/08 21:46:06 INFO mapreduce.Job:  map 92% reduce 30%
14/10/08 21:46:10 INFO mapreduce.Job:  map 92% reduce 31%
14/10/08 21:46:12 INFO mapreduce.Job:  map 93% reduce 31%
14/10/08 21:46:24 INFO mapreduce.Job:  map 94% reduce 31%
14/10/08 21:48:02 INFO mapreduce.Job:  map 95% reduce 31%
14/10/08 21:48:12 INFO mapreduce.Job:  map 95% reduce 32%
14/10/08 21:48:13 INFO mapreduce.Job:  map 96% reduce 32%
14/10/08 21:48:22 INFO mapreduce.Job:  map 97% reduce 32%
14/10/08 21:49:00 INFO mapreduce.Job:  map 98% reduce 32%
14/10/08 21:49:09 INFO mapreduce.Job:  map 98% reduce 33%
14/10/08 21:49:19 INFO mapreduce.Job:  map 99% reduce 33%
14/10/08 21:49:22 INFO mapreduce.Job:  map 100% reduce 33%
14/10/08 21:49:27 INFO mapreduce.Job:  map 100% reduce 34%
14/10/08 21:49:28 INFO mapreduce.Job:  map 100% reduce 37%
14/10/08 21:49:29 INFO mapreduce.Job:  map 100% reduce 40%
14/10/08 21:49:30 INFO mapreduce.Job:  map 100% reduce 42%
14/10/08 21:49:32 INFO mapreduce.Job:  map 100% reduce 44%
14/10/08 21:49:33 INFO mapreduce.Job:  map 100% reduce 46%
14/10/08 21:49:34 INFO mapreduce.Job:  map 100% reduce 48%
14/10/08 21:49:35 INFO mapreduce.Job:  map 100% reduce 49%
14/10/08 21:49:36 INFO mapreduce.Job:  map 100% reduce 54%
14/10/08 21:49:37 INFO mapreduce.Job:  map 100% reduce 55%
14/10/08 21:49:39 INFO mapreduce.Job:  map 100% reduce 58%
14/10/08 21:49:40 INFO mapreduce.Job:  map 100% reduce 59%
14/10/08 21:49:41 INFO mapreduce.Job:  map 100% reduce 60%
14/10/08 21:49:42 INFO mapreduce.Job:  map 100% reduce 61%
14/10/08 21:49:44 INFO mapreduce.Job:  map 100% reduce 62%
14/10/08 21:49:46 INFO mapreduce.Job:  map 100% reduce 63%
14/10/08 21:49:48 INFO mapreduce.Job:  map 100% reduce 64%
14/10/08 21:49:50 INFO mapreduce.Job:  map 100% reduce 65%
14/10/08 21:49:56 INFO mapreduce.Job:  map 100% reduce 66%
14/10/08 21:50:03 INFO mapreduce.Job:  map 100% reduce 67%
14/10/08 21:52:11 INFO mapreduce.Job:  map 100% reduce 68%
14/10/08 21:56:12 INFO mapreduce.Job:  map 100% reduce 69%
14/10/08 22:00:35 INFO mapreduce.Job:  map 100% reduce 71%
14/10/08 22:00:40 INFO mapreduce.Job:  map 100% reduce 72%
14/10/08 22:00:55 INFO mapreduce.Job:  map 100% reduce 74%
14/10/08 22:01:09 INFO mapreduce.Job:  map 100% reduce 75%
14/10/08 22:01:14 INFO mapreduce.Job:  map 100% reduce 77%
14/10/08 22:02:23 INFO mapreduce.Job:  map 100% reduce 78%
14/10/08 22:02:37 INFO mapreduce.Job:  map 100% reduce 80%
14/10/08 22:03:32 INFO mapreduce.Job:  map 100% reduce 82%
14/10/08 22:04:14 INFO mapreduce.Job:  map 100% reduce 83%
14/10/08 22:04:43 INFO mapreduce.Job:  map 100% reduce 85%
14/10/08 22:08:11 INFO mapreduce.Job:  map 100% reduce 86%
14/10/08 22:08:45 INFO mapreduce.Job:  map 100% reduce 88%
14/10/08 22:09:53 INFO mapreduce.Job:  map 100% reduce 89%
14/10/08 22:11:21 INFO mapreduce.Job:  map 100% reduce 91%
14/10/08 22:11:44 INFO mapreduce.Job:  map 100% reduce 92%
14/10/08 22:13:23 INFO mapreduce.Job:  map 100% reduce 94%
14/10/08 22:13:25 INFO mapreduce.Job:  map 100% reduce 95%
14/10/08 22:15:21 INFO mapreduce.Job:  map 100% reduce 97%
14/10/08 22:18:14 INFO mapreduce.Job:  map 97% reduce 98%
14/10/08 22:18:15 INFO mapreduce.Job:  map 97% reduce 100%
14/10/08 22:19:14 INFO mapreduce.Job:  map 97% reduce 95%
14/10/08 22:19:15 INFO mapreduce.Job:  map 97% reduce 90%
14/10/08 22:19:27 INFO mapreduce.Job:  map 97% reduce 91%
14/10/08 22:19:35 INFO mapreduce.Job:  map 97% reduce 92%
14/10/08 22:20:36 INFO mapreduce.Job:  map 97% reduce 93%
14/10/08 22:34:55 INFO mapreduce.Job:  map 94% reduce 93%
14/10/08 22:35:20 INFO mapreduce.Job:  map 95% reduce 93%
14/10/08 22:40:46 INFO mapreduce.Job:  map 96% reduce 93%
14/10/08 22:40:55 INFO mapreduce.Job:  map 97% reduce 93%
14/10/08 22:40:59 INFO mapreduce.Job:  map 97% reduce 94%
14/10/08 22:41:12 INFO mapreduce.Job:  map 97% reduce 95%
14/10/08 22:51:43 INFO mapreduce.Job:  map 98% reduce 95%
14/10/08 22:52:52 INFO mapreduce.Job:  map 99% reduce 95%
14/10/08 22:53:01 INFO mapreduce.Job:  map 100% reduce 95%
14/10/08 22:53:03 INFO mapreduce.Job:  map 100% reduce 96%
14/10/08 22:53:28 INFO mapreduce.Job:  map 100% reduce 97%
14/10/08 22:57:19 INFO mapreduce.Job:  map 100% reduce 98%
14/10/08 23:09:55 INFO mapreduce.Job:  map 100% reduce 100%
14/10/08 23:10:00 INFO mapreduce.Job: Job job_1412772687841_0019 completed successfully


此时，整个任务的状态很奇怪：



经过查看此task的具体状态，后面的map task明显是最后reduce大部分执行完成后才启动的，时间跨度大概1个小时。



其他人有遇到过这种情况吗？或者熟悉hadoop运行机制的能否帮忙简单分析一下什么原因？

谢谢！



  • 大小: 120.3 KB
  • 大小: 99.8 KB
返回顶楼
         
论坛首页 → 综合技术版

跳转论坛:
Global site tag (gtag.js) - Google Analytics