`
lookqlp
  • 浏览: 341906 次
  • 性别: Icon_minigender_1
  • 来自: 上海
社区版块
存档分类
最新评论

hive hive.optimize.ppd=false导致map数量很大

    博客分类:
  • hive
阅读更多
hive官方https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView

Prior to Hive 0.6.0, lateral view did not support the predicate push-down optimization. In Hive 0.5.0 and earlier, if you used a WHERE clause your query may not have compiled. A workaround was to add set hive.optimize.ppd=false; before your query. The fix was made in Hive 0.6.0; see https://issues.apache.org/jira/browse/HIVE-1056: Predicate push down does not work with UDTF's.
在0.5.0版本前使用hive.optimize.ppd=false是能解决where条件不识别的。但我在使用0.9.0版本时使用了该参数,导致where不识别。
不使用参数日志:
Starting Job = job_201312261829_7430, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7430
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:8021 -kill job_201312261829_7430
Hadoop job information for Stage-1: number of mappers: 90; number of reducers: 287

使用该参数日志:
Starting Job = job_201312261829_7433, Tracking URL = http://master:50030/jobdetails.jsp?jobid=job_201312261829_7433
Kill Command = /home/hadoop/hadoop-1.0.3/libexec/../bin/hadoop job  -Dmapred.job.tracker=master:8021 -kill job_201312261829_7433
Hadoop job information for Stage-1: number of mappers: 4363; number of reducers: 999


所以在新版本的hive中使用hive.optimize.ppd=true;默认值即可
分享到:
评论

相关推荐

Global site tag (gtag.js) - Google Analytics