- 浏览: 398484 次
- 性别:
- 来自: 北京
文章分类
最新评论
-
秦时明月黑:
深入浅出,楼主很有功底
hive编译部分的源码结构 -
tywo45:
感觉好多错误,但还是支持!
HDFS+MapReduce+Hive+HBase十分钟快速入门 -
xbbHistory:
解析的很棒!!
Linux-VFS -
darrendu:
执行这个命令,bin/hadoop fs -ls /home/ ...
Hadoop示例程序WordCount运行及详解 -
moudaen:
请问楼主,我执行总后一条语句时,执行的是自带的1.sql,你当 ...
TPC-H on Hive
http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore/
Configuring a Hadoop cluster is something akin to voodoo. There are a large number of variables in hadoop-default.xml that you can override in hadoop-site.xml . Some specify file paths on your system, but others adjust levers and knobs deep inside Hadoop’s guts. Unfortuately, there’s little or no documentation on how to set them well. Is there a single optimal configuration? Are there some settings that can just be “set to 11?”
At Cloudera, we’re working hard to make Hadoop easier to use and to make configuration less painful. Our Hadoop Configuration Tool gives you a web-based guide to help set up your cluster. Once it’s running, though, you might want to look under the hood and tune things a bit.
The rest of this post discusses why it’s a bad idea to just set all the limits as high as they’ll go, and gives you some pointers to get started on finding a happy medium.
Why can’t you just set all the limits to 1,000,000?
Increasing most settings has a direct impact on memory consumption. Increasing DataNode and TaskTracker settings, therefore, has an adverse impact on RAM available to individual MapReduce tasks. On large hardware, they can be set generously high. In general though, unless you have several dozen more more nodes working together, dialing up settings very high wastes system resources like RAM that could be better applied to running your mapper and reducer code.
That having been said, here’s a list of some things that can be cranked up higher than the defaults by a fair margin:
File descriptor limits
A busy Hadoop daemon might need to open a lot of files. The open fd ulimit in Linux defaults to 1024, which might be too low. You can set to something more generous — maybe 16384. Setting this an order of magnitude higher (e.g., 128K) is probably not a good idea. No individual Hadoop daemon is supposed to need hundreds of thousands of fds; if it’s consuming that many, then there’s probably an fd leak or other bug that needs fixing. This would just mask the true problem until errors started showing up somewhere else.
You can view your ulimits in bash by running:
$ ulimit -a
To set the fd ulimit for a process, you’ll need to be root. As root, open a shell, and run:
# ulimit -n 16384
You can then run the Hadoop daemon from that shell; the ulimits will be inherited. e.g.:
# sudo -u hadoop $HADOOP_HOME/bin/hadoop-daemon.sh start namenode
You can also set the ulimit for the hadoop user in /etc/security/limits.conf ; this mechanism will set the value persistently. Make sure pam_limits is enabled for whatever auth mechanism the hadoop daemon is using. The entry will look something like:
hadoop hard nofile 16384
If you’re running our distribution , we ship a modified version of Hadoop 0.18.3 that includes HADOOP-4346 , a fix for the “soft fd leak” that has affected Hadoop since 0.17, so this should be less critical for our users. Users of the official Apache Hadoop release are affected by the fd leak for all 0.17, 0.18, and 0.19 versions. (The fix is committed for 0.20.) For the curious, we’ve published a list of all differences between our release of Hadoop and the stock 0.18.3 release.
If you’re running Linux 2.6.27, you should also set the epoll limit to something generous; maybe 4096 or 8192.
# echo 4096 > /proc/sys/fs/epoll/max_user_instances
Then put the following text in /etc/sysctl.conf :
fs.epoll.max_user_instances = 4096
See http://pero.blogs.aprilmayjune.org/2009/01/22/hadoop-and-linux-kernel-2627-epoll-limits/ for more details.
Internal settings
If there is more RAM available than is consumed by task instances, set io.sort.factor to 25 or 32 (up from 10). io.sort.mb should be 10 * io.sort.factor . Don’t forget, multiply io.sort.mb by the number of concurrent tasks to determine how much RAM you’re actually allocating here, to prevent swapping. (So 10 task instances with io.sort.mb = 320 means you’re actually allocating 3.2 GB of RAM for sorting, up from 1.0 GB.) An open ticket on the Hadoop bug tracking database suggests making the default value here 100. This would likely result in a lower per-stream cache size than 10 MB.
io.file.buffer.size – this is one of the more “magic” parameters. You can set this to 65536 and leave it there. (I’ve profiled this in a bunch of scenarios; this seems to be the sweet spot.)
If the NameNode and JobTracker are on big hardware, set dfs.namenode.handler.count to 64 and same with mapred.job.tracker.handler.count . If you’ve got more than 64 GB of RAM in this machine, you can double it again.
dfs.datanode.handler.count defaults to 3 and could be set a bit higher. (Maybe 8 or 10.) More than this takes up memory that could be devoted to running MapReduce tasks, and I don’t know that it gives you any more performance. (An increased number of HDFS clients implies an increased number of DataNodes to handle the load.)
mapred.child.ulimit should be 2–3x higher than the heap size specified in mapred.child.java.opts and left there to prevent runaway child task memory consumption.
Setting tasktracker.http.threads higher than 40 will deprive individual tasks of RAM, and won’t see a positive impact on shuffle performance until your cluster is approaching 100 nodes or more.
Conclusions
Configuring Hadoop for “optimal performance” is a moving target, and depends heavily on your own applications. There are settings that need to be moved off their defaults, but finding the best value for each is difficult. Our configurator for Hadoop will do a reasonable job of getting you started.
We’d love to hear from you about your own configurations. Did you discover a combination of settings that really made your cluster sing? Please share in the comments.
发表评论
-
Hadoop的Secondary NameNode方案
2012-11-13 10:39 1252http://book.51cto.com/art/20120 ... -
hadoop
2011-10-08 12:20 1068hadoop job解决 ... -
hadoop作业调优参数整理及原理
2011-04-15 14:02 12791 Map side tuning 参数 ... -
Job运行流程分析
2011-03-31 11:04 1632http://www.cnblogs.com/forfutur ... -
hadoop作业运行部分源码
2011-03-31 10:51 1375一、客户端 Map-Reduce的过程首先是由客户端提交 ... -
eclipse中编译hadoop(hive)源码
2011-03-24 13:20 3379本人按照下面编译Hadoop 所说的方法在eclipse中编 ... -
7 Tips for Improving MapReduce Performance
2011-03-11 15:06 965http://www.cloudera.com/blog ... -
hadoop 源码分析一
2011-02-22 15:29 1172InputFormat : 将输入的 ... -
hadoop参数配置(mapreduce数据流)
2011-01-14 11:08 2857Hadoop配置文件设定了H ... -
混洗和排序
2011-01-05 19:33 3202在mapreduce过程中,map ... -
hadoop中每个节点map和reduce个数的设置调优
2011-01-05 19:28 8336map red.tasktracker.map.tasks. ... -
hadoop profiling
2010-12-20 20:52 2600和debug task一样,profiling一个运行在分布 ... -
关于JVM内存设置
2010-12-20 20:49 1315运行map、reduce任务的JVM内存调整:(我当时是在jo ... -
HADOOP报错Incompatible namespaceIDs
2010-12-14 12:56 973HADOOP报错Incomp ... -
node1-node6搭建hadoop
2010-12-13 18:42 1087环境: node1-node6 node1为主节点 ... -
hadoop启动耗时
2010-12-07 17:28 1287http://blog.csdn.net/AE86_FC/ar ... -
namenode 内部关键数据结构简介
2010-12-07 16:35 1235http://www.tbdata.org/archiv ... -
HDFS常用命令
2010-12-04 14:59 1269文件系统检查 bin/hadoop fsck [pa ... -
HDFS添加和删除节点
2010-12-04 14:45 1979From http://developer.yahoo.co ... -
hadoop 0.20 程式開發
2010-11-30 17:15 1265hadoop 0.20 程式開發 ecl ...
相关推荐
idea-gitignore:IntelliJ IDEA的.ignore支持插件
Publisher: Springer Language: English ISBN:3540731970 1 edition 342 pages ... Proactive IT service management can be thought of as risk management for IT: Can you afford to ignore it?
命令参数介绍:1) 参数: -p 2) 参数: -v 3)参数: -s 4) 参数: --throttle 毫秒> 5) 参数: --ignore-crashes 6) 参数: --ignore-timeouts 7) 参数: --ignore-security-exceptions 8) 参数: --...
好用的WMP的歌词插件: What can MiniLyrics do?... * Never expires, you can try it free and ignore the nagging registration notice. * Support saving/loading embedded ID3v2, Lyrics3v2 standard lyrics.
qdd是短当q uickly d ownload d ependencies。 这就是全部。 在不需要安装脚本的情况下,它可以代替npm ci 。 (这意味着也没有编译... 如果在安装依赖项时将--ignore-scripts传递给npm或yarn ,则项目运行正常。 您不
This isn't what you do in your day job, which means you can't showcase how good you are in an interview. These topics come up over and over again in Java interviews and you need to revise for them. ...
You can use the middle or right mouse button to resize windows. If you press the shift key while you drag or resize, the window will stick to other windows. You can double-click a window to maximize ...
With RegCure’s scheduling feature, you can just tell it when you want it to do its job, sit back and relax. You can set it to scan at system startup or assign it to work on a daily, weekly or monthly...
Did you know that you can not only change what is on UltraEdit's toolbars, you can also change the icon used, as well as create your own custom toolbars and tools? File tabs Understand how file tabs ...
What you’ll learn Debug shell scripts using existing debuggers, not inspection. Use and extend text–editing one–liners and learn to forget Perl. Manage files and filesystems using scripting, not ...
This can either just be ignored or, if it really bugs you, turned off with the following command: warning off MATLAB:dispatcher:nameConflict However, sometimes these kind of warnings can be usefull if...
obsolete please ignore it
Works well with Unity 2017.x and higher and also you can make it work with older Unity versions just ignore the warning before importing the package. Tested on the below Unity versions: - 2018.2, ...
如何去修改菜单ignore?
OmniThreadLibrary 版本 3.07, 发布时间 2017-02-... OTL doesn't limit you in any way - if it is not powerfull enough for you, you can ignore any part of its "smartness" and replace it with your own code.
忽略样式 一个babel/register样式钩子,用于在Node中运行时...要解决此问题,请在您的Mocha测试中要求使用“ ignore-styles : mocha --require ignore-styles 请参阅 ,以了解忽略的扩展程序的完整列表,并在需要
方案一:使用ignore关键字 如果是用主键primary或者唯一索引unique区分了记录的唯一性,避免重复插入记录可以使用: 代码如下: INSERT IGNORE INTO `table_name` (`email`, `phone`, `user_id`) VALUES (‘test9@...
Install fme_install.exe Install SafeFLEXlmInstaller.msi (if it complains about missing device, ignore the ...Note: You can re-run the licensing wizard if you wish to change the licensing type later.
# You can use this:deploy : type : git repo : <repository> branch : [branch] token : ' ' message : [message] name : [git user] email : [git email] extend_dirs : [extend directory] ignore_hidden : ...
you can have it by hand ,so ignore me