- 浏览: 214352 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (114)
- hbase (3)
- akka (7)
- hdfs (6)
- mapreduce (1)
- hive (0)
- zookeeper (8)
- storm (0)
- geese (0)
- leaf (0)
- stormbase (0)
- scala (2)
- oozie (11)
- zeromq (1)
- netty (3)
- mongodb (0)
- sqoop (2)
- flume (3)
- mahout (1)
- redis (0)
- lucene (1)
- solr (1)
- ganglia (3)
- 分布式理论 (2)
- hadoop (42)
- others (14)
- mq (1)
- clojure (3)
- flume ng (1)
- linux (1)
- esper (0)
最新评论
-
javalogo:
[b][i][u]引用[list]
[*][*][flash= ...
什么是Flume -
leibnitz:
what are they meanings
Hadoop Ganglia Metric Item -
di1984HIT:
没用过啊。
akka 介绍-Actor 基础 -
di1984HIT:
写的不错。
Hadoop管理-集群维护 -
developerinit:
很好,基本上介绍了
什么是Flume
Setting up Disks for Hadoop
Here are some recommendations for setting up disks in a Hadoop cluster. What we have here is anecdotal -hard evidence is very welcome, and everyone should expect a bit of trial and error work.
Key Points
Goals for a Hadoop cluster are normally massive amounts of data with high I/O bandwidth. Your MapReduce jobs may be IO bound or CPU/Memory bound -if you know which one is more important (effectively how many CPU cycles/RAM MB used per Map or Reduce), you can make better decisions.
Hardware
You don't need RAID disk controllers for Hadoop Data Node, as it copies data across multiple machines instead. This increase the likelihood that there is a free task slot near that data, and if the servers are on different PSUs and switches, eliminates some more points of failure in the data center.
While the Hadoop Name Node and Secondary Name Node can write to a list of drive locations, they will stop functioning if it can not write to ALL the locations. In this case a mirrored RAID is a good idea for higher availability.
Having lots of disks per server gives you more raw IO bandwidth than having one or two big disks. If you have enough that different tasks can be using different disks for input and output, disk seeking is minimized, which is one of the big disk performance killers. That said: more disks have a higher power budget; if you are power limited, you may want fewer but larger disks.
Configuring Hadoop
Pass a list of disks to the dfs.data.dir parameter, Hadoop will use all of the disks that are available. When one goes offline it is taken out of consideration. Hadoop does not check for the disk coming back -it assumes it is "gone".
How to limit Data node's disk usage?
Use dfs.datanode.du.reserved configuration value in $HADOOP_HOME/conf/hdfs-site.xml for limiting disk usage.
<property>
<name>dfs.datanode.du.reserved</name>
<!-- cluster variant -->
<value>182400</value>
<description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.
</description>
</property>
Logging
The environment variable, HADOOP_LOG_DIR sets the directory Hadoop logs to.
the log4j.properties file in your hadoop configuration dir controls logging in more detail
Don't log to the root directory, as having a machine that does not boot because the logs are overflowing can be inconvenient.
Have a plan to clean up log output, otherwise jobs that log too much to the console will fill up log directories.
Get your developers to use the commons-logging APIs in their MapReduce code, so that you can turn logging up or down without recompiling the code. They can run in debug mode on their test machines, you can run at WARN level in production.
Some JVMs (JRockit) seem to log more. Tune your Log4j settings for your JVM, and only capture the stuff you really want.
Do not keep stuff under /tmp
Hadoop defaults to keeping things under /tmp so that you can play with Hadoop without filling up your disk. This is dangerous in a production cluster, as any automated cleanup cron job will eventually delete stuff in /tmp, at which point your Hadoop cluster is in trouble.
You will need cron job to clean stuff in /tmp up eventually. Plan for it.
Configure Hadoop to store stuff in stable locations, preferably off that root disk.
Java stores the info for jps under /tmp/hsperfdata_${user} -after the cleanup jps won't work. Have your script leave those directories alone, or get used to using ps -ef | grep java to find Java processes instead.
Underlying File System Options
If mount the disks as noatime, then the file access times aren't written back; this speeds up reads. There is also relatime, which stores some access time information, but is not as slow as the classic atime attribute. Remember that any access time information kept by Hadoop is independent of the atime attribute of individual blocks, so Hadoop does not care what your settings are here. If you are mounting disks purely for Hadoop, use noatime.
Formatting and tuning options are important. Using tunefs to set the reserve to zero percent can save you over 25 GigaBytes on a 1 TeraByte disk. Also the underlying file system is going to have many large files, you can get more space by lowering the number of inodes at format time.
Ext3
Yahoo! has publicly stated they use ext3. Regardless of the merits of the filesystem, that means that HDFS-on-ext3 has been publicly tested at a bigger scale than any other underlying filesystem that we know of.
XFS
From Bryan on the core-user list on 19 May 2009:
We use XFS for our data drives, and we've had somewhat mixed results. One of the biggest pros is that XFS has more free space than ext3, even with the reserved space settings turned all the way to 0. Another is that you can format a 1TB drive as XFS in about 0 seconds, versus minutes for ext3. This makes it really fast to kickstart our worker nodes. We have seen some weird stuff happen though when machines run out of memory, apparently because the XFS driver does something odd with kernel memory. When this happens, we end up having to do some fscking before we can get that node back online. As far as outright performance, I actually *did* do some tests of xfs vs ext3 performance on our cluster. If you just look at a single machine's local disk speed, you can write and read noticeably faster when using XFS instead of ext3. However, the reality is that this extra disk performance won't have much of an effect on your overall job completion performance, since you will find yourself network bottlenecked well in advance of even ext3's performance. The long and short of it is that we use XFS to speed up our new machine deployment, and that's it.
Ext4
The Ext4 Linux filesystem has delayed allocation of data which makes it handle unplanned server shutdowns/power outages less well than classic ext3. Consider turning off the delalloc option in /etc/fstab unless you trust your UPS.
Here are some recommendations for setting up disks in a Hadoop cluster. What we have here is anecdotal -hard evidence is very welcome, and everyone should expect a bit of trial and error work.
Key Points
Goals for a Hadoop cluster are normally massive amounts of data with high I/O bandwidth. Your MapReduce jobs may be IO bound or CPU/Memory bound -if you know which one is more important (effectively how many CPU cycles/RAM MB used per Map or Reduce), you can make better decisions.
Hardware
You don't need RAID disk controllers for Hadoop Data Node, as it copies data across multiple machines instead. This increase the likelihood that there is a free task slot near that data, and if the servers are on different PSUs and switches, eliminates some more points of failure in the data center.
While the Hadoop Name Node and Secondary Name Node can write to a list of drive locations, they will stop functioning if it can not write to ALL the locations. In this case a mirrored RAID is a good idea for higher availability.
Having lots of disks per server gives you more raw IO bandwidth than having one or two big disks. If you have enough that different tasks can be using different disks for input and output, disk seeking is minimized, which is one of the big disk performance killers. That said: more disks have a higher power budget; if you are power limited, you may want fewer but larger disks.
Configuring Hadoop
Pass a list of disks to the dfs.data.dir parameter, Hadoop will use all of the disks that are available. When one goes offline it is taken out of consideration. Hadoop does not check for the disk coming back -it assumes it is "gone".
How to limit Data node's disk usage?
Use dfs.datanode.du.reserved configuration value in $HADOOP_HOME/conf/hdfs-site.xml for limiting disk usage.
<property>
<name>dfs.datanode.du.reserved</name>
<!-- cluster variant -->
<value>182400</value>
<description>Reserved space in bytes per volume. Always leave this much space free for non dfs use.
</description>
</property>
Logging
The environment variable, HADOOP_LOG_DIR sets the directory Hadoop logs to.
the log4j.properties file in your hadoop configuration dir controls logging in more detail
Don't log to the root directory, as having a machine that does not boot because the logs are overflowing can be inconvenient.
Have a plan to clean up log output, otherwise jobs that log too much to the console will fill up log directories.
Get your developers to use the commons-logging APIs in their MapReduce code, so that you can turn logging up or down without recompiling the code. They can run in debug mode on their test machines, you can run at WARN level in production.
Some JVMs (JRockit) seem to log more. Tune your Log4j settings for your JVM, and only capture the stuff you really want.
Do not keep stuff under /tmp
Hadoop defaults to keeping things under /tmp so that you can play with Hadoop without filling up your disk. This is dangerous in a production cluster, as any automated cleanup cron job will eventually delete stuff in /tmp, at which point your Hadoop cluster is in trouble.
You will need cron job to clean stuff in /tmp up eventually. Plan for it.
Configure Hadoop to store stuff in stable locations, preferably off that root disk.
Java stores the info for jps under /tmp/hsperfdata_${user} -after the cleanup jps won't work. Have your script leave those directories alone, or get used to using ps -ef | grep java to find Java processes instead.
Underlying File System Options
If mount the disks as noatime, then the file access times aren't written back; this speeds up reads. There is also relatime, which stores some access time information, but is not as slow as the classic atime attribute. Remember that any access time information kept by Hadoop is independent of the atime attribute of individual blocks, so Hadoop does not care what your settings are here. If you are mounting disks purely for Hadoop, use noatime.
Formatting and tuning options are important. Using tunefs to set the reserve to zero percent can save you over 25 GigaBytes on a 1 TeraByte disk. Also the underlying file system is going to have many large files, you can get more space by lowering the number of inodes at format time.
Ext3
Yahoo! has publicly stated they use ext3. Regardless of the merits of the filesystem, that means that HDFS-on-ext3 has been publicly tested at a bigger scale than any other underlying filesystem that we know of.
XFS
From Bryan on the core-user list on 19 May 2009:
We use XFS for our data drives, and we've had somewhat mixed results. One of the biggest pros is that XFS has more free space than ext3, even with the reserved space settings turned all the way to 0. Another is that you can format a 1TB drive as XFS in about 0 seconds, versus minutes for ext3. This makes it really fast to kickstart our worker nodes. We have seen some weird stuff happen though when machines run out of memory, apparently because the XFS driver does something odd with kernel memory. When this happens, we end up having to do some fscking before we can get that node back online. As far as outright performance, I actually *did* do some tests of xfs vs ext3 performance on our cluster. If you just look at a single machine's local disk speed, you can write and read noticeably faster when using XFS instead of ext3. However, the reality is that this extra disk performance won't have much of an effect on your overall job completion performance, since you will find yourself network bottlenecked well in advance of even ext3's performance. The long and short of it is that we use XFS to speed up our new machine deployment, and that's it.
Ext4
The Ext4 Linux filesystem has delayed allocation of data which makes it handle unplanned server shutdowns/power outages less well than classic ext3. Consider turning off the delalloc option in /etc/fstab unless you trust your UPS.
发表评论
-
Hadoop TestDFSIO
2013-04-21 21:02 2401@VM [bigdata@bigdata hadoo ... -
Hadoop NNBENCH
2013-04-21 20:46 1602@VM [bigdata@bigdata hadoop]$ ... -
Hadoop 安装手册
2013-04-08 15:47 1161Hadoop 安装手册 软件准备 ... -
What do real life hadoop workloads look like
2012-09-10 15:52 787http://www.cloudera.com/blog/20 ... -
CDH4 HA 切换时间
2012-09-05 15:15 4296blocksize:35M filesize 96M zk-s ... -
CDH4 HA 切换
2012-09-05 10:51 1334HA 切换问题 切换时间太长。。。 copy 0 ... ... -
Hadoop CDh4 Standby HA 启动过程
2012-08-02 11:40 2831根据日志: StandBy NN启动过程 1.获得Active ... -
CDH4 HA test
2012-08-01 14:55 2580场景: NN HA 设置成功,HA切换客户端出现异 ... -
Hadoop TextOutput
2012-07-29 21:08 871TextOutputFormat 分隔符参数: mapredu ... -
Hadoop SteamXMLRecordReader
2012-07-28 23:59 669StreamXmlRecordReader 设置属性 str ... -
Hadoop NLineInputFormat
2012-07-28 23:52 1600NLineInputFormat 重写了splits 设置 ... -
KeyValueTextInputFormat
2012-07-28 23:40 920key/value 分割符 mapreduce.input. ... -
Hadoop 控制split尺寸
2012-07-28 23:08 1294三个参数决定Map的Split尺寸 1.mapred.min ... -
Upgrade hadoop need think about it
2012-07-21 17:17 838Compatibility When movin ... -
Hadoop 0.23 config differ from 0.20.205
2012-07-21 17:14 897http://hadoop.apache.org/common ... -
Hadoop hdfs block 状态
2012-07-15 13:37 6901.In Service -
Hadoop 配置不当引起集群不稳
2012-07-05 15:35 985配置不当内容 资源配置不当:内存、文件句柄数量、磁盘空间 ... -
Hadoop管理-集群维护
2012-07-03 15:27 49551.检查HDFS状态 fsck命令 1)f ... -
Hadoop Ganglia Metric Item
2012-06-27 11:13 1990dfs.FSDirectory.files_delete ... -
Hadoop 参数
2012-06-27 10:05 986转发自:http://www.cnblogs.com/g ...
相关推荐
Apache Hadoop YARN is the modern distributed operating system for big data applications. It morphed the Hadoop compute layer to be a common resource-management platform that can host a wide variety of...
Hadoop2.4.0测试环境搭建 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_installing_manually_boo k/content/rpm-chap1-11.html [bigdata@namenode1 scripts]$ pwd /home/bigdata/software/...
1987年,加州大学伯克利分校的Patterson,Gibson和Katz发表了一篇题为“A Case for Redundant Array of Inexpensive Disks(RAID)”的论文,讲述了RAID
HP_when_good_disks_go_bad_wp1.pdf
The book discusses hard disks and their impact on data management, since Hard Disk Drives continue to be common in large data clusters. It also explores ways to store and retrieve data though primary...
- In Windows 3.1, set up a program item for Mask for Windows - PRWin98 in the StartUp group in Program Manager. When loaded, Mask redraws the desktop and all currently loaded, or selected, Windows...
Micro-disks embedded microring for optical filter
10.2.5 Setting Up a System for Static Routing 296 10.2.6 Con?guring the Corporate Domain 300 10.2.7 Testing the Network Con?guration 302 10.3 Monitoring Network Performance 304 10.3.1 dladm Command ...
a FTL for ssd need to be distributed on on chip sram or off chip dram
网上大部分教程是参照鸟哥的来,有些细节问题跳过了,这份文档较为全面
Novell - Netware Client 3.12 - Installation Disks
BIOS I O requests and caches data for up to 30 BIOS disks including A: or B: diskettes and including hard disks of any size UIDE can handle 48 bit LBA or 24 bit CHS I O calls by new or old DOS ...
# Allows up to 4 devices for simultaneous smart caching; # Cache file size of up to 4GB on each device (NTFS has no limit on the size); # Compatible with all “Enhanced for ReadyBoost” devices.
光度式多波长光学盘片读出信号MPRML检测,菅冀祁,,本文提出一种适用于光度式光度式多波长光学盘片的多通道PRML检测方法,讨论其PR模式及参数并给出一种较优算法。Matlab仿真表明优化算
Setting up User Mode Linux servers, step-by-step: small-scale and large-scale examples The future of virtualization and User Mode Linux Whether you're a netadmin, sysadmin, teacher, student, ...
Examine the command to back up the ASM metadata: ASMCMD>md_backup /backup/ASM_backup In which three situations can you use the backup? A. when one or more disks in an ASM disk group are lost B. when ...
现代操作系统课件:Chapter5-Disks.ppt
[DOS Application] Novell - Netware Lite 1.1 - Installation Disks.zip
SCGCQ00573091 (DFCT) - NCQ setting for SATA pd is showing disabled with TB fW. SCGCQ00576028 (DFCT) - No description for "setdelay" with help file of adppr. Installation Commands: =================...