- 浏览: 300713 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (165)
- hadoop (47)
- linux (11)
- nutch (7)
- hbase (7)
- solr (4)
- zookeeper (4)
- J2EE (1)
- jquery (3)
- java (17)
- mysql (14)
- perl (2)
- compass (4)
- suse (2)
- memcache (1)
- as (1)
- roller (1)
- web (7)
- MongoDB (8)
- struts2 (3)
- lucene (2)
- 算法 (4)
- 中文分词 (3)
- hive (17)
- noIT (1)
- 中间件 (2)
- maven (2)
- sd (0)
- php (2)
- asdf (0)
- kerberos 安装 (1)
- git (1)
- osgi (1)
- impala (1)
- book (1)
- python 安装 科学计算包 (1)
最新评论
-
dandongsoft:
你写的不好用啊
solr 同义词搜索 -
黎明lm:
meifangzi 写道楼主真厉害 都分析源码了 用了很久. ...
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker -
meifangzi:
楼主真厉害 都分析源码了
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker -
zhdkn:
顶一个,最近也在学习设计模式,发现一个问题,如果老是看别人的博 ...
Java观察者模式(Observer)详解及应用 -
lvwenwen:
木南飘香 写道
高并发网站的架构
转载:http://hi.baidu.com/hovlj_1130/blog/item/20200da530603af99052eed9.html
hadoop hdfs balance
最近集群的某些dump节点容量报警,hadoop dfsadmin -report可以发现,报警的节点,均为日志上传至hdfs的节点,在hadoop fs -put时,大部分hdfs文件都直接保存在了put的节点.
因此需要使用hadoop Balancer Administrator Guide.
下面是E文描述:
The balancer is a tool that balances disk space usage on an HDFS cluster when some
datanodes become full or when new empty nodes join the cluster. The tool is deployed as
an application program that can be run by the cluster administrator on a live HDFS
cluster while application adding and deleting files.
SYNOPSIS
To start:
bin/start-balancer.sh [-threshold <threshold>]
Example: bin/ start-balancer.sh
start the balancer with a default threshold of 10%
bin/ start-balancer.sh -threshold 5
start the balancer with a threshold of 5%
To stop:
bin/ stop-balancer.sh
DESCRIPTION
The threshold parameter is a fraction in the range of (0%, 100%) with a default value of
10%. The threshold sets a target for whether the cluster is balanced. A cluster is balanced
if for each datanode, the utilization of the node (ratio of used space at the node to total
capacity of the node) differs from the utilization of the cluster (ratio of used space in the
cluster to total capacity of the cluster) by no more than the threshold value. The smaller
the threshold, the more balanced a cluster will become. It takes more time to run the
balancer for small threshold values. Also for a very small threshold the cluster may not be
able to reach the balanced state when applications write and delete files concurrently.
The tool moves blocks from highly utilized datanodes to poorly utilized datanodes
iteratively. In each iteration a datanode moves or receives no more than the lesser of 10G
bytes or the threshold fraction of its capacity. Each iteration runs no more than 20
minutes. At the end of each iteration, the balancer obtains updated datanodes information
from the namenode.
A system property that limits the balancer’s use of bandwidth is defined in the default
configuration file:
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>1048576</value>
<description> Specifies the maximum bandwidth that each datanode can utilize for the
balancing purpose in term of the number of bytes per second. </description>
</property>
This property determines the maximum speed at which a block will be moved from one
datanode to another. The default value is 1MB/s. The higher the bandwidth, the faster a
cluster can reach the balanced state, but with greater competition with application
processes. If an administrator changes the value of this property in the configuration file,
the change is observed when HDFS is next restarted.
在使用start-balancer.sh时,如果在hdfs-site.xml里面没有配置dfs.balance.bandwidthPerSec,那么集群hdfs内部默认使用1M/S的速度移动数据(so slowly...),我们可以通过在hdfs-site.xml里面配置dfs.balance.bandwidthPerSec来加快balance的速度。
最开始我们配置的是20m/S
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>20971520</value>
<description> Specifies the maximum bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second. </description>
</property>
然后结果是导致job运行变得不稳定,出现一些意外的长map单元,某些reduce时间处理变长(整个集群负载满满的情况下,外加20m/s的balance),在前天的中国hadoop年会上听淘宝调整的为10m/s,需要调整后实验,看看情况如何。
另外再修改dfs.balance.bandwidthPerSec参数后,需要在namenode上运行stop-dfs.sh start-dfs.sh重启hdfs生效。
发表评论
-
博客地址变更
2013-08-16 10:29 1168all the guys of visiting the bl ... -
hadoop 源码分析(六)hadoop taskTracker 生成map 和reduce任务流程
2013-04-09 17:36 2718taskTracker 生成map reduce ... -
hadoop 源码分析(六)hadoop taskTracker 生成map 和reduce任务流程
2013-04-09 17:33 0taskTracker 生成map reduce ... -
hadoop 源码分析(五)hadoop 任务调度TaskScheduler
2013-04-01 11:07 3912hadoop mapreduce 之所有能够实现job的运行 ... -
hadoop 源码分析(四)JobTracker 添加job 到schduler 队列中
2013-03-29 18:37 2855启动 JobTracker 1. 进入main方法: ... -
hadoop 源码分析(三) hadoop RPC 机制
2013-03-28 15:13 2386Hadoop 通信机制采用自己编写的RPC. 相比于 ... -
hadoop 源码分析(二) jobClient 通过RPC 代理提交作业到JobTracker
2013-03-27 12:57 37021.JobClient 客户端类 通过 ... -
hadoop 源码分析(一) jobClient 提交到JobTracker
2013-03-26 13:41 3585Hadoop 用了2年多了.从最初一起创业的 ... -
RHadoop 安装教程
2013-02-01 17:18 1592RHadoop 环境安装 硬件: centos6 ... -
pig
2012-11-16 19:28 1183转自:http://www.hadoopor.c ... -
hadoop与hive的映射
2012-11-15 10:21 2346hadoop与hive的映射 ... -
hadoop distcp
2012-07-31 10:00 2793hadoop distcp 使用:distcp ... -
MapReduce中Mapper类和Reducer类4函数解析
2012-07-20 18:05 2100MapReduce中Mapper类和Reducer类4函数解析 ... -
hadoop metrics 各参数解释
2012-07-17 18:59 1487hadoop metrics 各参数解释 研究使用hadoo ... -
Hbase几种数据入库(load)方式比较
2012-07-17 14:52 13541. 预先生成HFile入库 这个地址有详细的说明http:/ ... -
Hadoop客户端环境配置
2012-05-11 14:59 1747Hadoop客户端环境配置 1. 安装客户端(通过端用户可以 ... -
hadoop 通过distcp进行并行复制
2012-05-02 15:25 2413通过distcp进行并行复制 前面的HDFS访问模型都集中于 ... -
linux crontab 执行hadoop脚本 关于hadoop环境变量引入
2012-04-10 12:11 0crontab问题 crontab的特点:PATH不全和无终 ... -
hadoop fs 命令封装
2012-04-09 09:39 0hadoop fs 命令封装 #!/usr/bin/env ... -
map-reduce编程核心问题
2012-02-22 13:38 12461-How do we break up a large p ...
相关推荐
深度剖析Hadoop HDFS-高清-完整目录-2017年3月。。。。
大数据 Hadoop HDFS 详解
java整合spring和hadoop HDFS全部jar
Hadoop HDFS和MapReduce架构浅析.pdf 更多资源请点击:https://blog.csdn.net/weixin_44155966
snapshot为hadoop2.1.0时新增加的功能。 主要为防止用户误删数据,和数据备份的作用 快照的对象为HDFS目录,前提是该目录是可以允许设置快照的(SnapShotable)
赠送jar包:hadoop-hdfs-2.7.3.jar; 赠送原API文档:hadoop-hdfs-2.7.3-javadoc.jar; 赠送源代码:hadoop-hdfs-2.7.3-sources.jar; 赠送Maven依赖信息文件:hadoop-hdfs-2.7.3.pom; 包含翻译后的API文档:hadoop...
python 操作 Hadoop hdfs 读read 写write 下载download
Prometheus Hadoop HDFS FSImage导出器 | 将Hadoop HDFS统计信息导出到包括 总数/每个用户/每个组/每个配置的目录路径/每个路径集 目录数 文件数 文件大小和大小分布(可选) 块数 文件复制(总体/每个用户摘要)...
《HDFS——Hadoop分布式文件系统深度实践》
Hadoop HDFS分布式文件系统常用命令汇总 Hadoop HDFS分布式文件系统是大数据存储和处理的核心组件。作为一个分布式文件系统,HDFS提供了高可靠性、高可扩展性和高性能的存储解决方案。在使用HDFS时,经常需要执行...
Hadoop hdfs.md
Hadoop HDFS_Shell命令详解
从hadoop hdfs中读取数据,进行groupby 显示统计结果count、avg、max,用文字和柱状图两种图形界面表示
赠送jar包:hadoop-hdfs-2.6.5.jar; 赠送原API文档:hadoop-hdfs-2.6.5-javadoc.jar; 赠送源代码:hadoop-hdfs-2.6.5-sources.jar; 赠送Maven依赖信息文件:hadoop-hdfs-2.6.5.pom; 包含翻译后的API文档:hadoop...
window客户端连接hadoop hdfs,完美解决障碍
赠送jar包:hadoop-hdfs-client-2.9.1.jar 赠送原API文档:hadoop-hdfs-client-2.9.1-javadoc.jar 赠送源代码:hadoop-hdfs-client-2.9.1-sources.jar 包含翻译后的API文档:hadoop-hdfs-client-2.9.1-javadoc-...
深入理解Hadoop HDFS高级教程.txt打包整理.zip
Hadoop(HDFS)
HadoopHDFS架构概述推荐系统框架图
Hadoop hdfs文件操作,单词统计MR代码 demo,topN求解编程代码 demo