HADOOP-HDFS 自动清除Trash以释放空间

HDFS should support Auto-Emptier to automatically expunge trash for releasing space.

HADOOP-HDFS需要有一个Auto-Emptier 线程来自动清除trash, 以释放HDFS的总使用空间, 该功能可以配置为可选项, 可以在Configuration下增加这两个参数以供配置.

1. fs.trash.autoemptier.interval  执行空间检查的时间时间隔, 设置为0时, 禁用该功能, 默认为20 Seconds.
2. fs.trash.max.percentused 当已使用空间率大于该值, 执行回收以释放空间. 默认为0.8f

这个功能只是个人的倾向而已, 并没有新建Feature到Apache, 欢迎同志们拍砖.

float percent = namenode.namesystem.getCapacityUsedPercent();
					if(percent < 1 && percent >= maxPercentUsed) {
						Path path = new Path(fs.getHomeDirectory(), TRASH);
						/*Trash trash = new Trash(path, conf);
						trash.expunge();
						trash.checkpoint();*/
						long space = 0;
						if (!fs.exists(path)
									|| (space = fs.getFileStatus(path).getLen()) == 0) {
								continue;
						}
						if(!fs.delete(path, true)){
							LOG.warn("Failed to expunge trash for releasing space.");
						} else {
						LOG.info("Auto-emptier, expunge trash for releasing space: " + FsShell.byteDesc(space));
}
					}

由于博客标题长度的限制, 标题里开头少写了HDFS字样. 源码见于附件.

评论

3 楼 david.org 2010-03-02  

1. 在fsnamesystem未进入安全模式之前, 不应该执行del操作, 当然会抛出异常, 应该避免这种异常的发生.

2. fsnamesystem#getCapacityUsedPercent方法, 返回的值不准确, 已使用空间比率 = 总配置空间 / 已使用空间 + 剩余空间大小.

if (namenode.namesystem.getFSState().equals("Operational") && now >= end) {
					long used = namenode.namesystem.getCapacityUsed();
					long remaining = namenode.namesystem.getCapacityRemaining();
					long presentCapacity = used + remaining;
					String pc = FsShell.limitDecimalTo2(((1.0 * used) / presentCapacity) * 100);
					float percent = Float.valueOf(pc)/100f;
					
					LOG.info("Present dfs used: " + percent);
					
					if(percent < 1 && percent >= maxPercentUsed) {
						Path path = new Path(fs.getHomeDirectory(), TRASH);
						long space = 0;
						if (!fs.exists(path)
									|| (space = fs.getContentSummary(path).getLength()) == 0) {
								continue;
						}
						if(!fs.delete(path, true)) {
							LOG.warn("Failed to expunge trash for releasing space.");
						} else {
							LOG.info("Auto-emptier, expunge trash for releasing space: " + FsShell.byteDesc(space));
						}
					}
				}

2 楼 david.org 2010-02-20  

if (!fs.exists(path)
									|| (space = fs.getContentSummary(path).getLength()) == 0) {
								continue;
						}

1 楼 david.org 2010-02-20  

更改一下 bug
getFileStatus方法无法获取目录的空间大小, 需要使用
[url]if (!fs.exists(path)
|| (space = fs.getContentSummary(path).getLength()) == 0) {
continue;
}[/url]