Map/Reduce 单元测试
world count 测试
package com.irwin.hadoop; import java.io.File; import java.io.IOException; import java.util.ArrayList; import java.util.List; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.TaskAttemptContext; import org.apache.hadoop.mapreduce.TaskAttemptID; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.input.LineRecordReader; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl; import org.apache.hadoop.mrunit.mapreduce.MapDriver; import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver; import org.apache.hadoop.mrunit.mapreduce.ReduceDriver; import org.apache.hadoop.util.ReflectionUtils; import org.junit.Assert; import org.junit.Before; import org.junit.Test; import com.irwin.hadoop.WordCount.TokenizerMapper; import com.irwin.hadoop.WordCount.IntSumReducer; public class WordCountTest extends Assert{ private MapDriver<Object, Text, Text, IntWritable> mapDriver; private ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver; private static final IntWritable one = new IntWritable(1); MapReduceDriver<Object, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver; @Before public void setUp(){ Mapper<Object, Text, Text, IntWritable> mapper = new TokenizerMapper(); mapDriver = MapDriver.newMapDriver(mapper); Reducer<Text, IntWritable, Text, IntWritable> reducer = new IntSumReducer(); reduceDriver = ReduceDriver.newReduceDriver(reducer); mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer); } @Test public void testReducer() throws Exception{ List<IntWritable> values = new ArrayList<IntWritable>(); values.add(new IntWritable(1)); values.add(new IntWritable(1)); reduceDriver.withInput(new Text("hello"), values); reduceDriver.withOutput(new Text("hello"), new IntWritable(2)); reduceDriver.runTest(); } @Test public void testMapReduce() throws IOException { mapReduceDriver.withInput(new LongWritable(1), new Text("cat cat dog")); mapReduceDriver.addOutput(new Text("cat"), new IntWritable(2)); mapReduceDriver.addOutput(new Text("dog"), new IntWritable(1)); mapReduceDriver.runTest(); } @Test public void testMapper() throws Exception { LineRecordReader reader = getBytesRecordReader(); int counter = 0; while (reader.nextKeyValue()) { mapDriver.withInput(reader.getCurrentKey(),reader.getCurrentValue()); StringTokenizer itr = new StringTokenizer(reader.getCurrentValue().toString()); while (itr.hasMoreElements()) { mapDriver.withOutput(new Text(itr.nextToken()), one); } counter++; } assertEquals(3, counter); // auto assert output mapDriver.runTest(); } private static LineRecordReader getBytesRecordReader()throws IOException, InterruptedException { Configuration conf = new Configuration(false); conf.set("fs.default.name", "file:///"); File testFile = new File("src\\test\\resources\\test_data\\text\\hello.text"); Path path = new Path(testFile.getAbsoluteFile().toURI()); FileSplit split = new FileSplit(path, 0, testFile.length(), null); TextInputFormat inputFormat = ReflectionUtils.newInstance(TextInputFormat.class, conf); TaskAttemptContext context = new TaskAttemptContextImpl(conf,new TaskAttemptID()); LineRecordReader reader = (LineRecordReader) inputFormat.createRecordReader(split, context); reader.initialize(split, context); return reader; } }
相关推荐
Hadoop MRUnit测试 工程用Maven构建,详细过程参见本人博客Maven构建一文
非常好用测试插件,在mapReduce下直接可以运行,本人亲测成功
MRUnit测试支持JAR包,它便于将已知的输入传递给mapper或者检查reducer的输出是否符合预期。MRUnit与标准的执行框架(JUnit)一起使用。
支持MapReduce MRUnit单元测试包,支持MapReduce MRUnit单元测试包
书中包括了更多的mapreduce资料,比如用maven打包MapReduce,设置java环境变量,写MRUnit测试单元(第五章介绍),还有一些更深入的特性,比如输出的提交,分布式缓存等(第8章),任务内存监控(第9章),第4章...
hadoop单元测试方法--使用和增强MRUnit.docx
hadoop本地测试的jar包 ,mrunit-1.1.0-hadoop2.jar,使用后可在本地进行MapReduce代码测试
用于hadoop单元测试的jar包 hadoop-mrunit-0.20.2-cdh3u4.jar
具有MRUnit测试的MapReduce WordCount应用程序 字符串对的MapReduce自定义可写实现 MapReduce自定义InputFormat和RecordReader实现 MapReduce自定义OutputFormat和RecordWriter实现 Pig自定义LoadFunc加载和解析...
官网下载的Hadoop2的单元测试工具apache-mrunit-1.1.0-hadoop2-bin.tar.gz,用法跟JUnit相似,需要的朋友拿走
使用MRUnit来编写HadoopMapReduce应用程序的JUnit测试2.使用PowerMock和Mockito模拟静态方法3.模拟其他类型中的业务逻辑(译注:也就是编写测试驱动模块)4.查看模拟的业务逻辑是否被调用(译注:测试驱动模块是否...
“ src / test / java” 样本输入日志文件路径: “输入” 样本输出路径: “已排序”特征LogAnalyzer的开发目的是演示以下功能: 编写MapReduce Java程序使用柜台使用排序比较器使用MRUnit Framework编写单元测试一...
13.1.2 MRUnit . 技术点79 MapReduce 函数、作业和管道的单元测试 13.1.3 LocalJobRunner 技术点80 用LocalJobRunner 进行重量级的作业测试 13.1.4 集成和QA 测试 13.2 调试用户空间的问题 13.2.1 ...
12 Crunch 及相关技术12.1 什么是Crunch12.1.1 背景和概念12.1.2 基本原理12.1.3 简单示例12.2 发现日志中最热门的URL技术点77 ...13.1 测试13.1.1 有效的单元测试的基本要素13.1.2 MRUnit ....
对于每个样本,我都包含了一些使用mrunit(map-reduce单元)的简单测试。 这是一个标准的Maven项目,因此您可以使用以下命令进行构建: mvn clean install字数程序包:com.sodonnel.Hadoop 这是wordcount示例的一种...