首先先引入hadoop2.7.1中的一个工具类GenericsUtil,顾名思义,是用来处理java泛型的一个工具类,这个类很简单,但很实用,不多说,直接上源码:
package org.apache.hadoop.util;
import java.lang.reflect.Array;
import java.util.List;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
/**
* Contains utility methods for dealing with Java Generics.
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
public class GenericsUtil {
/**
* Returns the Class object (of type <code>Class<T></code>) of the
* argument of type <code>T</code>.
* @param <T> The type of the argument
* @param t the object to get it class
* @return <code>Class<T></code>
*/
public static <T> Class<T> getClass(T t) {
@SuppressWarnings("unchecked")
Class<T> clazz = (Class<T>)t.getClass();
return clazz;
}
/**
* Converts the given <code>List<T></code> to a an array of
* <code>T[]</code>.
* @param c the Class object of the items in the list
* @param list the list to convert
*/
public static <T> T[] toArray(Class<T> c, List<T> list)
{
@SuppressWarnings("unchecked")
T[] ta= (T[])Array.newInstance(c, list.size());
for (int i= 0; i<list.size(); i++)
ta[i]= list.get(i);
return ta;
}
/**
* Converts the given <code>List<T></code> to a an array of
* <code>T[]</code>.
* @param list the list to convert
* @throws ArrayIndexOutOfBoundsException if the list is empty.
* Use {@link #toArray(Class, List)} if the list may be empty.
*/
public static <T> T[] toArray(List<T> list) {
return toArray(getClass(list.get(0)), list);
}
}
利用上面的工具类,下面举一个TestWritableSerialization的一个示例,源码如下:
import java.io.Serializable;
import java.util.EnumSet;
import org.apache.hadoop.io.DataInputBuffer;
import org.apache.hadoop.io.DataOutputBuffer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.ArrayWritable;
import org.apache.hadoop.io.EnumSetWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparator;
import org.apache.hadoop.util.GenericsUtil;
import org.junit.Test;
import static org.junit.Assert.*;
public class THT_TestWritableSerialization {
private static final Configuration conf = new Configuration();
enum TestEnumSet {
CREATE, OVERWRITE, APPEND;
}
@Test
public void testWritableSerialization() throws Exception {
IntWritable before1 = new IntWritable(123456789);
byte[] hadoop = "hadoop".getBytes();
BytesWritable before2 = new BytesWritable(hadoop);
String[] arraywr = { "zero", "one", "two" };
ArrayWritable arraywrA = new ArrayWritable(arraywr);
Writable[] arraywrW = arraywrA.get();
Writable before3 = arraywrW[0];
EnumSetWritable<TestEnumSet> before4 = new EnumSetWritable<TestEnumSet>(
EnumSet.of(TestEnumSet.APPEND));
Text before5=new Text("test for you");
IntWritable after1 = testSerialization(conf, before1);
BytesWritable after2 = testSerialization(conf, before2);
Writable after3 = testSerialization(conf, before3);
EnumSetWritable<TestEnumSet> after4 = testSerialization(conf, before4);
Text after5 = testSerialization(conf, before5);
assertEquals(before1, after1);
assertEquals(before2, after2);
assertEquals(before3, after3);
assertEquals(before4, after4);
assertEquals(before5, after5);
}
public static <K> K testSerialization(Configuration conf, K before)
throws Exception {
SerializationFactory factory = new SerializationFactory(conf);
Serializer<K> serializer = factory.getSerializer(GenericsUtil
.getClass(before));
Deserializer<K> deserializer = factory.getDeserializer(GenericsUtil
.getClass(before));
DataOutputBuffer out = new DataOutputBuffer();
serializer.open(out);
serializer.serialize(before);
serializer.close();
DataInputBuffer in = new DataInputBuffer();
in.reset(out.getData(), out.getLength());
deserializer.open(in);
K after = deserializer.deserialize(null);
deserializer.close();
return after;
}
}
分享到:
相关推荐
Hadoop安装教程_单机/伪分布式配置_Hadoop2.7.1/Ubuntu 16.04
hadoop2.7.1的eclipse插件,编译环境,eclipse 4.4(luna) ,jdk1.7,ant1.9.6,maven3.3,hadoop2.7.1,centos6.7,jdk1.7 要注意的是开发黄金下jdk版本必须是jdk1.7及以上,否则无法使用
eclipse hadoop2.7.1 plugin 配置,包括操作步骤说明,eclipse hadoop2.7.1的插件,还有hadoop.dll和winutils.exe等文件。
Hadoop2.7.1中文文档
hadoop2.7.1的linux下eclipse支持插件。已经经过检查,安装后在eclipse下,可以正常上传,下载,删除hdfs文件,以及跑mapreduce程序。 编译环境:ant1.9.6 jdk1.8 hadoop2.7.1
hadoop2.7.1平台搭建
Spark所需的hadoop2.7.1相关资源 hadoop2.7.1版本的hadoop.dll,winutils.exe 适用Spark2.0.0+版本
hadoop2.7.1版本的hadoop.dll,winutils.exe hadoop2.7.1版本的hadoop.dll,winutils.exe
本人用7个多小时成功编译 hadoop 2.7.1 64位编译包(JDK1.8 64),由于文件太大,分3卷压缩。 hadoop 2.7.1 相对于2.7.0修复了上百个Bug,是可用于生产环境的版本了。
hadoop2.7.1 windows 64位 相关 dll文件
在网上下了好多2.6版本的hadoop.dll,但是都不好使,昨天有个好心网友给我发了一份,实际测试通过。开发环境是64位win7+hadoop2.7.1+redhat版本的linux。
Hadoop-2.7.1-Windows-64-binaries, 预先编译,非官方的Hadoop 2.7.1的Win64二进制文件 用于 Windows 64位 平台的 2.7.1二进制文件这是针对 Windows 64位 平台的非官方预先编译的Apache Hadoop 2.7.1. 这里的tar.gz ...
Windows 7 or 10 eclipse hadoop2.7.1 配置需要文件和工具
Hadoop 2.7.1 中文文档 Hadoop 2.7.1 中文文档 Hadoop 2.7.1 中文文档
hadoop.dll winutils.exe for hadoop2.7.1 下载版本,
win32平台winutils,hadoop win32 native code
hadoop2.7.1运行Wordcount错误 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012) Exception in thread "main" java.lang.UnsatisfiedLinkError
hadoop2.7.1 windows7 32 位 hadoop.dll winutils.exe.在window7 32位下,编译的源码。
hadoop-2.7.1-64位编译包,本人亲测,绝对有用。
hadoop 2.7.1 hdfs-over-ftp http://yunpan.cn/cHaQvV4UdLxX3 访问密码 f09f