`
bnmnba
  • 浏览: 288930 次
  • 性别: Icon_minigender_1
社区版块
存档分类
最新评论

在Hbase表上运行mapreduce，Maven打包

 
阅读更多

 

运行了一个《HBase权威指南》的示例ch07。

代码：https://github.com/larsgeorge/hbase-book

 

不需要更改hadoop和hbase以及zookeeper的任何配置。只要它们正常运行即可。我用的是CDH5.8.0版本。

大概方法如下：

 

1.先命令行建表

bin/hbase shell
create 't2','c2'
put 't2','row1','c2:col1','value1'

 

2.用maven插件shade打包并拷贝jar到服务器。

具体方法是首先建立maven项目，拷贝代码，增加必要依赖，增加打包插件，编译。

 

3.配置好服务的环境变量HADOOP_HOME、HBASE_HOME、ZK_HOME。

例如命令，一行一行执行：

echo 'export HADOOP_PREFIX=/usr/local/app/cdh5.8.0/hadoop-2.6.0-cdh5.8.0/' >> /etc/profile
echo 'export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop/' >> /etc/profile
echo 'export HADOOP_HOME=$HADOOP_PREFIX' >> /etc/profile
echo 'export HBASE_HOME=/usr/local/app/cdh5.8.0/hbase-1.2.0-cdh5.8.0/' >> /etc/profile
echo 'export ZK_HOME=/usr/local/app/cdh5.8.0/zookeeper-3.4.5-cdh5.8.0/' >> /etc/profile
source /etc/profile

 

4.对hbase表进行mapreduce计算测试，运行命令：(注意：这个很长的是一个linux命令，其中重音符号(`)并不是单引号(')，写错成单引号将出现错误！)

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/lib/hbase-server-1.2.0-cdh5.8.0.jar rowcounter t2

 

5.运行自己打包的jar：

HADOOP_CLASSPATH=`${HBASE_HOME}/bin/hbase classpath` ${HADOOP_HOME}/bin/hadoop jar test.Maven-0.0.1-SNAPSHOT.jar AnalyzeData -t t2 -c c2:col1 -o analyze1

 

 

 

附录：

pom.xml：比较大，大部分是拷贝的。关键是依赖和shade打包插件。那个fatjar貌似不行。

 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.xxx</groupId>
  <artifactId>test.Maven</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>test.Maven</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <jdkLevel>1.7</jdkLevel>
    <requiredMavenVersion>[2.1,)</requiredMavenVersion>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.build.outputEncoding>UTF-8</project.build.outputEncoding>

    <hadoop.version>2.7.2</hadoop.version>
    <zookeeper.version>3.4.8</zookeeper.version>
    <hbase.version>1.2.1</hbase.version>
  </properties>

  <build>
  
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.3.1</version>
        <configuration>
          <archive>
            <index>true</index>
            <manifest>
              <mainClass>mapreduce/Driver</mainClass>
            </manifest>
          </archive>
        </configuration>
      </plugin>
      
  		<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
	        <configuration>
	          <!-- put your configurations here -->
	          <transformers>
	          	<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
	          		<mainClass>mapreduce.Driver</mainClass>
	          	</transformer>
				<transformer  
					implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
					<resource>META-INF/spring.handlers</resource>  
				</transformer>  
				<transformer  
					implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">  
					<resource>META-INF/spring.schemas</resource>  
				</transformer>
	          </transformers>
	        </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  <dependencies>
  
		<dependency>
		  <groupId>log4j</groupId>
		  <artifactId>log4j</artifactId>
		  <version>1.2.17</version>
		</dependency>
		<dependency>
  <groupId>commons-logging</groupId>
  <artifactId>commons-logging</artifactId>
  <version>1.1.3</version>
</dependency>
  	<dependency>
  		<groupId>junit</groupId>
  		<artifactId>junit</artifactId>
  		<version>4.9</version>
  		<scope>test</scope>
  	</dependency>
  	<dependency>
	  <groupId>org.apache.hbase</groupId>
	  <artifactId>hbase-client</artifactId>
	  <version>1.2.0</version>
	</dependency>
	<dependency>
	    <groupId>jdk.tools</groupId>
	    <artifactId>jdk.tools</artifactId>
	    <version>1.6</version>
	    <scope>system</scope>
	    <systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
	</dependency>
	
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-core</artifactId>
      <version>2.6.0</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-yarn-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>guice-servlet</artifactId>
          <groupId>com.google.inject.extensions</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>commons-cli</groupId>
      <artifactId>commons-cli</artifactId>
      <version>1.2</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>com.googlecode.json-simple</groupId>
      <artifactId>json-simple</artifactId>
      <version>1.1.1</version>
      <exclusions>
        <exclusion>
          <artifactId>hamcrest-core</artifactId>
          <groupId>org.hamcrest</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>commons-codec</groupId>
      <artifactId>commons-codec</artifactId>
      <version>1.10</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>log4j-over-slf4j</artifactId>
      <version>1.7.10</version>
    </dependency>
   
    <!-- HBase and Hadoop dependencies -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <groupId>commons-cli</groupId>
          <artifactId>commons-cli</artifactId>
        </exclusion>
        <exclusion>
          <groupId>xmlenc</groupId>
          <artifactId>xmlenc</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-httpclient</groupId>
          <artifactId>commons-httpclient</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-codec</groupId>
          <artifactId>commons-codec</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-net</groupId>
          <artifactId>commons-net</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty-util</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-compiler</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-api-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-el</groupId>
          <artifactId>commons-el</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.java.dev.jets3t</groupId>
          <artifactId>jets3t</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>servlet-api-2.5</artifactId>
        </exclusion>
        <exclusion>
          <groupId>net.sf.kosmosfs</groupId>
          <artifactId>kfs</artifactId>
        </exclusion>
        <exclusion>
          <groupId>hsqldb</groupId>
          <artifactId>hsqldb</artifactId>
        </exclusion>
        <exclusion>
          <groupId>oro</groupId>
          <artifactId>oro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.eclipse.jdt</groupId>
          <artifactId>core</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-configuration</artifactId>
          <groupId>commons-configuration</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsr305</artifactId>
          <groupId>com.google.code.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-shuffle</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-yarn-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-hdfs</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-app</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-annotations</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-yarn-api</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hadoop-annotations</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-cli</artifactId>
          <groupId>commons-cli</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>xmlenc</artifactId>
          <groupId>xmlenc</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math3</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-net</artifactId>
          <groupId>commons-net</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>servlet-api</artifactId>
          <groupId>javax.servlet</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-core</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-json</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-server</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-api</artifactId>
          <groupId>javax.servlet.jsp</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-el</artifactId>
          <groupId>commons-el</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jets3t</artifactId>
          <groupId>net.java.dev.jets3t</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-compiler</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-runtime</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-configuration</artifactId>
          <groupId>commons-configuration</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-api</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>avro</artifactId>
          <groupId>org.apache.avro</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsch</artifactId>
          <groupId>com.jcraft</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsr305</artifactId>
          <groupId>com.google.code.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>gson</artifactId>
          <groupId>com.google.code.gson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-client</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-recipes</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-auth</artifactId>
      <version>${hadoop.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>httpclient</artifactId>
          <groupId>org.apache.httpcomponents</groupId>
        </exclusion>
        <exclusion>
          <artifactId>apacheds-kerberos-codec</artifactId>
          <groupId>org.apache.directory.server</groupId>
        </exclusion>
        <exclusion>
          <artifactId>curator-framework</artifactId>
          <groupId>org.apache.curator</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-annotations</artifactId>
      <version>${hadoop.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.zookeeper</groupId>
      <artifactId>zookeeper</artifactId>
      <version>${zookeeper.version}</version>
      <exclusions>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>jline</groupId>
          <artifactId>jline</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>netty</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-api</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>slf4j-log4j12</artifactId>
          <groupId>org.slf4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.apache.avro</groupId>
          <artifactId>avro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-lang</groupId>
          <artifactId>commons-lang</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.guava</groupId>
          <artifactId>guava</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
        <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>avro</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.zookeeper</groupId>
          <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.thrift</groupId>
          <artifactId>thrift</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.jruby</groupId>
          <artifactId>jruby-complete</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.google.protobuf</groupId>
          <artifactId>protobuf-java</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-core</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-json</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.sun.jersey</groupId>
          <artifactId>jersey-server</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.ws.rs</groupId>
          <artifactId>jsr311-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>javax.xml.bind</groupId>
          <artifactId>jaxb-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>stax</groupId>
          <artifactId>stax-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-cli</groupId>
          <artifactId>commons-cli</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-codec</groupId>
          <artifactId>commons-codec</artifactId>
        </exclusion>
        <exclusion>
          <groupId>commons-httpclient</groupId>
          <artifactId>commons-httpclient</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jetty-util</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>jsp-api-2.1</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.mortbay.jetty</groupId>
          <artifactId>servlet-api-2.5</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-compiler</artifactId>
        </exclusion>
        <exclusion>
          <groupId>tomcat</groupId>
          <artifactId>jasper-runtime</artifactId>
        </exclusion>
        <exclusion>
          <groupId>com.github.stephenc.high-scale-lib</groupId>
          <artifactId>high-scale-lib</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.apache.thrift</groupId>
          <artifactId>libthrift</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.jamon</groupId>
          <artifactId>jamon-runtime</artifactId>
        </exclusion>
        <exclusion>
          <artifactId>jdk.tools</artifactId>
          <groupId>jdk.tools</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty-all</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-common</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-common</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-protocol</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>joni</artifactId>
          <groupId>org.jruby.joni</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jcodings</artifactId>
          <groupId>org.jruby.jcodings</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-common</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>protobuf-java</artifactId>
          <groupId>com.google.protobuf</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-protocol</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-annotations</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-server</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>hbase-protocol</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-prefix-tree</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-collections</artifactId>
          <groupId>commons-collections</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-hadoop-compat</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-auth</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-mapreduce-client-core</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-httpclient</artifactId>
          <groupId>commons-httpclient</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hbase-hadoop2-compat</artifactId>
          <groupId>org.apache.hbase</groupId>
        </exclusion>
        <exclusion>
          <artifactId>metrics-core</artifactId>
          <groupId>com.yammer.metrics</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-io</artifactId>
          <groupId>commons-io</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>hadoop-hdfs</artifactId>
          <groupId>org.apache.hadoop</groupId>
        </exclusion>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-cli</artifactId>
          <groupId>commons-cli</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-lang</artifactId>
          <groupId>commons-lang</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-math</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-sslengine</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-2.1</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>servlet-api-2.5</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-jaxrs</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-runtime</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>netty-all</artifactId>
          <groupId>io.netty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>disruptor</artifactId>
          <groupId>com.lmax</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jsp-api-2.1</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jasper-compiler</artifactId>
          <groupId>tomcat</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jamon-runtime</artifactId>
          <groupId>org.jamon</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jetty-util</artifactId>
          <groupId>org.mortbay.jetty</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-codec</artifactId>
          <groupId>commons-codec</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-core</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jersey-server</artifactId>
          <groupId>com.sun.jersey</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>htrace-core</artifactId>
          <groupId>org.apache.htrace</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-logging</artifactId>
          <groupId>commons-logging</groupId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-annotations</artifactId>
      <version>${hbase.version}</version>
      <exclusions>
        <exclusion>
          <artifactId>findbugs-annotations</artifactId>
          <groupId>com.github.stephenc.findbugs</groupId>
        </exclusion>
        <exclusion>
          <artifactId>log4j</artifactId>
          <groupId>log4j</groupId>
        </exclusion>
        <exclusion>
          <artifactId>junit</artifactId>
          <groupId>junit</groupId>
        </exclusion>
      </exclusions>
    </dependency>
   
    
		<dependency>
		  <groupId>commons-logging</groupId>
		  <artifactId>commons-logging</artifactId>
		  <version>1.1.3</version>
		</dependency>
		<dependency>
		  <groupId>commons-collections</groupId>
		  <artifactId>commons-collections</artifactId>
		  <version>3.0</version>
		</dependency>
		<dependency>
		  <groupId>commons-lang</groupId>
		  <artifactId>commons-lang</artifactId>
		  <version>2.6</version>
		</dependency>
   
	   <dependency>
		  <groupId>io.netty</groupId>
		  <artifactId>netty-all</artifactId>
		  <version>4.1.5.Final</version>
		</dependency>
  </dependencies>

  <profiles>
    <profile>
      <id>fatjar</id>
      <activation>
        <activeByDefault>false</activeByDefault>
        <property>
          <name>fatjar</name>
        </property>
      </activation>
      <!--dependencies>
        <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-snappy</artifactId>
          <version>${hadoop-snappy.version}</version>
        </dependency>
      </dependencies-->
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <version>2.5.3</version>
            <configuration>
              <tarLongFileMode>gnu</tarLongFileMode>
              <descriptors>
                <descriptor>src/main/assembly/job.xml</descriptor>
              </descriptors>
              <!--descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
              </descriptorRefs-->
              <archive>
                <index>true</index>
                <manifest>
                  <mainClass>mapreduce/Driver</mainClass>
                </manifest>
              </archive>
            </configuration>
            <executions>
              <execution>
                <id>tarball</id>
                <phase>package</phase>
                <goals>
                  <goal>single</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
</project>

 

 

 

 

 

 

分享到：
| java客户端连接操作Hbase示例代码
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

    HBase与MapReduce处理操作（基于JavaAPI）

    该案例中主要使用MapReduce作为处理组件进行数据处理，实现的案例有如通过javaapi实现hbase数据写入hdfs、hbase表数据复制到另一个表中等操作 对应（《HBase分布式存储系统应用》胡鑫喆 张志刚著）教材中案例

    Hbase中文文档

    2.4. HBase运行模式:单机和分布式 2.5. ZooKeeper 2.6. 配置文件 2.7. 配置示例 2.8. 重要配置 2.9. Bloom Filter 3. 升级 3.1. 从HBase 0.20.x or 0.89.x 升级到 HBase 0.90.x 3.2. 从 0.90.x 到 0.92.x 4. The ...

    大数据技术开发环境搭建.docx

    使用Eclipse打包自己的MapReduce程序 51 不用Hadoop-Eclipse-Plugin编写MapReduce程序 54 Hadoop集群安装配置教程 56 HBase安装 56 HBase伪分布式配置 57 HBase Shell编程实践 60 HBase JAVA API编程实践 64 ...

    HadoopLearning：完整的大数据基础学习教程，包含最基础的centos，maven。大数据主要包含hdfs，mr，yarn，hbase，kafka，scala，sparkcore，sparkstreaming，sparksql。

    maven相关2，大数据教程2.1，hdfs教程2.1，mapreduce教程3，剩余编写HDFS入门，深入，Shell访问，Java API操作MapReduce入门，深入，编程基础，编程进阶，实战分析和训练Yarn入门，原理剖解和应用场景Hbase存储原理...

    Thesis-HBase:硕士论文

    Distribution : Ubuntu 12.04 64bit, Apache Hadoop 2.5.0 Programming : Java SDK 8, Apache ...HBase是一个宽列存储，提供简单的get和set API以及完整的MapReduce支持。 在本硕士论文中，开发一种称为“ NotaQL”的语

    2017最新大数据架构师精英课程

    本资源为大数据基础到中高级教学资源，适合稍微有点大数据或者java基础的人群学习，资源过大，上传乃是下载链接，不多说，上目录： 1_java基础2 l3 a2 a$ t7 J2 b+ `- p 2_java引入ide-eclipse 3_java基础知识-循环...

    JavaNotes：Java重新学习笔记。包括Linux，maven，git，互联网架构，大数据体系等

    JavaNotes：Java重新学习笔记。包括Linux，maven，git，互联网架构，大数据体系等

    毕设项目基于SSM框架实现的二手交易平台项目源码+项目说明.zip

    毕设项目基于SSM框架实现的二手交易平台项目源码+项目说明.zip 首先该项目主题是二手交易系统，在该程序中可以实现两种角色权限进行登录 用户权限登录：使用用户权限...大数据技术：flume、kafka、MapReduce、HBASE等

    HadoopLearning:大数据学习教程

    Hbase存储原理、RowKey设计、协处理、Shell访问、Java API访问和Hbase性能调优 ElasticSearch入门、概念基础、基础原理、索引、映射、搜索、聚合和性能优化 Scala入门、函数、类、对象、特征、模式匹配、常用类和...

    hadoop-mini-clusters

    MapReduce-迷你MapReduce集群 HBase-迷你HBase群集 Zookeeper-基于策展人的本地集群 HiveServer2-本地HiveServer2实例 HiveMetaStore-Derby支持的HiveMetaStore 风暴-Storm LocalCluster Kafka-本地Kafka经纪人 ...

    pentaho-hadoop-shims:Hadoop配置

    Hadoop配置（也称为垫片和Pentaho大数据自适应层）是与特定版本的Hadoop（以及相关工具：Hive，HBase，Sqoop，Pig等）进行通信所需的Hadoop库的集合。 它们被设计为易于配置。 如何建造 pentaho-hadoop-shims使用...

    java查看函数源码-BigDataArchitect:大数据架构师

    学习到相关模块请在该项目目录中去到子目录查看具体的README描述！ 大数据架构师课程导读 1. bigdata_hadoop 2. bigdata_hive 3. bigdata_hbase 4. bigdata_hadoop_project 5. bigdata_redis 6. bigdata_zookeeper 7...

    matlab饼图程序代码-zaopingshujufenxi:招聘数据分析案例

    使用软件/工具：idea，webstom，maven，visual studio code 2 思路和实现流程 2.1整体思路 1), 先编写java程序（项目：yuchuli1）对源数据的空行，多行，多空白字符，进行消除，对字段位置错误（如公司人数）进行...

Magicbox
Global site tag (gtag.js) - Google Analytics