侯上校
- 浏览: 216525 次
- 性别:
- 来自: 上海
-
最新评论
-
chowqh：
#修改指向我的hadoop安装目录 org.apache.s ...
Sqoop 1.99.3 安装 -
chowqh：
#修改指向我的hadoop安装目录 org.apache.s ...
Sqoop 1.99.3 安装 -
wuzhongfei：
sqoop1.99.3以后是不是全部取消了sqoop命令,例如 ...
Sqoop 1.99.3 安装 -
cyj0421129：
sqoop:000> show version -all ...
Sqoop 1.99.3 安装 -
mypeterhero：
请问，我的服务端也起来了如下：sqoop.sh server ...
Sqoop 1.99.3 安装
文章列表
tensorflow安装
- 博客分类：
- AI
首先, 安装所有必备工具:
# 在 Mac 上:
$ sudo easy_install pip # 如果还没有安装 pip
$ sudo pip install --upgrade virtualenv
接下来, 建立一个全新的 virtualenv 环境. 为了将环境建在 ~/tensorflow 目录下, 执行:
$ virtualenv --system-site-packages ~/tensorflow
$
VCC=>3.3v GND=>地 IN=>GPIO 01 常开、公共端当开关
import com.pi4j.io.gpio.GpioController;
import com.pi4j.io.gpio.GpioFactory;
import com.pi4j.io.gpio.GpioPinDigitalOutput;
import com.pi4j.io.gpio.PinState;
import com.pi4j.io.gpio.RaspiPin;
public class Test{
public static void main(St ...
import org.apache.spark._
import scala.collection.mutable._
import scala.collection.mutable.HashMap
import org.apache.spark.Partitioner
import java.net.URL
val sc = SparkContext.getOrCreate
val rdd1 = sc.textFile("/home/hadoop/my.data")
val rdd2 = rdd1.map(x => {
val url = ...
小度小度，你几岁了？
- 博客分类：
- AI
<div class="iteye-blog-content-contain" style="font-size: 14px"></div>
SparkStreaming定时器生成Job
- 博客分类：
- spark
/**
* An interface to represent clocks, so that they can be mocked out in unit tests.
*/
private[spark] trait Clock {
def getTimeMillis(): Long
def waitTillTime(targetTime: Long): Long
}
/**
* A clock backed by the actual time from the OS as reported by the `System` API.
*/
priva ...
Hive的Base64使用
- 博客分类：
- hive
public static void main(String[] args) throws IOException {
BASE64Encoder base64=new BASE64Encoder();
base64.encode("http://www.xnhzj.com/".getBytes("utf-8"), System.out);
System.out.println();
String apache = new String(org.apache.commons.cod ...
......
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[WARNING] hadoop-yarn-common-2.6.0.jar, hadoop-yarn-api-2.6.0.jar define 3 overlapping classes:
[WARNING] - org.apache.hadoop.yarn.factories.package-info
[WARNING] - org.apache.hadoop.yarn.util.package-info
...
【转】Scala的协变和逆变
- 博客分类：
- spark
转自：http://my.oschina.net/scipio/blog/282794
一、概念
协变和逆变是描述在集合中的多态关系与原来类型多态关系之间的关系。如果T1是T的子类，那么Container[T1]是不是Container[T]的子类呢？Variance注释允许你在类继承和多态类型之间表达下面的这些关系：
* covariant(协变),[+T],C[T1]是 C[T]的子类
* contravariant(逆变)，[-T]，C[T] 是 C[T1]的子类
* invariant(不变)，[T], C[T] 与 C[T1]不相关
...
编译spark-project的hive
- 博客分类：
- hive
mvn clean compile package install -Phadoop-2 -DskipTests
main:
[delete] Deleting directory /usr/local/src/spark_hive/hive-release-1.2.1-spark/packaging/target/tmp
[delete] Deleting directory /usr/local/src/spark_hive/hive-release-1.2.1-spark/packaging/target/warehouse
[mkdir] Create ...
编译Spark1.5.1源码
- 博客分类：
- spark
[root@cluster spark-1.5.1]# mvn -Dhadoop.version=2.6.0-cdh5.4.0 -Dscala-2.10.4 -Phive -Dmaven.test.skip=true clean package
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /usr/local/src/spark-1.5.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.5 ...
编译Hive1.2.1
- 博客分类：
- hive
[root@cluster apache-hive-1.2.1-src]# mvn clean package -Pdist -Phadoop-2 -Dmaven.test.skip=true
....
Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.15/plexus-interpolation-1.15.jar (60 KB at 123.5 KB/sec)
Downloaded: https://repo1.maven.org/maven2/org/apach ...
Hadooop序列化框架使用
- 博客分类：
- hadoop
第一步:实现Writable接口(TestGenericWritable.java)
public static class Baz extends Bar {
@Override
public void readFields(DataInput in) throws IOException {
super.readFields(in);
//needs a configuration parameter
assertEquals("Configuration is not set for the wrapped ob ...
Flume写数据换行
- 博客分类：
- flume
在Flume写入HDFS的时候会自动换行问题， 项目要求写入HDFS中数据不需要换行， 查看源码及配置如下：
BodyTextEventSerializer.java
@Override
public void write(Event e) throws IOException {
out.write(e.getBody());
if (appendNewline) {
out.write('\n');
}
}
// for legacy reasons, by default, append a newline to ea ...