10-Oracle大数据OLH连接

pftzzg

浏览: 9749 次
性别:
来自: 北京

最近访客更多访客>>

asdasfa

donlianli

withcourageto

ivwsaiisrg

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

2013-07 ( 11)
更多存档...

博客分类：

hadoop

参考：

http://www.dataguru.cn/forum.php?mod=viewthread&tid=39857

http://blog.sina.com.cn/s/blog_701a48e7010189rc.html

http://www.chenjunlu.com/2012/12/trying-with-oracle-loader-for-hadoop/

http://f.dataguru.cn/thread-39092-1-1.html

os:rhel-server-5.4-x86_64-dvd

jdk版本为:jdk-6u31-linux-i586-rpm.bin

Hadoop版本为：hadoop-0.20.2.tar.gz hadoop-0.20.2-CDH3B4.tar

oraloader版本为：oraloader-2.0.0

oracle的版本为：Version 11.2.0.1.0

JDBC:ojdbc6.jar

解压：oraloader

Oracle大数据OLH连接

有二个文件

查看 README.TXT

所以我们要用CHD 的版本hadoop 现有的版本：hadoop-0.20.2-CDH3B4.tar

解压oraloader

oraloader-2.0.0-1

修改环境变量

[root@hadoop64 home]# vi /etc/profile

配置 Oracle 相关驱动包

ojdbc6.jar 包在/hadoop/oraloader-2.0.0-1/jlib里面有一个

[cloud@hadoop64 jlib]$ cp ojdbc6.jar /hadoop/hadoop-0.20.2/lib

启动hadoop集群

创建目标表

create table tab_lx(id number,name varchar2(50),address varchar2(100));

在Hadoop中放入示例文件

[cloud@hadoop64 bin]$ cat data.dat

1,zzg,湖南

2,zjy,北京

3,ldh,美国

拷贝国际化文件

cp /hadoop/oraloader-2.0.0-1/jlib/orai18n* /hadoop/hadoop-0.20.2/lib

不然报错：

配置 Oracle Loader for Hadoop

[cloud@hadoop64 bin]$ vi MyConf.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

        <property>
                <name>mapred.input.dir</name>
                <value>olh_exercise1_in/data.dat</value>
        </property>
        <property>
                <name>mapreduce.inputformat.class</name>
                <value>oracle.hadoop.loader.lib.input.DelimitedTextInputFormat</value>
        </property>

        <property>
                <name>mapreduce.outputformat.class</name>
                <value>oracle.hadoop.loader.lib.output.JDBCOutputFormat</value>
        </property>




        <property>
                <name>mapred.output.dir</name>
                <value>olh_exercise1_out</value>
        </property>


        <property>
                <name>oracle.hadoop.loader.connection.url</name>
                <value>jdbc:oracle:thin:@192.168.80.70:1521/hadoop64</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.connection.user</name>
                <value>SCOTT</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.connection.password</name>
                <value>oracle</value>
        </property>

        <property>
                <name>oracle.hadoop.loader.loaderMapFile</name>
                <value>file:////hadoop/hadoop-0.20.2/bin/loaderMap_exercise1.xml</value>
        </property>



</configuration>

~

[cloud@hadoop64 bin]$ vi loaderMap_exercise1.xml

<?xml version="1.0" encoding="UTF-8" ?>
<LOADER_MAP>
        <SCHEMA>SCOTT</SCHEMA>
        <TABLE>tab_lx</TABLE>
        <COLUMN field="F0">id</COLUMN>
        <COLUMN field="F1">name</COLUMN>
        <COLUMN field="F2">address</COLUMN>
</LOADER_MAP>

运行

[cloud@hadoop64 bin]$ hadoop jar ${OLH_HOME}/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf MyConf.xml

Oracle Loader for Hadoop Release 2.0.0 - Production

13/02/01 09:34:25 INFO loader.OraLoader: Oracle Loader for Hadoop Release 2.0.0 - Production

13/02/01 09:34:25 INFO loader.OraLoader: Built-Against: not available

13/02/01 09:34:28 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: TAB_LX is not partitioned

13/02/01 09:34:28 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided

13/02/01 09:34:28 INFO output.DBOutputFormat: Setting reduce tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat

13/02/01 09:34:28 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 1 . Job will continue without sampled information.

13/02/01 09:34:28 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:14ms (14 ms)

13/02/01 09:34:28 INFO loader.OraLoader: Submitting OraLoader job OraLoader

13/02/01 09:34:29 INFO input.FileInputFormat: Total input paths to process : 1

13/02/01 09:34:30 INFO loader.OraLoader: map 0% reduce 0%

13/02/01 09:34:38 INFO loader.OraLoader: map 100% reduce 0%

13/02/01 09:34:50 INFO loader.OraLoader: map 100% reduce 100%

13/02/01 09:34:52 INFO loader.OraLoader: Job complete: OraLoader (null)

13/02/01 09:34:52 INFO loader.OraLoader: Counters: 17

FileSystemCounters

FILE_BYTES_READ=114

FILE_BYTES_WRITTEN=260

HDFS_BYTES_READ=39