- 浏览: 151009 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (113)
- Java工具类 (3)
- Hibernate (1)
- Struts>1&2 (0)
- Spring (0)
- Web前端 (1)
- 数据库 (2)
- 中间件 (3)
- 搜索引擎 (11)
- EJB (1)
- Log日志 (1)
- OS操作系统 (7)
- Java文档 (4)
- Security (1)
- Exception&Error (7)
- 日志&心情 (7)
- 开心一下 (3)
- 云计算 (41)
- Cache缓存 (4)
- MongoDB (1)
- OS操作系统 Linux (7)
- jquery (1)
- 机器学习 (2)
- Plugins (2)
- AJAX (1)
- Java工具 (1)
- WebService (1)
最新评论
-
a420144030:
你好,我想计算n篇文章的相似度,用mahout能处理吗,如何做 ...
mahout使用 -
di1984HIT:
OK,真的很牛啊
hadoop常见错误 -
xwl1991:
还真是这个问题 !
quartz报错java.lang.NoSuchMethodError: org.apache.commons.collections.SetUtils.ord
用sqoop进行mysql和hdfs系统间的数据互导
- 博客分类:
- 云计算
转自:http://abloz.com/2012/07/19/data-between-the-mysql-and-hdfs-system-of-mutual-conductance-using-sqoop.html
sqoop
是apache下用于RDBMS和HDFS互相导数据的工具。
本文档是sqoop的使用实例,实现从mysql到hdfs互导数据,以及从Mysql导数据到HBase。
下载:
http://www.apache.org/dyn/closer.cgi/sqoop/
[zhouhh@Hadoop48 ~]$ wget http://labs.renren.com/apache-mirror/sqoop/1.4.1-incubating/sqoop-1.4.1-incubating__hadoop-1.0.0.tar.gz
最新用户手册
http://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html
一、从HBase库中直接导出到mysql中?
一开始我想从HBase库中直接导出到mysql中。
在mysql中创建一个库和表
mysql> create database toplists; Query OK, 1 row affected (0.06 sec) mysql> use toplists Database changed mysql> create table t1(id int not null primary key, name varchar(255),value int); Query OK, 0 rows affected (0.10 sec) hbase(main):011:0> scan 't1' ROW COLUMN+CELL 1001 column=info:count, timestamp=1340265059531, value=724988 1009 column=info:count, timestamp=1340265059533, value=108051 ... total column=info:count, timestamp=1340265059534, value=833039 total_user_count column=info:, timestamp=1340266656307, value=154516 11 row(s) in 0.0420 seconds [zhouhh@Hadoop48 ~]$ sqoop list-tables --connect jdbc:mysql://localhost/toplists --username root java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:657) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:473) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:496) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:194) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:178) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:114) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1235) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1060) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:97) at org.apache.sqoop.Sqoop.run(Sqoop.java:145) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at org.apache.sqoop.Sqoop.main(Sqoop.java:238) at com.cloudera.sqoop.Sqoop.main(Sqoop.java:57)
需下载 MySQL JDBC Connector 库,并将其复制到$SQOOP_HOME/lib
下载mysql
jdbc连接库
地址:
http://www.mysql.com/downloads/connector/j/
[zhouhh@Hadoop48 ~]$ wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.21.tar.gz/from/http://cdn.mysql.com/ [zhouhh@Hadoop48 mysql-connector-java-5.1.21]$ cp mysql-connector-java-5.1.21-bin.jar ../sqoop/lib/. [zhouhh@Hadoop48 ~]$ sqoop list-tables --connect jdbc:mysql://localhost/toplists --username root t1 [zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://localhost/toplists --username root --table t1 --export-dir /hbase java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:628) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:753) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
这是可能由jdbc版本引起的,换成5.1.18
[zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://localhost:3306/toplists --username root --table t1 --export-dir /hbase Error initializing attempt_201206271529_0006_r_000000_0: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for ttprivate/taskTracker/zhouhh/jobcache/job_201206271529_0006/jobToken at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127) at org.apache.hadoop.mapred.TaskTracker.localizeJobTokenFile(TaskTracker.java:4271) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1177) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430) at java.lang.Thread.run(Thread.java:722)
DiskErrorException ,定位半天,发现是另一台机器的空间满了,在mapreduce运行时会引起该异常。
[zhouhh@Hadoop46 ~]$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda3 28337624 26877184 0 100% /
[zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table t1 --export-dir /hbase Caused by: java.sql.SQLException: null, message from server: "Host 'Hadoop47' is not allowed to connect to this MySQL server"
这是权限问题,设置授权:
mysql> GRANT ALL PRIVILEGES ON *.* TO '%'@'%';#允许所有用户查看和修改databaseName数据库模式的内容,否则别的IP连不上本MYSQL Query OK, 0 rows affected (0.06 sec)
这是测试,所以权限没有限制。实际工作环境需谨慎授权。
[zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table t1 --export-dir /hbase Note: /tmp/sqoop-zhouhh/compile/fa1d1c042030b0ec8537c7a4cd02aab3/t1.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. java.lang.NumberFormatException: For input string: "7" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:481) at java.lang.Integer.valueOf(Integer.java:582) at t1.__loadFromFields(t1.java:218) at t1.parse(t1.java:170) at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:77) at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:183) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249)
这是由于/hbase是hbase的库表,根本不是可以导的格式,所以报错。
[zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table t1 --export-dir /hbase/t1 [zhouhh@Hadoop48 ~]$ sqoop-export --verbose --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table t1 --update-key id --input-fields-terminated-by '\t' --export-dir /hbase/t1 Note: /tmp/sqoop-zhouhh/compile/8ce6556eb13b3000550a9c864eaa6820/t1.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. [zhouhh@Hadoop48 ~]$
但将导出目录指到/hbase/t1表中,导出不会报错,而mysql中没有数据。后面才了解到,sqoop没有直接从hbase中将表导出到mysql的办法。必须先将hbase导出成平面文件,或者导出到hive中,才可以用sqoop将数据导出到mysql。
二、从mysql中导到hdfs。
创建mysql表,将其导入到hdfs
mysql> create table test(id int not null primary key auto_increment,name varchar(64) not null,price decimal(10,2), cdate date,version int,comment varchar(255)); Query OK, 0 rows affected (0.10 sec) mysql> insert into test values(null,'iphone',3900.00,'2012-7-18',1,'8g'); Query OK, 1 row affected (0.04 sec) mysql> insert into test values(null,'ipad',3200.00,'2012-7-16',2,'16g'); Query OK, 1 row affected (0.00 sec) mysql> select * from test; +----+--------+---------+------------+---------+---------+ | id | name | price | cdate | version | comment | +----+--------+---------+------------+---------+---------+ | 1 | iphone | 3900.00 | 2012-07-18 | 1 | 8g | | 2 | ipad | 3200.00 | 2012-07-16 | 2 | 16g | +----+--------+---------+------------+---------+---------+ 2 rows in set (0.00 sec)
导入:
[zhouhh@Hadoop48 ~]$ sqoop import --connect jdbc:mysql://Hadoop48/toplists --table test -m 1 java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Access denied for user ''@'Hadoop48' to database 'toplists' at org.apache.sqoop.manager.CatalogQueryManager.getColumnNames(CatalogQueryManager.java:162)
给空用户授权
mysql> GRANT ALL PRIVILEGES ON *.* TO ''@'%';
[zhouhh@Hadoop48 ~]$ sqoop import --connect jdbc:mysql://Hadoop48/toplists --username root --table test -m 1 12/07/18 11:10:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 12/07/18 11:10:16 INFO tool.CodeGenTool: Beginning code generation 12/07/18 11:10:16 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `index_mapping` AS t LIMIT 1 12/07/18 11:10:16 INFO orm.CompilationManager: HADOOP_HOME is /home/zhoulei/hadoop-1.0.0/libexec/.. 注: /tmp/sqoop-zhoulei/compile/2b04bdabb7043e4f75b215d72f65388e/index_mapping.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 12/07/18 11:10:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-zhoulei/compile/2b04bdabb7043e4f75b215d72f65388e/index_mapping.jar 12/07/18 11:10:18 WARN manager.MySQLManager: It looks like you are importing from mysql. 12/07/18 11:10:18 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 12/07/18 11:10:18 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 12/07/18 11:10:18 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 12/07/18 11:10:25 INFO mapreduce.ImportJobBase: Beginning import of index_mapping 12/07/18 11:10:26 INFO mapred.JobClient: Running job: job_201207101344_0519 12/07/18 11:10:27 INFO mapred.JobClient: map 0% reduce 0% 12/07/18 11:10:40 INFO mapred.JobClient: map 100% reduce 0% 12/07/18 11:10:45 INFO mapred.JobClient: Job complete: job_201207101344_0519 12/07/18 11:10:45 INFO mapred.JobClient: Counters: 18 12/07/18 11:10:45 INFO mapred.JobClient: Job Counters 12/07/18 11:10:45 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=12083 12/07/18 11:10:45 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 12/07/18 11:10:45 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 12/07/18 11:10:45 INFO mapred.JobClient: Launched map tasks=1 12/07/18 11:10:45 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 12/07/18 11:10:45 INFO mapred.JobClient: File Output Format Counters 12/07/18 11:10:45 INFO mapred.JobClient: Bytes Written=28 12/07/18 11:10:45 INFO mapred.JobClient: FileSystemCounters 12/07/18 11:10:45 INFO mapred.JobClient: HDFS_BYTES_READ=87 12/07/18 11:10:45 INFO mapred.JobClient: FILE_BYTES_WRITTEN=30396 12/07/18 11:10:45 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=28 12/07/18 11:10:45 INFO mapred.JobClient: File Input Format Counters 12/07/18 11:10:45 INFO mapred.JobClient: Bytes Read=0 12/07/18 11:10:45 INFO mapred.JobClient: Map-Reduce Framework 12/07/18 11:10:45 INFO mapred.JobClient: Map input records=2 12/07/18 11:10:45 INFO mapred.JobClient: Physical memory (bytes) snapshot=79167488 12/07/18 11:10:45 INFO mapred.JobClient: Spilled Records=0 12/07/18 11:10:45 INFO mapred.JobClient: CPU time spent (ms)=340 12/07/18 11:10:45 INFO mapred.JobClient: Total committed heap usage (bytes)=56623104 12/07/18 11:10:45 INFO mapred.JobClient: Virtual memory (bytes) snapshot=955785216 12/07/18 11:10:45 INFO mapred.JobClient: Map output records=2 12/07/18 11:10:45 INFO mapred.JobClient: SPLIT_RAW_BYTES=87 12/07/18 11:10:45 INFO mapreduce.ImportJobBase: Transferred 28 bytes in 20.2612 seconds (1.382 bytes/sec) 12/07/18 11:10:45 INFO mapreduce.ImportJobBase: Retrieved 2 records.
检查数据是否导入
[zhouhh@Hadoop48 ~]$ fs -cat /user/zhouhh/test/part-m-00000 1,iphone,3900.00,2012-07-18,1,8g 2,ipad,3200.00,2012-07-16,2,16g
或
[zhouhh@Hadoop48 ~]$ fs -cat test/part-m-00000 1,iphone,3900.00,2012-07-18,1,8g 2,ipad,3200.00,2012-07-16,2,16g
三、从hdfs导出到mysql
清空表
mysql> delete from test; Query OK, 2 rows affected (0.00 sec) mysql> select * from test; Empty set (0.00 sec)
导出
[zhouhh@Hadoop48 ~]$ sqoop-export --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table test --export-dir test Note: /tmp/sqoop-zhouhh/compile/7adaaa7ffe5f49ed9d794b1be8a9a983/test.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details.
导出时,–connect,–table, –export-dir是必须设置的。其中toplists是库名,–table是该库下的表名。 –export-dir是要导出的HDFS平面文件位置。如果不是绝对路径,指/user/username/datadir
检查mysql表
mysql> select * from test; +----+--------+---------+------------+---------+---------+ | id | name | price | cdate | version | comment | +----+--------+---------+------------+---------+---------+ | 1 | iphone | 3900.00 | 2012-07-18 | 1 | 8g | | 2 | ipad | 3200.00 | 2012-07-16 | 2 | 16g | +----+--------+---------+------------+---------+---------+ 2 rows in set (0.00 sec)
可见导出成功。
四、不执行mapreduce,但生成导入代码
[zhouhh@Hadoop48 ~]$ sqoop codegen --connect jdbc:mysql://192.168.10.48:3306/toplists --username root --table test --class-name Mycodegen Note: /tmp/sqoop-zhouhh/compile/104b871487669b89dcd5b9b2c61f905f/Mycodegen.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. [zhouhh@Hadoop48 ~]$ sqoop help codegen usage: sqoop codegen [GENERIC-ARGS] [TOOL-ARGS]
sqoop导入时,可以加选择语句,以过滤和综合多表,用–query.也可以只加条件,用–where。这样可以不必每次导入整张表。 如 –where ‘id
> 1000′
示例,采用join选择多表数据:
sqoop import –query ‘select a.*,b.* from a
join b on (a.id == b.id) where $conditions’ -m 1 –target-dir
/usr/foo/joinresults
五、将mysql表导入到HBase
虽然目前,sqoop没有将HBase直接导入mysql的办法,但将mysql直接导入HBase是可以的。需指定–hbase-table,用–hbase-create-table来自动在HBase中创建表。–column-family指定列族名。–hbase-row-key指定rowkey对应的mysql的键。
[zhouhh@Hadoop48
~]$ sqoop import –connect jdbc:mysql://Hadoop48/toplists –table test
–hbase-table a –column-family name –hbase-row-key id –hbase-create-table
–username ‘root’
检查hbase被导入的表:
hbase(main):002:0> scan 'a' ROW COLUMN+CELL 1 column=name:cdate, timestamp=1342601695952, value=2012-07-18 1 column=name:comment, timestamp=1342601695952, value=8g 1 column=name:name, timestamp=1342601695952, value=iphone 1 column=name:price, timestamp=1342601695952, value=3900.00 1 column=name:version, timestamp=1342601695952, value=1 2 column=name:cdate, timestamp=1342601695952, value=2012-07-16 2 column=name:comment, timestamp=1342601695952, value=16g 2 column=name:name, timestamp=1342601695952, value=ipad 2 column=name:price, timestamp=1342601695952, value=3200.00 2 column=name:version, timestamp=1342601695952, value=2 2 row(s) in 0.2370 seconds
关于导入的一致性:建议停止mysql表的写入再导入到HDFS或HIVE,否则,mapreduce可能会丢失新增的数据。
关于效率:mysql直接模式(–direct)导入的方式效率高。但不支持大对象数据,类型为CLOB或BLOB的列。用JDBC效率较低,但有专用API可以支持CLOB及BLOB。
六、从HBase导出数据到Mysql
目前没有直接的导出命令。但有两个方法可以将HBase数据导出到mysql。
其一,将HBase导出成HDFS平面文件,再导出到mysql.
其二,将HBase数据导出到HIVE,再导出到mysql,参见后续文章《从hive将数据导出到mysql
》
发表评论
-
用Sqoop把数据从HDFS导入到MYSQL
2012-10-27 23:30 2313转自:http://www.kaifajie.cn/mySQL ... -
大数据的储存:百度HDFS集群的数据压缩
2012-08-30 17:48 11502012年4月13日,由IT168(ITP ... -
从hive将数据导出到mysql
2012-07-31 16:31 1781转自:http://abloz.com/2012/07/20/ ... -
Hive SQL
2012-07-19 13:49 1397转自:http://www.tbdata.org/ ... -
Hive优化总结
2012-07-09 15:18 4261优化时,把hive sql当做ma ... -
hadoop/mapred 优化方法
2012-06-12 11:47 1368从三个方面着手优化 : 1. hadoop配置 2. ... -
hadoop集群dfs.data.dir目录权限问题导致节点无法启动
2012-06-11 18:32 6185安装使用Hive的时候发现文件无法从本地拷贝到hadoo ... -
hadoop中的balancer
2012-06-11 15:49 1100对于HDFS集群,经常长时间的运行,尤其是大量的delete操 ... -
Hadoop集群上使用Lzo压缩
2012-05-28 11:03 900转自:http://www.tech126.com/hadoo ... -
使用Hive读取Hbase中的数据
2012-05-23 13:12 2238转自:http://www.4ucode.com/Study/ ... -
在XP系统下配置eclipse集成hadoop开发
2012-05-05 11:09 1009一、目的:由于hadoop只能在Linux环境下运行,所以对于 ... -
使用Hive读取Hbase中的数据
2012-04-18 14:24 1053转自:http://www.4ucode.com/Study/ ... -
Web日志分析方法概述 让复杂的数据挖掘变得简单可行
2012-04-13 16:31 834志在计算机系统中是一个非常广泛的概念,任何程序都有可能输出日志 ... -
应用sqoop将MySQL数据库中的数据导入Hbase
2012-04-12 11:31 2033转自:http://www.bwxxkj.com/a/jish ... -
hadoop中的trash机制,恢复删除的文件
2012-03-31 13:46 2153Hadoop回收站trash,默认是关闭的。 1.修改con ... -
Hadoop 和DBMS 的互补性
2012-03-29 12:22 642随着Microsoft 也加入Hadoop 阵营,Hadoop ... -
Hadoop 中的两表join
2012-03-29 12:20 822转自:http://www.gemini52013 ... -
各版本编译hadoop eclipse plugin
2012-03-29 12:19 1349转自:http://www.gemini52013 ... -
hadoop常见错误
2012-03-24 13:04 1619错误1:bin/hadoop dfs 不能正常启动,持续提 ... -
hbase运维
2012-03-23 17:33 1281转自:http://blog.bluedavy.c ...
相关推荐
sqoop导入数据到hdfs,所有相关的操作命令均在更改相关地址就行。
使用sqoop从传统数据库mysql中抽取数据到大数据集群。
2、sqoop导入(RMDB-mysql、sybase到HDFS-hive) 网址:https://blog.csdn.net/chenwewi520feng/article/details/130572275 介绍sqoop从关系型数据库mysql、sybase同步到hdfs、hive中
sqoop导入数据到hive
05_Sqoop将MySQL数据导入到HDFS(一) 06_Sqoop将MySQL数据导入到HDFS(二) 07_Sqoop中的增量导入与Sqoop job 08_Sqoop将MySQL数据导入Hive表中 09_Sqoop的导出及脚本中使用的方式 10_案例分析-动态分区的实现 11_...
使用sqoop2从mysql导入hadoop时,导入的数据带单引号。是由于sqoop2的一个bug导致,该包上传到sqoop_home/server/lib替换原来的jar包后,重启sqoop server,即可解决。
其中包含Sqoop将SqlServer文件导入HDFS文件的所有jar包
3、Sqoop导出(HDFS到RMDB-mysql) 网址:https://blog.csdn.net/chenwewi520feng/article/details/130573318 介绍通过sqoop将hdfs数据同步到关系型数据库mysql中
使用Sqoop在Oracle数据库和HDFS之间进行导入和导出 内容 使用 操作系统-Ubuntu 20.04.1 LTS(VMware) 后端-Java (JDK 1.8), Sqoop(v1.4.7) , Hadoop(v3.3.0) 库-OJDBC(v7),commons-lang-2.6.jar ...
使用sqoop在mysql与hdfs之间进行数据迁移的java简单案例
除了代码还有总结内容,代码主要实现了从sqoop1.99.7从oracle导出数据到HDFS某目录;总结内容是一些踩坑的总结,还有一些坑的说明在代码注释里;灵活利用,可以帮助你跑通oracle到hdfs的过程。
Hadoop2.6伪分布安装sqoop1.4.6及hdfs与mysql导入导出数据 sqoop安装及简单测试 启动sqoop 启动mysql 从mysql导入数据到hdfs中 从HDFS导出数据到mysql
sqoop把mysql数据导入hbase2.1.6(提示错误及解决方案)
sqoop2 java API从oracle导数据到HDFS开发总结 有了这个文档,会为你节省很多时间,一些我踩坑趟路的时间;但是不代表你什么都不知道的前提就能把demo运行起来 。 建议操作顺序:sqoop官方环境搭建文档看一遍,然后...
hadoop环境搭建,涉及zookeeper、hbase、hive、mysql、kafka等服务的搭建测试运用。
■ 如何使用Flume 将数据导入HDFS ■ 如何使用Sqoop 将数据导入HDFS ■ Hadoop 提供的REST 接口 ■ 导入数据的最佳实践
hue平台oozie工作流操作sqoop,把mysql数据传输到hdfs上
1.集群配置,包含三个结点,主节点控制所有的子节点 2.集群已经搭建好了,解压之后...3.包含全套组件:hdfs,hbase,hive,sqoop,mysql,pig,spark等大数据组件 4.如果不知道如何使用,或者需要做Hadoop项目可以私信博主
sqoop的数据导入