问题:
[zkkafka@yanfabu2-37 ~]$ sqoop import \ > --connect jdbc:mysql://10.156.50.36:3306/mqh \ > --username root \ > --password root \ > --table device \ > --fields-terminated-by '\t' \ > --delete-target-dir \ > --num-mappers 1 \ > --hive-import \ > --hive-database test \ > --hive-table hive_bbs_product_snappy; Warning: /home/zkkafka/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/zkkafka/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 19/05/29 12:00:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 19/05/29 12:00:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 19/05/29 12:00:27 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 19/05/29 12:00:27 INFO tool.CodeGenTool: Beginning code generation Wed May 29 12:00:27 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:00:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:00:27 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:00:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/zkkafka/hadoop 注: /tmp/sqoop-zkkafka/compile/0edae8bed32dad40d1fe8ca424e30387/device.java使用或覆盖了已过时的 API。 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 19/05/29 12:00:31 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-zkkafka/compile/0edae8bed32dad40d1fe8ca424e30387/device.jar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/zkkafka/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/zkkafka/hbase/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/05/29 12:00:32 INFO tool.ImportTool: Destination directory device deleted. 19/05/29 12:00:32 WARN manager.MySQLManager: It looks like you are importing from mysql. 19/05/29 12:00:32 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 19/05/29 12:00:32 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 19/05/29 12:00:32 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 19/05/29 12:00:32 INFO mapreduce.ImportJobBase: Beginning import of device 19/05/29 12:00:32 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 19/05/29 12:00:32 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps Wed May 29 12:00:39 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:00:39 INFO db.DBInputFormat: Using read commited transaction isolation 19/05/29 12:00:40 INFO mapreduce.JobSubmitter: number of splits:1 19/05/29 12:00:40 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1558676658010_0008 19/05/29 12:00:41 INFO impl.YarnClientImpl: Submitted application application_1558676658010_0008 19/05/29 12:00:41 INFO mapreduce.Job: The url to track the job: http://master1:8088/proxy/application_1558676658010_0008/ 19/05/29 12:00:41 INFO mapreduce.Job: Running job: job_1558676658010_0008 19/05/29 12:00:49 INFO mapreduce.Job: Job job_1558676658010_0008 running in uber mode : false 19/05/29 12:00:49 INFO mapreduce.Job: map 0% reduce 0% 19/05/29 12:00:57 INFO mapreduce.Job: map 100% reduce 0% 19/05/29 12:00:59 INFO mapreduce.Job: Job job_1558676658010_0008 completed successfully 19/05/29 12:01:00 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=128603 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=87 HDFS: Number of bytes written=156 HDFS: Number of read operations=4 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=5465 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=5465 Total vcore-milliseconds taken by all map tasks=5465 Total megabyte-milliseconds taken by all map tasks=5596160 Map-Reduce Framework Map input records=6 Map output records=6 Input split bytes=87 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=103 CPU time spent (ms)=1380 Physical memory (bytes) snapshot=168935424 Virtual memory (bytes) snapshot=2104311808 Total committed heap usage (bytes)=93323264 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=156 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Transferred 156 bytes in 27.7324 seconds (5.6252 bytes/sec) 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Retrieved 6 records. 19/05/29 12:01:00 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table device Wed May 29 12:01:00 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 19/05/29 12:01:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `device` AS t LIMIT 1 19/05/29 12:01:00 WARN hive.TableDefWriter: Column register_time had to be cast to a less precise type in Hive 19/05/29 12:01:00 INFO hive.HiveImport: Loading uploaded data into Hive 19/05/29 12:01:00 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly. 19/05/29 12:01:00 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50) at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392) at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44) ... 12 more
解决方法:
cp /home/zkkafka/hive/lib/hive-exec-**.jar /home/zkkafka/sqoop/lib/
捐助开发者
在兴趣的驱动下,写一个免费
的东西,有欣喜,也还有汗水,希望你喜欢我的作品,同时也能支持一下。 当然,有钱捧个钱场(支持支付宝和微信 以及扣扣群),没钱捧个人场,谢谢各位。
个人主页:http://knight-black-bob.iteye.com/
谢谢您的赞助,我会做的更好!
相关推荐
运行Sqoop报错:Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/LogicalType,下载此资源放到Sqoop的lib目录下即可
出现此问题时使用:java.lang.NullPointerException at org.json.JSONObject.(JSONObject.java:144) at org.apache.sqoop.util.SqoopJsonUtil.... at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool...
编译Atlas用 sqoop-1.4.6.2.3.99.0-195.jar 内含安装jar包以及maven手动安装命令 详情可参考我的博客: https://blog.csdn.net/qq_26502245/article/details/108008070
sqoop-1.4.6.2.3.99.0-195.jar org.restlet-2.4.3.jar org.restlet.ext.servlet-2.4.3.jar
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
基于atlas2.0.0编译,需要而很难找到有效地址下载不到的jar包 sqoop-1.4.6.2.3.99.0-195.jar
apache-atlas-2.2.0-sqoop-hook.tar.gz
1.集群配置,包含三个结点,主节点控制所有的子节点 2.集群已经搭建好了,解压之后...3.包含全套组件:hdfs,hbase,hive,sqoop,mysql,pig,spark等大数据组件 4.如果不知道如何使用,或者需要做Hadoop项目可以私信博主
Apache Sqoop(SQL-to-Hadoop) 项目旨在协助 RDBMS 与 Hadoop 之间进行高效的大数据交流。 用户可以在 Sqoop 的帮助下,轻松地把关系型数据库的数据导入到 Hadoop 与其相关的系统 (如HBase 和Hive)中;同时也可以把...
sqoop框架开发工具使用的jar sqoop-1.4.6.jar 手动安装到maven <groupId>org.apache.sqoop <artifactId>sqoop <version>1.4.6 </dependency>
Hadoop HBbase HIVE Sqoop概念介绍说明,及和RDBMS的比较
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.zip
Hadoop hbase hive sqoop集群环境安装配置及使用文档
sqoop-1.4.7.bin__hadoop-2.6.0.tar 目前能下载到的最新版本 为不喜欢去官网下载的小伙伴提供便利
java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40 导致的原因: 在azkaban的server和executor中缺少一个叫derby.jar的包
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.zip。。。。。。。。。。。。。。。。
sqoop-1.4.6.jar,sqoop报错找不到加载类的时候 将该文件放入sqoop的lib文件夹下
sqoop-1.4.5.tar.gz
Sqoop is a command-line interface application for transferring data between relational databases and Hadoop.