- 浏览: 53726 次
- 性别:
- 来自: 广州
文章分类
最新评论
[hadoop@hadoopmaster ~]$ pig pig3.pig
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-08-30 01:34:26,086 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:27:58
2015-08-30 01:34:26,086 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1440923666085.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop260/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop260/hbase-0.98.13-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-08-30 01:34:26,741 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found
2015-08-30 01:34:26,893 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-08-30 01:34:26,893 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-08-30 01:34:26,893 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hadoopmaster:9000
2015-08-30 01:34:27,827 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
<file pig3.pig, line 1, column 53> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1712)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1420)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:364)
at org.apache.pig.PigServer.executeBatch(PigServer.java:389)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:608)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by:
<file pig3.pig, line 1, column 53> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1323)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1308)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5158)
at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3515)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 19 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:682)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1320)
... 27 more
2015-08-30 01:34:27,831 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /home/hadoop/pig_1440923666085.log
https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore
Running Pig with HCatalog
Pig does not automatically pick up HCatalog jars. To bring in the necessary jars, you can either use a flag in the pig command or set the environment variables PIG_CLASSPATH and PIG_OPTS as described below.
The -useHCatalog Flag
To bring in the appropriate jars for working with HCatalog, simply include the following flag:
pig -useHCatalog
Jars and Configuration Files
For Pig commands that omit -useHCatalog, you need to tell Pig where to find your HCatalog jars and the Hive jars used by the HCatalog client. To do this, you must define the environment variable PIG_CLASSPATH with the appropriate jars.
HCatalog can tell you the jars it needs. In order to do this it needs to know where Hadoop and Hive are installed. Also, you need to tell Pig the URI for your metastore, in the PIG_OPTS variable.
In the case where you have installed Hadoop and Hive via tar, you can do this:
export HADOOP_HOME=<path_to_hadoop_install>
export HIVE_HOME=<path_to_hive_install>
export HCAT_HOME=<path_to_hcat_install>
export PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-core*.jar:\
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:\
$HIVE_HOME/lib/slf4j-api-*.jar
export PIG_OPTS=-Dhive.metastore.uris=thrift://<hostname>:<port>
Or you can pass the jars in your command line:
<path_to_pig_install>/bin/pig -Dpig.additional.jars=\
$HCAT_HOME/share/hcatalog/hcatalog-core*.jar:\
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/lib/slf4j-api-*.jar <script.pig>
The version number found in each filepath will be substituted for *. For example, HCatalog release 0.5.0 uses these jars and conf files:
$HCAT_HOME/share/hcatalog/hcatalog-core-0.5.0.jar
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter-0.5.0.jar
$HIVE_HOME/lib/hive-metastore-0.10.0.jar
$HIVE_HOME/lib/libthrift-0.7.0.jar
$HIVE_HOME/lib/hive-exec-0.10.0.jar
$HIVE_HOME/lib/libfb303-0.7.0.jar
$HIVE_HOME/lib/jdo2-api-2.3-ec.jar
$HIVE_HOME/conf
$HADOOP_HOME/conf
$HIVE_HOME/lib/slf4j-api-1.6.1.jar
Authentication
If you are using a secure cluster and a failure results in a message like "2010-11-03 16:17:28,225 WARN hive.metastore ... - Unable to connect metastore with URI thrift://..." in /tmp/<username>/hive.log, then make sure you have run "kinit <username>@FOO.COM" to get a Kerberos ticket and to be able to authenticate to the HCatalog server.
Load Examples
This load statement will load all partitions of the specified table.
/* myscript.pig */
A = LOAD 'tablename' USING org.apache.hcatalog.pig.HCatLoader();
...
...
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
15/08/30 01:34:26 INFO pig.ExecTypeProvider: Picked MAPREDUCE as the ExecType
2015-08-30 01:34:26,086 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:27:58
2015-08-30 01:34:26,086 [main] INFO org.apache.pig.Main - Logging error messages to: /home/hadoop/pig_1440923666085.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop260/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop260/hbase-0.98.13-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-08-30 01:34:26,741 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/hadoop/.pigbootup not found
2015-08-30 01:34:26,893 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-08-30 01:34:26,893 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2015-08-30 01:34:26,893 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://hadoopmaster:9000
2015-08-30 01:34:27,827 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
<file pig3.pig, line 1, column 53> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1712)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1420)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:364)
at org.apache.pig.PigServer.executeBatch(PigServer.java:389)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:608)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by:
<file pig3.pig, line 1, column 53> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1323)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1308)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5158)
at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3515)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 19 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:682)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1320)
... 27 more
2015-08-30 01:34:27,831 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.apache.hcatalog.pig.HCatLoader using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /home/hadoop/pig_1440923666085.log
https://cwiki.apache.org/confluence/display/Hive/HCatalog+LoadStore
Running Pig with HCatalog
Pig does not automatically pick up HCatalog jars. To bring in the necessary jars, you can either use a flag in the pig command or set the environment variables PIG_CLASSPATH and PIG_OPTS as described below.
The -useHCatalog Flag
To bring in the appropriate jars for working with HCatalog, simply include the following flag:
pig -useHCatalog
Jars and Configuration Files
For Pig commands that omit -useHCatalog, you need to tell Pig where to find your HCatalog jars and the Hive jars used by the HCatalog client. To do this, you must define the environment variable PIG_CLASSPATH with the appropriate jars.
HCatalog can tell you the jars it needs. In order to do this it needs to know where Hadoop and Hive are installed. Also, you need to tell Pig the URI for your metastore, in the PIG_OPTS variable.
In the case where you have installed Hadoop and Hive via tar, you can do this:
export HADOOP_HOME=<path_to_hadoop_install>
export HIVE_HOME=<path_to_hive_install>
export HCAT_HOME=<path_to_hcat_install>
export PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-core*.jar:\
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:\
$HIVE_HOME/lib/slf4j-api-*.jar
export PIG_OPTS=-Dhive.metastore.uris=thrift://<hostname>:<port>
Or you can pass the jars in your command line:
<path_to_pig_install>/bin/pig -Dpig.additional.jars=\
$HCAT_HOME/share/hcatalog/hcatalog-core*.jar:\
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter*.jar:\
$HIVE_HOME/lib/hive-metastore-*.jar:$HIVE_HOME/lib/libthrift-*.jar:\
$HIVE_HOME/lib/hive-exec-*.jar:$HIVE_HOME/lib/libfb303-*.jar:\
$HIVE_HOME/lib/jdo2-api-*-ec.jar:$HIVE_HOME/lib/slf4j-api-*.jar <script.pig>
The version number found in each filepath will be substituted for *. For example, HCatalog release 0.5.0 uses these jars and conf files:
$HCAT_HOME/share/hcatalog/hcatalog-core-0.5.0.jar
$HCAT_HOME/share/hcatalog/hcatalog-pig-adapter-0.5.0.jar
$HIVE_HOME/lib/hive-metastore-0.10.0.jar
$HIVE_HOME/lib/libthrift-0.7.0.jar
$HIVE_HOME/lib/hive-exec-0.10.0.jar
$HIVE_HOME/lib/libfb303-0.7.0.jar
$HIVE_HOME/lib/jdo2-api-2.3-ec.jar
$HIVE_HOME/conf
$HADOOP_HOME/conf
$HIVE_HOME/lib/slf4j-api-1.6.1.jar
Authentication
If you are using a secure cluster and a failure results in a message like "2010-11-03 16:17:28,225 WARN hive.metastore ... - Unable to connect metastore with URI thrift://..." in /tmp/<username>/hive.log, then make sure you have run "kinit <username>@FOO.COM" to get a Kerberos ticket and to be able to authenticate to the HCatalog server.
Load Examples
This load statement will load all partitions of the specified table.
/* myscript.pig */
A = LOAD 'tablename' USING org.apache.hcatalog.pig.HCatLoader();
...
...
发表评论
-
spark snappy save text file
2016-08-28 21:54 1846SQL context available as sqlCon ... -
kerberos 5
2015-12-14 21:42 1331kadmin.local: listprincs K/M@J ... -
hadoop network
2015-11-21 23:09 640[root@hadoopmaster ~]# cat /etc ... -
distcp
2015-11-18 21:47 1897[hadoop@hadoopmaster test]$ had ... -
hadoop常用命令
2015-09-19 14:58 582namenode(hdfs)+jobtracker ... -
hive之jdbc
2015-09-06 16:08 613import java.sql.Connection; imp ... -
hadoop fs, hdfs dfs, hadoop dfs科普
2015-09-06 11:14 948stackoverflow Following are th ... -
hadoop fsck
2015-08-25 22:59 471hadoop fsck Usage: DFSck < ... -
hcatalog study
2015-08-25 22:41 561https://cwiki.apache.org/conflu ... -
pig
2015-08-22 15:34 794hdfs://hadoopmaster:9000/user ... -
pig
2015-08-19 23:03 494http://blog.csdn.net/zythy/ar ... -
hadoop jar command
2015-08-16 22:57 502hadoop jar /opt/jack.jar org.a ... -
mapreduce
2015-08-16 21:56 263ChainMapper 支持多个map reduce 参 ... -
sqoop command
2015-06-06 18:54 5771. list the database sqoop ... -
maven
2015-01-15 21:39 424xxxx -
hadoop 2.5.2安装实录
2014-12-09 23:35 8001. prepare the virtual enviro ...
相关推荐
Pig Hive 对比分享, Pig HCatalog 元数据组合使用
分享一下Hadoop、HBase、Hive、Pig、Zookeeper相关资料。
PIG(国内微服务热度最高的一个社区)
Learn to use Apache Pig to develop lightweight big data applications easily and quickly. This book shows you many optimization techniques and covers every context where Pig is used in big data ...
《Pig编程指南》不仅为初学者讲解ApachePig的基础知识,同时也向有一定使用经验的高级用户介绍更加综合全面的Pig重要特性,如PigLatin脚本语言、控制台shell交互命令以及用于对Pig进行拓展的用户自定义函数(UDF)等。...
Hadoop系统的pig工具包,很好用的
pig0.15源码,适合小白学习大数据参考和使用
apache pig 基础及应用,urldecode row_number web日志分析 根据 用户行为 做出 简易的 相似度 判断。
Beginning Apache Pig: Big Data Processing Made Easy English | 29 Dec. 2016 | ISBN: 1484223365 | 300 Pages | PDF | 4.9 MB Learn to use Apache Pig to develop lightweight big data applications easily ...
Pig-Toolbox_v1.0.7.6 谷歌浏览器插件,CRX可解压,用开发者模式安装
PIGO是一个采集引擎,开始的时候是按照采集各大搜索引擎而设计的。随着需求的增加,现在PIGO不单单能采集搜索引擎,而且能聚合结果,甚至,加载不同的核心,PIGO就变成不同的网站。所以PIGO不是任何类型的网站,但又...
来试试Pig安装,与学学Pig Latin语言,玩玩应用案例
大数据pig实战,大数据pig实战,大数据pig实战大数据pig实战大数据pig实战
Hadoop_HBase_Pig
pig将就的命令
pig源码包,最好的学习资料还是源码包。里边包括核心包,以及其他贡献的包,pigunit,还有源码
Pig编程指南.pdf Pig编程指南英文版pdf文档
pig官方的udf教程,介绍了 Writing Java UDFs Writing Python UDFs Writing JavaScript UDFs Writing Ruby UDFs Piggy Bank,一个开源pig的udf包,主要是java