When I submit a java job (include some map/reduce jobs) in the hue UI using oozie Editor, the third party jars are not loaded correctly.
1. the only success way i used is to build a fat jar which contains all dependency classes in thrid party jars.
2.If I export my application jar from eclipse with option including the thrid party jars in the jar's lib dir, hadoop still
can't find these jars, although some posts show that hadoop will auto extract these jars when run job.
And in oozie document, it says that if set related configuration using uber jar. The raw content is below, I followed
it, but fails too.
------------
For Map-Reduce jobs (not including streaming or pipes), additional jar files can also be included via an uber jar. An uber jar is a jar file that contains additional jar files within a "lib" folder (see Workflow Functional Specification for more information). Submitting a workflow with an uber jar requires at least Hadoop 2.2.0 or 1.2.0. As such, using uber jars in a workflow is disabled by default. To enable this feature, use the oozie.action.mapreduce.uber.jar.enable property in the oozie-site.xml (and make sure to use a supported version of Hadoop).
<configuration> <property> <name>oozie.action.mapreduce.uber.jar.enable</name> <value>true</value> </property> </configuration>
----------
3.I put the third party jars in oozie deploy dir, some works but anothers doesn't work.
4.when i design a job, add the third party jars in file option, see below. But the result likes 3
--------------
There are multiple ways to add the jars to the lib path. I would suggest you try one of the following 1) in your job.properties, add the path to your jar files (comma separated) to *oozie.libpath *property 2) create a *lib* directory under your application's directory and put your jars under lib so oozie can pick them up.
-------------------------------------------------
The same problem is http://qnalist.com/questions/4588838/how-to-manage-dependencies-for-m-r-job-which-is-executed-using-oozie-java-action
The solution is :http://pangool.net/executing_examples.html
a. in java driver class which sumbit a map/reduce job, You should use
Job job = HadoopUtil.prepareJob(inputPath, outputPath, inputFormat, mapper, mapperKey, mapperValue, reducer, reducerKey, reducerValue, outputFormat, getConf());
instead of using
// Path input = new Path(userContactNumberDir); // String redisServer = getOption("redisServer"); // String zsetkey = getOption("zsetkey") + "_user"; // // Configuration conf = new Configuration(); // Job job = new Job(conf, "push topk users to reids zset"); // job.setJarByClass(TopKItemHotToRedisJob.class); // job.setOutputKeyClass(NullWritable.class); // job.setOutputValueClass(ItemPropertyWritable.class); // // job.setMapperClass(TopKItemHotToRedisJob.Map.class); // // job.setReducerClass(TopKItemHotToRedisJob.Reduce.class); // job.setNumReduceTasks(1); // job.setInputFormatClass(TextInputFormat.class); // job.setOutputFormatClass(RedisZSetOutputFormat.class); // // RedisZSetOutputFormat.setRedisHosts(job,redisServer); // RedisZSetOutputFormat.setRedisKey(job, zsetkey); // // FileInputFormat.addInputPath(job,input); // // job.getConfiguration().setInt(TopKItemHotToRedisJob.NUMK, // Integer.parseInt(getOption("numKUser"))); // // job.waitForCompletion(true);The complete alternative code likes
Path input = new Path(userContactNumberDir); Path output = new Path(notUsedDir); HadoopUtil.delete(getConf(), output); Job getTopKUsersJob = prepareJob(input, output,TextInputFormat.class, TopKItemHotToRedisJob.Map.class, NullWritable.class, ItemPropertyWritable.class, TopKItemHotToRedisJob.Reduce.class, IntWritable.class, DoubleWritable.class,RedisZSetOutputFormat.class); getTopKUsersJob.setJobName("Get topk user and write to redis"); getTopKUsersJob.setNumReduceTasks(1); RedisZSetOutputFormat.setRedisHosts(getTopKUsersJob,getOption("redisServer")); RedisZSetOutputFormat.setRedisKey(getTopKUsersJob, getOption("zsetkey") + "_user"); getTopKUsersJob.getConfiguration().setInt(TopKItemHotToRedisJob.NUMK, Integer.parseInt(getOption("numKUser"))); getTopKUsersJob.waitForCompletion(true);
b. when design java action, you should add the following arg
-libjars hdfs://192.168.0.131:2014/user/inok/inok3/conf/thirdlib.jar
and add the thirdlib.jar as file
------
But I thought there otherway is to avoid step b is upload thirdlib.jar to path/to/wf-deployment-dir/lib/ .
and set
oozie.wf.application.path
but the actual oozie.wf.application.path is
This means that in hue, the oozie.wf.application.path cann't be owerwrited
References
http://hadoopi.wordpress.com/2014/06/05/hadoop-add-third-party-libraries-to-mapreduce-job/
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
相关推荐
官方版本,亲测可用
官方版本,亲测可用
javaEE6jar包
a set of apache jars
本篇文章对Eclipse add external jars导致运行出现java.lang.NoClassDefFoundError的解决方法进行了详细的分析介绍。需要的朋友参考下
What jars are required by JCS? As of verison 1.2.7.0, the core of JCS (the LRU memory cache, the indexed disk cache, the TCP lateral, and the RMI remote server) requires only two other jars. ...
In order to run the jars, you have to add the jars in the dependency folder to your classpath and define the main class. These jars are mainly intended for integration in your projects if you are not ...
hibernate-jars hibernate-jars hibernate-jars
classes12.jar、db2java.jar、mysql.jar、Opta.jar、SQL Server 2000 JDBC.jar
本文发表在国际知名遥感杂志JARS上,内容涉及合成孔径雷达(SAR)和地面运动目标检测(GMTI),对偏置天线相位中心技术(DPCA)和沿航迹干涉技术(ATI)进行了理论建模和系统全面地性能对比。具有相当的参考价值,欢迎...
jars_20221025.tar jars_20221025.tar
if you encounter some bugs on the third-party jars, the ClassPlugin will be the best way to solve it. Installation To use ClassPlugin in a module, add the plugin to your buildscript: buildscript { ...
内含很多java开发经常用到的jar包,比如,json数据的转换,把数据保留到excel中去,zxing扫描包。
RunCommunity Jars
jars 备份
j2ee-1.4.jar j2ee-1.4.jar j2ee-1.4.jar j2ee-1.4.jar
创建javaee6项目所需的jar包,用于在myeclipse10一下版本构建javaee6项目
官方版本,亲测可用
官方版本,亲测可用
官方版本,亲测可用