在配置好Hadoop Eclipse plugin 连接成功后,提交作业时会抛出下面异常:
2013-10-31 9:38:04 org.apache.hadoop.security.UserGroupInformation doAs 严重: PriviledgedActionException as:admin cause:java.io.IOException: Failed to set permissions of path: \home\hadoop\tmp\mapred\staging\admin-454528829\.staging to 0700 Exception in thread "main" java.io.IOException: Failed to set permissions of path: \home\hadoop\tmp\mapred\staging\admin-454528829\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:918) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at com.chenzehe.hadoop.action.WordCount.main(WordCount.java:86)
网上有些人的解决方法是把该异常的代码注释掉,如下:
这个是Windows下文件权限问题,在Linux下可以正常运行,不存在这样的问题。 解决方法是,修改/hadoop-1.0.2/src/core/org/apache/hadoop/fs/FileUtil.java里面的checkReturnValue,注释掉即可(有些粗暴,在Window下,可以不用检查): ...... private static void checkReturnValue(boolean rv, File p, FsPermission permission ) throws IOException { /** if (!rv) { throw new IOException("Failed to set permissions of path: " + p + " to " + String.format("%04o", permission.toShort())); } **/ } ......
该解决方法不尽美,通过跟踪Hadoop提交作业的代码可发现导致问题的原因,下面代码为在main方法中提交作业到hadoop中处理:
public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "my word count"); job.setJarByClass(WordCount.class); job.setMapperClass(MapClass.class); // job.setCombinerClass(ReduceClass.class); job.setReducerClass(ReduceClass.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }
提交作业的代码为job.waitForCompletion(true),跟踪进去代码:
public boolean waitForCompletion(boolean verbose) throws IOException, InterruptedException, ClassNotFoundException { if(state == JobState.DEFINE) submit(); if(verbose) jobClient.monitorAndPrintJob(conf, info); else info.waitForCompletion(); return isSuccessful(); }
submit()提交方法代码为:
public void submit() throws IOException, InterruptedException, ClassNotFoundException { ensureState(JobState.DEFINE); setUseNewAPI(); connect(); info = jobClient.submitJobInternal(conf); super.setJobID(info.getID()); state = JobState.RUNNING; }
里面还是调用了jobClient的方法jobClient.submitJobInternal(conf),在submitJobInternal()方法中使用status = jobSubmitClient.submitJob(jobId, submitJobDir.toString(), jobCopy.getCredentials())方法来提交job,jobSubmitClient有两个实现如下:
通过看JobClient中的init方法发现默认实例化的是LocalJobRunner对象:
public void init(JobConf conf) throws IOException { String tracker = conf.get("mapred.job.tracker", "local"); tasklogtimeout = conf.getInt("mapreduce.client.tasklog.timeout", 60000); ugi = UserGroupInformation.getCurrentUser(); if("local".equals(tracker)) { conf.setNumMapTasks(1); jobSubmitClient = new LocalJobRunner(conf); } else { rpcJobSubmitClient = createRPCProxy(JobTracker.getAddress(conf), conf); jobSubmitClient = createProxy(rpcJobSubmitClient, conf); } }
所以job默认是提交到本地的job tracker,所以运行失败,可以在提交job时设置下conf的mapred.job.tracker属性为集群,如下:
public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); conf.set("mapred.job.tracker", "master-2:49001"); ...... System.exit(job.waitForCompletion(true) ? 0 : 1); }
相关推荐
hadoop eclipse plugin for version 1.0.1
hadoop-eclipse-plugin-3.1.1, hadoop eclipse 插件 3.1.1
hadoop-eclipse-plugin-1.2.1hadoop-eclipse-plugin-1.2.1hadoop-eclipse-plugin-1.2.1hadoop-eclipse-plugin-1.2.1
Hadoop Eclipse是Hadoop开发环境的插件,用户在创建Hadoop程序时,Eclipse插件会自动导入Hadoop编程接口的jar文件,这样用户就可以在Eclipse插件的图形界面中进行编码、调试和运行Hadop程序,也能通过Eclipse插件...
hadoop eclipse plugin,可以集成Eclipse进行开发。
Eclipse集成Hadoop2.10.0的插件,使用`ant`对hadoop的jar包进行打包并...- `hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin/build.xml` 开源源地址: https://github.com/winghc/hadoop2x-eclipse-plugin
用来配置myeclipse或eclipse对应的hadoop 插件,方便开发
hadoop-eclipse-plugin-3.1.3,eclipse版本为eclipse-jee-2020-03
hadoop-eclipse-plugin-2.7.4.jar和hadoop-eclipse-plugin-2.7.3.jar还有hadoop-eclipse-plugin-2.6.0.jar的插件都在这打包了,都可以用。
hadoop-eclipse-plugin-2.7.2.jar完美兼容版 Tested with following eclipse version for hadoop2.7.2(http://pan.baidu.com/s/1i4plIfF): Eclipse Java EE IDE for Web Developers. Version: Mars.1 Release (4.5.1...
hadoop-eclipse-plugin.jar插件基于Ubuntu18.04和Hadoop-3.2.1编译的,最后可以在eclipse创建Map Reduce文件
该文档是hadoop-eclipse-plugin-2.7.6的下载地址及提取码
hadoop1.0.0没有提供eclipse插件的jar包,但是提供了源码,编译后,供大家下载。
hadoop-eclipse-plugin-2.7.3和2.7.7的jar包 hadoop-eclipse-plugin-2.7.3和2.7.7的jar包 hadoop-eclipse-plugin-2.7.3和2.7.7的jar包 hadoop-eclipse-plugin-2.7.3和2.7.7的jar包
最新的hadoop-eclipse-plugin-2.7.4.jar 很好用的hadoop的eclipse插件。自己编译的。 经过测试,使用没有任何问题。 请各位放心使用
网上没找到2.8.1的版本,自己编译,经测试可用。
hadoop官方自带的eclipse插件貌似不能使用 在连接hadoop集群的时候报错 我根据这篇文章对jar包进行了修改 http://hi.baidu.com/wangyucao1989/blog/item/279cef87c4b37c34c75cc315.html 亲测正常使用
eclipse中需要的hadoop插件,对应hadoop版本3.x。 注意:插件的版本要和用的hadoop版本保持一致
找不到与hadoop-2.9.2版本对应的插件,手动生成的hadoop-eclipse-plugin-2.9.2版本,
eclipse连接hadoop搭建mapreduce开发环境。本资源含有hadoop-2.8.5.tar和在eclipse配置mapreduce环境的plugin的jar包,版本为2.8.5