本博客属原创文章,转载请注明出处:http://guoyunsky.iteye.com/blog/1396801
欢迎加入Hadoop超级群: 180941958
最近开始监控Oozie所有workflow job,比如跟踪所有workflow job,有失败的job就邮件等通知.或者重启失败的job等.
这个异常就是发生在rerun的过程中.完整的异常如下:
Unknown exception in doAs java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:194) at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:283) at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:171) at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:110) at javax.servlet.http.HttpServlet.service(HttpServlet.java:640) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:123) at com.cloudera.alfredo.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:371) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:128) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: java.security.PrivilegedActionException: org.apache.oozie.service.HadoopAccessorException: E0903: Configuration does not have Jobtracker Kerberos name or rule at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115) ... 24 more Caused by: org.apache.oozie.service.HadoopAccessorException: E0903: Configuration does not have Jobtracker Kerberos name or rule at org.apache.oozie.service.KerberosHadoopAccessorService.validateConf(KerberosHadoopAccessorService.java:244) at org.apache.oozie.service.KerberosHadoopAccessorService.access$000(KerberosHadoopAccessorService.java:49) at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:202) at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:194) ... 27 more
网上并没有找到解决方案,也不清楚 Jobtracker Kerberos到底是啥玩意.
于是跟踪下源码看,终于搞定.
1.解决方案:
只要在你的workflow/wf_job.properties,也就是你的workflow配置文件加上以下代码就行:
mapreduce.jobtracker.kerberos.principal=mapred/_HOST@LOCALHOST
dfs.namenode.kerberos.principal=hdfs/_HOST@LOCALHOST
2.原理:
跟踪源码发现,有如下判断之后才会触发该异常:
private void validateConf(Configuration conf) throws HadoopAccessorException { if (conf.get(WorkflowAppService.HADOOP_JT_KERBEROS_NAME) == null) { throw new HadoopAccessorException(ErrorCode.E0903); } if (conf.get(WorkflowAppService.HADOOP_NN_KERBEROS_NAME) == null) { throw new HadoopAccessorException(ErrorCode.E0904); } }
从中可以发现只要Configuration中包含WorkflowAppService.HADOOP_JT_KERBEROS_NAME也就是
mapreduce.jobtracker.kerberos.principal和WorkflowAppService.HADOOP_NN_KERBEROS_NAME(dfs.namenode.kerberos.principal)即可.所以可以在配置文件,或者hadoop/conf/中配置.
更多技术文章、感悟、分享、勾搭,请用微信扫描:
相关推荐
Hadoop学习笔记,自己总结的一些Hadoop学习笔记,比较简单。
Hadoop常见异常,以及hadoop配置,等资料
Hadoop 学习笔记.md
Hadoop和Kerberos简介。
第4章 Oozie的使用 4.1 案例一:Oozie调度shell脚本 目标:使用Oozie调度Shell脚本 分步实现: 1)解压官方案例模板 [atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ tar -zxvf oozie-examples.tar.gz 2)创建工作目录 ...
云计算,hadoop,学习笔记, dd
hadoop学习笔记,hadoop简介,适用于hadoop入门,讲解hadoop安装,使用,基本原理,大数据,分布式等概念
hadoop 学习笔记,从搭建环境开始到具体实验。包括hdfs配置,yarn配置,分布式配置,如何编写mapreuduce 一步一步手把手,最后项目是hadoop 与 javaweb
hadoop oozie启动或运行报错:Table ‘oozie.VALIDATE_CONN’ doesn’t exist
赠送jar包:hadoop-mapreduce-client-common-2.6.5.jar; 赠送原API文档:hadoop-mapreduce-client-common-2.6.5-javadoc.jar; 赠送源代码:hadoop-mapreduce-client-common-2.6.5-sources.jar; 赠送Maven依赖信息...
大数据平台,hadoop集群学习笔记
Hadoop 适合初学者 Hadoop2.0 hbase 什么时候用HBase: 确信有足够多数据 确信可以不依赖所有RDBMS的额外特性(列数据类型/第二索引/事物/高级查询语言) 确信有足够硬件
在网上搜集的以及本人自己总结的hadoop集群常见问题及解决办法,融合了网上常常搜到的一些文档以及个人自己的经验。
踏入hadoop的世界,一个不一样的世界
hadoop权威指南代码 (Hadoop: The Definitive Guide code) http://www.hadoopbook.com
ERROR org.apache.hadoop.mapred.TaskTracker: Can not start task tracker because java.io.IOException: Failed to set permissions of path: \tmp\hadoop-admin \mapred\local\ttprivate to 0700 at org.apache...
Hadoop oozie报错:Table ‘oozie.VALIDATE_CONN’ doesn’t exist 。 oozie建表sql,直接下载执行
For the latest information about Hadoop, please visit our website at: http://hadoop.apache.org/ and our wiki, at: http://wiki.apache.org/hadoop/ This distribution includes cryptographic software. ...