- 浏览: 341758 次
- 性别:
- 来自: 上海
文章分类
最新评论
-
tpxcer:
不开启时可以的,而且开启以后各种坑。。。。
hue beeswax权限管理 -
yangze:
博主请教一个问题,hue 控制hive表的权限怎么弄? 怎么联 ...
cloudera新增用户权限配置 -
linux91:
楼主你好,我用CM配置LDAP用户组映射,进入impala时, ...
sentry配置 -
linux91:
版主:按你的步骤配置了,可是,执行 impala-shell ...
impala集成LDAP -
lookqlp:
super_a 写道你好!找不到表这个问题是如何解决的,可以描 ...
hcatalog读取hive数据并写入hive
cloudera manager
hive- sevice wide - advanced-Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml
<property>
<name>hive.server2.authentication</name>
<value>LDAP</value>
</property>
<property>
<name>hive.server2.authentication.ldap.url</name>
<value>ldap://master-71:389</value>
</property>
<property>
<name>hive.server2.authentication.ldap.baseDN</name>
<value>ou=ndpmedia,dc=yeahmobi,dc=com</value>
</property>
测试例子:https://github.com/firecodeman/Cloudera-Impala-Hive-JDBC-Example
奇怪现象:
http://community.cloudera.com/t5/CDH-Manual-Installation/hive-ldap-not-work/m-p/25461#U25461
hive运行一段时间后,或者添加LDAP配置后重启hive时,hive metastore启动异常,异常如下:
在cloudera manager中service wide 、gateway 、hivemetastore、hiveserver2中都添加ldap配置,都无法走LDAP,而报认证错误。
但有时不会报错,如我现在的线上CDH5.2.0版本就没有问题。
后来定位是否为版本问题,下载tar包,CDH5.2.4版本的hive,及CDH5.2.0版本的hive分别进行测试,发现5.2.4版本hiveLDAP运行正常,5.2.0版本hive报各种错,例如:
将5.2.0版本的hivejar替换成5.2.4版本的hive jar就完全ok。
即此奇怪现象确实版本问题,因为使用cloudera manager管理集群,为了不整体升级CDH,替换安装hive server2和metastore节点的jar包,例如5.2.0jar目录为/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib(查看hive执行脚本的HIVE_HOME)。在cm上,重启hive,问题解决。
hive- sevice wide - advanced-Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml
<property>
<name>hive.server2.authentication</name>
<value>LDAP</value>
</property>
<property>
<name>hive.server2.authentication.ldap.url</name>
<value>ldap://master-71:389</value>
</property>
<property>
<name>hive.server2.authentication.ldap.baseDN</name>
<value>ou=ndpmedia,dc=yeahmobi,dc=com</value>
</property>
测试例子:https://github.com/firecodeman/Cloudera-Impala-Hive-JDBC-Example
奇怪现象:
http://community.cloudera.com/t5/CDH-Manual-Installation/hive-ldap-not-work/m-p/25461#U25461
hive运行一段时间后,或者添加LDAP配置后重启hive时,hive metastore启动异常,异常如下:
2015-03-10 18:38:37,618 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:262) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more 2015-03-10 18:38:37,618 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:262) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more 2015-03-10 18:38:37,618 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:262) at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ... 10 more
在cloudera manager中service wide 、gateway 、hivemetastore、hiveserver2中都添加ldap配置,都无法走LDAP,而报认证错误。
但有时不会报错,如我现在的线上CDH5.2.0版本就没有问题。
后来定位是否为版本问题,下载tar包,CDH5.2.4版本的hive,及CDH5.2.0版本的hive分别进行测试,发现5.2.4版本hiveLDAP运行正常,5.2.0版本hive报各种错,例如:
org.apache.hive.service.ServiceException: Failed to Start HiveServer2 at org.apache.hive.service.CompositeService.start(CompositeService.java:80) at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:101) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.hive.service.ServiceException: Error setting stage directories at org.apache.hive.service.cli.CLIService.start(CLIService.java:132) at org.apache.hive.service.CompositeService.start(CompositeService.java:70) ... 8 more Caused by: org.apache.hadoop.security.AccessControlException: Permission denied at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:169) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:157) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:138) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6250) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6232) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6151) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1678) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1658) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:587) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setPermission(AuthorizationProviderProxyClientProtocol.java:148) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:437) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) 相关issue:https://issues.apache.org/jira/browse/HIVE-8606
将5.2.0版本的hivejar替换成5.2.4版本的hive jar就完全ok。
即此奇怪现象确实版本问题,因为使用cloudera manager管理集群,为了不整体升级CDH,替换安装hive server2和metastore节点的jar包,例如5.2.0jar目录为/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hive/lib(查看hive执行脚本的HIVE_HOME)。在cm上,重启hive,问题解决。
发表评论
-
hive相关元数据迁移(mysql)
2015-11-18 18:27 2519mysqldump -hhost -uroot -ppassw ... -
hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space
2015-10-26 18:03 3028动态分区问题,如果数据量大或者当动态分区大甚至只有十几个时 ... -
yarn NullPointerException
2015-03-26 17:03 1394yarn重启后,部分nm启动不了,报空指针问题 20 ... -
mapreduce mapper access security hbase
2015-03-17 14:42 1169环境: security cdh 5.2.0 secu ... -
sentry服务后,几个权限问题
2015-03-10 16:08 8902以账户bi为例 问题一:账户bi beeline ldap后 ... -
cloudera新增用户权限配置
2015-03-05 16:13 3897目标: 给各个业务组提供不同用户及用户组,并有限制的访问h ... -
sentry配置
2015-02-13 10:06 2294当前cdh版本为5.2.0,且通过cloudera mange ... -
impala HA
2014-12-11 17:36 1837目的: 为impala jdbc提供统一的接口,作用参照htt ... -
impala集成LDAP
2014-12-11 12:55 7040目的: 为解决kerberos安 ... -
security cdh mapreduce access hbase
2014-12-02 15:09 1005执行mapreduce的用户必须是可以访问hdfs相应目录和执 ... -
hive gateway(client) configuration
2014-12-02 14:32 7975配置hive gateway机器 Caused by: Met ... -
hcatalog读取hive数据并写入hive
2014-12-01 17:49 18035参考http://www.cloudera.com/cont ... -
CDH5.0.2升级至CDH5.2.0
2014-12-01 16:59 9737升级需求 1.为支持spark k ... -
hue beeswax权限管理
2014-08-05 17:54 10542http://www.cloudera.com/content ... -
cloudera client集群部署
2014-08-05 17:48 597一般我们使用使用client机器访问集群,而不会直接在hado ... -
cloudera manager kerberos配置
2014-08-05 17:37 1506CDH5.1.0前的版本,可以通过cloudera manag ... -
CDH5安装
2014-08-05 17:05 2238CDH安装有很多方式: ta ... -
hadoop集群数据迁移
2014-08-04 22:31 6512hadoop distcp hdfs://namenode1/ ... -
java.lang.OutOfMemoryError: unable to create new native thread
2014-05-23 17:29 164735227 2014-05-21 13:53:18,504 I ... -
hadoop-2.2.0编译import eclipse
2013-10-22 17:50 7753编译hadoop-2.2.0 下载hadoop-2.2.0-s ...
相关推荐
intellij idea集成apache Hive 用到的jar包
Hive;Hive;Hive;Hive;Hive;Hive;Hive;Hive;Hive;Hive;Hive;Hive
apache-hive-3.1.3 2.本文软件均安装在自建的目录/export/server/下 :sudo mkdir -p /export/server 创建好目录后要改变该目录的属主和属组 :sudo chown -R ljr:ljr /export 修改目录权限:sudo +771 -R /...
spring-boot集成mybatis+druid实现 hive/mysql多数据源切换,用mysql数据库作为用户验证库以及用户信息库,hive作为数据可视化源库。
19:Flume+HBase+Hive集成大数据项目离线分析
flink 集成 hive 所需要的依赖包,包括 antlr-runtime-3.5.2.jar flink-connector-hive_2.12-1.14.4.jar flink-shaded-hadoop-3-3.1.1.7.2.9.0-173-9.0.jar flink-sql-connector-hive-3.1.2_2.12-1.14.4.jar hive-...
SpagoBI集成Hive数据库:此文档主要介绍集成步骤,不包含Hive数据库部分。在做集成时Hive数据库老是不稳定,导致报表出问题。针对这种情况小伙伴可以写个简单的JDBC连接 循环测试一下。
flink1.12.0集成或者连接hive3.1.2的依赖包,pyflink也适用
本项目对如何在Springboot项目中整合hive-jdbc进行简单示例和介绍,亲测可用,请放心下载。
0. 重新编译依赖包编译过程参考:- Hive整合Hbase详解删除软链接的命令修改hive-site.xml的配置主要修改zookeeper的配置项,以便能够
hive hive hive hive hive hive hive hive hive hive hive hive
使用hive3.1.2和spark3.0.0配置hive on spark的时候,发现官方下载的hive3.1.2和spark3.0.0不兼容,hive3.1.2对应的版本是spark2.3.0,而spark3.0.0对应的hadoop版本是hadoop2.6或hadoop2.7。 所以,如果想要使用高...
此文档是本人在工作中用到的知识总结出来的整合过程,本人是菜鸟,希望得到大神们的建议。
hive-jdbc
Hive表生成工具,Hive表生成工具Hive表生成工具
hive0.8.1和hbase0.92.0集成的hive-hbase-handler.Jar包,里面包含:hbase-0.92.0.jar、hbase-0.92.0-tests.jar、hive-hbase-handler-0.9.0-SNAPSHOT.jar。经测试没有问题。
1 Hive 概念与连接使用: 2 2 Hive支持的数据类型: 2 2.1原子数据类型: 2 2.2复杂数据类型: 2 2.3 Hive类型转换: 3 3 Hive创建/删除数据库 3 3.1创建数据库: 3 3.2 删除数据库: 3 4 Hive 表相关语句 3 4.1 Hive ...
《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第5章 Hive数据操作.pdf《Hive数据仓库案例教程》教学课件 第...
hive0.10.0和hbase0.94.4集成的hive-hbase-handler.Jar包,经测试没有问题。