- 浏览: 341755 次
- 性别:
- 来自: 上海
文章分类
最新评论
-
tpxcer:
不开启时可以的,而且开启以后各种坑。。。。
hue beeswax权限管理 -
yangze:
博主请教一个问题,hue 控制hive表的权限怎么弄? 怎么联 ...
cloudera新增用户权限配置 -
linux91:
楼主你好,我用CM配置LDAP用户组映射,进入impala时, ...
sentry配置 -
linux91:
版主:按你的步骤配置了,可是,执行 impala-shell ...
impala集成LDAP -
lookqlp:
super_a 写道你好!找不到表这个问题是如何解决的,可以描 ...
hcatalog读取hive数据并写入hive
mysqldump -hhost -uroot -ppasswd sentry > /tmp/sentry.sql
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
flush PRIVILEGES;
source /tmp/sentry.sql
mysql参数
show variables like 'lower%';
注意迁移时查看系统大小写参数,如果大小写敏感参数不一致可能报如下错误
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
flush PRIVILEGES;
source /tmp/sentry.sql
mysql参数
[mysqld] #transaction-isolation=READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: # symbolic-links=0 socket=/var/lib/mysql/mysql.sock key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 16M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 # Important: see Configuring the Databases and Setting max_connections max_connections = 2000 default-storage-engine=InnoDB # log-bin should be on a disk with enough free space #log-bin=/var/log/mysql/binary/mysql_binary_log # For MySQL version 5.1.8 or later. Comment out binlog_format for older versions. #binlog_format = mixed #read_buffer_size = 2M #read_rnd_buffer_size = 16M #sort_buffer_size = 8M #join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M lower_case_table_names = 1 [mysqld_safe] socket=/var/lib/mysql/mysql.sock log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
show variables like 'lower%';
注意迁移时查看系统大小写参数,如果大小写敏感参数不一致可能报如下错误
Query for candidates of org.apache.hadoop.hive.metastore.model.MVersionTable and subclasses resulted in no possible candidates Required table missing : "`VERSION`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables" org.datanucleus.store.rdbms.exceptions.MissingTableException: Required table missing : "`VERSION`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.autoCreateTables" at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:485) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3380) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3190) at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2841) at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122) at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1605) at org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:954) at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:679) at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:408) at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:947) at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:370) at org.datanucleus.store.query.Query.executeQuery(Query.java:1744) at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672) at org.datanucleus.store.query.Query.execute(Query.java:1654) at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221) at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6936) at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6920) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6878) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6862) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:560) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:450) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5622) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5617) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:5850) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5775) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Nov 18, 4:16:37.046 PM ERROR org.apache.hadoop.hive.metastore.HiveMetaStore MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6881) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6862) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:560) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:450) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5622) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5617) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:5850) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5775) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Nov 18, 4:16:37.046 PM ERROR org.apache.hadoop.hive.metastore.HiveMetaStore Metastore Thrift Server threw an exception... MetaException(message:Version information not found in metastore. ) at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6881) at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6862) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:98) at com.sun.proxy.$Proxy0.verifySchema(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:560) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:608) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:450) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:66) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5622) at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5617) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:5850) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5775) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
发表评论
-
hive dynamic partitions insert java.lang.OutOfMemoryError: Java heap space
2015-10-26 18:03 3028动态分区问题,如果数据量大或者当动态分区大甚至只有十几个时 ... -
yarn NullPointerException
2015-03-26 17:03 1394yarn重启后,部分nm启动不了,报空指针问题 20 ... -
mapreduce mapper access security hbase
2015-03-17 14:42 1169环境: security cdh 5.2.0 secu ... -
sentry服务后,几个权限问题
2015-03-10 16:08 8902以账户bi为例 问题一:账户bi beeline ldap后 ... -
cloudera新增用户权限配置
2015-03-05 16:13 3897目标: 给各个业务组提供不同用户及用户组,并有限制的访问h ... -
hive集成LDAP
2015-02-13 10:09 9676cloudera manager hive- sevice ... -
sentry配置
2015-02-13 10:06 2293当前cdh版本为5.2.0,且通过cloudera mange ... -
impala HA
2014-12-11 17:36 1837目的: 为impala jdbc提供统一的接口,作用参照htt ... -
impala集成LDAP
2014-12-11 12:55 7040目的: 为解决kerberos安 ... -
security cdh mapreduce access hbase
2014-12-02 15:09 1005执行mapreduce的用户必须是可以访问hdfs相应目录和执 ... -
hive gateway(client) configuration
2014-12-02 14:32 7975配置hive gateway机器 Caused by: Met ... -
hcatalog读取hive数据并写入hive
2014-12-01 17:49 18035参考http://www.cloudera.com/cont ... -
CDH5.0.2升级至CDH5.2.0
2014-12-01 16:59 9737升级需求 1.为支持spark k ... -
hue beeswax权限管理
2014-08-05 17:54 10542http://www.cloudera.com/content ... -
cloudera client集群部署
2014-08-05 17:48 597一般我们使用使用client机器访问集群,而不会直接在hado ... -
cloudera manager kerberos配置
2014-08-05 17:37 1506CDH5.1.0前的版本,可以通过cloudera manag ... -
CDH5安装
2014-08-05 17:05 2238CDH安装有很多方式: ta ... -
hadoop集群数据迁移
2014-08-04 22:31 6512hadoop distcp hdfs://namenode1/ ... -
java.lang.OutOfMemoryError: unable to create new native thread
2014-05-23 17:29 164735227 2014-05-21 13:53:18,504 I ... -
hadoop-2.2.0编译import eclipse
2013-10-22 17:50 7753编译hadoop-2.2.0 下载hadoop-2.2.0-s ...
相关推荐
1.将Mysql中的数据迁移到Hdfs文件系统中,然后通过Hive加载HDFS文件系统中的数据值 2.将Hive中的数据迁移到指定Mysql数据库中 注意点: 1.数据迁移的过程中,由于hive的Null值存储为"\N",Mysql存储为NULL值,二者...
Hive.sql
datax数据从hive表导入mysql表,数据缺失解决
hive.ziphive数据迁移和数据分析
由于大数据里面涉及到非关系型数据库如hive、kudu、hbase等的数据迁移,目前涉及到的迁移工具都没有支持hive数据库的事务表的迁移,如果hive库里面存在大量的事务表的时候,目前的工具都是不支持的,例如华为的CDM,...
centos 下hive连接mysql驱动 ,mysql-connector-java-8.0.26-1.el7.noarch. 使用方法: 1.下载本rpm文件后,上传至你的服务器(虚拟机) 2. rpm -ivh mysql-connector-java-8.0.26-1.el7.noarch进行安装 3.安装完成...
通过mysql> source XXXXX\hive-schema-3.1.0.mysql.sql初始化元数据
可以从hive元数据生成建表语句的资源。包括表结构、分区等信息
apache-hive-1.2.1-bin.tar.gz mysql-connector-java-5.1.6-bin 用于安装Hive并使用MySQL作为元数据库
hadoop+hive+sqoop数据迁移+azkaban任务调度
Hive运维中通常会用到操作元数据,这里提供了常用的sql语句
spring-boot集成mybatis+druid实现 hive/mysql多数据源切换,用mysql数据库作为用户验证库以及用户信息库,hive作为数据可视化源库。
执行命令: java -cp hive-distcp-1.0-SNAPSHOT-shaded_sc.jar com.apex.distcp.HiveTable --config [库名] > test.sql
hive-mysqlhive-mysql
pyflink将mysql数据直接插入hive,由此可以延伸出pyflink实现hive关联mysql
hive仓库元数据管理系统 hive仓库元数据管理系统 ##有如下功能: 1.hive元数据信息的查看,包括表基本信息,数据库基本信息,字段信息,分区信息,索引信息等; 2.对hive元数据的检索,包括表、字段、数据库等内容...
mysql安装包,mysql外界数据库作为存储hive元数据的存储介质,它的存在,方便hive用户根据自身的需求对数据进行分析处理。
{ job: { setting: { speed: { channel: 1 }, errorLimit: { record: 0, percentage: 0.02 } }, content: [ { reader: {
(3)sqoop数据迁移,完成HIve与MySQL数据库中的数据交互 (4)Echarts搭建动态可视化大屏 (5)SpringBoot搭建可视化后台系统,完成前端与后台的数据传递与交互。 (6)基于Cenots7 搭建虚拟机,配置Hadoop、HDFS、...
新手学习大数据搭建环境所需要安装资料包,其中包括hive-mysql安装包,HIVE连接mysql驱动。其中包括 apache-hive-1.2.1-bin.tar.gz MySQL-client-5.6.24-1.el6.x86_64.rpm mysql-connector-java-5.1.27.tar.gz MySQL-...