续上一篇:
搭建Hadoop2.7.3+Hive2.1.1及MySQL(配置Hive)(二)
准备工作下载最新连接器地址
https://dev.mysql.com/downloads/connector/j/
例子:下载mysql-connector-java-5.1.41.tar
1、解压连接器 connector 文件
1.1、解压
[root@localhost Software]# tar xzfmysql-connector-java-5.1.41.tar.gz
[root@localhost Software]# cd mysql-connector-java-5.1.41/
1.2、查看文件夹
[root@localhostmysql-connector-java-5.1.41]# ll
1.3、Copy到hive/lib路径下
[root@localhost Software]# cpmysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar/usr/hive/lib/mysql-connector-java-5.1.41-bin.jar
2、登陆MySQL创建数据库:hive_db(注意配置hive-site.xml时有指定)
2.1、用户名:root 密码:password,另开一个终端登陆MySQL,创建数据库hive_db
[root@localhost hive]# mysql -u root -ppassword
mysql> create database hive_db;
3、改配置文件hive-site.xml
以下只列出 改动的 配置项,其它保留默认
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/usr/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>Use false if a production metastore server is used</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive_db?createDatabaseIfNoExist=true</value>
<description> Roy
JDBC connect string for a JDBC metastore.
To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.
For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.
</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>User-Defined(Roy) Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
<description>User-defined(Roy)Username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
<description>User-defined(Roy)password to use against metastore database</description>
</property>
</configuration>
4、使用schematool初始化
[root@localhost hive]# schematool -dbTypemysql -initSchema
--显示成功
schemaTool completed
5、 启动hive服务端程序
5.1、启动 hive服务端
[root@localhost hive]# hive --servicemetastore &
-- 屏幕提示信息不显示时,按ctrl+c退出
5.2、查看进程信息
[root@localhost hive]# jps
--显示进程信息多了(RunJar)
51280 Jps 5985 SecondaryNameNode 6226 ResourceManager 45766 DataNode 5753 NameNode 51194 RunJar 6348 NodeManager
5.3、有需要时,可启动hive 远程服务 (端口号10000)
[root@localhost hive]# hive --servicehiveserver &
6、测试环境配置是否成功
6.1、准备导入文本文件/root/桌面/Test/wc-in/a.txt
格式:
1,h
2,i
3,v
4,e
6.2、登陆hive成功后,测试创建表
root@localhost hadoop]# hive
6.2.1、创建表及指定逗号(,)为分隔符
hive> create table a(id int,name string)
> row format delimited fields terminated by ',';
--显示信息
OK
Time taken: 0.288 seconds
6.2.2、导入文件a.txt
hive> load data local inpath '/root/桌面/Test/wc-in/a.txt' into table a;
--显示信息
Loading data to table default.a
OK
Time taken: 0.763 seconds
6.2.3、查看效果
hive> select * from a;
--显示信息
OK
1 h
2 i
3 v
4 e
Time taken: 0.309 seconds, Fetched: 4row(s)
6.3、在Hive内使用dfs命令
6.3.1、查看a表dfs存储路径
hive> dfs -ls /usr/hive/warehouse/a;
--显示信息
Found 1 items
-rw-r--r-- 1 root supergroup 16 2017-03-08 17:46/usr/hive/warehouse/a/a.txt
6.3.2、查看文件内容
hive> dfs -cat /usr/hive/warehouse/a/*;
--显示信息
1,h
2,i
3,v
4,e
7、登陆MySQL查看创建表
[root@localhost conf]# mysql -u root -ppassword
mysql> use hive_db;
mysql> select TBL_ID, CREATE_TIME,DB_ID, OWNER, TBL_NAME,TBL_TYPE from TBLS;
--显示信息
+--------+-------------+-------+-------+----------+---------------+
| TBL_ID | CREATE_TIME | DB_ID | OWNER |TBL_NAME | TBL_TYPE |
+--------+-------------+-------+-------+----------+---------------+
| 37 | 1488966386 | 1 | root | a | MANAGED_TABLE |
+--------+-------------+-------+-------+----------+---------------+
1 row in set (0.03 sec)
8、在hdfs查看生成文件(同上步骤[6.3])
8.1、查看a表存储路径
[root@localhost hadoop]# hdfs dfs -ls/usr/hive/warehouse/a
-- 显示信息
Found 1 items
-rw-r--r-- 1 root supergroup 162017-03-08 17:46 /usr/hive/warehouse/a/a.txt
8.2、查看内容
[root@localhost hadoop]# hdfs dfs -cat /usr/hive/warehouse/a/*
--显示信息
1,h
2,i
3,v
4,e
常见问题处理:
1、启动hive时报错
[root@localhost hive]# hive
--显示报错信息
Caused by:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):Cannot create directory /tmp/hive/root/24f1d91f-f32b-47e1-824d-ba26b02bd13e.Name node is in safe mode.
原因:hadoop为安全模式
--解决方法:
关闭安全模式
[root@localhost hadoop]# hadoop dfsadmin-safemode leave
--显示信息
DEPRECATED: Use of this script to executehdfs command is deprecated.
Instead use the hdfs command for it.
Safe mode is OFF
2、在导入数据时出错信息
hive> load data local inpath '/root/桌面/Test/wc-in/a.txt' into table a;
--显示报错信息
FAILED: Execution Error, return code 1 fromorg.apache.hadoop.hive.ql.exec.MoveTask.org.apache.hadoop.ipc.RemoteException(java.io.IOException): File/usr/hive/warehouse/a/a_copy_2.txt could only be replicated to 0 nodes insteadof minReplication (=1). There are 0datanode(s) running and no node(s) are excluded in this operation.
原因:hadoop没有启动datanote
解决方法:
[root@localhost hive]# start-dfs.sh
[root@localhost hive]# jps
--显示信息
51152 Jps
5985 SecondaryNameNode
6226 ResourceManager
45766 DataNode
5753 NameNode
6348 NodeManager
http://www.tuicool.com/articles/fQZnAnf
相关推荐
hadoop2.7.3+hive1.2.1+spark2.0.1性能测试
1、内容概要:Hadoop+Spark+Hive+HBase+Oozie+Kafka+Flume+Flink+Elasticsearch+Redash等大数据集群及组件搭建指南(详细搭建步骤+实践过程问题总结)。 2、适合人群:大数据运维、大数据相关技术及组件初学者。 3、...
win10下搭建Hadoop(jdk+mysql+hadoop+scala+hive+spark),包括jdk的安装、mysql安装和配置,hadoop安装和配置,scala安装和配置,hive安装和配置,spark安装和配置。
hadoop 2.7.3 + Windows安装替换bin目录文件
Hive2.1.1+hadoop2.7.3 Hivedrive for kettle lib
适合新手,详细 01-Java环境安装 02- Eclipse下载与安装 03-VMware虚拟机的安装 04-在VMware中安装CentOS 05- Hadoop集群+ Hive+ MySQL搭建
Hadoop2.7.3+HBase1.2.5+ZooKeeper3.4.6 搭建分布式集群环境详解。 详细介绍了如何搭建分布式集群环境。
Hadoop2.7.3安装文档
毕业设计,采用Hadoop+Hive构建数据仓库,使用django+echarts构建前端web网站对业务指标进行可视化呈现 1. Hadoop+Hive构建数据仓库 2. django+echarts网站开发 3. 数据清洗,数据模型构建 毕业设计,采用Hadoop+...
hadoop-2.7.3+zookeeper-3.4.8+hadoop-2.7.3分布式环境搭建整理(王三旗亲试成功安装)
hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+hadoop配置文件hive+...
本文档详细介绍了如何用ZooKeeper和Hadoop、HBase搭建分布式大数据分析平台。
七月在线七月在线## Note, this file is written by cloud-init on first boot of an instance
jdk1.8+hadoop2.7.3+hbase1.2.6在ubuntu14.06上安装步骤,在这里,没有说明ubuntu的安装步骤,读者要先将ubuntu安装好,然后将需要的安装包分别到官网下载准备。
Centos+Hadoop+Hive+HBase
hive2.1.1 + hadoop2.6.0jdbc驱动,里边并没有包含所有的hive/lib 下的jar包。共有15个,自己在eclipse中测试过完全可用
基于hadoop的Hive数据仓库JavaAPI简单调用的实例,关于Hive的简介在此不赘述。hive提供了三种用户接口:CLI,JDBC/ODBC和 WebUI CLI,即Shell命令行 JDBC/ODBC 是 Hive 的Java,与使用传统数据库JDBC的方式类似 Web...
hadoop2.2+hbase0.96+hive0.12安装整合详细高可靠文档及经验总结
在windows环境下开发hadoop时,需要配置HADOOP_HOME环境变量,变量值D:\hadoop-common-2.7.3-bin-master,并在Path追加%HADOOP_HOME%\bin,有可能出现如下错误: org.apache.hadoop.io.nativeio.NativeIO$Windows....
mysql5.7.38、mysql jdbc驱动mysql-connector-java-8.0.8-dmr-bin.jar、 apache-hive-3.1.3 2.本文软件均安装在自建的目录/export/server/下 :sudo mkdir -p /export/server 创建好目录后要改变该目录的属主和属...