hive的安装简单一些,使用也比较简单,基础hadoop搭建好之后,只要初始化一些目录和数据库就好了
安装需要做几件事:
1.设立一个数据源作为元数据存储的地方,默认是derby内嵌数据库,不过不允许远程连接,所以换成mysql
2.配置java路径和classpath路径
下载地址: http://mirrors.shuosc.org/apache/hive/hive-2.3.2/
发现一个问题:该地址会变化,所以不一定有效,可以到官网选择: http://www.apache.org/dyn/closer.cgi/hive/
解压后先配置hive环境变量
vi /etc/profile
添加:
export HIVE_HOME=/home/sri_udap/app/apache-hive-2.3.2-bin
export PATH=$PATH:$HIVE_HOME/bin生效:
source /etc/profile
在conf目录下,拷贝模板进行配置:
mv hive-default.xml.template hive-site.xml mv hive-env.sh.template hive-env.sh
先修改其他两个配置文件:
修改hadoop的配置文件hadoop-env.sh，修改内容如下：
export HADOOP_CLASSPATH=.:$CLASSPATH:$HADOOP_CLASSPATH:$HADOOP_HOME/bin
这里配置的classpath后,在后面执行hive初始化时仍然一直报java的类错误,查阅资料后,把他改成另一种更可靠的方式:
for f in $HADOOP_HOME/hadoop-*.jar; do CLASSPATH=${CLASSPATH}:$f donefor f in $HADOOP_HOME/lib/*.jar; do CLASSPATH=${CLASSPATH}:$f donefor f in $HIVE_HOME/lib/*.jar; do CLASSPATH=${CLASSPATH}:$f done
在目录$HIVE_HOME/bin下面，修改文件hive-env.sh，增加以下内容：
export HADOOP_HOME=/home/sri_udap/app/hadoop-2.7.2
export HIVE_CONF_DIR=/home/sri_udap/app/apache-hive-2.3.2-bin/conf
export HIVE_AUX_JARS_PATH=/home/sri_udap/app/apache-hive-2.3.2-bin/lib
修改hive-site.xml文件，修改内容如下：
<property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://master:3306/hive?createDatabaseIfNotExist=true</value> </property> <property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.jdbc.Driver</value> </property> <property><name>javax.jdo.option.ConnectionUserName</name><value>hivetest</value> </property> <property><name>javax.jdo.option.ConnectionPassword</name><value>hivetest</value> </property>
拷贝一个mysql的连接jar包到lib目录下,我用的是 mysql-connector-java-5.1.30.jar
然后到hdfs上建立一些基础目录hive-site.xml中配置的仓库地址等,手工创建(包括配置的hive的数据目录,仓库地址,日志等,并赋权):
bin/hadoop fs -mkdir -p /user/hive/warehouse bin/hadoop fs -mkdir -p /user/hive/tmp bin/hadoop fs -mkdir -p /user/hive/log bin/hadoop fs -chmod -R 777 /user/hive/warehouse bin/hadoop fs -chmod -R 777 /user/hive/tmp bin/hadoop fs -chmod -R 777 /user/hive/log
这样就可以开始初始化了,先启动hadoop,然后在bin目录下执行命令
./schematool -initSchema -dbType mysql
此时应该有个错误:
Exception in thread "main"java.lang.RuntimeException: java.lang.IllegalArgumentException:java.net.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7Datorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)atjava.lang.reflect.Method.invoke(Method.java:606)atorg.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: java.lang.IllegalArgumentException:java.net.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7Datorg.apache.hadoop.fs.Path.initialize(Path.java:148)atorg.apache.hadoop.fs.Path.<init>(Path.java:126)atorg.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:487)atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)... 7more
Caused by: java.net.URISyntaxException:Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7Datjava.net.URI.checkPath(URI.java:1804)atjava.net.URI.<init>(URI.java:752)atorg.apache.hadoop.fs.Path.initialize(Path.java:145)... 10more这是因为无法识别"system:java.io.tmpdir",换成自己建立的临时目录就好,比如我的是:/home/sri_udap/app/apache-hive-2.3.2-bin/temp.
把hive-site.xml中有这个配置的都换掉.其实${system:user.name}这个变量也是不识别的,勤快的话把这个也替换一下,把system:去掉即可,否则会出现跟我一样的情况,会建立奇怪的目录:
[root@master temp]# ls
9c9855ee-f160-48d4-ab74-9d597c81bb13_resources c1d48876-f1c9-4f97-bc3a-f9743fecc417_resources ${system:user.name} 再进行一次初始化,然后可以看到mysql中建立了一些表,这样就完成了建立工作
简单使用:
建立几张表:(hive建立表后会在hdfs上多出一个和表明一样的目录,然后加载数据后会在目录下多出文件,在hive中,数据就是目录和文件)
新建两张表:
hive>CREATE TABLE t1(id int); // 创建内部表t1，只有一个int类型的id字段 hive>CREATE TABLE t2(id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'; // 创建内部表t2，有两个字段，它们之间通过tab分隔
然后,按照字段分隔要求弄两个txt文件,并加载到表里面:
[root@master temp]# cat t1.txt
1
2
3
4
5
6
7
9[root@master temp]# cat t2.txt 1 a 2 b 3 c 9 x
加载数据:
hive>LOAD DATA LOCAL INPATH '/t1.txt' INTO TABLE t1; // 从本地文件加载 hive>LOAD DATA INPATH 't2.txt' INTO TABLE t1; // 从HDFS中加载
此时可以用一些简单的查询语句来查询hive,但是为了生成MapReduce作业,我们将语句写得稍微复杂些:
hive> select t2.name from t1 left join t2 on t1.id = t2.id; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Query ID = root_20171228104347_a63966e5-d32a-41c9-a363-79aef39cac63 Total jobs = 1 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/sri_udap/app/apache-hive-2.3.2-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/sri_udap/app/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2017-12-28 10:43:53 Starting to launch local task to process map join; maximum memory = 932184064 2017-12-28 10:43:54 Dump the side-table for tag: 1 with group count: 4 into file: file:/home/sri_udap/app/apache-hive-2.3.2-bin/temp/${system:user.name}/9c9855ee-f160-48d4-ab74-9d597c81bb13/hive_2017-12-28_10-43-47_556_6806677688398200490-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile31--.hashtable 2017-12-28 10:43:54 Uploaded 1 File to: file:/home/sri_udap/app/apache-hive-2.3.2-bin/temp/${system:user.name}/9c9855ee-f160-48d4-ab74-9d597c81bb13/hive_2017-12-28_10-43-47_556_6806677688398200490-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile31--.hashtable (364 bytes) 2017-12-28 10:43:54 End of local task; Time Taken: 1.103 sec. Execution completed successfully MapredLocal task succeeded Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1514424221956_0004, Tracking URL = http://master:8088/proxy/application_1514424221956_0004/ Kill Command = /home/sri_udap/app/hadoop-2.7.2/bin/hadoop job -kill job_1514424221956_0004 Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0 2017-12-28 10:44:10,516 Stage-3 map = 0%, reduce = 0% 2017-12-28 10:44:16,416 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.88 sec MapReduce Total cumulative CPU time: 1 seconds 880 msec Ended Job = job_1514424221956_0004 MapReduce Jobs Launched: Stage-Stage-3: Map: 1 Cumulative CPU: 1.88 sec HDFS Read: 5568 HDFS Write: 205 SUCCESS Total MapReduce CPU Time Spent: 1 seconds 880 msec OK a b c
完,有问题欢迎交流
