安装环境:
机器 只需要安装一台机器
操作系统:Ubuntu 11.04 64操作系统
hadoop:版本是1.0.2,安装在/usr/local/hadoop
sun jdk:版本是1.6.0_31 64bit,安装在/usr/local/jdk
hive:版本是0.8.1,安装在/usr/local/hive
安装步骤:
1.下载
下载hive:http://labs.mop.com/apache-mirror/hive/hive-0.8.1/hive-0.8.1.tar.gz
2.安装
(1)上传hive安装包到机器上,使用root用户登陆:
tar -xvf hive-0.8.1.tar.gz
(2)将解压的hive分别移动并改名为/usr/local/hive
rm -rf /usr/local/<span style="color: #000000;">hive</p><p>mv hive</span>-0.8.1 /usr/local/hive
3.配置hive
(1)修改/usr/local/hive/bin/hive-config.sh
在文件末尾加入
export JAVA_HOME=/usr/local/<span style="color: #000000;">jdk </p><p>export HIVE_HOME</span>=/usr/local/<span style="color: #000000;">hive </p><p>export HADOOP_HOME</span>=/usr/local/hadoop
(2) 根据hive-default.xml复制hive-site.xml
cp /usr/local/hive/conf/hive-<span style="color: #0000ff;">default</span>.xml /usr/local/hive/conf/hive-site.xml
(3)配置hive-site.xml,主要配置项如下:
hive.metastore.warehouse.dir：（HDFS上的）数据目录
hive.exec.scratchdir：（HDFS上的）临时文件目录
hive.metastore.warehouse.dir默认值是/user/hive/warehouse
hive.exec.scratchdir默认值是/tmp/hive-${user.name}
以上是默认值，暂时不改。
(4)改变 /usr/local/hive的目录所有者为hadoop
chown -R hadoop:hadoop /usr/local/hive
(5)配置hive的log4j:
cp /usr/loca/hive/conf/hive-log4j.properties.template /usr/loca/hive/conf/hive-log4j.properties
修改/usr/loca/hive/conf/hive-log4j.properties将org.apache.hadoop.metrics.jvm.EventCounter改为org.apache.hadoop.log.metrics.EventCounter
(6)启动hive
使用hadoop用户登陆,执行/usr/local/hive/bin/hive
(7)测试hive
hive><span style="color: #000000;"> create TABLE pokes( id INT, name string); </p><p>hive</span>><span style="color: #000000;"> SHOW TABLES; </p><p>hive</span>> select *<span style="color: #000000;"> from pokes; </p><p>hive</span>> drop table pokes;
4.优化hive
默认meta数据库为derby ，为了避免使用默认的Derby数据库（有并发访问和性能的问题），通常还需要配置元数据库为MySQL
修改配置文件conf/hive-site.xml
<span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.metastore.local<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>false<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>javax.jdo.option.ConnectionURL<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>jdbc:mysql://mysql_server_host:3306/hivedb?createDatabaseIfNotExist=true&amp;<span style="color: #ff0000;">useUnicode</span>=true&amp;<span style="color: #ff0000;">characterEncoding</span>=latin1<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>JDBC connect string for a JDBC metastore<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>javax.jdo.option.ConnectionDriverName<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>com.mysql.jdbc.Driver<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>Driver class name for a JDBC metastore<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>javax.jdo.option.ConnectionUserName<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>mysql_username<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>username to use against metastore database<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>javax.jdo.option.ConnectionPassword<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>mysql_password<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>password to use against metastore database<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.stats.dbconnectionstring<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;<span style="color: #ff0000;">characterEncoding</span>=latin1&amp;<span style="color: #ff0000;">user</span>=mysql_username&amp;<span style="color: #ff0000;">password</span>=mysql_password&amp;<span style="color: #ff0000;">createDatabaseIfNotExist</span>=true<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>The default connection string for the database that stores temporary hive statistics.<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.stats.dbconnectionstring<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>jdbc:mysql://mysql_server_host:3306/hive_stats?useUnicode=true&amp;<span style="color: #ff0000;">characterEncoding</span>=utf8&amp;<span style="color: #ff0000;">user</span>=mysql_username&amp;<span style="color: #ff0000;">password</span>=mysql_password&amp;<span style="color: #ff0000;">createDatabaseIfNotExist</span>=true<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>The default connection string for the database that stores temporary hive statistics.<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.stats.dbclass<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>jdbc:mysql<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>The default database that stores temporary hive statistics.<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.stats.jdbcdriver<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>com.mysql.jdbc.Driver<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span>The JDBC driver for the database that stores temporary hive statistics.<span style="color: #0000ff;"></</span><span style="color: #800000;">description</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"><</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span>hive.metastore.uris<span style="color: #0000ff;"></</span><span style="color: #800000;">name</span><span style="color: #0000ff;">></span></p><p> <span style="color: #0000ff;"><</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span>thrift://127.0.0.1:9083<span style="color: #0000ff;"></</span><span style="color: #800000;">value</span><span style="color: #0000ff;">></span></p><p><span style="color: #0000ff;"></</span><span style="color: #800000;">property</span><span style="color: #0000ff;">></span>
添加metastore启动脚本bin/hive-metastore.sh
<span style="color: #008000;">#</span><span style="color: #008000;">!/bin/sh</span></p><p>nohup ./hive --service metastore >> metastore.log 2>&1 &<span style="color: #000000;"></p><p>echo </span>$! > hive-metastore.pid
添加hive server启动脚本bin/hive-server.sh
nohup ./hive --service hiveserver >> hiveserver.log 2>&1 &<span style="color: #000000;"></p><p>echo </span>$! > hive-server.pid
启动metastore和hive server
./hive-<span style="color: #000000;">metastore.sh</p><p>.</span>/hive-server.sh
使用客户端连接和测试
参考之前的hive测试步骤一一执行。
如果出现
FAILED: Error in metadata: javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.
NestedThrowables:
com.mysql.jdbc.exceptions.MySQLSyntaxErrorException: Specified key was too long; max key length is 1000 bytes
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
或者是
FAILED: Error in metadata: MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
以上错误，主要是在select操作、drop操作时，就会出错。
这样的错误，可以修改hivedb数据库编码:
mysql> alter database hivedb character set latin1;
然后重启hive metastore和hive server就可以了
mysql数据库的编码目前最好设置为latin1,否则使用hive会出现莫名其妙的问题
来源URL：http://www.cnblogs.com/ggjucheng/archive/2012/04/26/2471341.html