背景：欧拉操作系统下安装CDH集群的时候，需要安装python2.7.5，但是本身欧拉系统对python2的支持可能没有那么好，所以考虑搭建原生的hadoop集群。
基础环境如下
| 组件名称 | 组件版本 |
|---|---|
| 欧拉 | VERSION=“22.03 (LTS-SP4)” |
| jdk | openjdk version “1.8.0_44” |
| mysql | 8.0.42 |
| hadoop | |
| hive |
一、jdk的安装
openjdk的下载地址
https://jdk.java.net/java-se-ri/8-MR6
二、mysql的安装
mysql安装包的下载地址
https://dev.mysql.com/downloads/mysql/
三、hadoop的安装
hadoop安装的前置条件是系统中已经有Java的环境
还需要将免密登录配置好
创建Hadoop用户
#创建Hadoop的用户
sudo useradd hadoop#设置Hadoop用户的密码
sudo passwd hadoop #设置Hadoop的用户有sudo权限
sudo usermod -aG sudo hadoop
解压Hadoop的安装包
#解压这个压缩包到/opt的目录下，这里最好别放在/root下
sudo tar -xzf hadoop-3.4.1.tar.gz -C /opt#给这个解压完的目录，修改一个目录名
sudo mv /opt/hadoop-3.4.1 /opt/hadoop#设置这个路径的用户、用户组
sudo chown -R hadoop:hadoop /opt/hadoop
创建环境变量在配置文件中
#将配置写入到配置文件中
echo 'export HADOOP_HOME=/opt/hadoop' | sudo tee -a /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' | sudo tee -a /etc/profile#让配置文件立即生效
source /etc/profile
修改这几个配置文件
这几个配置文件的位置如下；
$HADOOP_HOME/etc/hadoop/
1.hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
2.core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value> # 默认文件系统地址</property><property><name>hadoop.tmp.dir</name><value>/opt/hadoop/tmp</value> # 临时文件目录:</property>
</configuration>3.hdfs-site.xml
<property><name>dfs.replication</name><value>1</value> # 数据副本数（单机设置为1）
</property>
<property><name>dfs.namenode.name.dir</name><value>/opt/hadoop/hdfs/namenode</value> # NameNode数据存储路径
</property>
4.mapred-site.xml
<property><name>mapreduce.framework.name</name><value>yarn</value> # 启用YARN框架
</property>
5.yarn-site.xml
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value> # 指定NodeManager附加服务
</property>
格式化HDFS文件系统
hdfs namenode -format # 初始化NameNode
启动hadoop
1.启动hdfs
start-dfs.sh2.启动yarn
start-yarn.sh
查看集群启动状态及进程
jpshdfs dfsadmin -report
启动hadoop报错
./start-dfs.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation
解决方案，在配置文件中配置如下，即可用root用户进行启动了
四、hive的安装
https://dlcdn.apache.org/hive/hive-4.0.1/
在MySQL中创建Hive元数据库及用户
CREATE DATABASE hive_meta;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Jky1234!@#$';
GRANT ALL ON hive_meta.* TO 'hive'@'%';
FLUSH PRIVILEGES;
修改Hive配置文件
cd $HIVE_HOME/conf
cp hive-default.xml.template hive-site.xml
vi hive-site.xml # 替换以下关键参数
hive的配置文件内容添加如下
<property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true</value>
</property>
<property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.cj.jdbc.Driver</value>
</property>
<property><name>javax.jdo.option.ConnectionUserName</name><value>hive</value>
</property>
<property><name>javax.jdo.option.ConnectionPassword</name><value>Jky1234!@#$</value>
</property>下载mysql-connector-java-8.0.27.jar并拷贝到Hive的lib目录
cp mysql-connector-java-8.0.27.jar /opt/hive/lib/opt/hadoop/share/hadoop/common/lib
cp guava-27.0-jre.jar /opt/hive/lib
在hive的bin目录下执行初始化数据库的操作
./schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 4.0.0
Metastore connection URL: jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true
Metastore connection Driver : com.mysql.cj.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 4.0.0
Initialization script hive-schema-4.0.0.mysql.sqlInitialization script completed
是否成功？ 去MySQL中
use hive_meta;
show tables;
启动hive的客户端，显示如下；
仔细看日志打印的信息，应该是提示日志冲突了
./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.1 by Apache Hive
beeline> show databases;
No current connectionll log4j
-rw-r–r–. 1 root root 349845 9月 25 2024 log4j-1.2-api-2.18.0.jar
-rw-r–r–. 1 root root 315115 9月 25 2024 log4j-api-2.18.0.jar
-rw-r–r–. 1 root root 1861441 9月 25 2024 log4j-core-2.18.0.jar
-rw-r–r–. 1 root root 24801 9月 25 2024 log4j-slf4j-impl-2.18.0.jar
-rw-r–r–. 1 root root 36166 9月 25 2024 log4j-web-2.18.0.jar
mv log4j-slf4j-impl-2.18.0.jar log4j-slf4j-impl-2.18.0-bak.jar0526
然后在执行，发现多余的日志信息没有了
执行连接hive的命令，报错如下；
!connect jdbc:hive2://localhost:10000
./beeline --verbose
Default hs2 connection config file not found
Beeline version 4.0.1 by Apache Hive
查看1000的端口是否被监听
ss -tulnp | grep 9870
tcp LISTEN 0 500 0.0.0.0:9870 0.0.0.0:* users:((“java”,pid=13987,fd=329))
./hive --service hiveserver2 --hiveconf hive.log.dir=/var/log/hive --hiveconf hive.log.file=hiveserver2-daemon.log
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/java-se-8u44-ri/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/hive/bin:/root/bin)
2025-05-26 18:16:09: Starting HiveServer2
Exception in thread “main” java.lang.IllegalArgumentException: Logs will be split in two files if the commandline argument hive.log.file is used. To prevent this use to HADOOP_CLIENT_OPTS -Dhive.log.file=hiveserver2-daemon.log or use the set the value in the configuration file (see HIVE-19886)
at org.apache.hive.service.server.HiveServer2$ServerOptionsProcessor.parse(HiveServer2.java:1366)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:330)
at org.apache.hadoop.util.RunJar.main(RunJar.java:245)
./beeline --verbose
Overriding connection url property url_prefix from user connection configuration file
Connecting to jdbc:hive2://localhost:10000/default
Enter username for jdbc:hive2://localhost:10000/default: hive
Enter password for jdbc:hive2://localhost:10000/default: ****
25/05/26 18:23:26 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive
解决方案
hadoop.proxyuser.root.hosts
hadoop.proxyuser.root.groups
的使用)
