Hadoop 3.3.6 完全分布式集群搭建VMware 5节点实战与SSH免密配置1. 环境准备与规划在开始搭建Hadoop集群之前我们需要先规划好整个集群的架构和资源配置。本次搭建的是一个5节点的完全分布式集群包括1个主节点Master、1个备用节点Standby和3个工作节点Worker。硬件资源建议宿主机配置CPU≥8核内存≥16GB推荐32GB磁盘空间≥150GBSSD更佳软件需求VMware Workstation Pro 16CentOS 7/8或Ubuntu 20.04 LTSJDK 8推荐OpenJDK 1.8.0Hadoop 3.3.6网络规划表节点角色主机名IP地址核心服务主节点master192.168.10.101NameNode, ResourceManager备用节点standby192.168.10.102SecondaryNameNode工作节点1worker1192.168.10.103DataNode, NodeManager工作节点2worker2192.168.10.104DataNode, NodeManager工作节点3worker3192.168.10.105DataNode, NodeManager提示在实际生产环境中建议将SecondaryNameNode和ResourceManager部署在不同的节点上以提高可用性。2. 虚拟机创建与基础配置2.1 创建母机虚拟机在VMware中新建虚拟机选择自定义安装方式操作系统选择CentOS 7或Ubuntu Server 20.04 LTS硬件配置建议CPU2核后续可调整内存2GB安装后可根据需要增加磁盘40GB精简配置网络适配器NAT模式关键配置注意事项取消勾选加速3D图形选项安装时务必选择最小安装并勾选以下组件OpenSSH Server标准系统工具2.2 系统初始化配置安装完成后首先进行系统更新和基础配置# 更新系统 sudo yum update -y # CentOS sudo apt update sudo apt upgrade -y # Ubuntu # 创建专用用户 sudo useradd -m hadoop sudo passwd hadoop sudo usermod -aG wheel hadoop # CentOS sudo usermod -aG sudo hadoop # Ubuntu # 配置sudo权限 echo hadoop ALL(ALL) NOPASSWD:ALL | sudo tee /etc/sudoers.d/hadoop2.3 网络与主机名配置修改主机名以master节点为例sudo hostnamectl set-hostname master配置静态IP以CentOS为例sudo vi /etc/sysconfig/network-scripts/ifcfg-ens33 # 修改以下内容 BOOTPROTOstatic ONBOOTyes IPADDR192.168.10.101 NETMASK255.255.255.0 GATEWAY192.168.10.2 DNS18.8.8.8配置所有节点的hosts文件sudo vi /etc/hosts # 添加以下内容 192.168.10.101 master 192.168.10.102 standby 192.168.10.103 worker1 192.168.10.104 worker2 192.168.10.105 worker3重启网络服务sudo systemctl restart network # CentOS sudo netplan apply # Ubuntu3. JDK安装与环境配置Hadoop运行需要Java环境我们选择OpenJDK 8# CentOS sudo yum install -y java-1.8.0-openjdk-devel # Ubuntu sudo apt install -y openjdk-8-jdk-headless验证安装java -version # 应输出类似openjdk version 1.8.0_312配置环境变量echo export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk ~/.bashrc source ~/.bashrc4. Hadoop安装与基础配置4.1 下载并安装Hadoopwget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz sudo tar -zxvf hadoop-3.3.6.tar.gz -C /usr/local sudo mv /usr/local/hadoop-3.3.6 /usr/local/hadoop sudo chown -R hadoop:hadoop /usr/local/hadoop4.2 配置Hadoop环境变量编辑~/.bashrc文件vi ~/.bashrc # 添加以下内容 export HADOOP_HOME/usr/local/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop export HADOOP_MAPRED_HOME$HADOOP_HOME export HADOOP_COMMON_HOME$HADOOP_HOME export HADOOP_HDFS_HOME$HADOOP_HOME export YARN_HOME$HADOOP_HOME使配置生效source ~/.bashrc验证安装hadoop version # 应输出Hadoop 3.3.6版本信息5. SSH免密登录配置集群节点间的SSH免密登录是Hadoop正常运行的基础。5.1 生成SSH密钥对在master节点执行su - hadoop ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys5.2 分发公钥到所有节点ssh-copy-id hadoopmaster ssh-copy-id hadoopstandby ssh-copy-id hadoopworker1 ssh-copy-id hadoopworker2 ssh-copy-id hadoopworker35.3 测试免密登录ssh standby # 应能直接登录而不需要密码 exit6. Hadoop集群配置6.1 修改hadoop-env.shvi $HADOOP_HOME/etc/hadoop/hadoop-env.sh # 设置JAVA_HOME export JAVA_HOME/usr/lib/jvm/java-1.8.0-openjdk6.2 配置core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/usr/local/hadoop/data/tmp/value /property property namehadoop.http.staticuser.user/name valuehadoop/value /property /configuration6.3 配置hdfs-site.xmlconfiguration property namedfs.namenode.name.dir/name value/usr/local/hadoop/data/namenode/value /property property namedfs.datanode.data.dir/name value/usr/local/hadoop/data/datanode/value /property property namedfs.replication/name value3/value /property property namedfs.namenode.secondary.http-address/name valuestandby:9868/value /property property namedfs.client.use.datanode.hostname/name valuetrue/value /property /configuration6.4 配置mapred-site.xmlconfiguration property namemapreduce.framework.name/name valueyarn/value /property property namemapreduce.jobhistory.address/name valuestandby:10020/value /property property namemapreduce.jobhistory.webapp.address/name valuestandby:19888/value /property property nameyarn.app.mapreduce.am.env/name valueHADOOP_MAPRED_HOME/usr/local/hadoop/value /property property namemapreduce.map.env/name valueHADOOP_MAPRED_HOME/usr/local/hadoop/value /property property namemapreduce.reduce.env/name valueHADOOP_MAPRED_HOME/usr/local/hadoop/value /property /configuration6.5 配置yarn-site.xmlconfiguration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.resourcemanager.hostname/name valuemaster/value /property property nameyarn.nodemanager.env-whitelist/name valueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value /property property nameyarn.log-aggregation-enable/name valuetrue/value /property /configuration6.6 配置workers文件vi $HADOOP_HOME/etc/hadoop/workers # 添加以下内容 worker1 worker2 worker37. 集群同步与启动7.1 同步配置到所有节点在master节点执行for node in standby worker1 worker2 worker3; do rsync -avz $HADOOP_HOME/ hadoop$node:$HADOOP_HOME/ ssh hadoop$node mkdir -p /usr/local/hadoop/data/{tmp,namenode,datanode} done7.2 格式化HDFS仅在第一次启动时执行hdfs namenode -format7.3 启动Hadoop集群# 在主节点启动HDFS start-dfs.sh # 在主节点启动YARN start-yarn.sh # 在备用节点启动历史服务器 mapred --daemon start historyserver7.4 验证集群状态检查各节点进程# 在master节点应看到 # NameNode # ResourceManager # 在standby节点应看到 # SecondaryNameNode # JobHistoryServer # 在工作节点应看到 # DataNode # NodeManager访问Web UIHDFS NameNode: http://master:9870YARN ResourceManager: http://master:8088JobHistory: http://standby:198888. 集群测试与验证8.1 运行测试作业# 创建HDFS目录 hdfs dfs -mkdir -p /user/hadoop/input # 上传测试文件 hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /user/hadoop/input # 运行WordCount示例 hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar \ wordcount /user/hadoop/input /user/hadoop/output # 查看结果 hdfs dfs -cat /user/hadoop/output/*8.2 检查集群健康状态hdfs dfsadmin -report # 应显示3个活动的DataNode yarn node -list # 应显示3个活动的NodeManager9. 常见问题排查问题1DataNode无法启动解决方案检查workers文件配置是否正确检查DataNode日志$HADOOP_HOME/logs/hadoop-hadoop-datanode-*.log确保所有节点的防火墙已关闭或相关端口已开放问题2YARN作业无法运行解决方案检查NodeManager日志验证yarn-site.xml配置检查资源分配是否合理问题3Web UI无法访问解决方案检查相关服务是否正常运行验证防火墙设置检查端口是否被占用提示所有节点的系统时间必须同步否则可能导致各种奇怪的问题。建议配置NTP服务sudo yum install -y ntp # CentOS sudo apt install -y ntp # Ubuntu sudo systemctl enable ntpd sudo systemctl start ntpd