sanyisheng
10/16/2019 - 7:45 AM

CentOS7安装hadoop-1.2.1

CentOS7安装hadoop-1.2.1

CentOS7安装hadoop-1.2.1

分布式环境:

masterslave1slave2
centos7centos7centos7
1c1c1c
2G1G1G
192.168.18.30192.168.18.31192.168.18.32

0.初始化三个节点

配置环境变量

  • 更改主机名分别为master、slave1、slave2
  • 配置ssh秘钥验证,达到任一节点无需密码验证登录其他站点root用户的效果
  • 配置/etc/hosts文件,使任一节点通过master、slave1、slave2即可域名解析到对应节点IP

关闭防火墙

[root@master ~]# vim /etc/selinux/config
...
SELINUX=disabled
...
[root@master ~]# setenforce 0
[root@master ~]# getenforce
Permissive
[root@master ~]# systemctl stop firewalld
[root@master ~]# systemctl disable firewalld
另外从节点也要执行

1.安装java

下载hadoop和jdk安装包到/usr/local/src目录下

https://archive.apache.org/dist/hadoop/core/hadoop-1.2.1/hadoop-1.2.1-bin.tar.gz https://repo.huaweicloud.com/java/jdk/6u45-b06/jdk-6u45-linux-x64.bin

[root@master src]# ls
hadoop-1.2.1-bin.tar.gz  jdk-6u45-linux-x64.bin

给予jdk-6u45-linux-x64.bin可执行权限,执行如下命令:

[root@master src]# ./jdk-6u45-linux-x64.bin

可得如下目录:

[root@master src]# ls -ld jdk1.6.0_45
drwxr-xr-x. 8 root root 176 3月  27 2013 jdk1.6.0_45

将该目录移动到/usr/local下(个人习惯):

[root@master jdk1.6.0_45]# pwd
/usr/local/jdk1.6.0_45

此目录即为java的安装目录,现在需要将java的相关变量声明命令添加到对于脚本中。在此选择添加到/etc/bashrc文件中。

[root@master jdk1.6.0_45]# tail -n 5 /etc/bashrc

export JAVA_HOME=/usr/local/jdk1.6.0_45
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

[root@master jdk1.6.0_45]# source /etc/bashrc
[root@master jdk1.6.0_45]# which java
/usr/local/jdk1.6.0_45/bin/java
[root@master jdk1.6.0_45]# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

如法炮制,在slave1和slave2节点上安装java

[root@slave1 ~]# which java
/usr/local/jdk1.6.0_45/bin/java
[root@slave2 ~]# which java
/usr/local/jdk1.6.0_45/bin/java

2.安装hadoop

解压安装包

[root@master src]# ls
hadoop-1.2.1-bin.tar.gz  jdk-6u45-linux-x64.bin
[root@master src]# tar zxf hadoop-1.2.1-bin.tar.gz
[root@master src]# ls
hadoop-1.2.1  hadoop-1.2.1-bin.tar.gz  jdk-6u45-linux-x64.bin

移动新生成的目录到/usr/local/

[root@master hadoop-1.2.1]# pwd
/usr/local/hadoop-1.2.1

生成存放运行过程中临时文件的tmp目录

[root@master hadoop-1.2.1]# mkdir tmp

进入conf目录下,修改配置文件:

[root@master conf]# pwd
/usr/local/hadoop-1.2.1/conf

修改masters文件,写入master节点的自定义域名

[root@master conf]# cat masters
master

修改slaves文件,写入slave节点的自定义域名

[root@master conf]# cat slaves
slave1
slave2

修改core-site.xml文件

[root@master conf]# vim core-site.xml
...
<configuration>
        <!--用来指定使用hadoop时产生文件的存放目录-->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/hadoop-1.2.1/tmp</value>
        </property>
        <!--指定namenode的地址-->
        <property>
                <name>fs.default.name</name>
                <value>hdfs://master:9000</value>
        </property>
</configuration>

修改mapred-site.xml文件

[root@master conf]# vim mapred-site.xml
...
<configuration>
        <!--作业跟踪管理器的HTTP服务器访问端口和地址-->
        <property>
                <name>mapred.job.tracker</name>
                <value>http://master:9001</value>
        </property>
</configuration>

修改hdfs-site.xml文件

[root@master conf]# vim hdfs-site.xml
...
<configuration>
        <!--指定hdfs保存数据的副本数量-->
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
</configuration>

修改hadoop-env.sh脚本

[root@master conf]# tail -n2 hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.6.0_45

将/usr/local/hadoop-1.2.1目录拷贝到另外两个从节点:

[root@master hadoop-1.2.1]# rsync -azvP /usr/local/hadoop-1.2.1 root@slave1:/usr/local/
[root@master hadoop-1.2.1]# rsync -azvP /usr/local/hadoop-1.2.1 root@slave2:/usr/local/

3.启动集群

初始化

[root@master bin]# pwd
/usr/local/hadoop-1.2.1/bin
[root@master bin]# ./hadoop namenode -format

启动

[root@master bin]# ./start-all.sh
[root@master bin]# jps
13144 JobTracker
12906 NameNode
13254 Jps
13067 SecondaryNameNode

尝试执行

[root@master bin]# ./hadoop fs -ls /
Found 1 items
drwxr-xr-x   - root supergroup          0 2019-02-06 14:51 /usr
[root@master bin]# ./hadoop fs -put /etc/passwd /
[root@master bin]# ./hadoop fs -ls /
Found 2 items
-rw-r--r--   3 root supergroup        892 2019-02-06 14:53 /passwd
drwxr-xr-x   - root supergroup          0 2019-02-06 14:51 /usr
[root@master bin]# ./hadoop fs -cat /passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
...