安装伪集群版Hadoop-HDFS

发表于 | 分类于

网上搜了一些中文资料，大多前言不搭后语，按照步骤做了几次都失败了。现在记录下成功记录。

官方文档

尽量参照官方文档

安装Java

安装Java-1.8

1
$ apt install -y openjdk-8-jre-headless openjdk-8-jdk-headless

配置JAVA_HOME

1
2
$ echo 'export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64' >> ~/.bashrc
$ . ~/.bashrc

配置SSH Key & Host

配置hostname

1
2
$ echo hadoop > /etc/hostname
$ hostname hadoop

配置sshKey

1
2
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

下载Hadoop并解压

1
2
3
$ wget 'https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz'
$ tar xvf hadoop-3.3.0.tar.gz -C /usr/local
$ ln -s /usr/local/hadoop-3.3.0 /usr/local/hadoop

修改配置文件

etc/hadoop/core-site.xml

1
2
3
4
5
6
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml

1
2
3
4
5
6
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

etc/hadoop/hadoop-env.sh

1
2
3
4
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root

启动HDFS

1
2
3
4
5
6
7
8
# format filesystem
$ bin/hdfs namenode -format

# start namenode & datanode
$ sbin/start-dfs.sh

# test
$ curl http://localhost:9870