VPC 설정하고, route table은 자동으로 붙는데, internet gateway는 안붙는다. 하나 만들어서 route table과 붙여줘야 ssh connect가 된다.

1. Docker 설치

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release

# Add Docker's official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up the stable repository
echo \\
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] <https://download.docker.com/linux/ubuntu> \\
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Login to Docker
docker login -u chayhyeon
# enter password

2. 미리 만들어둔 이미지 pull

하려 했는데.. 이미지가 날라가버림 ㅋㅋ; 어쩔 수 없이 W3M2a에서 했던 이미지를 일단 그대로 가져와봐야겠다ㅠ

sudo docker pull chayhyeon/softeer:w3m2

# 열어둔 포트가 많아서 바인딩도 많이 해줘야 한다
# 네임노드는 네임노드 볼륨만, 데이터노드는 데이터노드 볼륨만
sudo docker run -it -d \\
	-v namenode_volume:/mnt/dfs/name \\
	-v datanode_volume:/mnt/dfs/data \\
	-p 19888:19888 \\
	-p 10020:10020 \\
	-p 9870:9870 \\
	-p 9866:9866 \\
	-p 9864:9864 \\
	-p 9000:9000 \\
	-p 8088:8088 \\
	-p 8042:8042 \\
	-p 8040:8040 \\
	-p 8032:8032 \\
	-p 8030:8030 \\
	--name hadoop_w3m2_ec2 \\
	chayhyeon/softeer:w3m2

<!-- core-site.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.31.0.xx:9000</value>
    </property>
</configuration>

<!-- hdfs-site.xml for namenode -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>0.0.0.0:9870</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address</name>
        <value>0.0.0.0:9000</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///mnt/dfs/name</value>
    </property>
</configuration>

<!-- hdfs-site.xml for datanode -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.datanode.http-address</name>
        <value>0.0.0.0:9864</value>
    </property>
    <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:9866</value>
    </property>
    <property>
        <name>dfs.datanode.ipc.address</name>
        <value>0.0.0.0:9867</value>
    </property>
    <property>
        <name>dfs.datanode.name.dir</name>
        <value>file:///mnt/dfs/data</value>
    </property>
</configuration>

<!-- mapred-site.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.application.classpath</name>
        <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
</configuration>

<!-- yarn-site.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>172.31.0.9</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>0.0.0.0:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>0.0.0.0:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>0.0.0.0:8030</value>
    </property>
    <property>
        <name>yarn.nodemanager.localizer.address</name>
        <value>0.0.0.0:8040</value>
    </property>
    <property>
        <name>yarn.nodemanager.webapp.address</name>
        <value>0.0.0.0:8042</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

야심차게 빌드하고 pull하는 순간!

failed to register layer: write /workspace/hadoop-3.4.1/share/hadoop/tools/lib/bundle-2.24.6.jar: no space left on device

디스크 용량이 부족합니다… 돈을 더 내세요…. 이런……

다음에 꼭 다시 돌아올거야 긴장해