참고: https://base-on.tistory.com/category/%ED%81%B4%EB%9D%BC%EC%9A%B0%EB%93%9C/AWS#google_vignette
가시다 님의 스터디에 참여한 수강생의 블로그를 참조하며, Amazon EKS STUDY 를 진행하고자 한다.
1. 실습 환경 아키텍처
bastion-ec2를 활용하여 Worke
- VPC 1 ea, Subnet (Public 3ea, Private 3ea)
- EKS 클러스터(Control Plane)
- 관리형 노드 그룹(EC2 3대)
- Add-on
2. CloudFormation을 활용한 실습 환경 배포
2.0 사전 수행 작업
- 관리자 권한을 가진 IAM User
- 관리자 권한을 가진 IAM Role
- IAM User의 AccessKey 및 SecretKey
용도
IAM User - Root 계정은 보안적으로 사용 지양하므로 IAM User를 사용
IAM Role - CloudFormation이 리소스 생성을 위해 사용하는 Role
AccessKey 및 SecretKey - 관리자 수준의 권한을 가진 IAM User의 액세스 키ID 입력
KeyPair - 작업용 bastion ec2에 SSH 접속을 위한 SSH 키페어 선택
위의 준비물은 미리 생성해둬야 한다.
CloudFormation 스택 배포 시 위의 정보를 사용해야하기 때문이다.
2.1 스택 생성
브라우저를 통해 아래의 링크를 접속하면 CloudFormation 스택 YAML을 다운받을 수 있다.
https://ap-northeast-2.console.aws.amazon.com/cloudformation/home?region=ap-northeast-2#/stacks/create?stackName=myeks&templateURL=https:%2F%2Fs3.ap-northeast-2.amazonaws.com%2Fcloudformation.cloudneta.net%2FK8S%2Feks-oneclick.yaml
CloudFormation yaml 내용
앞서 다운받은 CloudFormation 스택 YAML의 내용이다.
AWSTemplateFormatVersion: '2010-09-09'
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: "<<<<< Deploy EC2 >>>>>"
Parameters:
- KeyName
- MyIamUserAccessKeyID
- MyIamUserSecretAccessKey
- SgIngressSshCidr
- MyInstanceType
- LatestAmiId
- Label:
default: "<<<<< EKS Config >>>>>"
Parameters:
- ClusterBaseName
- KubernetesVersion
- WorkerNodeInstanceType
- WorkerNodeCount
- WorkerNodeVolumesize
- Label:
default: "<<<<< Region AZ >>>>>"
Parameters:
- TargetRegion
- AvailabilityZone1
- AvailabilityZone2
- AvailabilityZone3
- Label:
default: "<<<<< VPC Subnet >>>>>"
Parameters:
- VpcBlock
- PublicSubnet1Block
- PublicSubnet2Block
- PublicSubnet3Block
- PrivateSubnet1Block
- PrivateSubnet2Block
- PrivateSubnet3Block
Parameters:
KeyName:
Description: Name of an existing EC2 KeyPair to enable SSH access to the instances. Linked to AWS Parameter
Type: AWS::EC2::KeyPair::KeyName
ConstraintDescription: must be the name of an existing EC2 KeyPair.
MyIamUserAccessKeyID:
Description: IAM User - AWS Access Key ID (won't be echoed)
Type: String
NoEcho: true
MyIamUserSecretAccessKey:
Description: IAM User - AWS Secret Access Key (won't be echoed)
Type: String
NoEcho: true
SgIngressSshCidr:
Description: The IP address range that can be used to communicate to the EC2 instances
Type: String
MinLength: '9'
MaxLength: '18'
Default: 0.0.0.0/0
AllowedPattern: (\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})
ConstraintDescription: must be a valid IP CIDR range of the form x.x.x.x/x.
MyInstanceType:
Description: Enter t2.micro, t2.small, t2.medium, t3.micro, t3.small, t3.medium. Default is t2.micro.
Type: String
Default: t3.medium
AllowedValues:
- t2.micro
- t2.small
- t2.medium
- t3.micro
- t3.small
- t3.medium
LatestAmiId:
Description: (DO NOT CHANGE)
Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
AllowedValues:
- /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
ClusterBaseName:
Type: String
Default: myeks
AllowedPattern: "[a-zA-Z][-a-zA-Z0-9]*"
Description: must be a valid Allowed Pattern '[a-zA-Z][-a-zA-Z0-9]*'
ConstraintDescription: ClusterBaseName - must be a valid Allowed Pattern
KubernetesVersion:
Description: Enter Kubernetes Version, 1.23 ~ 1.26
Type: String
Default: 1.28
WorkerNodeInstanceType:
Description: Enter EC2 Instance Type. Default is t3.medium.
Type: String
Default: t3.medium
WorkerNodeCount:
Description: Worker Node Counts
Type: String
Default: 3
WorkerNodeVolumesize:
Description: Worker Node Volumes size
Type: String
Default: 30
TargetRegion:
Type: String
Default: ap-northeast-2
AvailabilityZone1:
Type: String
Default: ap-northeast-2a
AvailabilityZone2:
Type: String
Default: ap-northeast-2b
AvailabilityZone3:
Type: String
Default: ap-northeast-2c
VpcBlock:
Type: String
Default: 192.168.0.0/16
PublicSubnet1Block:
Type: String
Default: 192.168.1.0/24
PublicSubnet2Block:
Type: String
Default: 192.168.2.0/24
PublicSubnet3Block:
Type: String
Default: 192.168.3.0/24
PrivateSubnet1Block:
Type: String
Default: 192.168.11.0/24
PrivateSubnet2Block:
Type: String
Default: 192.168.12.0/24
PrivateSubnet3Block:
Type: String
Default: 192.168.13.0/24
Resources:
# VPC
EksVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcBlock
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-VPC
# PublicSubnets
PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone1
CidrBlock: !Ref PublicSubnet1Block
VpcId: !Ref EksVPC
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PublicSubnet1
- Key: kubernetes.io/role/elb
Value: 1
PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone2
CidrBlock: !Ref PublicSubnet2Block
VpcId: !Ref EksVPC
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PublicSubnet2
- Key: kubernetes.io/role/elb
Value: 1
PublicSubnet3:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone3
CidrBlock: !Ref PublicSubnet3Block
VpcId: !Ref EksVPC
MapPublicIpOnLaunch: true
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PublicSubnet3
- Key: kubernetes.io/role/elb
Value: 1
InternetGateway:
Type: AWS::EC2::InternetGateway
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref EksVPC
PublicSubnetRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PublicSubnetRouteTable
PublicSubnetRoute:
Type: AWS::EC2::Route
Properties:
RouteTableId: !Ref PublicSubnetRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PublicSubnet1RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet1
RouteTableId: !Ref PublicSubnetRouteTable
PublicSubnet2RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet2
RouteTableId: !Ref PublicSubnetRouteTable
PublicSubnet3RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet3
RouteTableId: !Ref PublicSubnetRouteTable
# PrivateSubnets
PrivateSubnet1:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone1
CidrBlock: !Ref PrivateSubnet1Block
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PrivateSubnet1
- Key: kubernetes.io/role/internal-elb
Value: 1
PrivateSubnet2:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone2
CidrBlock: !Ref PrivateSubnet2Block
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PrivateSubnet2
- Key: kubernetes.io/role/internal-elb
Value: 1
PrivateSubnet3:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Ref AvailabilityZone3
CidrBlock: !Ref PrivateSubnet3Block
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PrivateSubnet3
- Key: kubernetes.io/role/internal-elb
Value: 1
PrivateSubnetRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-PrivateSubnetRouteTable
PrivateSubnet1RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet1
RouteTableId: !Ref PrivateSubnetRouteTable
PrivateSubnet2RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet2
RouteTableId: !Ref PrivateSubnetRouteTable
PrivateSubnet3RouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet3
RouteTableId: !Ref PrivateSubnetRouteTable
# EKSCTL-Host
EKSEC2SG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: eksctl-host Security Group
VpcId: !Ref EksVPC
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-HOST-SG
SecurityGroupIngress:
- IpProtocol: '-1'
#FromPort: '22'
#ToPort: '22'
CidrIp: !Ref SgIngressSshCidr
EKSEC2:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref MyInstanceType
ImageId: !Ref LatestAmiId
KeyName: !Ref KeyName
Tags:
- Key: Name
Value: !Sub ${ClusterBaseName}-bastion-EC2
NetworkInterfaces:
- DeviceIndex: 0
SubnetId: !Ref PublicSubnet1
GroupSet:
- !Ref EKSEC2SG
AssociatePublicIpAddress: true
PrivateIpAddress: 192.168.1.100
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeType: gp3
VolumeSize: 30
DeleteOnTermination: true
UserData:
Fn::Base64:
!Sub |
#!/bin/bash
hostnamectl --static set-hostname "${ClusterBaseName}-bastion-EC2"
# Config Root account
echo 'root:qwe123' | chpasswd
sed -i "s/^#PermitRootLogin yes/PermitRootLogin yes/g" /etc/ssh/sshd_config
sed -i "s/^PasswordAuthentication no/PasswordAuthentication yes/g" /etc/ssh/sshd_config
rm -rf /root/.ssh/authorized_keys
systemctl restart sshd
# Config convenience
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/ec2-user/.bashrc
sed -i "s/UTC/Asia\/Seoul/g" /etc/sysconfig/clock
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
# Install Packages
yum -y install tree jq git htop
# Install kubectl & helm
cd /root
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.28.5/2024-01-04/bin/linux/amd64/kubectl
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
# Install eksctl
curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_Linux_amd64.tar.gz" | tar xz -C /tmp
mv /tmp/eksctl /usr/local/bin
# Install aws cli v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip >/dev/null 2>&1
./aws/install
complete -C '/usr/local/bin/aws_completer' aws
echo 'export AWS_PAGER=""' >>/etc/profile
export AWS_DEFAULT_REGION=${AWS::Region}
echo "export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION" >> /etc/profile
# Install YAML Highlighter
wget https://github.com/andreazorzetto/yh/releases/download/v0.4.0/yh-linux-amd64.zip
unzip yh-linux-amd64.zip
mv yh /usr/local/bin/
# Install krew
curl -L https://github.com/kubernetes-sigs/krew/releases/download/v0.4.4/krew-linux_amd64.tar.gz -o /root/krew-linux_amd64.tar.gz
tar zxvf krew-linux_amd64.tar.gz
./krew-linux_amd64 install krew
export PATH="$PATH:/root/.krew/bin"
echo 'export PATH="$PATH:/root/.krew/bin"' >> /etc/profile
# Install kube-ps1
echo 'source <(kubectl completion bash)' >> /root/.bashrc
echo 'alias k=kubectl' >> /root/.bashrc
echo 'complete -F __start_kubectl k' >> /root/.bashrc
git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1
cat <<"EOT" >> /root/.bashrc
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=false
function get_cluster_short() {
echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT
# Install krew plugin
kubectl krew install ctx ns get-all neat # ktop df-pv mtail tree
# Install Docker
amazon-linux-extras install docker -y
systemctl start docker && systemctl enable docker
# Create SSH Keypair
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa
# IAM User Credentials
export AWS_ACCESS_KEY_ID=${MyIamUserAccessKeyID}
export AWS_SECRET_ACCESS_KEY=${MyIamUserSecretAccessKey}
export AWS_DEFAULT_REGION=${AWS::Region}
export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID" >> /etc/profile
echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY" >> /etc/profile
echo "export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION" >> /etc/profile
echo "export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)" >> /etc/profile
# CLUSTER_NAME
export CLUSTER_NAME=${ClusterBaseName}
echo "export CLUSTER_NAME=$CLUSTER_NAME" >> /etc/profile
# K8S Version
export KUBERNETES_VERSION=${KubernetesVersion}
echo "export KUBERNETES_VERSION=$KUBERNETES_VERSION" >> /etc/profile
# VPC & Subnet
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" | jq -r .Vpcs[].VpcId)
echo "export VPCID=$VPCID" >> /etc/profile
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
echo "export PubSubnet1=$PubSubnet1" >> /etc/profile
echo "export PubSubnet2=$PubSubnet2" >> /etc/profile
echo "export PubSubnet3=$PubSubnet3" >> /etc/profile
export PrivateSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PrivateSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PrivateSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PrivateSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PrivateSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-PrivateSubnet3" --query "Subnets[0].[SubnetId]" --output text)
echo "export PrivateSubnet1=$PrivateSubnet1" >> /etc/profile
echo "export PrivateSubnet2=$PrivateSubnet2" >> /etc/profile
echo "export PrivateSubnet3=$PrivateSubnet3" >> /etc/profile
# Create EKS Cluster & Nodegroup
eksctl create cluster --name $CLUSTER_NAME --region=$AWS_DEFAULT_REGION --nodegroup-name=ng1 --node-type=${WorkerNodeInstanceType} --nodes ${WorkerNodeCount} --node-volume-size=${WorkerNodeVolumesize} --vpc-public-subnets "$PubSubnet1","$PubSubnet2","$PubSubnet3" --version ${KubernetesVersion} --ssh-access --ssh-public-key /root/.ssh/id_rsa.pub --with-oidc --external-dns-access --full-ecr-access --dry-run > myeks.yaml
sed -i 's/certManager: false/certManager: true/g' myeks.yaml
sed -i 's/ebs: false/ebs: true/g' myeks.yaml
cat <<EOT >> myeks.yaml
addons:
- name: vpc-cni # no version is specified so it deploys the default version
version: latest # auto discovers the latest available
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
configurationValues: |-
enableNetworkPolicy: "true"
- name: kube-proxy
version: latest
- name: coredns
version: latest
EOT
cat <<EOT > precmd.yaml
preBootstrapCommands:
- "yum install nvme-cli links tree tcpdump sysstat -y"
EOT
sed -i -n -e '/instanceType/r precmd.yaml' -e '1,$p' myeks.yaml
nohup eksctl create cluster -f myeks.yaml --verbose 4 --kubeconfig "/root/.kube/config" 1> /root/create-eks.log 2>&1 &
echo 'cloudinit End!'
Outputs:
eksctlhost:
Value: !GetAtt EKSEC2.PublicIp
2.2 스택 세부정보 지정
위의 CloudFormation 스택을 생성하면 아래의 파라미터 정보를 입력 받는다.
앞서 사전 작업으로 생성해둔 KeyPair, AccessKey 및 Secret Key 정보를 기입한다.
SgIngressSshCidr는 접속하고자 하는 사용자의 Public IP(ex_ A.B.C.D/32)를 기입하면 된다.
사용자 PC의 공인 IP 확인 방법
사용자 PC의 cmd 창에서 curl ipconfig.me 명령어를 통해 사용자 PC의 공인 IP를 확인한다.
2.2.1 EKS Config 변경사항
EKS Version
- 평균 4개월에 한 번씩 새로운 Kubernetes 마이너 버전(예: 1.30 )을 릴리스
- v1.28는 2024년 12월 EOL 되었고, v1.29는 2025년 3월 만료 예정
KuberenetesVersion의 기본값은 1.28이다. 버전이 만료되었으므로 v1.31으로 변경해서 실습 진행할 예정이다.
2.2.2 CloudFormation이 배포할 때 사용할 IAM Role 생성
- AdministratorAccess
테스트 용도니... Admin 권한을 줘버렸다..
2.3 스택 옵션 구성
앞서 생성한 IAM Role은 CloudFormation 스택 옵션 구성에서 권한 부분에 지정하면 된다.
2.4 배포 확인
총 스택은 4개의 스택이 배포된다.
이어서 각 스택별 용도를 설명하겠다.
2.4.1 myeks 스택 배포 확인
CloudFormation에서 배포되는 스택은 myeks 스택이다.
VPC, Subnet, Route Table, EC2(Bastion-ec2) 생성 정의가 되어있다.
Output 값으로 bastion-ec2 Public IP를 확인할 수 있다.
2.4.2 eksctl-myeks-cluster 스택 배포 확인
앞서 CloudFormation에 정의된 EC2 Userdata를 통해 배포된 eksctl-myeks-cluster 스택이다.
서버 내에서 eksctl 명령어를 실행하면 내부적으로는 CloudFormation 스택이 생성되어 배포된다.
eksctl을 통해 생성된 스택은 이름에 'eksctl-{cluster이름}- ~~'이 붙는다.
EKS Cluster, SG, IAM Role 생성 정의가 되어있다.
2.4.3 eksctl-myeks-addon-vpc-cni 스택 배포 확인
앞서 CloudFormation에 정의된 EC2 Userdata를 통해 배포된 eksctl-myeks-addon-vpc-cni 스택이다.
addon-vpc-cni 관련 IAM Role 생성 정의가 되어있다.
2.4.4 eksctl-myeks-nodegroup-ng1 스택 배포 확인
앞서 CloudFormation에 정의된 EC2 Userdata를 통해 배포된 eksctl-myeks-nodegroup-ng1 스택이다.
Nodegroup, LaunchTemplate, IAM Role, Policy, SG 생성 정의가 되어있다.
3. 배포 정상 확인
앞서 배포한 CloudFormation에서 초기화한 로그인 정보를 통해 접속한다.
ssh root@<bastion의 public-ip>
# id: root, pw: qwe123
MobaXterm을 통해 bastion-ec2에 접속했다. 개인별 편리한 ssh 접속 Tool을 사용하여 접속하면 되겠다.
UserData에 정의된 Kubectl, AWS CLI, Docker 설치 되었는지 확인하였다.
'세미나 & 교육 & Tech' 카테고리의 다른 글
Google Cloud Summit 2024 후기 (1) | 2024.07.06 |
---|---|
[교육] DevOps Engineering on AWS 3일차 (1) | 2024.06.24 |
[교육] DevOps Engineering on AWS 2일차 (0) | 2024.06.24 |
[교육] DevOps Engineering on AWS 1일차 (0) | 2024.06.24 |
[세미나] RedHat container Day 2024 (0) | 2024.03.16 |