클라우드/AWS
[AWS] EKS backup (velero)
Cloud_Park
2023. 5. 13. 18:53
2023.05.13 - [클라우드/AWS] - [AWS] EKS EBS Snapshots
EBS 스냅샷의 기능도 있지만 클러스터 단위의 전체 백업이 필요하거나 네임스페이스 단위의 백업이 필요한 경우 velero를 사용하게 된다.
velero는 pv의 내용도 백업하기 때문에 EBS Snapshot 기능보다 백업 범위가 넓은 점 참고하여 테스트 진행해보자.
참고 (https://hanhorang31.github.io/post/pkos2-2-localstorage/)
생성 순서
1. s3 버킷 접근을 윈한 IAM USER ID 와 키 생성
aws s3 mb s3://<bucket-name> --region ap-northeast-22. 정책 추가
# 버킷 변수 설정
export BUCKET=<bucket-name>
# IAM Policy 생성
cat > velero-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVolumes",
"ec2:DescribeSnapshots",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:CreateSnapshot",
"ec2:DeleteSnapshot"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::${BUCKET}/*"
]
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::${BUCKET}"
]
}
]
}
EOF
# IAM Policy Attach
aws iam put-user-policy \
--user-name velero \
--policy-name velero \
--policy-document file://velero-policy.json
# IAM user 정보 가져오기
aws iam create-access-key --user-name velero
---------------------------------
{
"AccessKey": {
"UserName": "velero",
"AccessKeyId": "{ID}", # 밑의 credentials-velero ID에 저장
"Status": "Active",
"SecretAccessKey": "{KEY}", # 밑의 credentials-velero KEY에 저장
"CreateDate": "2023-03-16T04:31:23+00:00"
}
}
# credentials-velero 생성 및 IAM 정보 저장
cat << EOF > credentials-velero
[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
EOF3. velero 설치 (버전은 현재 날짜에 맞는 버전을 확인하여 진행한다.)
# arch 확인
uname -m
---------------------------------
x86_64
# velero CLI 설치
wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.6/velero-v1.9.6-linux-amd64.tar.gz
tar xzvf velero-v1.9.6-linux-amd64.tar.gz
cp velero-v1.9.6-linux-amd64/velero ~/bin
# CLI 확인
velero
---------------------------------
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.
If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.
export BUCKET=<bucket-name>
export REGION=ap-northeast-2
velero install \
--provider aws \
--bucket $BUCKET \
--secret-file ./credentials-velero \
--backup-location-config region=$REGION \
--use-volume-snapshots=false \
--plugins velero/velero-plugin-for-aws:v1.3.0 \
--use-restic
---------------------------------
...
Deployment/velero: created
DaemonSet/restic: attempting to create resource
DaemonSet/restic: attempting to create resource client
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
# Velero 확인
kubectl get all -n velero
NAME READY STATUS RESTARTS AGE
pod/restic-f5ngz 1/1 Running 0 38s
pod/restic-x9sk9 1/1 Running 0 37s
pod/velero-5f6657d4c8-jttxv 1/1 Running 0 38s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/restic 2 2 2 2 2 <none> 38s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/velero 1/1 1 1 38s
NAME DESIRED CURRENT READY AGE
replicaset.apps/velero-5f6657d4c8 1 1 1 38s
...
4. 백업
pod의 데이터의 백업을 원한다면 annotate을 등록하는 작업이 필요하다.
# 주석 추가 pod에 볼륨 정보를 추가
kubectl annotate pod/mysql-0 backup.velero.io/backup-volumes=data
# 백업
velero backup create mysql --include-namespaces default --wait
---------------------------------
Backup request "mysql" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
..................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe mysql` and `velero backup logs mysql`.
# 백업 목록 확인
velero get backup
---------------------------------
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
mysql Completed 0 0 2023-03-16 14:22:39 +0900 KST 29d default <none>
복원
#mysql 지우기
kubectl delete -f ./
kubectl delete pvc/<PVC 볼륨>
#velero 복원
velero restore create --from-backup mysql --wait
---------------------------------
Restore request "mysql-20230316155542" submitted successfully.
Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background.
...........
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe mysql-20230316155542` and `velero restore logs mysql-20230316155542`.
# 쿠버네티스 리소스 복원 확인
kubectl get all
---------------------------------
NAME READY STATUS RESTARTS AGE
pod/mysql-0 2/2 Running 0 39s
pod/mysql-1 0/2 Init:CrashLoopBackOff 2 (18s ago) 39s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 4h39m
service/mysql ClusterIP None <none> 3306/TCP 39s
service/mysql-read ClusterIP 100.69.52.194 <none> 3306/TCP 39s
kubectl get pv
---------------------------------
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-601b919a-cf20-4478-9f28-10d541c66844 10Gi RWO Delete Bound default/data-mysql-0 local-path 71s
pvc-b8a766a6-411f-47df-a548-d6b0ee091ea1 10Gi RWO Delete Bound default/data-mysql-1 local-path 70s
# Mysql data 확인
kubectl exec -it pod/mysql-0 -- /bin/bash
---------------------------------
Defaulted container "mysql" out of: mysql, xtrabackup, restic-wait (init), init-mysql (init), clone-mysql (init)
bash-4.2# mysql -u root -p
---------------------------------
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 73
Server version: 5.7.41-log MySQL Community Server (GPL)
mysql> use testdb;
---------------------------------
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
---------------------------------
mysql> select * from test;
+------+--------------------+
| name | testdata |
+------+--------------------+
| han | mysql example test |
+------+--------------------+
1 row in set (0.01 sec)스케줄 넣기
velero schedule create mysql-crontab --include-namespaces default --schedule="*/5 * *
복구 시 고려사항
파일시스템 레벨의 마이그레이션을 한다.
복원은 버전이 낮은 곳으로는 불가능하다.
마이그레이션시 crd와 같은 호환성을 사전에 체크해야된다.
CSP에 따라 리전간 마이그레이션은 불가능하다. (AWS, Azure)