Calico 설치 시 CustomResourceDefinition 오류 해결

발생환경

• 운영체제 : Amazon Linux 2023
• 쿠버네티스 엔터프라이즈 2/e, ch2, page 93 - 사용자 지정 KinD 만들기 중 Calico 설치 중

git clone https://github.com/AcornPublishing/kubernetes-enterprise.git

# kubernetes-enterprise/chapter2 폴더로 이동
cd ~/kubernetes-enterprise/chapter2

# cluster01-kind.yaml 파일이 있는지 확인
ls cluster01-kind.yaml

# 클러스터 생성
kind create cluster --name cluster01 --config cluster01-kind.yaml

# 노드 상태 확인
kubectl get nodes

# 워커 노드에 라벨 추가 (인그레스 컨트롤러용)
kubectl label node cluster01-worker ingress-ready=true

# Calico Operator 설치
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml

# Error 발생 지점
kubectl create -f calico.yaml

 

 

발생한 오류 메시지

resource mapping not found for name: "felixconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first

[ec2-user@ip-10-0-3-220 chapter2]$ kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
namespace/tigera-operator created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created
[ec2-user@ip-10-0-3-220 chapter2]$ ls calico.yaml
calico.yaml
[ec2-user@ip-10-0-3-220 chapter2]$ kubectl create -f calico.yaml
configmap/calico-config created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
Warning: spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead
Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
resource mapping not found for name: "felixconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamblocks.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "blockaffinities.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamhandles.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ipamconfigs.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "bgppeers.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "bgpconfigurations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "ippools.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "hostendpoints.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "clusterinformations.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "globalnetworkpolicies.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "globalnetworksets.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "networkpolicies.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "networksets.crd.projectcalico.org" namespace: "" from "calico.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first

 

 

 

오류 원인

1. API 버전 불일치

• 레포지토리의 calico.yaml은 apiextensions.k8s.io/v1beta1 API 버전을 사용
• 현재 쿠버네티스는 이 버전을 더 이상 지원하지 않고 v1을 사용

 

2. 오래된 Calico 매니페스트

• 레포지토리에 있는 매니페스트가 최신 쿠버네티스 버전과 호환되지 않는 옛 버전

https://github.com/AcornPublishing/kubernetes-enterprise/blob/main/chapter2/calico.yaml

 

해결 방안

# 직접 설치한 매니페스트 제거
kubectl delete -f calico.yaml

# Operator 제거
kubectl delete -f https://docs.projectcalico.org/manifests/tigera-operator.yaml

# tigera-operator 네임스페이스 제거 확인
kubectl get namespace | grep tigera
# calico-system 네임스페이스 제거 확인
kubectl get namespace | grep calico
# Calico 관련 CRD 제거 확인
kubectl get crds | grep projectcalico
# Tigera Operator 관련 CRD 제거 확인
kubectl get crds | grep tigera
# 모든 네임스페이스에서 calico 관련 파드 제거 확인
kubectl get pods -A | grep calico
# 모든 네임스페이스에서 tigera 관련 파드 제거 확인
kubectl get pods -A | grep tigera
# Calico 관련 클러스터롤 제거 확인
kubectl get clusterrole | grep calico
# Calico 관련 클러스터롤바인딩 제거 확인
kubectl get clusterrolebinding | grep calico
# Tigera 관련 클러스터롤과 바인딩 제거 확인
kubectl get clusterrole | grep tigera
kubectl get clusterrolebinding | grep tigera

# 최신 Calico 매니페스트 다운로드
curl https://raw.githubusercontent.com/projectcalico/calico/v3.25.0/manifests/calico.yaml -O

# 매니페스트 적용
kubectl apply -f calico.yaml

# 정상 동작 확인
kubectl get pods -n kube-system | grep calico

 

 

•