Preface
I've wanted to play with K8s clusters for a long time, but always felt that without sufficient knowledge, it would be too difficult. Recently, I spent some time studying DN42 and routing protocols like BGP and OSPF, and found that I could now understand them without much difficulty. So I decisively started with K3s (
The main reason for choosing K3s over K8s is its lightweight nature: low resource requirements, no need to pull a bunch of images during deployment, availability of domestic mirrors... In short, K3s fits my needs better.
I'm a beginner just starting to explore K3s. Please go easy on me if there are any mistakes~
Network premise: This article assumes that readers have a basic understanding of BGP concepts (AS, iBGP, Route Reflector) and have a configurable BGP router environment. If you don't have these foundations, you can read my DN42 series articles first...?
Analysis
Choice of CNI Component
My current network architecture looks like this:
graph TD
subgraph ZeroTier Domestic
subgraph WDS
Gateway <--> VM1
Gateway <--> VM2
end
NGB <--> Gateway
HFE-NAS <--> Gateway
NGB <--> HFE-NAS
end
subgraph IEPL
Global-NIC <==OSPF==> CN-NIC
end
subgraph ZeroTier Global
HKG02 <--> HKG04
TYO <--> HKG04
TYO <--> HKG02
end
CN-NIC <--> NGB
CN-NIC <--> HFE-NAS
CN-NIC <--OSPF--> Gateway
Global-NIC <--OSPF--> TYO
Global-NIC <--OSPF--> HKG02
Global-NIC <--OSPF--> HKG04
%% Styling: orange background, bold border for routers
classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
class Global-NIC,CN-NIC,Gateway router;
Among these, the WDS node is a Proxmox VE hosting multiple VMs. It broadcasts the IPv4 prefixes of its VMs via OSPF. When a Hong Kong node needs to access a VM under the WDS node, it can do so by joining the OSPF internal network and achieving multi-hop reachability. This results in only one layer of encapsulation, so there's no worry about MTU shrinking.
I plan to create two new VMs under WDS as the master and one node (tentatively named KubeMaster and KubeNode-WDS1), and then also use HKG04 (tentatively named KubeNode-HKG04) as a node to join the K3s cluster.
The simplest way would be to use K3s's default Flannel as the CNI. However, Flannel is based on VXLAN, and adding it on top of my existing internal network would cause MTU shrinkage like this:
Data packet -> Flannel VXLAN encapsulation -> ZeroTier encapsulation -> physical link
The actual usable MTU for inter-container communication would be compressed to around 1350 or even lower. Therefore, I tried to find a CNI solution that can work directly on this internal network, and then I found Calico. From my understanding, Calico uses BGP as its underlying routing protocol and supports starting in No-Encapsulation mode, where data packets are directly routed by the upper-layer routers. So I chose Calico as the CNI component.
Routing Design
To ensure that routers at intermediate nodes know how to route Pod IPs, KubeMaster and KubeNode-WDS1 are under the Proxmox VE host. They need to establish BGP with HKG04 across the entire internal network. This means every intermediate router must learn the full BGP routes to enable a routing path like this:
graph LR
subgraph WDS
KubeMaster
KubeNode-WDS1
Gateway
end
subgraph IEPL
CN-Namespace
Global-Namespace
end
KubeNode-WDS1 <--> Gateway
KubeMaster <--> Gateway <--> CN-Namespace <--> Global-Namespace <--> HKG04
%% Styling: highlight nodes with routing capability
classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
class Gateway,CN-Namespace,Global-Namespace router;
Otherwise, any hop in the middle would drop packets because it doesn't recognize the source/destination IP. Additionally, due to the iBGP property that routes learned from an iBGP neighbor cannot be propagated to another iBGP neighbor, all BGP sessions between Gateway, CN-Namespace, Global-Namespace, and the nodes need to enable Route Reflector; otherwise, nodes cannot correctly learn each other's routes.
That said, this architecture is actually better suited for BGP Confederation. However, my existing network is already quite complex, and adding BGP Confederation would complicate maintenance later. Moreover, my number of nodes is small, so the overhead of iBGP Full Mesh is acceptable. Definitely not because I'm lazy (
So the final network routing structure looks like this:
graph TD
subgraph WDS
KubeMaster
KubeNode-WDS1
Gateway
end
subgraph IEPL
CN-Namespace
Global-Namespace
end
%% BGP logical connections
KubeMaster <-. Calico iBGP Full Mesh .-> KubeNode-WDS1
KubeMaster <-- iBGP Route Reflector --> Gateway
KubeNode-WDS1 <-- iBGP Route Reflector --> Gateway
Gateway <-- iBGP --> CN-Namespace
CN-Namespace <-- iBGP --> Global-Namespace
Global-Namespace <-- iBGP Route Reflector --> HKG04
%% Redundant and cross-domain connections
Gateway <-- iBGP --> Global-Namespace
HKG04 <-. Calico iBGP Full Mesh .-> KubeMaster
KubeNode-WDS1 <-. Calico iBGP Full Mesh .-> HKG04
%% Styling
classDef router fill:#f96,stroke:#333,stroke-width:2px,font-weight:bold;
%% Mark nodes with routing or RR functions as Router
class Gateway,CN-Namespace,Global-Namespace router;
The dashed BGP sessions are automatically created by Calico; the solid ones need to be manually configured.
Keeping Calico's own iBGP Full Mesh is for future scalability, allowing nodes to establish direct P2P connections via ZeroTier whenever possible, rather than routing through the Route Reflector aggregation router.
Deployment
After clarifying the structure, deployment becomes simple.
Enable Kernel Forwarding and Disable rp_filter
Standard practice.
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.forwarding=1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf
echo "net.ipv4.conf.default.rp_filter=0" >> /etc/sysctl.conf
echo "net.ipv4.conf.all.rp_filter=0" >> /etc/sysctl.conf
sysctl -p
Install K3s
Master
Since the KubeMaster control plane node is in China, it's best to configure image acceleration:
mkdir -p /etc/rancher/k3s
cat <<EOF > /etc/rancher/k3s/registries.yaml
mirrors:
docker.io:
endpoint:
- "https://docker.m.daocloud.io"
quay.io:
endpoint:
- "https://quay.m.daocloud.io"
EOF
Install using the mirror:
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | \
INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC=" \
--flannel-backend=none \
--disable-network-policy \
--cluster-cidr=10.42.0.0/16" sh -
Note that you need to specify --flannel-backend=none and --disable-network-policy to disable the default CNI components.
View the token with cat /var/lib/rancher/k3s/server/node-token and record it.
Worker Nodes
Configure image acceleration for nodes in China:
mkdir -p /etc/rancher/k3s
cat <<EOF > /etc/rancher/k3s/registries.yaml
mirrors:
docker.io:
endpoint:
- "https://docker.m.daocloud.io"
quay.io:
endpoint:
- "https://quay.m.daocloud.io"
EOF
Then install K3s using the mirror and join the cluster:
export INSTALL_K3S_MIRROR=cn
export K3S_URL=https://<master node IP>:6443 # Replace with your master node's actual IP
export K3S_TOKEN=K10...your token...::server:xxx # Replace with the full token from the first step
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | sh -
At this point, the nodes should be in NotReady state because the CNI component is missing.
Install Calico and Configure No-Encapsulation Mode
Download https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml on the master, then install the Calico operator:
kubectl create -f tigera-operator.yaml
Create a custom resource file custom-resource.yaml:
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Add image registry configuration
registry: quay.m.daocloud.io
calicoNetwork:
ipPools:
- blockSize: 26
cidr: 10.42.0.0/16
encapsulation: None
natOutgoing: Enabled
nodeSelector: all()
Here, encapsulation: None enables No-Encapsulation mode. You can also modify the IPv4 CIDR here. Then:
kubectl apply -f custom-resource.yaml
This executes the installation. Check pod status with:
kubectl get pods -A -o wide
Wait for each node to finish pulling images.
Configure BGP Topology
Label Nodes
Label nodes to specify that nodes under WDS should connect to the WDS Gateway's BGP, and overseas nodes should connect to the Global Namespace's BGP:
kubectl label nodes kubemaster region=WDS
kubectl label nodes kubenode-wds-1 region=WDS
kubectl label nodes kubenode-hkg04 region=Global
Calico Configuration
Write a YAML configuration file:
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: route-reflector-domestic
spec:
nodeSelector: region == 'Domestic' # This part isn't actually used; I originally designed a domestic aggregation router
peerIP: 100.64.0.108
asNumber: 64512
---
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: route-reflector-wds
spec:
nodeSelector: region == 'WDS'
peerIP: 192.168.100.1
asNumber: 64512
---
apiVersion: crd.projectcalico.org/v1
kind: BGPPeer
metadata:
name: route-reflector-global
spec:
nodeSelector: region == 'Global'
peerIP: 100.64.1.106
asNumber: 64512
This means:
- All nodes with label
region=Domesticget a BGP session to100.64.0.108(domestic aggregation router) using AS64512. - All nodes with label
region=WDSget a BGP session to192.168.100.1(gateway for all VMs under the WDS node) using AS64512. - All nodes with label
region=Globalget a BGP session to100.64.1.106(overseas aggregation router) using AS64512.
This achieves the diagram above: all VMs under WDS, including the master and KubeNode-WDS1, connect to the WDS Gateway aggregation router, and all nodes in the overseas region connect to the overseas aggregation router.
Configure Aggregation Router iBGP
This part is straightforward: just write Bird configuration files. Here are some examples:
k3s/ibgp.conf:
function is_insider_as(){
if bgp_path.len > 0 && !(bgp_path ~ [= 64512 =]) then {
return false;
}
if net ~ [ 10.42.0.0/16{16,32} ] then {
return true;
}
return false;
}
template bgp k3sbackbone{
local as K3S_AS;
router id INTRA_ROUTER_ID;
neighbor as K3S_AS;
ipv4{
table intra_table_v4;
import filter{
if is_insider_as() then accept;
reject;
};
export filter{
if is_insider_as() then accept;
reject;
};
next hop self;
extended next hop;
};
ipv6{
table intra_table_v6;
import filter{
if is_insider_as() then accept;
reject;
};
export filter{
if is_insider_as() then accept;
reject;
};
next hop self;
};
};
template bgp k3speers{
local as K3S_AS;
neighbor as K3S_AS;
router id INTRA_ROUTER_ID;
rr client;
rr cluster id INTRA_ROUTER_ID;
ipv4{
table intra_table_v4;
import filter{
if is_insider_as() then accept;
reject;
};
export filter{
if is_insider_as() then accept;
reject;
};
next hop self;
};
ipv6{
table intra_table_v6;
import filter{
if is_insider_as() then accept;
reject;
};
export filter{
if is_insider_as() then accept;
reject;
};
next hop self;
};
};
include "ibgpeers/*";
ibgpeers/backbone-cn.conf:
protocol bgp 'k3s_backbone_cn_v4' from k3sbackbone{
neighbor fd18:3e15:61d0:cafe:f001::1;
};
ibgpeers/master.conf:
protocol bgp 'k3s_master_v4' from k3speers{
neighbor 192.168.100.251;
};
The key points: aggregation routers should not enable Route Reflector among themselves, and remember to enable next hop self.
After everything is done, kubectl get nodes should show all nodes as Ready:
NAME STATUS ROLES AGE VERSION
kubemaster Ready control-plane 2d23h v1.34.5+k3s1
kubenode-hkg04 Ready <none> 11h v1.34.6+k3s1
kubenode-wds-1 Ready <none> 2d7h v1.34.5+k3s1
Check pods with kubectl get pods -A -o wide:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-system calico-kube-controllers-64fc874957-6bdlz 1/1 Running 0 5h38m 10.42.253.136 kubenode-hkg04 <none> <none>
calico-system calico-node-2qz82 1/1 Running 0 4h24m 10.2.5.7 kubenode-hkg04 <none> <none>
calico-system calico-node-dhl2c 1/1 Running 0 4h24m 192.168.100.251 kubemaster <none> <none>
calico-system calico-node-nbpkj 1/1 Running 0 4h23m 192.168.100.252 kubenode-wds-1 <none> <none>
calico-system calico-typha-7bb5db4bdc-rfpwg 1/1 Running 0 5h38m 10.2.5.7 kubenode-hkg04 <none> <none>
calico-system calico-typha-7bb5db4bdc-rwwr5 1/1 Running 0 5h38m 192.168.100.251 kubemaster <none> <none>
calico-system csi-node-driver-jglwp 2/2 Running 0 5h38m 10.42.64.68 kubenode-wds-1 <none> <none>
calico-system csi-node-driver-jqjsc 2/2 Running 0 5h38m 10.42.253.137 kubenode-hkg04 <none> <none>
calico-system csi-node-driver-vk26s 2/2 Running 0 5h38m 10.42.141.16 kubemaster <none> <none>
kube-system coredns-695cbbfcb9-8fx4p 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.14 kubemaster <none> <none>
kube-system helm-install-traefik-crd-5bkwx 0/1 Completed 0 2d23h <none> kubemaster <none> <none>
kube-system helm-install-traefik-m9fgj 0/1 Completed 1 2d23h <none> kubemaster <none> <none>
kube-system local-path-provisioner-546dfc6456-dmn4g 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.15 kubemaster <none> <none>
kube-system metrics-server-c8774f4f4-2wkwh 1/1 Running 1 (7h27m ago) 2d23h 10.42.141.12 kubemaster <none> <none>
kube-system svclb-traefik-999cddce-hpmcm 2/2 Running 6 (7h26m ago) 11h 10.42.253.134 kubenode-hkg04 <none> <none>
kube-system svclb-traefik-999cddce-q4225 2/2 Running 2 (7h27m ago) 2d22h 10.42.141.9 kubemaster <none> <none>
kube-system svclb-traefik-999cddce-xmd64 2/2 Running 2 (7h26m ago) 2d6h 10.42.64.66 kubenode-wds-1 <none> <none>
kube-system traefik-788bc4688c-vbbhj 1/1 Running 1 (7h27m ago) 2d22h 10.42.141.13 kubemaster <none> <none>
tigera-operator tigera-operator-6b95bbf4db-vl46l 1/1 Running 1 (7h27m ago) 2d23h 192.168.100.251 kubemaster <none> <none>
Use kubectl exec -it -n calico-system <calico-node-xxxx> -- birdcl s p to check Bird status:
root@KubeMaster:~/kube/calico# kubectl exec -it -n calico-system calico-node-2qz82 -- birdcl s p
Defaulted container "calico-node" out of: calico-node, flexvol-driver (init), install-cni (init)
BIRD v0.3.3+birdv1.6.8 ready.
name proto table state since info
static1 Static master up 08:58:17
kernel1 Kernel master up 08:58:17
device1 Device master up 08:58:17
direct1 Direct master up 08:58:17
Mesh_192_168_100_251 BGP master up 08:58:33 Established
Mesh_192_168_100_252 BGP master up 08:59:00 Established
Node_100_64_1_106 BGP master up 12:57:44 Established
ip r shows the system routing table:
root@KubeMaster:~/kube/calico# ip r
default via 192.168.100.1 dev eth0 proto static
10.42.64.64/26 proto bird
nexthop via 192.168.100.1 dev eth0 weight 1
nexthop via 192.168.100.252 dev eth0 weight 1
blackhole 10.42.141.0/26 proto bird
10.42.141.9 dev caliac6501d3794 scope link
10.42.141.12 dev calib07c23291bb scope link
10.42.141.13 dev caliab16e60bd19 scope link
10.42.141.14 dev calid5959219080 scope link
10.42.141.15 dev cali026d8f1ddb7 scope link
10.42.141.16 dev califa657ba417a scope link
10.42.253.128/26 via 192.168.100.1 dev eth0 proto bird
192.168.100.0/24 dev eth0 proto kernel scope link src 192.168.100.251
Ping a Pod IP; if everything is fine, it should work directly:
root@KubeMaster:~/kube/calico# ping 10.42.253.137
PING 10.42.253.137 (10.42.253.137) 56(84) bytes of data.
64 bytes from 10.42.253.137: icmp_seq=1 ttl=60 time=33.7 ms
64 bytes from 10.42.253.137: icmp_seq=2 ttl=60 time=33.5 ms
^C
--- 10.42.253.137 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 33.546/33.632/33.718/0.086 ms
Tune MTU
This step is actually for stability...?
Testing shows that although my ZeroTier MTU is 1420, packet fragmentation starts occurring around 1380 bytes (test with ping -M do -s <size> <Pod_IP>). Therefore, force the Pod MTU to 1370:
root@KubeMaster:~/kube/calico# cat patch-mtu.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
calicoNetwork:
mtu: 1370
nodeAddressAutodetectionV4:
firstFound: true
root@KubeMaster:~/kube/calico# kubectl apply -f patch-mtu.yaml
installation.operator.tigera.io/default configured
Comments (0)