An Experience of Manually Installing Proxmox VE, Configuring Multipath iSCSI, and NAT Forwarding The reason was that I rented a physical server, but the IDC did not provide Proxmox VE or Debian system images, only Ubuntu, CentOS, and Windows series. Additionally, the data disk was provided via multipath iSCSI. I wanted to use PVE for isolating different usage scenarios, so I attempted to reinstall the system and migrate the aforementioned configurations. Backup Configuration First, perform a general check of the system, which reveals: The system has two Network Interfaces: enp24s0f0 is connected to a public IP address for external access; enp24s0f1 is connected to the private network address 192.168.128.153. The data disk is mapped to /dev/mapper/mpatha. Under /etc/iscsi, there are configurations for two iSCSI Nodes: 192.168.128.250:3260 and 192.168.128.252:3260, both corresponding to the same target iqn.2024-12.com.ceph:iscsi. It can be inferred that the data disk is mounted by configuring two iSCSI Nodes and then merging them into a single device using multipath. Check the system's network configuration: network: version: 2 renderer: networkd ethernets: enp24s0f0: addresses: [211.154.[REDACTED]/24] routes: - to: default via: [REDACTED] match: macaddress: ac:1f:6b:0b:e2:d4 set-name: enp24s0f0 nameservers: addresses: - 114.114.114.114 - 8.8.8.8 enp24s0f1: addresses: - 192.168.128.153/17 match: macaddress: ac:1f:6b:0b:e2:d5 set-name: enp24s0f1 It's found to be very simple static routing. The internal network interface doesn't even have a default route; just binding the IP is sufficient. Then, save the iSCSI configuration files from /etc/iscsi, which include account and password information. Reinstall Debian Used the bin456789/reinstall script for this reinstallation. Download the script: curl -O https://cnb.cool/bin456789/reinstall/-/git/raw/main/reinstall.sh || wget -O ${_##*/} $_ Reinstall as Debian 13 (Trixie): bash reinstall.sh debian 13 Then, enter the password you want to set as prompted. If all goes well, wait about 10 minutes, and it will automatically complete and reinstall into a clean Debian 13. You can connect via SSH during the process using the set password to check the installation progress. After reinstalling, perform a source change and apt upgrade as usual to get a clean Debian 13. For changing sources, directly refer to the USTC Mirror Site tutorial. Install Proxmox VE This step mainly refers to the Proxmox official tutorial. Note: The Debian installed by the above script sets the hostname to localhost. If you want to change it, please modify it before configuring the Hostname and change the hostname in hosts to your modified hostname, not localhost. Configure Hostname Proxmox VE requires the current hostname to be resolvable to a non-loopback IP address: The hostname of your machine must be resolvable to an IP address. This IP address must not be a loopback one like 127.0.0.1 but one that you and other hosts can connect to. For example, my server IP is 211.154.[CENSORED], I need to add the following record in /etc/hosts: 127.0.0.1 localhost +211.154.[CENSORED] localhost ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters After saving, use hostname --ip-address to check if it outputs the set non-loopback address: ::1 127.0.0.1 211.154.[CENSORED]. Add Proxmox VE Software Repository Debian 13 uses the Deb822 format (though you can use sources.list if you want), so just refer to the USTC Proxmox Mirror Site: cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF Types: deb URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Here, a keyring needs to be migrated but I couldn't find one after searching online, so I chose to pull a copy from an existing Proxmox VE server. It's available here: proxmox-keyrings.zip Extract the public key file and place it in /usr/share/keyrings/, then run: apt update apt upgrade -y This will sync the Proxmox VE software repository. Install Proxmox VE Kernel Use the following command to install the PVE kernel and reboot to apply the new kernel: apt install proxmox-default-kernel reboot Afterwards, uname -r should show a kernel version ending with pve, like 6.17.2-2-pve, indicating the new kernel is successfully applied. Install Proxmox VE Related Packages Use apt to install the corresponding packages: apt install proxmox-ve postfix open-iscsi chrony During configuration, you will need to set up the postfix mail server. Official explanation: If you have a mail server in your network, you should configure postfix as a satellite system. Your existing mail server will then be the relay host which will route the emails sent by Proxmox VE to their final recipient. If you don't know what to enter here, choose local only and leave the system name as is. After this, you should be able to access the Web console at https://<your server address>:8006. The account is root, and the password is your root password, i.e., the password configured during the Debian reinstallation. Remove Old Debian Kernel and os-prober Use the following commands: apt remove linux-image-amd64 'linux-image-6.1*' update-grub apt remove os-prober to remove the old Debian kernel, update grub, and remove os-prober. Removing os-prober is not mandatory, but it is recommended by the official guide because it might mistakenly identify VM boot files as multi-boot files, adding incorrect entries to the boot list. At this point, the installation of Proxmox VE is complete and ready for normal use! Configuring Internal Network Interface Because the iSCSI network interface and the public network interface are different, and the reinstallation lost this configuration, the internal network interface needs to be manually configured. Open the Proxmox VE Web interface, go to Datacenter - localhost (hostname) - Network, edit the internal network interface (e.g., ens6f1 here), enter the backed-up IPv4 in CIDR format: 192.168.128.153/17, and check Autostart, then save. Then use the command to set the interface state to UP: ip link set ens6f1 up Now you should be able to ping the internal iSCSI server's IP. Configure Data Disk iSCSI In the previous step, we should have installed the open-iscsi package required for iscsiadm. We just need to reset the nodes according to the backed-up configuration. First, discover the iSCSI storage: iscsiadm -m discovery -t st -p 192.168.128.250:3260 This should yield the two original LUN Targets: 192.168.128.250:3260,1 iqn.2024-12.com.ceph:iscsi 192.168.128.252:3260,2 iqn.2024-12.com.ceph:iscsi Transfer the backed-up configuration files to the server, overwriting the existing configuration in /etc/iscsi. Also, in my backed-up config, I found the authentication configuration: # /etc/iscsi/nodes/iqn.2024-12.com.ceph:iscsi/192.168.128.250,3260,1/default # BEGIN RECORD 2.1.5 node.name = iqn.2024-12.com.ceph:iscsi ... # Some unimportant configurations omitted node.session.auth.authmethod = CHAP node.session.auth.username = [CENSORED] node.session.auth.password = [CENSORED] node.session.auth.chap_algs = MD5 ... # Some unimportant configurations omitted # /etc/iscsi/nodes/iqn.2024-12.com.ceph:iscsi/192.168.128.252,3260,2/default # BEGIN RECORD 2.1.5 node.name = iqn.2024-12.com.ceph:iscsi ... # Some unimportant configurations omitted node.session.auth.authmethod = CHAP node.session.auth.username = [CENSORED] node.session.auth.password = [CENSORED] node.session.auth.chap_algs = MD5 ... # Some unimportant configurations omitted Write these configurations to the new system using: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.authmethod -v CHAP iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.username -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.password -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.chap_algs -v MD5 iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.authmethod -v CHAP iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.username -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.session.auth.password -v [CENSORED] iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.session.auth.chap_algs -v MD5 (I don't know why the auth info needs to be written separately, but testing shows it won't log in without rewriting it.) Then, use: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 --login iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 --login to log into the Targets. Then use: iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.250:3260 -o update -n node.startup -v automatic iscsiadm -m node -T iqn.2024-12.com.ceph:iscsi -p 192.168.128.252:3260 -o update -n node.startup -v automatic to enable automatic mounting on boot. At this point, checking disks with tools like lsblk should reveal two additional hard drives; in my case, sdb and sdc appeared. Configure Multipath To identify if it's a multipath device, I tried: /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdb /usr/lib/udev/scsi_id --whitelisted --device=/dev/sdc Checking the scsi_id of the two disk devices revealed they were identical, confirming they are the same disk using multi-path for load balancing and failover. Install multipath-tools using apt: apt install multipath-tools Then, create /etc/multipath.conf and add: defaults { user_friendly_names yes find_multipaths yes } Configure multipathd to start on boot: systemctl start multipathd systemctl enable multipathd Then, use the following command to scan and automatically configure the multipath device: multipath -ll It should output: mpatha(360014056229953ef442476e85501bfd7)dm-0LIO-ORG,TCMU device size=500G features='1 queue_if_no_path' hwhandler='1 alua'wp=rw |-+- policy='service-time 0' prio=50 status=active | `- 14:0:0:152 sdb 8:16 active ready running `-+- policy='service-time 0' prio=50 status=active `- 14:0:0:152 sdc 8:16 active ready running This shows the two disks have been recognized as a single multipath device. Now, you can find the multipath disk under /dev/mapper/: root@localhost:/dev/mapper# ls control mpatha mpatha is the multipath aggregated disk. If it's not scanned, try using: rescan-scsi-bus.sh to rescan the SCSI bus and try again. If the command is not found, install it via apt install sg3-utils. If all else fails, just reboot. Configure Proxmox VE to Use the Data Disk Because we used multipath, we cannot directly add an iSCSI type storage. Use the following commands to create the PV and VG: pvcreate /dev/mapper/mpatha vgcreate <vg name> /dev/mapper/mpatha Here, I configured the entire disk as a PV. You could also create a separate partition for this. After completion, open the Proxmox VE management interface, go to Datacenter - Storage, click Add - LVM, select the name of the VG you just created for Volume group, give it an ID (name), and click Add. At this point, all configurations from the original system should have been migrated. Configure NAT and Port Forwarding NAT Because only one IPv4 address was purchased, NAT needs to be configured to allow all VMs to access the internet normally. Open /etc/network/interfaces and add the following content: auto vmbr0 iface vmbr0 inet static address 192.168.100.1 netmask 255.255.255.0 bridge_ports none bridge_stp off bridge_fd 0 post-up echo 1 > /proc/sys/net/ipv4/ip_forward post-up iptables -t nat -A POSTROUTING -s 192.168.100.0/24 -o ens6f0 -j MASQUERADE post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-up iptables -A FORWARD -i vmbr0 -j ACCEPT post-down iptables -t nat -D POSTROUTING -s 192.168.100.0/24 -o ens6f0 -j MASQUERADE post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -D FORWARD -i vmbr0 -j ACCEPT Here, vmbr0 is the NAT bridge, with the IP segment 192.168.100.0/24. Traffic from this segment will be translated to the IP of the external network interface ens6f0 for outgoing traffic, and translated back to the original IP upon receiving replies, enabling IP sharing. Then, use: ifreload -a to reload the configuration. Now, the VMs should be able to access the internet. Just configure a static IP within the 192.168.100.0/24 range during installation, set the default gateway to 192.168.100.1, and configure the DNS address. Port Forwarding Got lazy, directly prompted an AI. Had an AI write a configuration script /usr/local/bin/natmgr: #!/bin/bash # =================Configuration Area================= # Public network interface name (Please modify according to your actual situation) PUB_IF="ens6f0" # ==================================================== ACTION=$1 ARG1=$2 ARG2=$3 ARG3=$4 ARG4=$5 # Check if running as root if [ "$EUID" -ne 0 ]; then echo "Please run this script with root privileges" exit 1 fi # Generate random ID (6 characters) generate_id() { # Introduce nanoseconds and random salt to ensure ID uniqueness even if the script runs quickly echo "$RANDOM $(date +%s%N)" | md5sum | head -c 6 } # Show help information usage() { echo "Usage: $0 {add|del|list|save} [parameters]" echo "" echo "Commands:" echo " add <Public Port> <Internal IP> <Internal Port> [Protocol] Add forwarding rule" echo " [Protocol] optional: tcp, udp, both (default: both)" echo " del <ID> Delete forwarding rule by ID" echo " list View all current forwarding rules" echo " save Save current rules to persist after reboot (Must run!)" echo "" echo "Examples:" echo " $0 add 8080 192.168.100.101 80 both" echo " $0 save" echo "" } # Internal function: add single protocol rule _add_single_rule() { local PROTO=$1 local L_PORT=$2 local T_IP=$3 local T_PORT=$4 local RULE_ID=$(generate_id) local COMMENT="NAT_ID:${RULE_ID}" # 1. Add DNAT rule (PREROUTING chain) iptables -t nat -A PREROUTING -i $PUB_IF -p $PROTO --dport $L_PORT -j DNAT --to-destination $T_IP:$T_PORT -m comment --comment "$COMMENT" # 2. Add FORWARD rule (Allow packet passage) iptables -A FORWARD -p $PROTO -d $T_IP --dport $T_PORT -m comment --comment "$COMMENT" -j ACCEPT # Output result printf "%-10s %-10s %-10s %-20s %-10s\n" "$RULE_ID" "$PROTO" "$L_PORT" "$T_IP:$T_PORT" "Success" # Remind user to save echo "Please run '$0 save' to ensure rules persist after reboot." } # Main add function add_rule() { local L_PORT=$1 local T_IP=$2 local T_PORT=$3 local PROTO_REQ=${4:-both} # Default to both if [[ -z "$L_PORT" || -z "$T_IP" || -z "$T_PORT" ]]; then echo "Error: Missing parameters" usage exit 1 fi # Convert to lowercase PROTO_REQ=$(echo "$PROTO_REQ" | tr '[:upper:]' '[:lower:]') echo "Adding rule..." printf "%-10s %-10s %-10s %-20s %-10s\n" "ID" "Protocol" "Public Port" "Target Address" "Status" echo "------------------------------------------------------------------" if [[ "$PROTO_REQ" == "tcp" ]]; then _add_single_rule "tcp" "$L_PORT" "$T_IP" "$T_PORT" elif [[ "$PROTO_REQ" == "udp" ]]; then _add_single_rule "udp" "$L_PORT" "$T_IP" "$T_PORT" elif [[ "$PROTO_REQ" == "both" ]]; then _add_single_rule "tcp" "$L_PORT" "$T_IP" "$T_PORT" _add_single_rule "udp" "$L_PORT" "$T_IP" "$T_PORT" else echo "Error: Unsupported protocol '$PROTO_REQ'. Please use tcp, udp, or both." exit 1 fi echo "------------------------------------------------------------------" } # Delete rule (Delete in reverse line number order) del_rule() { local RULE_ID=$1 if [[ -z "$RULE_ID" ]]; then echo "Error: Please provide rule ID" usage exit 1 fi echo "Searching for rule with ID [${RULE_ID}]..." local FOUND=0 # --- Clean NAT table (PREROUTING) --- LINES=$(iptables -t nat -nL PREROUTING --line-numbers | grep "NAT_ID:${RULE_ID}" | awk '{print $1}' | sort -rn) if [[ ! -z "$LINES" ]]; then for line in $LINES; do iptables -t nat -D PREROUTING $line echo "Deleted NAT table PREROUTING chain line $line" FOUND=1 done fi # --- Clean Filter table (FORWARD) --- LINES=$(iptables -t filter -nL FORWARD --line-numbers | grep "NAT_ID:${RULE_ID}" | awk '{print $1}' | sort -rn) if [[ ! -z "$LINES" ]]; then for line in $LINES; do iptables -t filter -D FORWARD $line echo "Deleted Filter table FORWARD chain line $line" FOUND=1 done fi if [[ $FOUND -eq 0 ]]; then echo "No rule found with ID $RULE_ID." else echo "Delete operation completed." echo "Please run '$0 save' to update the persistent configuration file." fi } # Save rules to disk (New feature) save_rules() { echo "Saving current iptables rules..." # netfilter-persistent is the service managing iptables-persistent in Debian/Proxmox if command -v netfilter-persistent &> /dev/null; then netfilter-persistent save if [ $? -eq 0 ]; then echo "✅ Rules successfully saved to /etc/iptables/rules.v4, will be automatically restored after system reboot." else echo "❌ Failed to save rules. Please check the status of the 'netfilter-persistent' service." fi else echo "Warning: 'netfilter-persistent' command not found." echo "Please ensure the 'iptables-persistent' package is installed." echo "Install command: apt update && apt install iptables-persistent" fi } # List rules list_rules() { echo "Current Port Forwarding Rules List:" printf "%-10s %-10s %-10s %-20s %-10s\n" "ID" "Protocol" "Public Port" "Target Address" "Target Port" echo "------------------------------------------------------------------" # Parse iptables output iptables -t nat -nL PREROUTING -v | grep "NAT_ID:" | while read line; do id=$(echo "$line" | grep -oP '(?<=NAT_ID:)[^ ]*') # Extract protocol if echo "$line" | grep -q "tcp"; then proto="tcp"; else proto="udp"; fi # Extract port after dpt: l_port=$(echo "$line" | grep -oP '(?<=dpt:)[0-9]+') # Extract IP:Port after to: target=$(echo "$line" | grep -oP '(?<=to:).*') t_ip=${target%:*} t_port=${target#*:} printf "%-10s %-10s %-10s %-20s %-10s\n" "$id" "$proto" "$l_port" "$t_ip" "$t_port" done } # Main logic case "$ACTION" in add) add_rule "$ARG1" "$ARG2" "$ARG3" "$ARG4" ;; del) del_rule "$ARG1" ;; list) list_rules exit 0 ;; save) save_rules ;; *) usage exit 1 ;; esac save_rules This script automatically adds/deletes iptables rules for port forwarding. Remember to chmod +x. Use iptables-persistent to save the configuration and load it automatically on boot: apt install iptables-persistent During configuration, you will be asked whether to save the current rules; Yes or No is fine. When adding a forwarding rule, use natmgr add <host listen port> <VM internal IP> <VM port> [tcp/udp/both]. The script will automatically assign a unique ID. Use natmgr del <ID> to delete. Use natmgr list to view the existing forwarding list. Reference Articles: bin456789/reinstall: 一键DD/重装脚本 (One-click reinstall OS on VPS) - GitHub Install Proxmox VE on Debian 12 Bookworm - Proxmox VE PVE连接 TrueNAS iSCSI存储实现本地无盘化_pve iscsi-CSDN博客 ProxmoxVE (PVE) NAT 网络配置方法 - Oskyla 烹茶室

DN42&OneManISP - Troubleshooting OSPF Source Address in a Coexistence Environment Backstory As mentioned in the previous post of this series, because the VRF solution was too isolating, the DNS service I deployed on the HKG node (172.20.234.225) became inaccessible from the DN42 network. Research indicated this could be achieved by setting up veth or NAT forwarding, but due to the scarcity of available documentation, I ultimately abandoned the VRF approach. Structure Analysis This time, I planned to place both DN42 and clearnet BGP routes into the system's main routing table, then separate them for export using filters to distinguish which should be exported. For clarity, I stored the configuration for the DN42 part and the clearnet part (hereinafter referred to as inet) separately, and then included them from the main configuration file. Also, since there should ideally only be one kernel configuration per routing table, I merged the DN42 and inet kernel parts, keeping only one instance. After multiple optimizations and revisions, my final directory structure is as follows: /etc/bird/ ├─envvars ├─bird.conf: Main Bird config file, defines basic info (ASN, IP, etc.), includes sub-configs below ├─kernel.conf: Kernel config, imports routes into the system routing table ├─dn42 | ├─defs.conf: DN42 function definitions, e.g., is_self_dn42_net() | ├─ibgp.conf: DN42 iBGP template | ├─rpki.conf: DN42 RPKI route validation | ├─ospf.conf: DN42 OSPF internal network | ├─static.conf: DN42 static routes | ├─ebgp.conf: DN42 Peer template | ├─ibgp | | └<ibgp configs>: DN42 iBGP configs for each node | ├─ospf | | └backbone.conf: OSPF area | ├─peers | | └<ibgp configs>: DN42 Peer configs for each node ├─inet | ├─peer.conf: Clearnet Peer | ├─ixp.conf: Clearnet IXP connection | ├─defs.conf: Clearnet function definitions, e.g., is_self_inet_v6() | ├─upstream.conf: Clearnet upstream | └static.conf: Clearnet static routes I separated the function definitions because I needed to reference them in the filters within kernel.conf, so I isolated them for early inclusion. After filling in the respective configurations and setting up the include relationships, I ran birdc configure and it started successfully. So, case closed... right? Problems occurred After running for a while, I suddenly found that I couldn't ping the HKG node from my internal devices, nor could I ping my other internal nodes from the HKG node. Strangely, external ASes could ping my other nodes or other external ASes through my HKG node, and my internal nodes could also ping other non-directly connected nodes (e.g., 226(NKG)->225(HKG)->229(LAX)) via the HKG node. Using ip route get <other internal node address> revealed: root@iYoRoyNetworkHKG:/etc/bird# ip route get 172.20.234.226 172.20.234.226 via 172.20.234.226 dev dn42_nkg src 23.149.120.51 uid 0 cache See the problem? The src address should have been the HKG node's own DN42 address (configured on the OSPF stub interface), but here it showed the HKG node's clearnet address instead. Attempting to read the route learned by Bird using birdc s r for 172.20.234.226: root@iYoRoyNetworkHKGBGP:/etc/bird/dn42/ospf# birdc s r for 172.20.234.226 BIRD 2.17.1 ready. Table master4: 172.20.234.226/32 unicast [dn42_ospf_iyoroynet_v4 00:30:29.307] * I (150/50) [172.20.234.226] via 172.20.234.226 on dn42_nkg onlink Looks seemingly normal...? Theoretically, although the DN42 source IP is different from the usual, DN42 rewrites krt_prefsrc when exporting to the kernel to inform the kernel of the correct source address, so this issue shouldn't occur: protocol kernel kernel_v4{ ipv4 { import none; export filter { if source = RTS_STATIC then reject; + if is_valid_dn42_network() then krt_prefsrc = DN42_OWNIP; accept; }; }; } protocol kernel kernel_v6 { ipv6 { import none; export filter { if source = RTS_STATIC then reject; + if is_valid_dn42_network_v6() then krt_prefsrc = DN42_OWNIPv6; accept; }; }; } Regarding krt_prefsrc, it stands for Kernel Route Preferred Source. This attribute doesn't manipulate the route directly but instead attaches a piece of metadata to it. This metadata directly instructs the Linux kernel to prioritize the specified IP address as the source address for packets sent via this route. I was stuck on this for a long time. The Solution Finally, during an unintentional attempt, I added the krt_prefsrc rewrite to the OSPF import configuration as well: protocol ospf v3 dn42_ospf_iyoroynet_v4 { router id DN42_OWNIP; ipv4 { - import where is_self_dn42_net() && source != RTS_BGP; + import filter { + if is_self_dn42_net() && source != RTS_BGP then { + krt_prefsrc=DN42_OWNIP; + accept; + } + reject; + }; export where is_self_dn42_net() && source != RTS_BGP; }; include "ospf/*"; }; protocol ospf v3 dn42_ospf_iyoroynet_v6 { router id DN42_OWNIP; ipv6 { - import where is_self_dn42_net_v6() && source != RTS_BGP; + import filter { + if is_self_dn42_net_v6() && source != RTS_BGP then { + krt_prefsrc=DN42_OWNIPv6; + accept; + } + reject; + }; export where is_self_dn42_net_v6() && source != RTS_BGP; }; include "ospf/*"; }; After running this, the src address became correct, and mutual pinging worked. Configuration files for reference: KaguraiYoRoy/Bird2-Configuration

DN42&OneManISP - Using VRF to Run Clearnet BGP and DN42 on the Same Machine Background Currently, clearnet BGP and DN42 each use a separate VPS in the same region, meaning two machines are required per region. After learning about VRF from a group member, I explored using VRF to enable a single machine to handle both clearnet BGP and DN42 simultaneously. Note: Due to its isolation nature, the VRF solution will prevent DN42 from accessing services on the host. If you need to run services (like DNS) on the server for DN42, you might need additional port forwarding or veth configuration, which is beyond the scope of this article. (This is also the reason why I ultimately did not adopt VRF in my production environment). Advantages of VRF Although DN42 uses private IP ranges and internal ASNs, which theoretically shouldn't interfere with clearnet BGP, sharing the same routing table can lead to issues like route pollution and management complexity. VRF (Virtual Routing and Forwarding) allows creating multiple routing tables on a single machine. This means we can isolate DN42 routes into a separate routing table, keeping them apart from the clearnet routing table. The advantages include: Absolute Security and Policy Isolation: The DN42 routing table is isolated from the clearnet routing table, fundamentally preventing route leaks. Clear Operation and Management: Use commands like birdc show route table t_dn42 and birdc show route table t_inet to view and debug two completely independent routing tables, making things clear at a glance. Fault Domain Isolation: If a DN42 peer flaps, the impact is confined to the dn42 routing table. It won't consume routing computation resources for the clearnet instance nor affect clearnet forwarding performance. Alignment with Modern Network Design Principles: Using VRF for different routing domains (production, testing, customer, partner) is standard practice in modern network engineering. It logically divides your device into multiple virtual routers. Configuration System Part Creating the VRF Interface Use the following commands to create a VRF device named dn42-vrf and associate it with the system's routing table number 1042: ip link add dn42-vrf type vrf table 1042 ip link set dev dn42-vrf up # Enable it You can change the routing table number according to your preference, but avoid the following reserved routing table IDs: Name ID Description unspec 0 Unspecified, rarely used main 254 Main routing table, where most ordinary routes reside default 253 Generally unused, reserved local 255 Local routing table, contains 127.0.0.1/8, local IPs, broadcast addresses, etc. Cannot be modified Associating Existing Network Interfaces with VRF In my current DN42 setup, several WireGuard interfaces and a dummy interface are used for DN42. Therefore, associate these interfaces with the VRF: ip link set dev <interface_name> master dn42-vrf Note: After associating an interface with a VRF, it might lose its IP addresses. Therefore, you need to readd the addresses, for example: ip addr add 172.20.234.225 dev dn42 After completion, ip a should show the corresponding interface's master as dn42-vrf: 156: dn42: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master dn42-vrf state UNKNOWN group default qlen 1000 link/ether b6:f5:28:ed:23:04 brd ff:ff:ff:ff:ff:ff inet 172.20.234.225/32 scope global dn42 valid_lft forever preferred_lft forever inet6 fd18:3e15:61d0::1/128 scope global valid_lft forever preferred_lft forever inet6 fe80::b4f5:28ff:feed:2304/64 scope link valid_lft forever preferred_lft forever Persistence I use ifupdown2 to automatically load the dummy interface and VRF device on boot. auto dn42-vrf iface dn42-vrf inet manual vrf-table 1042 auto dn42 iface dn42 inet static pre-up ip link add $IFACE type dummy || true vrf dn42-vrf address <IPv4 Address>/32 address <IPv6 Address>/128 post-down ip link del $IFACE My dummy interface is named dn42; modify accordingly if yours is different. After creation, use ifup dn42-vrf && ifup dn42 to start the dummy interface. Note: The number prefix for the VRF device file should be smaller than that of the dummy interface file, ensuring the VRF device starts first. WireGuard Tunnels Add PostUp commands to associate them with the VRF and readd their addresses. Example: [Interface] PrivateKey = [Data Redacted] ListenPort = [Data Redacted] Table = off Address = fe80::2024/64 + PostUp = ip link set dev %i master dn42-vrf + PostUp = ip addr add fe80::2024/64 dev %i PostUp = sysctl -w net.ipv6.conf.%i.autoconf=0 [Peer] PublicKey = [Data Redacted] Endpoint = [Data Redacted] AllowedIPs = 10.0.0.0/8, 172.20.0.0/14, 172.31.0.0/16, fd00::/8, fe00::/8 Then restart the tunnel. Bird2 Part First, define two routing tables for DN42's IPv4 and IPv6: ipv4 table dn42_table_v4; ipv6 table dn42_table_v6 Then, specify the VRF and system routing table number in the kernel protocol, and specify the previously created v4/v6 routing tables in the IPv4/IPv6 sections: protocol kernel dn42_kernel_v6{ + vrf "dn42-vrf"; + kernel table 1042; scan time 20; ipv6 { + table dn42_table_v6; import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = DN42_OWNIPv6; accept; }; }; }; protocol kernel dn42_kernel_v4{ + vrf "dn42-vrf"; + kernel table 1042; scan time 20; ipv4 { + table dn42_table_v4; import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = DN42_OWNIP; accept; }; }; } For protocols other than kernel, add the VRF and the independent IPv4/IPv6 tables, but do not specify the system routing table number: protocol static dn42_static_v4{ + vrf "dn42-vrf"; route DN42_OWNNET reject; ipv4 { + table dn42_table_v4; import all; export none; }; } protocol static dn42_static_v6{ + vrf "dn42-vrf"; route DN42_OWNNETv6 reject; ipv6 { + table dn42_table_v6; import all; export none; }; } In summary: Configure a VRF and the previously defined routing tables for everything related to DN42. Only the kernel protocol needs the system routing table number specified; others do not. Apply the same method to BGP, OSPF, etc. However, I chose to use separate Router IDs for the clearnet and DN42, so a separate Router ID needs to be configured: # /etc/bird/dn42/ospf.conf protocol ospf v3 dn42_ospf_iyoroynet_v4 { + vrf "dn42-vrf"; + router id DN42_OWNIP; ipv4 { + table dn42_table_v4; import where is_self_dn42_net() && source != RTS_BGP; export where is_self_dn42_net() && source != RTS_BGP; }; include "ospf/*"; }; protocol ospf v3 dn42_ospf_iyoroynet_v6 { + vrf "dn42-vrf"; + router id DN42_OWNIP; ipv6 { + table dn42_table_v6; import where is_self_dn42_net_v6() && source != RTS_BGP; export where is_self_dn42_net_v6() && source != RTS_BGP; }; include "ospf/*"; }; # /etc/bird/dn42/ebgp.conf ... template bgp dnpeers { + vrf "dn42-vrf"; + router id DN42_OWNIP; local as DN42_OWNAS; path metric 1; ipv4 { + table dn42_table_v4; ... }; ipv6 { + table dn42_table_v6; ... }; } include "peers/*"; After completion, reload the configuration with birdc c. Now, we can view the DN42 routing table separately using ip route show vrf dn42-vrf: root@iYoRoyNetworkHKGBGP:~# ip route show vrf dn42-vrf 10.26.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 10.29.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 10.37.0.0/16 via inet6 fe80::ade0 dev dn42_4242423914 proto bird src 172.20.234.225 metric 32 ... You can also ping through the VRF using the -I dn42-vrf parameter: root@iYoRoyNetworkHKGBGP:~# ping 172.20.0.53 -I dn42-vrf ping: Warning: source address might be selected on device other than: dn42-vrf PING 172.20.0.53 (172.20.0.53) from 172.20.234.225 dn42-vrf: 56(84) bytes of data. 64 bytes from 172.20.0.53: icmp_seq=1 ttl=64 time=3.18 ms 64 bytes from 172.20.0.53: icmp_seq=2 ttl=64 time=3.57 ms 64 bytes from 172.20.0.53: icmp_seq=3 ttl=64 time=3.74 ms 64 bytes from 172.20.0.53: icmp_seq=4 ttl=64 time=2.86 ms ^C --- 172.20.0.53 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3006ms rtt min/avg/max/mdev = 2.863/3.337/3.740/0.341 ms Important Notes If the VRF device is reloaded, all devices originally associated with the VRF need to be reloaded as well, otherwise they won't function correctly. Currently, DN42 cannot access services inside the host configured with VRF. A future article might explain how to allow traffic within the VRF to access host services (Adding to the TODO list). Reference Articles:： Run your MPLS network with BIRD

Configuring Multi-Exit Routing for a Dual-NIC VPS Using PBR Warning: This is a low-value/watered-down post Background I got a Shenzhen-Hong Kong IEPL machine from a friend. It has two network interfaces, eth0 and eth1, but by default, all traffic goes through eth0. eth1 had no routing configured. I planned to use metrics for basic traffic splitting initially, and then use PBR (Policy-Based Routing) to implement rule-based routing configuration. Configuration The machine's network was initially configured by cloud-init: # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: version: 2 ethernets: eth0: addresses: - 10.10.1.31/16 gateway4: 10.10.0.1 match: macaddress: bc:24:11:f8:42:7a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth0 eth1: addresses: - 10.20.1.31/16 - [REDACTED]/64 gateway4: 10.20.0.1 gateway6: fe80::be24:11ff:fe80:66bb match: macaddress: bc:24:11:50:96:0a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth1 After backing up the config, I added metrics: # This file is generated from information provided by the datasource. Changes # to it will not persist across an instance reboot. To disable cloud-init's # network configuration capabilities, write a file # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following: # network: {config: disabled} network: version: 2 ethernets: eth0: addresses: - 10.10.1.31/16 gateway4: 10.10.0.1 match: macaddress: bc:24:11:f8:42:7a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth0 + routes: + - to: "default" + via: "10.10.0.1" + # Set metric=50 as the backup exit + metric: 50 eth1: addresses: - 10.20.1.31/16 - [REDACTED]/64 gateway4: 10.20.0.1 gateway6: fe80::be24:11ff:fe80:66bb match: macaddress: bc:24:11:50:96:0a nameservers: addresses: - 223.5.5.5 - 119.29.29.29 search: - [REDACTED] set-name: eth1 + routes: + - to: "default" + via: "10.20.0.1" + # Set metric=25 as the preferred exit + metric: 25 Create the PBR configuration: # /etc/netplan/90-pbr.yaml network: version: 2 ethernets: eth0: routes: - to: default via: 10.10.0.1 table: 10 routing-policy: - from: 10.10.0.0/16 table: 10 - to: 202.46.[REDACTED]/32 table: 10 eth1: routes: - to: default via: 10.20.0.1 table: 20 routing-policy: - from: 10.20.0.0/16 table: 20 - to: 38.47.[REDACTED]/32 table: 20 - to: 23.149.[REDACTED]/32 table: 20 For IPs that need to be accessed via a specific exit, just add a 'to' type rule and bind it to the corresponding routing table. After finishing, run netplan apply to update the configuration.

OneManISP - Ep.2 Announcing Our Own IP Prefix to the World Preface In the previous article, we successfully registered an ASN and obtained an IPv6 address block. Now, we will announce this block to the world. Setting Up the Subnet Object in the RIPE Database It's important to note that the minimum IPv6 prefix allowed for announcement on the public internet is /48. This means if you only have a single /48 block, you cannot break it down into smaller segments. Therefore, I later leased a separate /40 block, intending to split it into multiple /48s for announcement. The IPv6 block I obtained is 2a14:7583:f200::/40, and I plan to split out 2a14:7583:f203::/48 for use with Vultr. If you don't need to split your block, please skip directly to the "Creating the Route Object" section. Splitting the Prefix First, go to Create "inet6num" object - RIPE Database and fill in the following: inet6num: The IP block you want to split out, in CIDR format. netname: Network name. country: The country to which the IP block belongs, must conform to the ISO 3166 standard (can be selected directly in the RIPE DB). admin-c: The primary key value of the Role object created earlier. tech-c: The primary key value of the Role object created earlier. status: Keep ASSIGNED This step splits a smaller /48 address block from your obtained allocation. Creating the Route Object Go to Create "route6" object - RIPE Database and fill in the following: route6: The IPv6 address block you intend to announce, in CIDR format. origin: The ASN you applied for, including the 'AS' prefix. This step declares that your ASN is permitted to use this address block for originating BGP routes. Applying for BGP Session with a VPS Provider This time I'm using a machine from Vultr. Their BGP Session setup is very beginner-friendly, with their own validation system. Furthermore, their upstream has good filters ensuring that incorrect route advertisements generally won't affect the public internet. (I forgot to take screenshots during my configuration, but you can refer to the section 申请 Vultr 的 BGP 广播功能 in Bao Shuo's article 年轻人的第一个 ASN for reference.) Go to BGP - Vultr.com, select Get Started, and fill in your ASN and IPv6 block information as required. For the LOA (Letter Of Authorization), you can refer to this template: LOA-template.docx (I rewrote one for individuals as most templates found online are for companies). After submission, the system will automatically create a ticket, and you will see your ASN and IP block in a pending verification state: Click Start, and the system will send a verification email to the abuse-mailbox email address registered with your Role object: The received email looks like this: The top link represents approving the authorization for Vultr to announce your IP block, and the bottom one is for disapproval. Click the top link, which will take you to Vultr's webpage: Then click Approve Announcement. Both the ASN and the IP block need to be verified once. Next, wait for the Vultr staff to review and complete the process. Then, in your VPS control panel, you will see the BGP tab, where you can find the upstream information: I must commend Vultr's ticket efficiency here; it took me an average of only about 10 minutes from creating the ticket requesting authorization to completion. (In contrast, the average weekday ticket response time at iFog GmbH was around 1 day, which is much slower in comparison). The process with other VPS providers is generally similar. You need to inform their staff of the ASN and IP block you want to announce. After verifying ownership, the staff will configure the corresponding BGP Session for you. Advertisement! You should have received the following information from your upstream: Upstream's ASN Upstream's IP address for the BGP Session (Optional) Password The operating system I use is Debian 12 Bookworm, using Bird2 as the routing software. I updated Bird2 to the latest version following the section "Update Bird2 to v2.16 or above" in this article. The upstream ASN Vultr gave me is 64515, the upstream BGP Session address is 2001:19f0:ffff::1, and the VPS's BGP Session address is 2001:19f0:0006:0ff5:5400:05ff:fe96:881f. My Bird2 configuration file is modified from the configuration file used in DN42: log syslog all; define OWNAS = 205369; define OWNIPv6 = 2a14:7583:f203::1; define OWNNETv6 = 2a14:7583:f203::/48; define OWNNETSETv6 = [ 2a14:7583:f203::/48+ ]; router id 45.77.x.x; protocol device { scan time 10; } function is_self_net_v6() { return net ~ OWNNETSETv6; } protocol kernel { scan time 20; ipv6 { import none; export filter { if source = RTS_STATIC then reject; krt_prefsrc = OWNIPv6; accept; }; }; }; protocol static { route OWNNETv6 reject; ipv6 { import all; export none; }; } template bgp upstream { local as OWNAS; path metric 1; multihop; ipv6 { import filter { if net ~ [::/0] then reject; accept; }; export filter { if is_self_net_v6() then accept; reject; }; import limit 1000 action block; }; graceful restart; } protocol bgp 'Vultr_v6' from upstream{ local 2001:19f0:0006:0ff5:5400:05ff:fe96:881f as OWNAS; password "123456"; neighbor 2001:19f0:ffff::1 as 64515; } A few noteworthy points: The import rule in the upstream template here rejects the default route. This prevents the routing table sent by the upstream from overwriting local default gateway routes and other routing information. If we have multiple BGP neighbors, this could cause detours or even routing loops. The upstream template specifies multihop (multihop;) because Vultr's BGP peer is not directly reachable. Without setting multihop, the BGP session would get stuck in the Idle state. If your BGP upstream is directly connected, you can omit this line or set it to direct;. After filling in the configuration file, run birdc configure to load the configuration. Run birdc show protocols to check the status. If all goes well, you should see the BGP session state as Established: At this point, you can take a break and wait for global routing convergence. After about half an hour, open bgp.tools and query your /48 block. You should see that it has been successfully received by the global internet, and you can see our upstream information: Next, we create a dummy interface on the VPS and assign a single IPv6 address from the block allocated for this machine. For example, I assigned 2a14:7583:f203::1 to my machine: ip link add dummy0 type dummy ip addr add 2a14:7583:f203::1/128 dev dummy0 Then, using your own PC, you should be able to ping this address, and traceroute will show the complete routing path: Thanks to Mi Lu for the technical support! Reference Articles: 自己在家开运营商 Part.2 - 向世界宣告 IP 段 (BGP Session & BIRD) 年轻人的第一个 ASN - 宝硕博客 BGPlayer 从零开始速成指北 - 开通 Vultr 的 BGP 广播功能 - AceSheep BGP (2) 在 Vultr 和 HE 使用自己的 IPV6 地址 - 131's Blog