VeilNet Conflux and K3s

Prerequisites

• Ubuntu/Debian-based Linux system
• Root or sudo access
• VeilNet Conflux binary (veilnet-conflux)
• VeilNet registration token

Setup K3s Cluster

This guide walks you through setting up a K3s Kubernetes cluster using VeilNet for networking across multiple nodes.

Step 1: Install VeilNet

First, prepare the VeilNet Conflux binary and register the node:

chmod +x ./veilnet-conflux

Register the node with VeilNet:

sudo ./veilnet-conflux register \
    -t <YOUR_VEILNET_TOKEN> \
    --ip <YOUR_NODE_IP> \
    --tag <YOUR_TAG> \
    -p

Replace the placeholders:

• <YOUR_VEILNET_TOKEN>: Your VeilNet registration token
• <YOUR_NODE_IP>: The VeilNet IP for this node (e.g., 10.128.0.1) — this is the IP other nodes will use to reach this node over VeilNet
• <YOUR_TAG>: A tag to identify this node (e.g., master-node-1)

Optional: use -r to enable rift mode instead of (or in addition to) -p for portal mode. Omit both for default mode.

Check the VeilNet service logs:

journalctl -u veilnet -f

Step 2: Install K3s Control Node

Update the system:

sudo apt update
sudo apt upgrade -y

Install K3s on the control node with VeilNet network configuration. Replace <YOUR_NODE_IP> with the VeilNet IP address assigned to this node (same as the one used in register --ip):

curl -sfL https://get.k3s.io | sh -s - server --cluster-init \
    --node-ip <YOUR_NODE_IP> \
    --bind-address <YOUR_NODE_IP> \
    --advertise-address <YOUR_NODE_IP> \
    --tls-san <YOUR_NODE_IP> \
    --flannel-iface veilnet \
    --node-name <YOUR_NODE_NAME>

Replace the placeholders:

• <YOUR_NODE_IP>: The VeilNet IP address of this node (e.g., 10.128.0.1)
• <YOUR_NODE_NAME>: A name for this node (e.g., master-node-1)

Get the node token for joining additional nodes:

sudo cat /var/lib/rancher/k3s/server/node-token

Step 3: Join Additional Server Nodes

To join additional server nodes to form a HA cluster, first register each node with VeilNet (as in Step 1), then run:

curl -sfL https://get.k3s.io | K3S_TOKEN=<NODE_TOKEN> sh -s - server \
    --server https://<CONTROL_NODE_IP>:6443 \
    --node-ip <NEW_NODE_IP> \
    --bind-address <NEW_NODE_IP> \
    --advertise-address <NEW_NODE_IP> \
    --tls-san <NEW_NODE_IP> \
    --flannel-iface veilnet \
    --node-name <NEW_NODE_NAME>

Replace the placeholders:

• <NODE_TOKEN>: The token from the control node
• <CONTROL_NODE_IP>: The VeilNet IP of the control node (e.g., 10.128.0.1)
• <NEW_NODE_IP>: The VeilNet IP of the new server node (e.g., 10.128.0.2)
• <NEW_NODE_NAME>: A name for the new server node (e.g., master-node-2)

Step 4: Join Worker Nodes

To join worker nodes to the cluster, first register each node with VeilNet (as in Step 1), then run:

curl -sfL https://get.k3s.io | K3S_TOKEN=<NODE_TOKEN> sh -s - agent \
    --server https://<CONTROL_NODE_IP>:6443 \
    --node-ip <WORKER_NODE_IP> \
    --flannel-iface veilnet \
    --node-name <WORKER_NODE_NAME>

Replace the placeholders:

• <NODE_TOKEN>: The token from the control node
• <CONTROL_NODE_IP>: The VeilNet IP of the control node (e.g., 10.128.0.1)
• <WORKER_NODE_IP>: The VeilNet IP of the worker node (e.g., 10.128.0.3)
• <WORKER_NODE_NAME>: A name for the worker node (e.g., worker-node-1)

Verification

Verify your cluster is running correctly:

kubectl get nodes
kubectl get pods --all-namespaces

Updating VeilNet

To update VeilNet on a node, download the new binary and follow these steps:

1. Download the new VeilNet Conflux binary
2. Make it executable:

chmod +x ./veilnet-conflux

3. Remove the existing VeilNet service (config in conflux.json is preserved):

sudo ./veilnet-conflux remove

4. Install and start the service again (uses existing config):

sudo ./veilnet-conflux install

5. Reboot the node:

sudo reboot

After rebooting, the node will reconnect to the VeilNet network with the updated binary, as well as the K3s cluster.

For running Conflux as a sidecar in pods (shared network namespace, VeilNet IPs), see Kubernetes – sidecar.

FAQ

Do I need to configure a sub-router?

No, you do not need to configure a sub-router. VeilNet handles all the networking automatically, including routing between nodes across different regions.

Do I need to configure firewall rules or Flannel VXLAN settings?

No, you do not need to configure firewall rules or Flannel VXLAN settings. VeilNet manages the network layer, and by specifying --flannel-iface veilnet during K3s installation, Flannel will use the VeilNet interface automatically without requiring additional VXLAN configuration.

Can I use Longhorn for distributed storage?

We do not recommend using Longhorn for distributed storage unless all nodes are in the same local network. Longhorn has strict latency requirements that may not be met when nodes are distributed across different regions or have higher network latency. For multi-region deployments, consider using other storage solutions that are designed for higher latency environments.

Should I use VeilNet even if all my nodes are local?

Yes, you can still use VeilNet for your cluster even if all nodes are on the same local network. VeilNet provides additional security by encrypting all traffic between nodes and can help isolate your cluster traffic from other network traffic on the same physical network.