mirror of
git://git.proxmox.com/git/pve-docs.git
synced 2025-01-10 01:17:51 +03:00
634 lines
12 KiB
Plaintext
634 lines
12 KiB
Plaintext
|
[[chapter_pvesdn]]
|
|||
|
Software Defined Network
|
|||
|
========================
|
|||
|
ifndef::manvolnum[]
|
|||
|
:pve-toplevel:
|
|||
|
endif::manvolnum[]
|
|||
|
|
|||
|
The SDN feature allow to create virtual networks (vnets)
|
|||
|
at datacenter level.
|
|||
|
|
|||
|
To enable SDN feature, you need to install "libpve-network-perl" package
|
|||
|
|
|||
|
----
|
|||
|
apt install libpve-network-perl
|
|||
|
----
|
|||
|
|
|||
|
A vnet is a bridge with a vlan or vxlan tag.
|
|||
|
|
|||
|
The vnets are deployed locally on each node after configuration
|
|||
|
commit at datacenter level.
|
|||
|
|
|||
|
You need to have "ifupdown2" package installed on each node to manage local
|
|||
|
configuration reloading.
|
|||
|
|
|||
|
----
|
|||
|
apt install ifupdown2
|
|||
|
----
|
|||
|
|
|||
|
Main configuration
|
|||
|
------------------
|
|||
|
|
|||
|
The configuration is done at datacenter level.
|
|||
|
|
|||
|
The sdn feature have 4 main sections for the configuration
|
|||
|
|
|||
|
* SDN
|
|||
|
|
|||
|
* Zones
|
|||
|
|
|||
|
* Vnets
|
|||
|
|
|||
|
* Controller
|
|||
|
|
|||
|
|
|||
|
SDN
|
|||
|
~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-status.png"]
|
|||
|
|
|||
|
This is the Main panel, where you can see deployment of zones on differents nodes.
|
|||
|
|
|||
|
They are an "apply" button, to push && reload local configuration on differents nodes.
|
|||
|
|
|||
|
|
|||
|
Zones
|
|||
|
~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-zone.png"]
|
|||
|
|
|||
|
A zone will defined the kind of virtual network you want to defined.
|
|||
|
|
|||
|
it can be
|
|||
|
|
|||
|
* vlan
|
|||
|
|
|||
|
* QinQ (stacked vlan)
|
|||
|
|
|||
|
* vxlan (layer2 vxlan)
|
|||
|
|
|||
|
* bgp-evpn (vxlan with layer3 routing)
|
|||
|
|
|||
|
You can restrict a zone to specific nodes.
|
|||
|
|
|||
|
It's also possible to add permissions on a zone, to restrict user
|
|||
|
to use only a specific zone and the vnets in this zone
|
|||
|
|
|||
|
Vnets
|
|||
|
~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-vnet-evpn.png"]
|
|||
|
|
|||
|
A vnet is a bridge that will be deployed locally on the node,
|
|||
|
for vm communication. (Like a classic vmbrX).
|
|||
|
|
|||
|
Vnet properties are:
|
|||
|
|
|||
|
* ID: a 8 characters ID
|
|||
|
|
|||
|
* Alias: Optionnal bigger name
|
|||
|
|
|||
|
* Zone: The associated zone of the vnet
|
|||
|
|
|||
|
* Tag: unique vlan or vxlan id
|
|||
|
|
|||
|
* ipv4: an anycast ipv4 address (same bridge ip deployed on each node), for bgp-evpn routing only
|
|||
|
|
|||
|
* ipv6: an anycast ipv6 address (same bridge ip deployed on each node), for bgp-evpn routing only
|
|||
|
|
|||
|
|
|||
|
Controllers
|
|||
|
~~~~~~~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-controller.png"]
|
|||
|
|
|||
|
Some zone plugins (Currently : bgp-evpn only),
|
|||
|
need an external controller to manage the vnets control-plane.
|
|||
|
|
|||
|
|
|||
|
|
|||
|
Zones Plugins
|
|||
|
-------------
|
|||
|
common zone options:
|
|||
|
|
|||
|
* nodes: restrict deploy of the vnets of theses nodes only
|
|||
|
|
|||
|
|
|||
|
Vlan
|
|||
|
~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-zone-vlan.png"]
|
|||
|
|
|||
|
This is the most simple plugin, it'll reuse an existing local bridge or ovs,
|
|||
|
and manage vlan on it.
|
|||
|
The benefit of using sdn module, is that you can create different zones with specific
|
|||
|
vnets vlan tag, and restrict your customers on their zones.
|
|||
|
|
|||
|
specific qinq configuration options:
|
|||
|
|
|||
|
* bridge: a local vlan-aware bridge or ovs switch already configured on each local node
|
|||
|
|
|||
|
QinQ
|
|||
|
~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-zone-qinq.png"]
|
|||
|
|
|||
|
QinQ is stacked vlan.
|
|||
|
you have the first vlan tag defined on the zone (service-vlan), and
|
|||
|
the second vlan tag defined on the vnets
|
|||
|
|
|||
|
Your physical network switchs need to support stacked vlans !
|
|||
|
|
|||
|
specific qinq configuration options:
|
|||
|
|
|||
|
* bridge: a local vlan-aware bridge already configured on each local node
|
|||
|
* service vlan: The main vlan tag of this zone
|
|||
|
* mtu: you need 4 more bytes for the double tag vlan.
|
|||
|
You can reduce the mtu to 1496 if you physical interface mtu is 1500.
|
|||
|
|
|||
|
Vxlan
|
|||
|
~~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-zone-vxlan.png"]
|
|||
|
|
|||
|
The vxlan plugin will established vxlan tunnel (overlay) on top of an existing network (underlay).
|
|||
|
you can for example, create a private ipv4 vxlan network on top of public internet network nodes.
|
|||
|
This is a layer2 tunnel only, no routing between different vnets is possible.
|
|||
|
|
|||
|
Each vnet will have a specific vxlan id ( 1 - 16777215 )
|
|||
|
|
|||
|
|
|||
|
Specific evpn configuration options:
|
|||
|
|
|||
|
* peers address list: an ip list of all nodes where you want to communicate (could be also external nodes)
|
|||
|
|
|||
|
* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
|
|||
|
than the outgoing physical interface.
|
|||
|
|
|||
|
evpn
|
|||
|
~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-zone-evpn.png"]
|
|||
|
|
|||
|
This is the most complex plugin.
|
|||
|
|
|||
|
BGP-evpn allow to create routable layer3 network.
|
|||
|
The vnet of evpn can have an anycast ip address/mac address.
|
|||
|
The bridge ip is the same on each node, then vm can use
|
|||
|
as gateway.
|
|||
|
The routing is working only across vnets of a specific zone through a vrf.
|
|||
|
|
|||
|
Specific evpn configuration options:
|
|||
|
|
|||
|
* vrf vxlan tag: This is a vxlan-id used for routing interconnect between vnets,
|
|||
|
it must be different than vxlan-id of vnets
|
|||
|
|
|||
|
* controller: an evpn need to be defined first (see controller plugins section)
|
|||
|
|
|||
|
* mtu: because vxlan encapsulation use 50bytes, the mtu need to be 50 bytes lower
|
|||
|
than the outgoing physical interface.
|
|||
|
|
|||
|
|
|||
|
Controllers Plugins
|
|||
|
-------------------
|
|||
|
|
|||
|
evpn
|
|||
|
~~~~
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-controller-evpn.png"]
|
|||
|
|
|||
|
For bgp-evpn, we need a controller to manage the control plane.
|
|||
|
The software controller is "frr" router.
|
|||
|
You need to install it on each node where you want to deploy the evpn zone.
|
|||
|
|
|||
|
----
|
|||
|
apt install frr
|
|||
|
----
|
|||
|
|
|||
|
configuration options:
|
|||
|
|
|||
|
*asn: a unique bgp asn number.
|
|||
|
It's recommended to use private asn number (64512 – 65534, 4200000000 – 4294967294)
|
|||
|
|
|||
|
*peers: an ip list of all nodes where you want to communicate (could be also external nodes or route reflectors servers)
|
|||
|
|
|||
|
If you want to route traffic from the sdn bgp-evpn network to external world:
|
|||
|
|
|||
|
* gateway-nodes: The proxmox nodes from where the bgp-evpn traffic will exit to external through the nodes default gateway
|
|||
|
|
|||
|
If you want that gateway nodes don't use the default gateway, but for example, sent traffic to external bgp routers
|
|||
|
|
|||
|
* gateway-external-peers: 192.168.0.253,192.168.0.254
|
|||
|
|
|||
|
|
|||
|
Local deployment Monitoring
|
|||
|
---------------------------
|
|||
|
|
|||
|
[thumbnail="screenshot/gui-sdn-local-status.png"]
|
|||
|
|
|||
|
After apply configuration on the main sdn section,
|
|||
|
the local configuration is generated locally on each node,
|
|||
|
in /etc/network/interfaces.d/sdn, and reloaded.
|
|||
|
|
|||
|
You can monitor the status of local zones && vnets through the main tree.
|
|||
|
|
|||
|
|
|||
|
|
|||
|
Vlan setup example
|
|||
|
------------------
|
|||
|
node1: /etc/network/interfaces
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet manual
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
bridge-vlan-aware yes
|
|||
|
bridge-vids 2-4094
|
|||
|
|
|||
|
#management ip on vlan100
|
|||
|
auto vmbr0.100
|
|||
|
iface vmbr0.100 inet static
|
|||
|
address 192.168.0.1/24
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
|
|||
|
----
|
|||
|
|
|||
|
node2: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet manual
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
bridge-vlan-aware yes
|
|||
|
bridge-vids 2-4094
|
|||
|
|
|||
|
#management ip on vlan100
|
|||
|
auto vmbr0.100
|
|||
|
iface vmbr0.100 inet static
|
|||
|
address 192.168.0.2/24
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
create an vlan zone
|
|||
|
|
|||
|
----
|
|||
|
id: mylanzone
|
|||
|
bridge: vmbr0
|
|||
|
----
|
|||
|
|
|||
|
create a vnet1 with vlan-id 10
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet1
|
|||
|
zone: myvlanzone
|
|||
|
tag: 10
|
|||
|
----
|
|||
|
|
|||
|
Apply the configuration on the main sdn section, to create vnets locally on each nodes,
|
|||
|
and generate frr config.
|
|||
|
|
|||
|
|
|||
|
create a vm1, with 1 nic on vnet1 on node1
|
|||
|
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.100/24
|
|||
|
----
|
|||
|
|
|||
|
create a vm2, with 1 nic on vnet1 on node2
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.101/24
|
|||
|
----
|
|||
|
|
|||
|
Then, you should be able to ping between between vm1 && vm2
|
|||
|
|
|||
|
|
|||
|
QinQ setup example
|
|||
|
------------------
|
|||
|
node1: /etc/network/interfaces
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet manual
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
bridge-vlan-aware yes
|
|||
|
bridge-vids 2-4094
|
|||
|
|
|||
|
#management ip on vlan100
|
|||
|
auto vmbr0.100
|
|||
|
iface vmbr0.100 inet static
|
|||
|
address 192.168.0.1/24
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
node2: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet manual
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
bridge-vlan-aware yes
|
|||
|
bridge-vids 2-4094
|
|||
|
|
|||
|
#management ip on vlan100
|
|||
|
auto vmbr0.100
|
|||
|
iface vmbr0.100 inet static
|
|||
|
address 192.168.0.2/24
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
create an qinq zone1 with service vlan 20
|
|||
|
|
|||
|
----
|
|||
|
id: qinqzone1
|
|||
|
bridge: vmbr0
|
|||
|
service vlan: 20
|
|||
|
----
|
|||
|
|
|||
|
create an qinq zone2 with service vlan 30
|
|||
|
|
|||
|
----
|
|||
|
id: qinqzone2
|
|||
|
bridge: vmbr0
|
|||
|
service vlan: 30
|
|||
|
----
|
|||
|
|
|||
|
create a vnet1 with customer vlan-id 100 on qinqzone1
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet1
|
|||
|
zone: qinqzone1
|
|||
|
tag: 100
|
|||
|
----
|
|||
|
|
|||
|
create a vnet2 with customer vlan-id 100 on qinqzone2
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet2
|
|||
|
zone: qinqzone1
|
|||
|
tag: 100
|
|||
|
----
|
|||
|
|
|||
|
Apply the configuration on the main sdn section, to create vnets locally on each nodes,
|
|||
|
and generate frr config.
|
|||
|
|
|||
|
|
|||
|
create a vm1, with 1 nic on vnet1 on node1
|
|||
|
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.100/24
|
|||
|
----
|
|||
|
|
|||
|
create a vm2, with 1 nic on vnet1 on node2
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.101/24
|
|||
|
----
|
|||
|
|
|||
|
create a vm3, with 1 nic on vnet2 on node1
|
|||
|
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.102/24
|
|||
|
----
|
|||
|
|
|||
|
create a vm4, with 1 nic on vnet2 on node2
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.103/24
|
|||
|
----
|
|||
|
|
|||
|
Then, you should be able to ping between between vm1 && vm2
|
|||
|
vm3 && vm4 could ping together
|
|||
|
|
|||
|
but vm1 && vm2 couldn't ping vm3 && vm4,
|
|||
|
as it's a different zone, with different service vlan
|
|||
|
|
|||
|
|
|||
|
Vxlan setup example
|
|||
|
-------------------
|
|||
|
node1: /etc/network/interfaces
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.1/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
node2: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.2/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
node3: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.3/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
create an vxlan zone
|
|||
|
|
|||
|
----
|
|||
|
id: myvxlanzone
|
|||
|
peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
|
|||
|
mtu: 1450
|
|||
|
----
|
|||
|
|
|||
|
create first vnet
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet1
|
|||
|
zone: myvxlanzone
|
|||
|
tag: 100000
|
|||
|
----
|
|||
|
|
|||
|
Apply the configuration on the main sdn section, to create vnets locally on each nodes,
|
|||
|
and generate frr config.
|
|||
|
|
|||
|
|
|||
|
create a vm1, with 1 nic on vnet1 on node2
|
|||
|
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.100/24
|
|||
|
mtu 1450
|
|||
|
----
|
|||
|
|
|||
|
create a vm2, with 1 nic on vnet1 on node3
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.3.101/24
|
|||
|
mtu 1450
|
|||
|
----
|
|||
|
|
|||
|
Then, you should be able to ping between between vm1 && vm2
|
|||
|
|
|||
|
|
|||
|
|
|||
|
EVPN setup example
|
|||
|
------------------
|
|||
|
node1: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.1/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
node2: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.2/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
node3: /etc/network/interfaces
|
|||
|
|
|||
|
----
|
|||
|
auto vmbr0
|
|||
|
iface vmbr0 inet static
|
|||
|
address 192.168.0.3/24
|
|||
|
gateway 192.168.0.254
|
|||
|
bridge-ports eno1
|
|||
|
bridge-stp off
|
|||
|
bridge-fd 0
|
|||
|
mtu 1500
|
|||
|
|
|||
|
source /etc/network/interfaces.d/*
|
|||
|
----
|
|||
|
|
|||
|
create a evpn controller
|
|||
|
|
|||
|
----
|
|||
|
id: myevpnctl
|
|||
|
asn: 65000
|
|||
|
peers: 192.168.0.1,192.168.0.2,192.168.0.3
|
|||
|
gateway nodes: node1,node2
|
|||
|
----
|
|||
|
|
|||
|
create an evpn zone
|
|||
|
|
|||
|
----
|
|||
|
id: myevpnzone
|
|||
|
vrf vxlan tag: 10000
|
|||
|
controller: myevpnctl
|
|||
|
mtu: 1450
|
|||
|
----
|
|||
|
|
|||
|
create first vnet
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet1
|
|||
|
zone: myevpnzone
|
|||
|
tag: 11000
|
|||
|
ipv4: 10.0.1.1/24
|
|||
|
mac address: 8C:73:B2:7B:F9:60 #random generate mac addres
|
|||
|
----
|
|||
|
|
|||
|
create second vnet
|
|||
|
|
|||
|
----
|
|||
|
id: myvnet2
|
|||
|
zone: myevpnzone
|
|||
|
tag: 12000
|
|||
|
ipv4: 10.0.2.1/24
|
|||
|
mac address: 8C:73:B2:7B:F9:61 #random mac, need to be different on each vnet
|
|||
|
----
|
|||
|
|
|||
|
Apply the configuration on the main sdn section, to create vnets locally on each nodes,
|
|||
|
and generate frr config.
|
|||
|
|
|||
|
|
|||
|
|
|||
|
create a vm1, with 1 nic on vnet1 on node2
|
|||
|
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.1.100/24
|
|||
|
gateway 10.0.1.1 #this is the ip of the vnet1
|
|||
|
mtu 1450
|
|||
|
----
|
|||
|
|
|||
|
create a vm2, with 1 nic on vnet2 on node3
|
|||
|
----
|
|||
|
auto eth0
|
|||
|
iface eth0 inet static
|
|||
|
address 10.0.2.100/24
|
|||
|
gateway 10.0.2.1 #this is the ip of the vnet2
|
|||
|
mtu 1450
|
|||
|
----
|
|||
|
|
|||
|
|
|||
|
Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
|
|||
|
|
|||
|
from vm2 on node3, if you ping an external ip, the packet will go
|
|||
|
to the vnet2 gateway, then will be routed to gateway nodes (node1 or node2)
|
|||
|
then the packet will be routed to the node1 or node2 default gw.
|
|||
|
|
|||
|
Of course you need to add reverse routes to 10.0.1.0/24 && 10.0.2.0/24 to node1,node2 on your external gateway.
|
|||
|
|
|||
|
If you have configured an external bgp router, the bgp-evpn routes (10.0.1.0/24 && 10.0.2.0/24),
|
|||
|
will be announced dynamically.
|
|||
|
|