Architecture III - Dedicated Data Center Layer 3 Spine/Leaf Topology

This workflow provisions switches in a Spine/Leaf topology using eBGP EVPN for the L3 fabric and L2 VXLAN with VSX based on the validated reference design. This workflow does not configure the centralized L3 gateway.

Workflow Prerequisites

All prerequisites defined in Project Prerequisites
Ensure the provided Ansible inventory file has been modified to suit your environment, according to the instructions in Inventory Setup:
- YAML Inventory File : inventory_spine_leaf.yml
- Excel Template File : dedicated_spine_leaf_ebgp_DCN_Settings.xlsx
  - Example inventory using aoscx_dcn_plugin: dynamic_ebgp_spine_leaf_inventory.yml
Spine switches = 2 AOS-CX switches
- Spine switches should be in a VSX pair and must support EVPN (8325/8400 required)
Leaf switches = 4 or more AOS-CX switches
- Leaf switches should be in VSX pairs and must support EVPN/VXLAN (8325 required)
Out-of-Band Management (OOBM) connections to the management ports on AOS-CX switches
- Ansible control machine should be reachable via each device's OOBM

Files Used

Playbook : deploy_ebgp_evpn_vxlan.yml
Inventory :
- YAML Inventory File : inventory_spine_leaf.yml
- Excel Template File : dedicated_spine_leaf_ebgp_DCN_Settings.xlsx
  - Example inventory using aoscx_dcn_plugin : dynamic_ebgp_spine_leaf_inventory.yml
Jinja2 Templates :
- templates/eBGP/spine.j2
- templates/eBGP/leaf.j2

Workflow Walkthrough

Prior to executing the Ansible playbook, the environment must be in this initial state:

Zone1-Spine<1/2> + Zone1-Rack1-Leaf<1a/1b> + Zone1-Rack3-Leaf<3a/3b> - These devices each have a default configuration with an IP address (DHCP/Static) assigned to the management interface. This IP address should match the the value of ansible_host for each device in the inventory.
Zone1-Rack1-Leaf<1a/1b> - These devices are in a VSX pair with their physical links matching the values defined in the inventory
Zone1-Rack3-Leaf<3a/3b> - These devices are in a VSX pair with their physical links matching the values defined in the inventory

The playbook will perform the following actions on every device in the inventory file using SSH:

Generate a configuration based on the template file templates/eBGP/spine.j2 or templates/eBGP/leaf.j2 and values defined in the inventory

Push the generated configuration to the device using the AOS-CX SSH Ansible module aoscx_config
Enable 10g speed interface groups (if defined in the inventory using the AOS-CX SSH Ansible module aoscx_config

The playbook will perform the following actions on every spine device in the inventory file
using SSH:

Configure BGP neighbors and EVPN address families for every leaf's loopback IP address on every leaf's BGP ASN

The playbook will perform the following actions on every leaf device in the inventory file
using SSH:

Configure BGP neighbors and EVPN address families for every spine's loopback IP address

Create all VLANs defined as server_vlans in the inventory
Create an EVPN instance and map VLANs defined in server_vlans

Because of path requirements, you must run this workflow from the root level of the cloned repository:

ansible-playbook deploy_ebgp_evpn_vxlan.yml -i inventory_spine_leaf.yml

ansible-playbook deploy_ebgp_evpn_vxlan.yml -i dynamic_ebgp_spine_leaf_inventory.yml

Final Sample Configs

!
!Version ArubaOS-CX GL.10.04.0040
!export-password: default
hostname Zone1-Spine1
user admin group administrators password ciphertext AQBapeeuZ6Nw+Phok7vJbD6r75PivsY6o/r0QfxdpH1h3fQYYgAAACrisdLluFaTV+Fj1JfL0WsZPS8LBYsoE/N6qohz8bziNZQvKts2XD+d+Hgx+qrd64f4Htq7A/1mAvqetP90ljtfIOX27j/ZvVwqV6ewUQyQ7V7rFCe8BIXyVCXZD5QhqRdg
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
interface mgmt
    no shutdown
    ip static 10.10.10.54/24
    default-gateway 10.10.10.254
interface 1/1/23 
    no shutdown
    mtu 9198
    description rack3-Downlink
    ip mtu 9198
    ip address 192.168.2.13/31
interface 1/1/24 
    no shutdown
    mtu 9198
    description rack3-Downlink
    ip mtu 9198
    ip address 192.168.2.9/31
interface 1/1/27 
    no shutdown
    mtu 9198
    description rack1-Downlink
    ip mtu 9198
    ip address 192.168.2.1/31
interface 1/1/28 
    no shutdown
    mtu 9198
    description rack1-Downlink
    ip mtu 9198
    ip address 192.168.2.5/31
interface loopback 0
    ip address 192.168.1.11/32
router bgp 65101
    bgp router-id 192.168.1.11
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.1 remote-as 65001
    neighbor 192.168.1.1 ebgp-multihop 3
    neighbor 192.168.1.1 update-source loopback 0
    neighbor 192.168.1.2 remote-as 65001
    neighbor 192.168.1.2 ebgp-multihop 3
    neighbor 192.168.1.2 update-source loopback 0
    neighbor 192.168.1.5 remote-as 65003
    neighbor 192.168.1.5 ebgp-multihop 3
    neighbor 192.168.1.5 update-source loopback 0
    neighbor 192.168.1.6 remote-as 65003
    neighbor 192.168.1.6 ebgp-multihop 3
    neighbor 192.168.1.6 update-source loopback 0
    neighbor 192.168.2.0 remote-as 65001
    neighbor 192.168.2.4 remote-as 65001
    neighbor 192.168.2.8 remote-as 65003
    neighbor 192.168.2.12 remote-as 65003
    address-family ipv4 unicast
        neighbor 192.168.2.0 activate
        neighbor 192.168.2.4 activate
        neighbor 192.168.2.8 activate
        neighbor 192.168.2.12 activate
        redistribute connected
        network 192.168.1.11/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.1 activate
        neighbor 192.168.1.1 next-hop-unchanged
        neighbor 192.168.1.1 send-community extended
        neighbor 192.168.1.2 activate
        neighbor 192.168.1.2 next-hop-unchanged
        neighbor 192.168.1.2 send-community extended
        neighbor 192.168.1.5 activate
        neighbor 192.168.1.5 next-hop-unchanged
        neighbor 192.168.1.5 send-community extended
        neighbor 192.168.1.6 activate
        neighbor 192.168.1.6 next-hop-unchanged
        neighbor 192.168.1.6 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0040
!export-password: default
hostname Zone1-Spine2
user admin group administrators password ciphertext AQBapW41EEHA+zskBXcBrm9Rr+euZNH+d4Q5BiGgeNrIvw1gYgAAANRxoQcfTTV7tma79JXNUOqrhv2y0xM21jrQxXeufK4O4aCwIfhuPJA7SPvno9iqnFJ9ehwrWSd5HdcyT1eb80glx6No9vqdDGKvOUfi6IOlDg6rcdaJcQWJ+tO7bZYVz5uz
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
interface mgmt
    no shutdown
    ip static 10.10.10.55/24
    default-gateway 10.10.10.254
interface 1/1/23 
    no shutdown
    mtu 9198
    description rack3-Downlink
    ip mtu 9198
    ip address 192.168.2.15/31
interface 1/1/24 
    no shutdown
    mtu 9198
    description rack3-Downlink
    ip mtu 9198
    ip address 192.168.2.11/31
interface 1/1/27 
    no shutdown
    mtu 9198
    description rack1-Downlink
    ip mtu 9198
    ip address 192.168.2.3/31
interface 1/1/28 
    no shutdown
    mtu 9198
    description rack1-Downlink
    ip mtu 9198
    ip address 192.168.2.7/31
interface loopback 0
    ip address 192.168.1.12/32
router bgp 65101
    bgp router-id 192.168.1.12
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.1 remote-as 65001
    neighbor 192.168.1.1 ebgp-multihop 3
    neighbor 192.168.1.1 update-source loopback 0
    neighbor 192.168.1.2 remote-as 65001
    neighbor 192.168.1.2 ebgp-multihop 3
    neighbor 192.168.1.2 update-source loopback 0
    neighbor 192.168.1.5 remote-as 65003
    neighbor 192.168.1.5 ebgp-multihop 3
    neighbor 192.168.1.5 update-source loopback 0
    neighbor 192.168.1.6 remote-as 65003
    neighbor 192.168.1.6 ebgp-multihop 3
    neighbor 192.168.1.6 update-source loopback 0
    neighbor 192.168.2.2 remote-as 65001
    neighbor 192.168.2.6 remote-as 65001
    neighbor 192.168.2.10 remote-as 65003
    neighbor 192.168.2.14 remote-as 65003
    address-family ipv4 unicast
        neighbor 192.168.2.2 activate
        neighbor 192.168.2.6 activate
        neighbor 192.168.2.10 activate
        neighbor 192.168.2.14 activate
        redistribute connected
        network 192.168.1.12/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.1 activate
        neighbor 192.168.1.1 next-hop-unchanged
        neighbor 192.168.1.1 send-community extended
        neighbor 192.168.1.2 activate
        neighbor 192.168.1.2 next-hop-unchanged
        neighbor 192.168.1.2 send-community extended
        neighbor 192.168.1.5 activate
        neighbor 192.168.1.5 next-hop-unchanged
        neighbor 192.168.1.5 send-community extended
        neighbor 192.168.1.6 activate
        neighbor 192.168.1.6 next-hop-unchanged
        neighbor 192.168.1.6 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0020
!export-password: default
hostname Zone1-Rack1-Leaf1a
user admin group administrators password ciphertext AQBapZ4yCW+QbkkvhYYoSS0WaqDVKw88SZxmgXHIxwMipV9EYgAAAMAuiAnGsQwvlI3bNifJth6elIQWykn7bGlAq+byxaItlAZQiZom10jqCFTailvy80jwvoNQdgLf6Ie6XIqed9Jzxk3X14GujvBxfL4XFHit14RQIALWT12Cj1o9TE55wRck
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
vlan 11
    name VLAN 11
    description Server VLAN
evpn
    vlan 11
        rd auto
        route-target export 1:11
        route-target import 1:11
interface mgmt
    no shutdown
    ip static 10.10.10.56/24
    default-gateway 10.10.10.254
system interface-group 1 speed 10g
    !interface group 1 contains ports 1/1/1-1/1/12
system interface-group 3 speed 10g
    !interface group 3 contains ports 1/1/25-1/1/36
interface lag 1
    no shutdown
    description VSX ISL LAG
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface 1/1/1 
    no shutdown
    no routing
    vlan access 11
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.110/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface 1/1/49 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.0/31
interface 1/1/50 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.2/31
interface loopback 0
    ip address 192.168.1.1/32
interface loopback 1
    ip address 192.168.100.1/32
interface vxlan 1
    source ip 192.168.100.1
    no shutdown
    vni 11
        vlan 11
vsx
    inter-switch-link lag 1
    role primary
    keepalive peer 192.168.1.111 source 192.168.1.110
    no split-recovery
router bgp 65001
    bgp router-id 192.168.1.1
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.11 remote-as 65101
    neighbor 192.168.1.11 ebgp-multihop 3
    neighbor 192.168.1.11 update-source loopback 0
    neighbor 192.168.1.12 remote-as 65101
    neighbor 192.168.1.12 ebgp-multihop 3
    neighbor 192.168.1.12 update-source loopback 0
    neighbor 192.168.2.1 remote-as 65101
    neighbor 192.168.2.3 remote-as 65101
    neighbor 192.168.2.5 remote-as 65101
    neighbor 192.168.2.7 remote-as 65101
    address-family ipv4 unicast
        neighbor 192.168.2.1 activate
        neighbor 192.168.2.3 activate
        neighbor 192.168.2.5 activate
        neighbor 192.168.2.7 activate
        redistribute connected
        network 192.168.1.1/32
        network 192.168.100.1/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.11 activate
        neighbor 192.168.1.11 next-hop-unchanged
        neighbor 192.168.1.11 send-community extended
        neighbor 192.168.1.12 activate
        neighbor 192.168.1.12 next-hop-unchanged
        neighbor 192.168.1.12 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0020
!export-password: default
hostname Zone1-Rack1-Leaf1b
user admin group administrators password ciphertext AQBapd3Qg7OPKcjRayIQyuxOPabPIbT8bvU05pOk8sc+vAXyYgAAAGtM+A5APROROs6l56dpUdXic8SskYkcBHqp0rxFPtTqgmXoEzI21Mk5T3CR023fONvCpIZGpS4WUmReFVaiMR2XKnitYUhfkJLCK19Kl9uBL85jHFsthncP+X7/1q0bs/RG
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
vlan 11
    name VLAN 11
    description Server VLAN
evpn
    vlan 11
        rd auto
        route-target export 1:11
        route-target import 1:11
interface mgmt
    no shutdown
    ip static 10.10.10.57/24
    default-gateway 10.10.10.254
system interface-group 1 speed 10g
    !interface group 1 contains ports 1/1/1-1/1/12
system interface-group 3 speed 10g
    !interface group 3 contains ports 1/1/25-1/1/36
system interface-group 4 speed 10g
    !interface group 4 contains ports 1/1/37-1/1/48
interface lag 1
    no shutdown
    description VSX ISL LAG
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.111/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface 1/1/49 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.4/31
interface 1/1/50 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.6/31
interface loopback 0
    ip address 192.168.1.2/32
interface loopback 1
    ip address 192.168.100.2/32
interface vxlan 1
    source ip 192.168.100.2
    no shutdown
    vni 11
        vlan 11
vsx
    inter-switch-link lag 1
    role secondary
    keepalive peer 192.168.1.110 source 192.168.1.111
    no split-recovery
router bgp 65001
    bgp router-id 192.168.1.2
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.11 remote-as 65101
    neighbor 192.168.1.11 ebgp-multihop 3
    neighbor 192.168.1.11 update-source loopback 0
    neighbor 192.168.1.12 remote-as 65101
    neighbor 192.168.1.12 ebgp-multihop 3
    neighbor 192.168.1.12 update-source loopback 0
    neighbor 192.168.2.1 remote-as 65101
    neighbor 192.168.2.3 remote-as 65101
    neighbor 192.168.2.5 remote-as 65101
    neighbor 192.168.2.7 remote-as 65101
    address-family ipv4 unicast
        neighbor 192.168.2.1 activate
        neighbor 192.168.2.3 activate
        neighbor 192.168.2.5 activate
        neighbor 192.168.2.7 activate
        redistribute connected
        network 192.168.1.2/32
        network 192.168.100.2/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.11 activate
        neighbor 192.168.1.11 next-hop-unchanged
        neighbor 192.168.1.11 send-community extended
        neighbor 192.168.1.12 activate
        neighbor 192.168.1.12 next-hop-unchanged
        neighbor 192.168.1.12 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0020
!export-password: default
hostname Zone1-Rack3-Leaf3a
user admin group administrators password ciphertext AQBapd0lfpkb1JQ/PeM7VAdLaPTFpCWvep8Ky+FcXCXZQjzuYgAAABM81mz9TPm9mRgJCcs5jU94yotLIjlGFmqRd7CPrUe2I/hhn9STUUgq5O+A0aM94fIRyUcLkDyRzgAKnzk1HmDNlT1yWnxptSrdw2lh0C9wBxf/UpiQFZe+RU1NxIoXC18J
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
vlan 11
    name VLAN 11
    description Server VLAN
evpn
    vlan 11
        rd auto
        route-target export 1:11
        route-target import 1:11
interface mgmt
    no shutdown
    ip static 10.10.10.60/24
    default-gateway 10.10.10.254
system interface-group 1 speed 10g
    !interface group 1 contains ports 1/1/1-1/1/12
system interface-group 3 speed 10g
    !interface group 3 contains ports 1/1/25-1/1/36
system interface-group 4 speed 10g
    !interface group 4 contains ports 1/1/37-1/1/48
interface lag 1
    no shutdown
    description VSX ISL LAG
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.112/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface 1/1/49 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.8/31
interface 1/1/50 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.10/31
interface loopback 0
    ip address 192.168.1.5/32
interface loopback 1
    ip address 192.168.100.5/32
interface vxlan 1
    source ip 192.168.100.5
    no shutdown
    vni 11
        vlan 11
vsx
    inter-switch-link lag 1
    role primary
    keepalive peer 192.168.1.113 source 192.168.1.112
    no split-recovery
router bgp 65003
    bgp router-id 192.168.1.5
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.11 remote-as 65101
    neighbor 192.168.1.11 ebgp-multihop 3
    neighbor 192.168.1.11 update-source loopback 0
    neighbor 192.168.1.12 remote-as 65101
    neighbor 192.168.1.12 ebgp-multihop 3
    neighbor 192.168.1.12 update-source loopback 0
    neighbor 192.168.2.9 remote-as 65101
    neighbor 192.168.2.11 remote-as 65101
    neighbor 192.168.2.13 remote-as 65101
    neighbor 192.168.2.15 remote-as 65101
    address-family ipv4 unicast
        neighbor 192.168.2.9 activate
        neighbor 192.168.2.11 activate
        neighbor 192.168.2.13 activate
        neighbor 192.168.2.15 activate
        redistribute connected
        network 192.168.1.5/32
        network 192.168.100.5/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.11 activate
        neighbor 192.168.1.11 next-hop-unchanged
        neighbor 192.168.1.11 send-community extended
        neighbor 192.168.1.12 activate
        neighbor 192.168.1.12 next-hop-unchanged
        neighbor 192.168.1.12 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0020
!export-password: default
hostname Zone1-Rack3-Leaf3b
user admin group administrators password ciphertext AQBapaHRO1zdYAmv8jyi6BEy2EdGo7mXog4SaUdBnB6VBVmrYgAAAMswpUXLpjUGA0QadW9dCf7EgZkDyT6oT740N0z8ey2PTAAz8DT02vzpz1sAo27jMoqJ3YCXA0bW05qG+CWqweUfanbUEccqyrEu8SpcQjUoYdHYFZFHFtniXxA7d9wFijPV
!
!
!
ssh server vrf mgmt
!
!
!
!
!
vlan 1
vlan 11
    name VLAN 11
    description Server VLAN
evpn
    vlan 11
        rd auto
        route-target export 1:11
        route-target import 1:11
interface mgmt
    no shutdown
    ip static 10.10.10.61/24
    default-gateway 10.10.10.254
system interface-group 1 speed 10g
    !interface group 1 contains ports 1/1/1-1/1/12
system interface-group 3 speed 10g
    !interface group 3 contains ports 1/1/25-1/1/36
system interface-group 4 speed 10g
    !interface group 4 contains ports 1/1/37-1/1/48
interface lag 1
    no shutdown
    description VSX ISL LAG
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface 1/1/1 
    no shutdown
    no routing
    vlan access 11
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.113/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface 1/1/49 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.12/31
interface 1/1/50 
    no shutdown
    mtu 9198
    description Spine-Uplink
    ip mtu 9198
    ip address 192.168.2.14/31
interface loopback 0
    ip address 192.168.1.6/32
interface loopback 1
    ip address 192.168.100.6/32
interface vxlan 1
    source ip 192.168.100.6
    no shutdown
    vni 11
        vlan 11
vsx
    inter-switch-link lag 1
    role secondary
    keepalive peer 192.168.1.112 source 192.168.1.113
    no split-recovery
router bgp 65003
    bgp router-id 192.168.1.6
    bgp fast-external-fallover
    bgp bestpath as-path multipath-relax
    neighbor 192.168.1.11 remote-as 65101
    neighbor 192.168.1.11 ebgp-multihop 3
    neighbor 192.168.1.11 update-source loopback 0
    neighbor 192.168.1.12 remote-as 65101
    neighbor 192.168.1.12 ebgp-multihop 3
    neighbor 192.168.1.12 update-source loopback 0
    neighbor 192.168.2.9 remote-as 65101
    neighbor 192.168.2.11 remote-as 65101
    neighbor 192.168.2.13 remote-as 65101
    neighbor 192.168.2.15 remote-as 65101
    address-family ipv4 unicast
        neighbor 192.168.2.9 activate
        neighbor 192.168.2.11 activate
        neighbor 192.168.2.13 activate
        neighbor 192.168.2.15 activate
        redistribute connected
        network 192.168.1.6/32
        network 192.168.100.6/32
    exit-address-family
    address-family l2vpn evpn
        neighbor 192.168.1.11 activate
        neighbor 192.168.1.11 next-hop-unchanged
        neighbor 192.168.1.11 send-community extended
        neighbor 192.168.1.12 activate
        neighbor 192.168.1.12 next-hop-unchanged
        neighbor 192.168.1.12 send-community extended
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt