Architecture I - Campus Attached DC ToR

This workflow provisions a campus attached data center set of top-of-rack AOS-CX switches in a VSX pair based on the validated reference design.

Workflow Prerequisites

All prerequisites defined in Project Prerequisites
Ensure the provided Ansible inventory file has been modified to suit your environment, according to the instructions in Inventory Setup:
- YAML Inventory File : inventory_2tier_dedicated_dc.yml
- Excel Template File : campus_attached_dcToR_DCN_Settings.xlsx
  - Example inventory using aoscx_dcn_plugin: dynamic_campus_attached_dc_tor_inventory.yml
L3 Campus Core
DC Core switches = 2 AOS-CX switches (8xxx series, using the latest available firmware if possible)
- DC Core switches should be in a VSX pair
Out-of-Band Management (OOBM) connections to the management ports on AOS-CX switches
- Ansible control machine should be reachable via each device's OOBM

Files Used

Playbook : deploy_campus_attached_dc_tor.yml
Inventory :
YAML Inventory File : inventory_2tier_dedicated_dc.yml
- Note: This inventory file is also used for the Architecture II workflow, so it contains additional information for access switches (Zone1-Rack<1/3>-Access<1/2/3/4>). Valid values for variables relating to those devices are not necessary for this workflow.
Excel Template File : campus_attached_dcToR_DCN_Settings.xlsx
Jinja2 Templates : templates/2Tier/core.j2

Workflow Walkthrough

Prior to executing the Ansible playbook, the environment must be in this initial state:

Zone1-Core<1a/1b> - These devices each have a default configuration with an IP address (DHCP/Static) assigned to the management interface. This IP address should match the value of ansible_host for each device in the inventory.

The playbook will perform the following actions on every core device in the inventory file, using REST API unless otherwise noted:

Generate a configuration based on the template file templates/2Tier/core.j2 and values defined in the inventory

Push the generated configuration to the device using the AOS-CX Ansible SSH module aoscx_config
Enable 10g speed interface groups (if defined in the inventory) using the AOS-CX Ansible SSH module aoscx_config
Create VSX Keepalive L3 Interface
Create VSX Inter-switch link
Configure VSX attributes on the switch and specify VSX role as outlined in the inventory
Create all VLANs defined as server_vlans in the inventory
Create SVIs for all VLANs defined as core_vlan_interfaces in the inventory
Configure the multi-chassis LAGs that connect to each access switch and trunk the VLANs in trunk_vlans.
- Note: vsx_pair_mclags is a list of VSX Pair (rack# grouping) information for the core devices to use for configuring downlink interfaces. You should modify these values appropriately to match your environment.
Configure BGP neighbor for iBGP peering between the core switches

Because of path requirements, you must run this workflow from the root level of the cloned repository:

ansible-playbook deploy_campus_attached_dc_tor.yml -i inventory_2tier_dedicated_dc.yml

ansible-playbook deploy_campus_attached_dc_tor.yml -i dynamic_campus_attached_dc_tor_inventory.yml

Final Sample Configs

!
!Version ArubaOS-CX GL.10.04.0040
!export-password: default
hostname Zone1-Core1a
user admin group administrators password ciphertext AQBapeeuZ6Nw+Phok7vJbD6r75PivsY6o/r0QfxdpH1h3fQYYgAAACrisdLluFaTV+Fj1JfL0WsZPS8LBYsoE/N6qohz8bziNZQvKts2XD+d+Hgx+qrd64f4Htq7A/1mAvqetP90ljtfIOX27j/ZvVwqV6ewUQyQ7V7rFCe8BIXyVCXZD5QhqRdg
!
!
!
ssh server vrf mgmt
!
!
!
!
!
router ospf 1
    router-id 192.168.1.11
    area 0.0.0.0
vlan 1
vlan 11
    name VLAN 11
vlan 4000
spanning-tree
spanning-tree priority 1
spanning-tree config-name Zone1
spanning-tree config-revision 1
spanning-tree instance 1 vlan 11
interface mgmt
    no shutdown
    ip static 10.10.10.54/24
    default-gateway 10.10.10.254
interface lag 1
    no shutdown
    description VSX ISL
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface lag 11 multi-chassis
    no shutdown
    description Zone1-Rack1-Access1/2
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed 11
    lacp mode active
interface lag 12 multi-chassis
    no shutdown
    description Zone1-Rack3-Access3/4
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed 11
    lacp mode active
interface 1/1/23 
    no shutdown
    mtu 9198
    description Zone1-Rack3-Access3/4
    lag 12
interface 1/1/24 
    no shutdown
    mtu 9198
    description Zone1-Rack3-Access3/4
    lag 12
interface 1/1/27 
    no shutdown
    mtu 9198
    description Zone1-Rack1-Access1/2
    lag 11
interface 1/1/28 
    no shutdown
    mtu 9198
    description Zone1-Rack1-Access1/2
    lag 11
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.100/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface loopback 0
    ip address 192.168.1.11/32
    ip ospf 1 area 0.0.0.0
interface vlan11 
    ip address 11.1.1.3/24
    active-gateway ip mac 00:00:00:00:01:01
    active-gateway ip 11.1.1.1
interface vlan4000 
    description IBGP transit
    ip mtu 9198
    ip address 10.1.2.2/31
    ip ospf 1 area 0.0.0.0
    ip ospf network point-to-point
vsx
    system-mac 00:00:00:00:01:01
    inter-switch-link lag 1
    role primary
    keepalive peer 192.168.1.101 source 192.168.1.100
router bgp 65101
    bgp router-id 192.168.1.11
    neighbor 192.168.1.12 remote-as 65101
    neighbor 192.168.1.12 update-source loopback 0
    address-family ipv4 unicast
        neighbor 192.168.1.12 activate
        neighbor 192.168.1.12 next-hop-self
        network 11.1.1.0/24
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt

!
!Version ArubaOS-CX GL.10.04.0040
!export-password: default
hostname Zone1-Core1b
user admin group administrators password ciphertext AQBapW41EEHA+zskBXcBrm9Rr+euZNH+d4Q5BiGgeNrIvw1gYgAAANRxoQcfTTV7tma79JXNUOqrhv2y0xM21jrQxXeufK4O4aCwIfhuPJA7SPvno9iqnFJ9ehwrWSd5HdcyT1eb80glx6No9vqdDGKvOUfi6IOlDg6rcdaJcQWJ+tO7bZYVz5uz
!
!
!
ssh server vrf mgmt
!
!
!
!
!
router ospf 1
    router-id 192.168.1.12
    area 0.0.0.0
vlan 1
vlan 11
    name VLAN 11
vlan 4000
spanning-tree
spanning-tree priority 1
spanning-tree config-name Zone1
spanning-tree config-revision 1
spanning-tree instance 1 vlan 11
interface mgmt
    no shutdown
    ip static 10.10.10.55/24
    default-gateway 10.10.10.254
interface lag 1
    no shutdown
    description VSX ISL
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed all
    lacp mode active
interface lag 11 multi-chassis
    no shutdown
    description Zone1-Rack1-Access1/2
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed 11
    lacp mode active
interface lag 12 multi-chassis
    no shutdown
    description Zone1-Rack3-Access3/4
    no routing
    vlan trunk native 1 tag
    vlan trunk allowed 11
    lacp mode active
interface 1/1/23 
    no shutdown
    mtu 9198
    description Zone1-Rack3-Access3/4
    lag 12
interface 1/1/24 
    no shutdown
    mtu 9198
    description Zone1-Rack3-Access3/4
    lag 12
interface 1/1/27 
    no shutdown
    mtu 9198
    description Zone1-Rack1-Access1/2
    lag 11
interface 1/1/28 
    no shutdown
    mtu 9198
    description Zone1-Rack1-Access1/2
    lag 11
interface 1/1/31 
    no shutdown
    description VSX KA
    ip address 192.168.1.101/31
interface 1/1/32 
    no shutdown
    mtu 9198
    description VSX ISL
    lag 1
interface loopback 0
    ip address 192.168.1.12/32
    ip ospf 1 area 0.0.0.0
interface vlan11 
    ip address 11.1.1.4/24
    active-gateway ip mac 00:00:00:00:01:01
    active-gateway ip 11.1.1.1
interface vlan4000 
    description IBGP transit
    ip mtu 9198
    ip address 10.1.2.3/31
    ip ospf 1 area 0.0.0.0
    ip ospf network point-to-point
vsx
    system-mac 00:00:00:00:01:01
    inter-switch-link lag 1
    role secondary
    keepalive peer 192.168.1.100 source 192.168.1.101
router bgp 65101
    bgp router-id 192.168.1.12
    neighbor 192.168.1.11 remote-as 65101
    neighbor 192.168.1.11 update-source loopback 0
    address-family ipv4 unicast
        neighbor 192.168.1.11 activate
        neighbor 192.168.1.11 next-hop-self
        network 11.1.1.0/24
    exit-address-family
!
https-server rest access-mode read-write
https-server vrf mgmt