Before Maintenance Window Checklist
Things to capture before doing maintenance on the SD-WAN fabric. The After Maintenance Window Checklist repeats every section below in the same order, so the two can be worked side by side.
Take these captures immediately before the window opens, not days ahead. A snapshot from last week is not a baseline.
Preparation
- Make a new folder named with the date of the outage and the change control number(s)
- Inside it, create a before folder and an after folder
- Find the organization's critical application list
- Have application tester(s) on hand
- Depending on the scope of the change you may be affecting one or many org apps
- Identify critical hubs/headends
- Identify 2-5 critical spoke locations
- Schedule the users and testers below for the outage window
Identify critical apps in the organization
- Voice
- Video
- App 1
- App 2
- App 3
Following best practice rules, making Custom Applications for your organization will make this step much easier. You can easily filter for one application or application group if that is all the particular outage window is covering. Another advantage is you do not have to research the application for traffic flows, ports, protocols, IP addresses, URLs etc. This information will be in the Application engine.
Identify critical users and SME testers
- USER 1
- USER 2
- USER 3
- USER 4
- SME APP 1
- SME APP 2
- SME APP 3
- SME APP 4
What goes in the folder
Use the same filenames in the before and after folders so the compare is folder to folder. Where a filename starts with hostname, use that EdgeConnect's hostname.
| Artifact | Where | Filename |
|---|---|---|
| Orchestrator active alarms | Alarms > Active | orch-alarms (export or screenshot) |
| Reachability | Monitoring > Reachability | reachability (export or screenshot) |
| NMS alarms and down devices | Enterprise NMS | nms-alarms, nms-down-devices |
| Tunnels - overlay | Configuration > Tunnels > Overlay | tunnels-report-overlay.csv |
| Tunnels - underlay | Configuration > Tunnels > Underlay | tunnels-report-underlay.csv |
| Tunnels - passthrough | Configuration > Tunnels > Passthrough | tunnels-report-passthrough.csv |
| Routes - Method 1 (all gateways) | Configuration > Routes | routes-report.csv |
| Routes - Method 2 (per gateway) | Configuration > Routes | hostname_routes_report.csv |
| BGP summary | Configuration > BGP > Summary | bgp-summary-report.csv |
| BGP peers | Configuration > BGP > Peers | bgp-peers-report.csv |
| OSPF state | Configuration > OSPF | ospf-report (export or screenshot) |
| Flows - hubs and critical spokes | Monitoring > Active & Recent Flows | hostname_flows.csv |
| Configuration backup | Administration > Backup Now | Held in Orchestrator, comment Before outage (change control number) |
| Application test evidence | Flow details, app performance stats | hostname_app_flows.csv, screenshots |
Orchestrator Alarms and Reachability
- Navigate to: Alarms > Active
- Export or screenshot, save to before as orch-alarms
- Navigate to: Monitoring > Reachability
- Note any "Not connected" EdgeConnect gateways
- Verify connected gateways are reachable (via both Orchestrator and Portal websockets)
- Export or screenshot, save to before as reachability
Enterprise NMS Tools - State Capture
Log in to any enterprise network monitoring system(s) and capture the following:
- Current/active alarms
- Current down/offline devices
- Export or screenshot the state details
- Save to before as nms-alarms and nms-down-devices
- Put the affected devices into maintenance mode in the NMS so the window does not page the on-call team
Tunnel States
Note: use exports and/or screenshots
- Select all gateways
- Navigate to: Configuration > Tunnels
- Select "Overlay", click "Export", save to before as tunnels-report-overlay
- Select "Underlay", click "Export", save to before as tunnels-report-underlay
- Select "Passthrough", click "Export", save to before as tunnels-report-passthrough
Edge Connect Route Snapshot
There are generally two methods to create a snapshot of the SD-WAN route table via the Orchestrator UI:
Method #1 - Global report including all gateways (single CSV file)
- Select all gateways in the navigation tree
- Navigate to Configuration > Routes
- Ensure "All" is selected and all filters are cleared
- Choose "Export"
- Save to before as routes-report.csv
Method #2 - Single report for each gateway (multiple CSVs)
- Select one gateway in the navigation tree
- Navigate to Configuration > Routes
- Ensure "All" is selected and all filters are cleared
- Choose "Export"
- Save to before as hostname_routes_report.csv
- Repeat for each gateway in the SD-WAN fabric
Method #1 is easier to complete, but is more work to diagnose a problem if there is a change in route table data.
Method #2 is more tedious, but provides an easier way to diagnose a route table change.
Whichever method you choose, use the same one after the window. The two exports must be like for like.
BGP and OSPF State Snapshot
- Select all gateways in the navigation tree
- Navigate to Configuration > BGP
- Choose "Summary" and click "Export", save to before as bgp-summary-report.csv
- Choose "Peers" and click "Export", save to before as bgp-peers-report.csv
- Navigate to Configuration > OSPF
- Export or screenshot the neighbor state, save to before as ospf-report
Flow Table Snapshot (Hubs and Critical Spokes)
- Select a hub gateway
- Navigate to Monitoring > Active & Recent Flows
- Ensure all filters are cleared and the search box is empty
- Click Export
- Save to before as hostname_flows.csv
- Repeat for each hub gateway
- Repeat for each critical spoke gateway identified in Preparation
Active flow analysis
- With no flow filter applied and Bytes set to "Total"
- Sort the flow table by bytes IN - note the number of flows with 0 bytes
- Sort the flow table by bytes OUT - note the number of flows with 0 bytes
Configuration Backup
- Select all appliances
- Navigate to: Administration > Backup Now
- Add comment "Before outage (change control number)"
- Click Backup
Test Applications for Function
Test all used applications for voice/video, or at least those "critical" to the organization. This is the baseline the testers will compare against after the window.
Teams, Zoom, Google, VoIP phones, etc.
- Voice
- Can call in both directions and get a ring
- Flows and direction (one method is fine - flow export is easier for large volumes)
- Flow export - save as EC name - save to before
- Screenshot - save as EC name - save to before
- Both parties can hear each other
- Test all used organization voice applications
- Note performance - quality and any existing issues
- Find internal application performance statistics, screenshot and save to before
- Video
- Able to connect a session on organization apps
- Both parties can see each other
- Note performance of video - quality and any existing issues
- Find internal application performance statistics, screenshot and save to before
- App 1-2-3
- Test application for functions
- Filter for application flows, export and save to before
- The flow snapshot above may cover this; having a snapshot for these critical apps while in use can be very useful. This is up to the org/engineers of the organization
- Screenshot the "Performance" tab in flow details
- Screenshot the "General" tab in flow details
- Save the above to before
- Find internal application performance statistics, screenshot and save to before
- Filter for application flows, export and save to before
- Ask tester(s) for feedback on performance
- Document for reference later
- User should do a "full transaction" in the application
- Test application for functions
Updated about 2 months ago