Systems Administration

This section documents the infrastructure behind this website and related work (summarized in the 'Projects' section).

Cisco Catalyst 3850 Network Segmentation Lab

This project is a hands-on enterprise-style networking lab built around a Cisco Catalyst 3850 running Cisco IOS XE. It covers VLAN segmentation, Layer 3 switching, inter-VLAN routing, ACL-based access control, DHCP relay, Layer 2 security, secure administration, centralized infrastructure services, and network monitoring.

Two laptops serve as user and administrative endpoints, while a Raspberry Pi provides centralized DHCP, NTP, syslog collection, and SNMPv3-based management. The lab is intentionally segmented into user, administrative, and server networks so routing, security controls, service availability, and troubleshooting can be tested on physical hardware.

IP addresses and subnets shown on this page have been replaced with representative private addressing. Credentials and other environment-specific details are omitted.

    Home Router / Existing LAN (VLAN 1)
                    |
                Gi1/0/1
                    |
            Cisco Catalyst 3850
            Layer 2 + Layer 3
                    |
    +---------------+--------------+                Switch Security Controls:
    |               |              |                 - DHCP Snooping
 Gi1/0/3         Gi1/0/4         Gi1/0/5             - Dynamic ARP Inspection
 VLAN 10         VLAN 20         VLAN 30             - IP Source Guard
 USERS            ADMIN          SERVERS
    |               |              |
 Laptop 1        Laptop 2       Raspberry Pi
 10.100.10.10   10.100.20.10    10.100.30.10
    |               |              |
    +--------- Inter-VLAN ---------+
                 Routing
                    | 
                   ACLs
                  
 Raspberry Pi Infrastructure Services:
   +-------------+-------------+
   |             |             |
   v             v             v
  DHCP          NTP          Syslog
  dnsmasq       chrony       rsyslog
   ^             ^             ^
   |             |             |
 DHCP relay     Time           +---------- SNMPv3 Polling
                                           Pi <------> 3850
            
  • Switch: Cisco Catalyst 3850-48P running Cisco IOS XE 16.12
  • Segmentation: Separate VLANs for USERS, ADMIN, and SERVERS, alongside the existing home LAN on VLAN 1
  • Layer 2 Security: DHCP Snooping, Dynamic ARP Inspection, and IP Source Guard
  • Routing: Layer 3 SVIs on the 3850 provide default gateways and inter-VLAN routing
  • Access Control: Extended ACLs restrict traffic initiated from the USERS VLAN
  • DHCP: Raspberry Pi DHCP server with three scopes and Cisco DHCP relay
  • NTP: Raspberry Pi running Chrony as the lab time source for consistent timestamps across network-management services
  • Administration: SSH-only switch management with unnecessary HTTP/HTTPS management disabled
  • Monitoring: Centralized Cisco syslog collection and authenticated/encrypted SNMPv3 polling from the Raspberry Pi
  • Testing: Physical laptops and a Raspberry Pi used to verify DHCP, routing, ACL behavior, and infrastructure services

Network Design

The 3850 performs Layer 3 routing for the three lab VLANs while VLAN 1 remains connected to the existing home network. The lab VLANs are kept logically separate so that traffic policy and infrastructure services can be tested without depending on the normal home LAN.

VLAN Role Subnet Gateway Primary Device
1 Existing home LAN 10.100.1.0/24 Home router Desktop / website Pi
10 USERS 10.100.10.0/24 10.100.10.1 User laptop
20 ADMIN 10.100.20.0/24 10.100.20.1 Admin laptop
30 SERVERS 10.100.30.0/24 10.100.30.1 Raspberry Pi

Physical Port Assignment

Port Purpose VLAN
Gi1/0/1 Uplink to home router 1
Gi1/0/2 Unmanaged switch for desktop / website Pi 1
Gi1/0/3 USERS laptop 10
Gi1/0/4 ADMIN laptop 20
Gi1/0/5 Raspberry Pi infrastructure server 30

Layer 2 Security & Network Management

The lab was extended with Layer 2 security controls and centralized event logging to practice switch hardening, endpoint validation, and operational troubleshooting.

  • DHCP Snooping: Builds trusted IP-to-MAC bindings from observed DHCP leases
  • Dynamic ARP Inspection: Validates ARP traffic against the DHCP snooping binding table
  • IP Source Guard: Restricts access-port traffic to valid source IP and MAC bindings
  • Centralized Syslog: Cisco switch events are forwarded to a Raspberry Pi and retained using rsyslog
  • SNMPv3: The Raspberry Pi polls switch status and interface information using authenticated and encrypted SNMPv3
  • NTP: Chrony provides a shared time source so switch and server logs can be correlated correctly

DHCP Snooping & Dynamic ARP Inspection

DHCP snooping provides the trust data used by the other Layer 2 security controls. Dynamic ARP Inspection compares observed ARP messages against the snooping binding table, while IP Source Guard applies similar source validation to traffic entering configured access ports.

During testing, Dynamic ARP Inspection correctly generated %SW_DAI-4-DHCP_SNOOPING_DENY messages when a client's configured IP address did not match the IP address associated with its MAC address in the DHCP snooping table.

This provided a useful example of why DHCP lease state, host configuration, and switch security bindings must agree. The event was visible both on the switch and through the centralized syslog collector.

SNMPv3 Monitoring

SNMPv3 was configured on the Catalyst 3850 so the Raspberry Pi can query operational information from the switch without relying on plaintext SNMP community strings. The configuration uses SNMPv3 authentication and privacy, providing both message authentication and encryption.

The Raspberry Pi uses command-line SNMP tools to retrieve system state, uptime, interface descriptions, interface status, traffic counters, and error counters. This provides a lightweight headless monitoring workflow without requiring a graphical monitoring platform.

snmp-server view NMS-VIEW iso included
snmp-server group NMS-GROUP v3 priv read NMS-VIEW
snmp-server user <user> NMS-GROUP v3 auth sha <auth-secret> priv aes 128 <privacy-secret>

SNMP is restricted to the Pi with an ACL:

ip access-list standard SNMP-MANAGERS
 permit host 10.100.30.10
 deny any log

snmp-server group NMS-GROUP v3 priv read NMS-VIEW access SNMP-MANAGERS

Example SNMP queries from the management Pi:

# System uptime
snmpget -v3 -l authPriv \
  -u <user> \
  -a SHA -A '<auth-secret>' \
  -x AES -X '<privacy-secret>' \
  10.100.30.1 \
  1.3.6.1.2.1.1.3.0

# Interface descriptions
snmpwalk -v3 -l authPriv \
  -u <user> \
  -a SHA -A '<auth-secret>' \
  -x AES -X '<privacy-secret>' \
  10.100.30.1 \
  1.3.6.1.2.1.2.2.1.2

Additional interface OIDs were used to inspect operational status, inbound/outbound traffic counters, and interface errors. This allowed switch state to be checked remotely from the headless Raspberry Pi.

VLANs and Inter-VLAN Routing

Access ports place each lab endpoint into its intended VLAN. Switch Virtual Interfaces (SVIs) provide the Layer 3 default gateways, and ip routing enables the 3850 to route between the connected VLAN networks.

vlan 10
 name USERS

vlan 20
 name ADMIN

vlan 30
 name SERVERS

interface Vlan10
 ip address 10.100.10.1 255.255.255.0
 no shutdown

interface Vlan20
 ip address 10.100.20.1 255.255.255.0
 no shutdown

interface Vlan30
 ip address 10.100.30.1 255.255.255.0
 no shutdown

ip routing

Endpoint ports use PortFast and BPDU Guard because they connect to end devices rather than downstream switches.

interface GigabitEthernet1/0/3
 switchport mode access
 switchport access vlan 10
 spanning-tree portfast
 spanning-tree bpduguard enable

interface GigabitEthernet1/0/4
 switchport mode access
 switchport access vlan 20
 spanning-tree portfast
 spanning-tree bpduguard enable

interface GigabitEthernet1/0/5
 switchport mode access
 switchport access vlan 30
 spanning-tree portfast
 spanning-tree bpduguard enable

Access Control

The USERS VLAN has an inbound extended ACL. It permits required services while preventing an ordinary user endpoint from freely initiating connections into the ADMIN and SERVERS networks.

ip access-list extended USERS-IN
 5 permit udp any eq 68 any eq 67
 10 permit icmp 10.100.10.0 0.0.0.255 host 10.100.10.1 echo
 20 permit icmp 10.100.10.0 0.0.0.255 host 10.100.30.10 echo
 30 permit tcp 10.100.10.0 0.0.0.255 host 10.100.30.10 eq 8080
 40 permit tcp 10.100.10.0 0.0.0.255 10.100.20.0 0.0.0.255 established
 50 permit icmp 10.100.10.0 0.0.0.255 10.100.20.0 0.0.0.255 echo-reply
 60 deny tcp 10.100.10.0 0.0.0.255 host 10.100.30.10 eq 22 log
 70 deny ip 10.100.10.0 0.0.0.255 10.100.20.0 0.0.0.255 log
 80 deny ip 10.100.10.0 0.0.0.255 10.100.30.0 0.0.0.255 log
 90 deny ip 10.100.10.0 0.0.0.255 10.100.0.0 0.0.0.255 log
 100 permit ip 10.100.10.0 0.0.0.255 any

interface Vlan10
 ip access-group USERS-IN in

The DHCP rule appears before the subnet-specific entries because a client without a lease initially sends from 0.0.0.0:68. ACL match counters were used to confirm which rules were handling test traffic.

Centralized DHCP and DHCP Relay

The Raspberry Pi uses the static address 10.100.30.10/24 with 10.100.30.1 as its gateway. It runs dnsmasq as the authoritative DHCP server for all three lab VLANs.

VLAN Lease Range Default Gateway
10 10.100.10.100 - 10.100.10.199 10.100.10.1
20 10.100.20.100 - 10.100.20.199 10.100.20.1
30 10.100.30.100 - 10.100.30.199 10.100.30.1
# /etc/dnsmasq.d/lab-dhcp.conf

port=0
interface=eth0
dhcp-authoritative

dhcp-range=set:vlan10,10.100.10.100,10.100.10.199,255.255.255.0,12h
dhcp-option=tag:vlan10,option:router,10.100.10.1

dhcp-range=set:vlan20,10.100.20.100,10.100.20.199,255.255.255.0,12h
dhcp-option=tag:vlan20,option:router,10.100.20.1

dhcp-range=set:vlan30,10.100.30.100,10.100.30.199,255.255.255.0,12h
dhcp-option=tag:vlan30,option:router,10.100.30.1

DHCP broadcasts do not normally cross routed VLAN boundaries, so VLANs 10 and 20 use the 3850 as a DHCP relay:

interface Vlan10
 ip helper-address 10.100.30.10

interface Vlan20
 ip helper-address 10.100.30.10

VLAN 30 does not require a helper because the DHCP server is directly attached to that broadcast domain. DHCP operation was verified by watching dnsmasq logs while clients released and renewed their leases.

NTP Service

The Raspberry Pi also runs chrony as the lab NTP service. Keeping the switch and infrastructure host on a common time source makes event logs and troubleshooting much easier to correlate.

# /etc/chrony/chrony.conf
allow 10.100.30.0/24

# Local fallback for the isolated lab when upstream time is unavailable
local stratum 10

The 3850 uses the Pi as its preferred NTP server:

ntp source Vlan30
ntp server 10.100.30.10 prefer

Synchronization can be checked with chronyc tracking and chronyc sources -v on the Pi, and show ntp associations, show ntp status, and show clock detail on the switch.

Secure Administration

Remote management uses local authentication over SSH. Telnet is not used, and the unnecessary built-in HTTP/HTTPS management service is disabled.

ip ssh version 2

line vty 0 15
 login local
 transport input ssh

no ip http server
no ip http secure-server

Credentials, secrets, private keys, and other sensitive configuration values are intentionally omitted from public documentation.

Verification and Testing

The lab was tested from both the Cisco and Linux sides instead of considering configuration complete as soon as commands were accepted. Common verification commands included:

# Cisco 3850
show vlan brief
show interfaces status
show ip interface brief
show ip route
show ip arp
show access-lists USERS-IN
show running-config interface Vlan10
show running-config interface Vlan20
show running-config interface Vlan30
show ntp associations
show ntp status
show snmp
show running-config | include snmp
show access-lists SNMP-MANAGERS

# Raspberry Pi
ip addr
ip route
nmcli connection show --active
systemctl status dnsmasq
systemctl status chrony
sudo journalctl -u dnsmasq -f
cat /var/lib/misc/dnsmasq.leases
snmpget ...
snmpwalk ...

# Windows clients
ipconfig /all
ipconfig /release
ipconfig /renew
ping <gateway-or-test-host>

One useful failure test was temporarily removing ip helper-address from an SVI. DHCP then failed for that routed VLAN, demonstrating why a relay is required. Restoring the helper restored DHCP operation.

Monitoring was tested by querying switch uptime and interface state over SNMPv3 while simultaneously watching centralized syslog. For example, disconnecting and reconnecting the USERS laptop generated link-state events in syslog while SNMP reported the current interface state.

This demonstrated the complementary roles of the management services: syslog records events as they occur, SNMP provides current device state and counters, and NTP keeps timestamps consistent between systems.

Operational Notes

The Catalyst switch is intentionally powered down when the lab is not in use. At night, the home-router uplink is moved directly back to the unmanaged switch so the desktop and website Raspberry Pi are not dependent on the lab switch.

The infrastructure Pi maintains two NetworkManager profiles. The lab profile gives it its static VLAN 30 address, while the temporary home profile can be used when Internet access is required for package installation or maintenance.

lab-static
 10.100.30.10/24
 gateway 10.100.30.1

home-dhcp
 automatic DHCP from the normal home LAN
 used temporarily for package installation / maintenance

Runbooks

These runbooks document repeatable troubleshooting procedures for the lab. They focus on verifying the failed layer before changing configuration.

This is a learning environment. Credentials, passwords, secrets, and other private network details are intentionally omitted from public documentation.

Verifying Layer 2 security controls

Use this procedure to confirm that DHCP Snooping, Dynamic ARP Inspection, and IP Source Guard are active and operating with valid client bindings.

Verify DHCP Snooping status and the VLANs on which it is enabled:

show ip dhcp snooping

Inspect the learned DHCP binding table:

show ip dhcp snooping binding

A valid client should have a binding containing its MAC address, assigned IP address, VLAN, interface, and remaining lease information.

Verify Dynamic ARP Inspection:

show ip arp inspection

Review DAI counters and statistics:

show ip arp inspection statistics

Verify IP Source Guard on access interfaces:

show ip verify source

Compare the switch binding information with the client's actual network configuration. The client's IP address, MAC address, VLAN, and switch port should agree with the snooping table.

If they do not match, investigate the DHCP lease or client configuration before modifying the security policy.

Troubleshooting a Dynamic ARP Inspection denial

Use this procedure when the switch reports a message similar to:

%SW_DAI-4-DHCP_SNOOPING_DENY: Invalid ARP...

First identify the affected VLAN and access interface from the log message.

Check the DHCP Snooping binding table:

show ip dhcp snooping binding

Locate the affected MAC address and record the IP address, VLAN, and interface associated with the binding.

On the client, verify its current IP and MAC configuration.

Compare the client state with the switch binding:

  • Does the client IP match the snooping binding?
  • Does the MAC address match?
  • Is the client connected to the expected VLAN and access port?
  • Was the client previously using a different DHCP lease?

If the host configuration does not match the binding table, correct the addressing or renew the DHCP lease before changing DAI configuration.

After correcting the mismatch, verify that ARP traffic is no longer being denied:

show ip arp inspection statistics

Confirm end-to-end connectivity by testing the client's gateway and permitted destinations.

Troubleshooting SNMPv3 monitoring

Use this procedure when the Raspberry Pi cannot retrieve switch information using SNMPv3.

  1. Confirm basic Layer 3 connectivity from the Pi to the switch:
    ping -c 4 10.100.30.1
  2. Confirm that SNMP is configured on the switch:
    show running-config | include snmp
    show snmp
  3. Verify the SNMP management ACL:
    show access-lists SNMP-MANAGERS
    The Raspberry Pi management address should be permitted.
  4. Test a simple system OID from the Pi:
    snmpget -v3 -l authPriv \
    -u <user> \
    -a SHA -A '<auth-secret>' \
    -x AES -X '<privacy-secret>' \
    10.100.30.1 \
    1.3.6.1.2.1.1.3.0
  5. If authentication fails, verify that the SNMPv3 username, authentication algorithm, authentication secret, privacy algorithm, and privacy secret match the switch configuration.
  6. If authentication succeeds but a requested OID fails, confirm that the configured SNMP view permits access to that OID.
  7. Test interface discovery:
    snmpwalk -v3 -l authPriv \
    -u <user> \
    -a SHA -A '<auth-secret>' \
    -x AES -X '<privacy-secret>' \
    10.100.30.1 \
    1.3.6.1.2.1.2.2.1.2
  8. Review Cisco logs for authentication or access failures if necessary.
Checking centralized Cisco syslog collection

Use this procedure to confirm that switch events are reaching the Raspberry Pi management host.

On the Cisco switch, review the current logging configuration:

show logging

Confirm that a remote logging host is configured and that the expected logging level is being forwarded.

On the Raspberry Pi, confirm that the syslog service is running:

sudo systemctl status rsyslog

Follow incoming log activity:

sudo tail -f /var/log/cisco.log

Generate a harmless test event by disconnecting and reconnecting a lab access-port device.

Confirm that corresponding interface down/up events appear in the centralized logs.

Layer 2 security events such as DHCP Snooping or Dynamic ARP Inspection violations should also appear when generated.

Troubleshooting VLAN and inter-VLAN connectivity

Use this procedure when a client cannot reach another device, its default gateway, or a service on another VLAN.

  1. Confirm physical interface state:
    show interfaces status
    show ip interface brief
  2. Verify the client-facing port belongs to the expected VLAN:
    show vlan brief
    show interfaces switchport
  3. Verify the client IP address, subnet mask, and default gateway.
  4. Ping the client's own SVI before testing remote networks.
  5. Confirm the required connected routes:
    show ip route
  6. If Layer 3 connectivity works but traffic is blocked, inspect the ACL and its counters:
    show access-lists USERS-IN
    show ip interface Vlan10
Troubleshooting DHCP on VLAN 10 or VLAN 20
  1. Verify the Pi is reachable:
    ping 10.100.30.10
  2. Check the DHCP service:
    sudo systemctl status dnsmasq
    sudo dnsmasq --test
  3. Watch DHCP activity:
    sudo journalctl -u dnsmasq -f
  4. On the switch, confirm the helper address exists on the affected SVI:
    show running-config interface Vlan10
    show running-config interface Vlan20
  5. For VLAN 10, confirm the ACL permits DHCP:
    show access-lists USERS-IN
  6. Renew the client lease:
    ipconfig /release
    ipconfig /renew
    ipconfig /all
  7. Confirm the resulting lease on the Pi:
    cat /var/lib/misc/dnsmasq.leases
Troubleshooting the Raspberry Pi infrastructure server
  1. Check the active NetworkManager profile:
    nmcli connection show --active
    nmcli device status
  2. Verify the lab address and route:
    ip -4 addr show eth0
    ip route
  3. Ping the VLAN 30 gateway:
    ping -c 4 10.100.30.1
  4. Check infrastructure services:
    systemctl status dnsmasq
    systemctl status chrony
    systemctl status ssh
  5. Confirm expected listening ports:
    sudo ss -lunp | grep ':67'
    sudo ss -lunp | grep ':123'
    sudo ss -lntp | grep ':22'
Troubleshooting NTP synchronization
  1. Check Chrony:
    systemctl status chrony
    chronyc tracking
    chronyc sources -v
  2. Confirm UDP 123 is listening:
    sudo ss -lunp | grep ':123'
  3. Verify the switch can reach 10.100.30.10.
  4. Check the Cisco NTP state:
    show running-config | include ntp
    show ntp associations
    show ntp status
    show clock detail
Troubleshooting SSH access to the switch
  1. Verify the management SVI is up and reachable.
  2. Check SSH:
    show ip ssh
  3. Check VTY configuration:
    show running-config | section line vty
  4. Confirm local authentication is configured without exposing secret values:
    show running-config | include username
  5. Check current sessions:
    show users
Nightly lab shutdown and startup

Shutdown

  1. Save the switch:
    copy running-config startup-config
  2. Gracefully shut down lab hosts if necessary.
  3. Power down the Catalyst 3850.
  4. Move the home-router uplink directly back to the unmanaged switch for normal overnight connectivity.

Startup

  1. Move the uplink back to the Cisco topology, making sure there is only one Layer 2 path.
  2. Power on the 3850 and reconnect the Raspberry Pi.
  3. Verify configuration and interfaces:
    show vlan brief
    show ip interface brief
    show ip route
  4. Confirm Pi services:
    systemctl is-active dnsmasq
    systemctl is-active chrony
    systemctl is-active ssh
  5. Renew one client lease and confirm the expected subnet.

Future Extensions

  • Interface utilization graphs
  • SPAN / Wireshark packet-capture exercises
  • Automated switch configuration backups
  • Additional routed lab segments and OSPF experimentation
GitHub YouTube in