Sunday, July 14, 2019

Cisco IOS & SNMP: A backdoor into devices you can't access.





We've all been there. You desparately need to get into a router or a switch, but every username and password you try doesn't seem to work. Or even worse, you can ping the device, see it in CDP but SSH isn't replying and telnet is disabled (presumably because the device doesn't have an RSA key). If you were close to the device, sure, you could walk over to it and try to console in. What if that's not an option? Reload and do password recovery?? Well, maybe not. In today's super quick post I want to give you an alternative if SNMP is enabled and you know the rw community string.

Ingredients:

  1. Router you can't log into.
    1. SNMP read-write community string
  2. SNMP tools installed on your machine. (e.g. net-snmp for Windows)
  3. TFTP Server running on your machine.
  4. Deep breaths. Seriously, calm down. You're locked out of a router, it's not the end of the world or anything Eddie. Yeah, I'm calling you out Eddie.

Don't Panic

Now that you're definitely not panicking, let's open a command prompt on our machine. Since this is a lab, I'm actually using an Ubuntu docker image in GNS3, however the actual process and syntax doesn't change at all if you're running net-snmp in Windows. Where I'm in bash, you'll be in CMD issuing the exact same commands. My TFTP server is also running on this Ubuntu docker image, but again, this works absolutely fine with SolarWind's tftp server or tftpd64. Finally my device is a router named R1 running at 192.168.122.211, and my Ubuntu machine is 192.168.122.93. Let's first confirm the state of things.

root@LX1:~# ssh admin@192.168.122.211     
Password:
Password:
Password:
admin@192.168.122.211's password:
Connection to 192.168.122.211 closed by remote host.
Connection to 192.168.122.211 closed.
root@LX1:~#  



Yup. Good and locked out. So the first thing I'll need to do is create a file in my TFTP directory that contains the changes I wish to make. For this post I'm just going to change the username admin's password to cisco123. However, get creative with this. You could just as easily modify vty lines to permit telnet temporarily, or disable AAA if it's pointing to TACACS/RADIUS for authentication. So I'll create a small file called 'configme.txt' and put said file in my TFTP root directory. Important note here, you don't need to put the full running config in this file, only the changes you wish to merge with existing config on the router. That is to say, this doesn't overwrite and replace the entire running-config, say it with me Eddie "merge". Here's what my file looks like.

root@LX1:~# cat /var/TFTP-ROOT/configme.txt
username admin privi 15 secret cisco123
!
end
root@LX1:~# 


Putting the word "end" on a line all by itself let's the router know it's reached the end of the config file. You don't have to do this, but if you don't the router generates an error saying the config file ended unexpectedly. Alright, now that we have all that setup, and our tftp server is running, let's send our router some commands via SNMP and tell it to download 'configme.txt' and merge it into our running-config. Lucky for me, I just so happen to know this device has 'write' as a read-write community string.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1

Created directory: /var/lib/snmp/mib_indexes
iso.3.6.1.4.1.9.9.96.1.1.1.1.2.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 1


iso.3.6.1.4.1.9.9.96.1.1.1.1.3.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 4


iso.3.6.1.4.1.9.9.96.1.1.1.1.4.111 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93


iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s configme.txt


iso.3.6.1.4.1.9.9.96.1.1.1.1.6.111 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 1


iso.3.6.1.4.1.9.9.96.1.1.1.1.14.111 = INTEGER: 1


Now let's quickly confirm everything works as expected.

root@LX1:~# ssh admin@192.168.122.211 
Password:


R1#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.122.211 YES DHCP   up                    up     

R1#show logging | inc tftp
Jul 14 15:40:49.808: %SYS-5-CONFIG_I: Configured from tftp://192.168.122.93/configme.txt by console
R1#exit
Connection to 192.168.122.211 closed.



What just happened?

All in all we sent (6) SNMP strings to the router, I'll really quickly break those out here.


  1. "1.3.6.1.4.1.9.9.96.1.1.1.1.2.[random_number] i 1" sets the transfer protocol to TFTP.
  2. "1.3.6.1.4.1.9.9.96.1.1.1.1.3.[random_number] i 1" sets the source file as a network file
  3. "1.3.6.1.4.1.9.9.96.1.1.1.1.4.[random_number] i 4" sets the destination as running-config
  4. "1.3.6.1.4.1.9.9.96.1.1.1.1.5.[random_number] a 192.168.122.93" sets the TFTP Server Address
  5. "1.3.6.1.4.1.9.9.96.1.1.1.1.6.[random_number] s configme.txt" sets the file name
  6. "1.3.6.1.4.1.9.9.96.1.1.1.1.14.[random_number] i 1" initiates the transfer.

If you look above, my [random_number] was 111, now if I wanted to issues these commands again, maybe to change the file name or TFTP server address, I'd need to pick a new random number. Just re-running those MIBs with the same random number as before will error out as seen here:

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93
Error in packet.
Reason: inconsistentValue (The set value is illegal or unsupported in some way)
Failed object: iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111


However, if I change .111 to .112 we're back in business.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.2.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.112 i 1


iso.3.6.1.4.1.9.9.96.1.1.1.1.3.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.112 i 4


iso.3.6.1.4.1.9.9.96.1.1.1.1.4.112 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.112 a 192.168.122.93


iso.3.6.1.4.1.9.9.96.1.1.1.1.5.112 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.112 s configme.txt


iso.3.6.1.4.1.9.9.96.1.1.1.1.6.112 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.112 i 1


iso.3.6.1.4.1.9.9.96.1.1.1.1.14.112 = INTEGER: 1

root@LX1:~#



Best of luck getting back into your devices, that's it for this post.








Thursday, July 11, 2019

FlexVPN: Spoke-2-Spoke PSK

So in my last post I covered configuring FlexVPN with dynamic spoke to spoke communication using certificates for authentication. What if you wanted to use pre-shared-keys though? We can do that, it's not nearly as secure as certificate based authentication, but we can do that.


We'll be using the exact same topology as the last post, and seeing as I went through the config in a good bit of detail in the last post, I'll keep this one short and sweet trying to only spend time on the differences in config.

Ingredients:

  • (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
  • Patience. 
Estimate lab time: ~15min

FlexVPN Topology

FlexVPN Configuration

In the interest of exploring some more interesting features in IKEv2 with respect to pre-shared-keys, I thought we'd up the ante a bit and do both asymmetric authentication between Hub and Spokes, as well as symmetric authentication between spokes. That means we'll need an ikev2 keyring.  Without further to do, let's look at the Hub configuration. I'll call out the biggest differences in red from my last post.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common

!
ip local pool flexpool 10.254.0.10 10.254.0.254
!

ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255

!
crypto ikev2 authorization policy flexauthzpol
 pool flexpool
 route set access-list flexroutes

!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20

!
crypto ikev2 policy flexpol
 proposal flexprop

!
crypto ikev2 keyring flexkeys
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key local HubsLikeCerts
  pre-shared-key remote SoDoSpokes
 !

crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share

 keyring local flexkeys
 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1
!

crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!

crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof

!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1337
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect


So not too shabby at all right? No PKI to worry about, all we have to do here is define a keyring and call that keyring in our IKEv2 profile. On the spokes we'll see a litte more configuration within the keyring to meet the requirements I defined above (asymm for hub<->spokes and symm spoke<->spoke). Now let's look at the spoke config, this configuration is identical on both S2 and S3 routers, except the ACL changes. S2 has permit 10.0.2.0 0.0.0.255 and S3 has permit 10.0.3.0 0.0.0.255.

Spoke Routers:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common

!
ip access-list standard flexroutes
 permit 10.0.x.0 0.0.0.255

 # ACL is the only part of the config that differs spoke to spoke.
!
crypto ikev2 authorization policy flexauthzpol
 route set access-list flexroutes

!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20

!
crypto ikev2 policy flexpol
 proposal flexprop

!
crypto ikev2 keyring flexkeys
 peer hub
  address 169.254.0.5
  pre-shared-key local SoDoSpokes
  pre-shared-key remote HubsLikeCerts
 !
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key Spokes<3Certs
 !

crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local flexkeys

 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1

!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel

!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof

!
interface Tunnel0
 ip address negotiated
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect

!
interface Virtual-Template1 type tunnel
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect


So as you can see, using pre-shared-keys isn't too different from a configuration stand point than certificate based authentication. However, in the real world, what you're giving up is the ability to revoke a single router's creditials. Following a certificate auth model, if a device was lost or compromised, we could revoke it's certificate and once all devices got an up to date CRL, they'd reject any authentication attempt from that device. With pre-shared-keys we'd have to touch each one of our devices to update PSKs to achieve the same goal. No bueno. That's enough lecturing, let's do some quick verfication from S2's router to make sure it can both talk to the Hub and build a dynamic tunnel with S3.

Verification on S2

S2#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       169.254.2.5     YES NVRAM  up                    up     
GigabitEthernet2       10.0.2.1        YES NVRAM  up                    up     
Tunnel0                10.254.0.10     YES manual up                    up     
Virtual-Template1      10.254.0.10     YES unset  up                    down   
!

!
S2#show ip route static | beg ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0

!
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1224 sec
!

S2#ping 10.0.3.1 source 10.0.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/7 ms
S2#
*Jul 11 19:25:26.917: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
S2#
*Jul 11 19:25:27.160: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
S2#show ip route static | beg ^Gateway
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
3         169.254.2.5/500       169.254.3.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/19 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1262 sec


And boom goes the dynamite. That's all for this one, happy labbing.

FlexVPN: Spoke-2-Spoke Tunnels


Let's talk about FlexVPN, a prime contender as a DMVPN replacement and sometimes referred to as DMVPN phase 4. In this post, I'm going to explore the nuts and bolts of getting FlexVPN up and running between (3) routers and, for added flare, I'm going to also configure dynamic spoke-2-spoke tunnels. The dynamic spoke-2-spoke portion of the config will look very similar to those of you familiar with DMVPN, with some key differences. Before diving in, let's take a look at our diagram. I'd like to also point out in this lab, the Hub is also our certificate authority (because pre-shared keys are for suckers).

Ingredients

  • (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
  • Patience.

Estimate lab time: ~30min

 

Configuring IOS CA & Provision Certificates

 

Since we'll be using RSA for authentication, as opposed to pre-shared keys, we're going to need a trusted certificate authority and for each router to have it's own certificate issued from said CA. Now, in real life this can be either an online or offline CA running on IOS, Linux, or Windows. However, to keep things simple, in this lab I'm just going to configure the Hub router as the CA, then have all routers request a certificate from the CA.

IOS CA Configuration:

ip domain name hop16.com
!
! Replace 12:00:00 10 JUL 2019 with current time and date
do clock set 12:00:00 10 JUL 2019
!
!
ntp master 3
!
crypto pki server flexvpnca
 database level complete
 no database archive
 issuer-name CN=Hub.hop16.com,OU=NOC,L=FOO,C=US
 hash sha256
 database url flash:
 grant auto
 no shutdown
%Some server settings cannot be changed after CA certificate generation.
% Please enter a passphrase to protect the private key
% or type Return to exit
Password: 
Re-enter password: 
% Generating 1024 bit RSA keys, keys will be non-exportable...
!

All Routers (yes, even the Hub/IOS CA from above)

ip domain name hop16.com
!
crypto key generate rsa modulus 2048 label FLEXKEYS
!
crypto pki trustpoint flexvpn-tp
 enrollment url http://169.254.0.5:80
 hash sha256
 subject-name CN={ChangeForEachSite}.hop16.com,OU=NOC,L=FOO,C=US
 fqdn {ChangeForEachSite}.hop16.com
 # e.g. subject-name CN=S2.hop16.com,OU=NOC,L=FOO,C=US for S2
 # e.g. subject-name CN=S3.hop16.com,OU=NOC,L=FOO,C=US for S3
 revocation-check crl
 rsakeypair FLEXKEYS
!
crypto pki authenticate flexvpn-tp
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll flexvpn-tp
 # This command will prompt you to create a password for revocation,
 # and some additional options about including router serial number,
 # and IP address in the certification subject name. Both are
 # optional, I opted to include the router's IP address but
 #
exclude the serial number.

After a couple seconds, you'll see a prompt on each router confirming they've received their cert from the CA "%PKI-6-CERTRET: Certificate received from Certificate Authority". At this point, assuming you haven't hit any errors, we should be able to verify we have a valid certificate from our CA with 'show crypto pki certificates flexvpn-tp'.


S2#show crypto pki certificates 
Certificate
  Status: Available
  Certificate Serial Number (hex): 03
  Certificate Usage: General Purpose
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    Name: S3.hop16.com
    IP Address: 169.254.3.5
    ipaddress=169.254.3.5+hostname=S3.hop16.com
    cn=S3.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:26:30 UTC Jul 10 2019
    end   date: 14:26:30 UTC Jul 9 2020
  Associated Trustpoints: flexvpn-tp
  Storage: nvram:Hubhop16com#3.cer

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 01
  Certificate Usage: Signature
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:20:00 UTC Jul 10 2019
    end   date: 14:20:00 UTC Jul 9 2022
  Associated Trustpoints: flexvpn-tp


Configuring FlexVPN - Hub and Spoke

The biggest downside to FlexVPN in my humble opinion is that it's ssoooo config intensive. The benefit is that once you get the configs ironed out, it's incredibly easy to template. Most everything (including the vast majority of the Hub configs) can be copy/paste to other routers. That said, there are a couple bits of config unique to the Hub, and I'll call those out as we go.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common

!
! Defining a local pool is only needed on the Hub router, and only
! required if you wish to have the spokes get their tunnel IP

! dynamically which is preferred.
!
ip local pool flexpool 10.254.0.2 10.254.0.254
!
ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255

 #
 # Note here, normally your route ACL only has local routes, but we
 # need to be more general on the Hub so S2 knows how to reach S3. 
 #
!
crypto ikev2 authorization policy flexauthzpol
 # Using an address pool here so our spokes can negotiate a tunnel
 # IP from the Hub.

 pool flexpool
 route set access-list flexroutes

!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20

!
crypto ikev2 policy flexpol
 proposal flexprop

!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn Hub.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol
 virtual-template 1

#
# Calling out a virtual-template within our IKEv2 profile is only
#
needed if the router has a DVTI interface, which in our initial
#
configurations will only be the Hub. Also, if you forget to
# specify this here, your Hub won't work and you'll start
# pulling your hair out troubleshooting IKEv2/IPsec messages 

# (which will all report successful authentications). Maddening.
#
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel

!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile

!
interface Loopback1
 ip address 10.254.0.1 255.255.255.0

!
interface Virtual-Template1 type tunnel
 # Again, DVTI interface is only required on the Hub
 # while we're doing a Hub and spoke topology.
 ip unnumbered Loopback1
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

!

Easy peasy right? It's a lot of config... like all of the config lol. We could trim this down a little if we were doing PSK with an IKEv2 keyring (follow up blog post for that). However, I like doing certificate based authentication so here we are.
Moving on to the spoke router configs, let's look to S2 as an example. I'll call out the bits in red that will need to change before copy/pasting this config to S3's router.

Spoke Router (S2):

ip access-list standard flexroutes
 permit 10.0.2.0 0.0.0.255

!
aaa new-model
aaa authorization network flexauthz local
aaa session-id common

!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20

!
crypto ikev2 policy flexpol
 proposal flexprop

!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn S2.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol

!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile

!
interface Tunnel0
 ip address negotiated
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect



See how easy that is to template out though? Absolutely nuts, and we could potentially make this even more generic with only injecting routes to our local interfaces (route set interface instead of route set access-list) and running EIGRP or BGP over the tunnels. Moving on with this config, router S3 has an identical config, except the IKEv2 local identity is S3.hop16.com and my access-list flexroutes is permitting 10.0.3.0 0.0.0.255. Now that our Hub and Spoke FlexVPN is up and running, let's see what it takes to convert that to a mesh style VPN topology like DMVPN.

Configuring FlexVPN - Mesh (DMVPN Phase 4)

Finally we're getting at the subject of this post, spoke to spoke dynamic tunnels. The mechanism for getting spoke to spoke communication is the very same used in DMVPN deployments, NHRP. With some key changes.
  • No NHRP Next-Hop Server (NHS)
    • Instead mapping NBMA to Tunnel IP is derived from IKEv2
  • Spokes use SVTI (Tunnel0 interface) for communication to the Hub
  • Spokes use DVTI (Virtual-Template) for dynamic tunnels between one another. 
Now, in order to make the required changes to enable NHRP we have to shutdown our SVTI (Tunnel0) interfaces on the spokes. Otherwise the Hub's virtual-template will be locked and IOS will prevent us from making any changes. After shutting down Tunnel0 on both routers, we can add the following to our Hub.

Hub Router:

interface Virtual-Template1 type tunnel
 ip nhrp network-id 1337
 ip nhrp redirect 


Believe it or not, that's actually we need to add on the Hub side. We're enabling NHRP redirects, just like DMVPN Phase 3, so the Hub can signal the Spokes to build dynamic tunnels. Also just like DMVPN, the first packet or two will transit the Hub while the Spokes are receiving NHRP information via the redirect message and building their dynamic tunnels. Alright, let's look at the required config on our Spokes.

Spoke Routers:

crypto ikev2 profile flexprofile
 virtual-template 1
!
interface Virtual-Template1 type tunnel
 no ip address
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

!
interface Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect


Much like most things FlexVPN, while easily templated and generic, it's configuration intensive. That said, this is a super interesting configuration. Where in DMVPN all the traffic happens on our multipoint Tunnel interface, in FlexVPN the tunnel interface remains a static P2P tunnel destined to the Hub. Then we're pushing our NHRP shortcut to a seperate virtual-template (DVTI) that is will build P2P virtual-access interfaces to other spokes. If we had multiple dynamic tunnels to other spokes each would get their own Virtual-Access interface. Alright, let's bring the tunnels on the spokes back up, and do some verification.

S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S        10.254.0.1/32 is directly connected, Tunnel0
S2#show crypto ikev2 sa             
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/26 sec


Now I'm going to enable NHRP debugs, and try to ping 10.0.3.1 which is connected to S3.


S2#debug nhrp
NHRP protocol debugging is on
S2#ping 10.0.3.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/11 ms
S2#

 [...]
*Jul 11 11:53:55.363: NHRP: Sending NHRP Resolution Request for dest: 10.0.3.1 to nexthop: 10.0.3.1 using our src: 10.254.0.9 vrf:global(0x0)
*Jul 11 11:53:55.363: NHRP: Attempting to send packet through interface Tunnel0 via DEST  dst 10.0.3.1
*Jul 11 11:53:55.364: NHRP: Send Resolution Request via Tunnel0 vrf global(0x0), packet size: 72
*Jul 11 11:53:55.364:       src: 10.254.0.9, dst: 10.0.3.1

 [...]
*Jul 11 11:53:55.364: NHRP: 96 bytes out Tunnel0
*Jul 11 11:53:55.372: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 92
*Jul 11 11:53:55.372: NHRP: Route lookup for destination 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us. Process the NHRP Resolution Request.
*Jul 11 11:53:55.373: NHRP: nhrp_rtlookup for 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us, responding with ouraddress
*Jul 11 11:53:55.373: NHRP: Checking for delayed event 10.254.0.8/10.254.0.9 on list (Tunnel0 vrf: global(0x0))
*Jul 11 11:53:55.373: NHRP: No delayed event node found.
*Jul 11 11:53:55.374: NHRP: Enqueued Delaying resolution request nbma src:169.254.2.5 nbma dst:169.254.3.5 reason:IPSEC-IFC: need to wait for IPsec SAs.
*Jul 11 11:53:55.374: NHRP: Interface: Tunnel0 configured with FlexVPN. Deferringcache creation for nhop 10.254.0.8
*Jul 11 11:53:55.374: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down

 [...]
*Jul 11 11:53:55.387:  Instructing NHRP to create Virtual-Access from Virtual template 1 for interface Virtual-Access1
*Jul 11 11:53:55.387: NHRP: NHRP Redirect Feature PI-code Initialized
*Jul 11 11:53:55.387: NHRP: Redirect Feature Initialized - Attempting Platform Init
*Jul 11 11:53:55.393: NHRP: Updating delayed event with destination 169.254.3.5 on interfaceTunnel0 with the new interface Virtual-Access1
*Jul 11 11:53:55.595: NHRP:
*Jul 11 11:53:55.595:  Fetched address from underlying IKEv2 for interfaceVirtual-Access1. Pre-NATed = 169.254.2.5, Post-NATed = UNKNOWN
*Jul 11 11:53:55.595: NHRP: Processing delayed event on interface Tunnel0 with NBMA 169.254.3.5

 [...]
*Jul 11 11:53:55.598: NHRP: No need to delay processing of resolution event nbma src:169.254.2.5 nbma dst:169.254.3.5
*Jul 11 11:53:55.598: NHRP: Attempting to send packet through interface Virtual-Access1 via DEST  dst 10.254.0.8
*Jul 11 11:53:55.598: NHRP: Send Resolution Reply via Virtual-Access1 vrf global(0x0), packet size: 120
*Jul 11 11:53:55.598:       src: 10.254.0.9, dst: 10.254.0.8
*Jul 11 11:53:55.598: NHRP: 144 bytes out Virtual-Access1
*Jul 11 11:53:55.604: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up

 [...]
!
!
S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 7 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S        10.254.0.1/32 is directly connected, Tunnel0
S   %    10.254.0.8/32 is directly connected, Virtual-Access1

S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
2         169.254.2.5/500       169.254.3.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/399 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/443 sec

 IPv6 Crypto IKEv2  SA



I did my best to trim down the NHRP debug messages, but there's a lot of interesting things happening here. Now looking at our static routes, we have the original summary to 10.0.0.0/8 via Tunnel0 to the Hub, but we also have a specific route to 10.0.3.0/24  and a route to 10.254.0.8/32 via Virtual-Access1 directly to router S3 with the % indicating next hop override (due to NHRP). Well that's about it for now, I'll kick out a quick follow up post for those interested in using pre-shared-keys for authentication.

Friday, July 5, 2019

CCIE Security: Troubleshooting (Ticket #1) - Solution


Alright it's been a couple of days since the original post, so after much fanfare and exactly 0 people attempting to solve, let's break this one down.


SPOILER ALERT

Issue #1

Since BGP is relying on OSPF for connectivity between peering interfaces (Loopback1), this seems like a natural place to start. The first thing we'll notice on R1 and R2, is when running debug ip ospf packet, we're only seeing traffic leave each respective router. We're not actually seeing either router receive ospf traffic.

R1#debug ip ospf adj  
OSPF adjacency debugging is on
R1#
*Jul  3 16:14:56.959: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:06.512: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:15.996: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 16:16:34.653: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:44.567: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:53.574: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0

So let's go over to the ASA and look at our logs to see if anything looks amiss.

ASAv1(config)# logging buffered 7
ASAv1(config)# logging enable
ASAv1(config)# end
ASAv1# show logging
ASAv1# show logging             
[...]
%ASA-3-106010: Deny inbound protocol 89 src INSIDE:10.0.0.2 dst OUTSIDE:224.0.0.5
[...]

Alright, that explains part of what's going on. The ASA is dropping OSPF traffic from INSIDE host R2. This is because, even though our INSIDE interface has security-level 100, multicast traffic is blocked by default. The easiest solution here would be to create a new ACL with 'permit ip any any' and apply it ingress on our INSIDE interface. This shouldn't break any of our rules as technically (multicast aside) INSIDE->OUTSIDE traffic was already allowing everything. Alternatively, would could permit ospf from 10.0.0.2 to [224.0.0.5, 224.0.0.6, 10.0.0.1] and ICMP between Loopbacks. 

ASAv1(config)# access-list INSIDE_in permit ip any any
ASAv1(config)# access-group INSIDE_in in interface INSIDE
ASAv1(config)# end
ASAv1# show access-list INSIDE_in
access-list INSIDE_in; 1 elements; name hash: 0x52aada44
access-list INSIDE_in line 1 extended permit ip any any (hitcnt=1) 0x4c917233
ASAv1# show conn
2 in use, 3 most used

OSPF OUTSIDE 10.0.0.1 INSIDE  224.0.0.5, idle 0:00:07, bytes 22780, flags 
OSPF OUTSIDE 224.0.0.5 INSIDE  10.0.0.2, idle 0:00:01, bytes 1088, flags  

That looks better, but looking at debug ip ospf packet, we're not actually seeing much of a change from the routers' perspective.

R1#debug ip ospf packet
OSPF packet debugging is on
R1#
*Jul  3 17:04:40.021: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:49.492: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:59.313: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 17:04:25.424: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:34.534: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:43.792: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#




Issue #2

Which brings us to issue #2, and this one is kind of dirty. Unless you have a keene eye looking through the config, or using capture with trace you might not catch this one, before I give it away, let's look at a capture with trace output (especially since we can't use packet tracer on a transparent firewall). *Note 'clear conn' is actually pretty important, otherwise our trace will only have fast-path information, and we won't see the full packet flow.*

ASAv1# capture in interface INSIDE trace trace-count 1 match ospf any any
ASAv1# show cap
capture in type raw-data trace trace-count 1 interface INSIDE [Capturing - 0 bytes]
  match ospf any any  
!
ASAv1# clear conn
1 connection(s) deleted.
!
ASAv1# show cap in trace detail

2 packets captured

   1: 22:10:39.930249 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20491)
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ACCESS-LIST
Subtype: log 
Result: ALLOW
Config:
access-group INSIDE_IN in interface INSIDE
access-list INSIDE_IN extended permit ip any any
Additional Information:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 20, packet dispatched to next module

Result:
input-interface: INSIDE
input-status: up
input-line-status: up
Action: allow



   2: 22:10:49.253740 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20492)
2 packets shown

Now, that probably looks good at first glance... but take a closer look. A big giagantic hint? Look at the TTL of that OSPF hello, then walk through each phase in the trace.
*Pause for dramatic effect*
Yup. Phase 4, the global policy is set to decrement-ttl. Which means that OSPF packet, while allowed, is going to get dropped due to the ttl being 0. Now, in a perfect world, 'show asp drop frame ttl-exceeded' would show this too. However, at least on my ASAv, it does not. So let's change the global policy.

ASAv1#  conf t
ASAv1(config)# policy-map global_policy
ASAv1(config-pmap)#  class class-default
ASAv1(config-pmap-c)# no   set connection decrement-ttl
ASAv1(config-pmap-c)# end
ASAv1# clear conn
2 connection(s) deleted.


Now let's check the output on R1 and R2.

R1#
*Jul  5 22:23:12.245: %OSPF-5-ADJCHG: Process 1, Nbr 10.2.2.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R1#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:01:51, GigabitEthernet1


R2#
*Jul  5 22:23:12.217: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.1.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R2#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:02:17, GigabitEthernet1



Much better, OSPF is up but now we have a brand new error related to BGP.

R1#
*Jul  5 22:25:12.344: %TCP-6-BADAUTH: Invalid MD5 digest from 10.2.2.1(31914) to 10.1.1.1(179) tableid - 0


R2#
*Jul  5 22:25:29.900: %TCP-6-BADAUTH: Invalid MD5 digest from 10.1.1.1(43305) to 10.2.2.1(179) tableid - 0



Saturday, June 29, 2019

CCIE Security: Troubleshooting (Ticket #1)

So in getting ready for the CCIE Security lab this year, I've been spending some time trying to come up with my own troubleshooting scenarios. The process for this, if you're curious, is normally born from practicing config and noting issues that come up when I misconfigure something. Then I'll try to layer it with other similar or related issues. Alright, so let's write this one up, standard sort of CCIE-ish rules apply.

  • Changes must be specific to issue you're troubleshooting (i.e. Make your changes as specific as possible)
  • Do not remove any security related configuration. Only adjust or add configuration to correct issues.



Scenario:

 

R1 and R2 are connected via ASAv1, running in transparent mode. The two routers have (2) loopback interfaces, Loopback1 and Loopback2 addressed as 10.x.x.1/32 and 10.x.x.2/32 where X is the router number. Loopback1 should be learned via OSPF, and is used for BGP peering whereas Loopback2 is advertised by BGP. Neither OSPF or BGP peerings are successfully forming, troubleshoot and resolve so that both routers establish OSPF and BGP peerings. Ping both of R1's loopback interfaces from both of R2's loopback interfaces to confirm you've resolved the issue. Additionally, match the output below to both routers.







Super Complex Network Diagram

Download Intitial Configs


R1 Output:

R1#show ip route | inc ^B|^O                        
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:09:48, GigabitEthernet1
B        10.2.2.2/32 [20/0] via 10.2.2.1, 00:08:54

!
R1#show bgp ipv4 unicast neighbors 10.2.2.1 | inc md5
Option Flags: nagle, path mtu capable, md5, Retrans timeout 


R2 Output:

R2#show ip route | inc ^B|^O                        
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:11:47, GigabitEthernet1
B        10.1.1.2/32 [20/0] via 10.1.1.1, 00:10:52

!
R2#show bgp ipv4 unicast neighbors 10.1.1.1 | inc md5
Option Flags: nagle, path mtu capable, md5 



Alright interwebs, have at and let me know what you think. Find the solution(s) here.

Tuesday, June 25, 2019

This Blog is getting an overhaul


I'm changing the name, and theme of my blog and website. NetworkKnerd is no more, and I'm (slowly) moving everything to Hop16. Mostly because I was tired of NetworkKnerd, but also because my focus on Routing and Switching has been less and less over the years as I've transitioned into a more network security focused role professionally. I'll keep this post short and sweet, just to serve as a heads up so when re-directs go into effect in the next couple of weeks there's some reference as to what's going on.


Tuesday, November 20, 2018

Firepower Threat Defense AVC and SmartCLI


Congratulations, you've purchased one of Cisco's shiny new Next-Generation Firewalls. It's going inspect more packets, push bits at blazing speed, and finally lockdown your disgraceful network. You've built your access control policy and added a ton of rules allowing traffic outside to your inside zone. Maybe you even took this opportunity to build out that DMZ you've always been talking about. Good for you, I'm happy for you. But, uh, did you restrict which ports were open inside to outside? Did you use AVC to restrict which applications were allowed on the ports you opened or did you just open port 80 and 443 like a child?!


Well, that's why we're here, Derek, to talk about both this underused feature and also about some of the new features in Firepower Device Manager (standalone manager for non-FMC deployments). Time for us all to start firewalling better, all of us... Derek. To start things off, let's look aoutt our starting environment. Built-in GNS3, I'll be using FTDv for KVM running 6.2.3 and a layer3 switch running whocare.png. Honestly, the L3 switch is just so I have another OSPF speaker on net, use whatever you like.

Super Complex Topology

All I've done on the FTD is assign a management IP on the same subnet as FDM-Client and FDM-Server. GigabitEthernet 0/1, which will be my inside interface, connected to SW1 on a separate routed link.

> show network
===============[ System Information ]===============
Hostname                  : FTDv
DNS Servers               : 208.67.222.222
                            208.67.220.220
Management port           : 8305
IPv4 Default route
  Gateway                 : 172.31.45.1

======================[ br1 ]=======================
State                     : Enabled
Channels                  : Management & Events
Mode                      : Non-Autonegotiation 
MDI/MDIX                  : Auto/MDIX 
MTU                       : 1500
MAC Address               : 0C:ED:AF:17:A6:01
----------------------[ IPv4 ]----------------------
Configuration             : Manual
Address                   : 172.31.45.45
Netmask                   : 255.255.255.0
Broadcast                 : 172.31.45.255
----------------------[ IPv6 ]----------------------
Configuration             : Disabled

> show managers 
Managed locally.


So from here, before diving into AVC, I'm just going to get basic addressing knocked out then get connectivity to the outside setup. Along the way, I'll show off using SmartCLI to configure OSPF.

FDM - Basic Setup and Addressing

Of note, I did have some initial issues getting into FDM on this GNS3 appliance. When trying to access the WebUI I was receiving 503 Service Unavailable. To resolve this, I went to the CLI and issued 'configure manager delete' followed by 'configure manager local'.

When you first log into FDM, you're prompted to complete Device Setup which configures the outside interface, a default policy rule, DNS servers for your management interface, and also gives you the option to change your hostname. The next two pages you can configure time settings, and add Smart Licensing. This is a lab, so I'm just going to use a 90-day trial. Once complete you'll arrive at a page that looks similar to below.

FDM Quick Setup complete




Next, we're going to configure our inside interface with an IP address 172.31.99.1/29. To do that I'll click on "3 Enabled" under Interfaces. Then select edit for interface GigabitEthernet0/1. From here I'll need to both change the IP address as well as delete the DHCP configurations to successfully modify this interface. After clicking OK, I'll then deploy my changes (top right, the icon with an 🠟, second from the left).

Change inside IP and delete associated DHCP scope

SmartCLI - Configuring OSPF in FDM

I know, I know... that was really boring. Thanks for hanging in there, Derek, I promise we're going to start getting into the meat and potatoes of this post now. Also, sorry for making you hungry. First up, to finish establishing connectivity, let's configure OSPF on our Firepower device. Super basic, we're going to drop Gig0/1 (inside) into area 0.0.0.0 and have it originate a default route. The L3 switch behind the firewall also has a basic OSPF configuration looking for a neighbor on this subnet. From the Device tab in FDM, in the bottom right you'll see 'Advanced Configuration', and that's where we're headed.

FDM Advanced Config - SmartCLI and FlexConfig

At present, SmartCLI serves a single purpose, which is configuring OSPF and OSPF alone (what no ODR?! #ObscureRoutingReference). From the OSPF template, the initial view leaves much to be desired. So we're going to click on 'Show Disabled' and then change enable 'setup ospf advanced'. From here you'll have a ton of OSPF options to configure, however, I'm going to disable most of them by clicking on the + to the left of each option I want to disable. Then I'll fill out all my relevant information.


SmartCLI - Advanced options hidden
SmartCLI - Advanced OSPF enabled


Additionally, there's a SmartCLI template called "Interface" that allows you to configure interface-specific OSPF options such as authentication, cost, and network type. After deploying, we can go to the CLI to confirm everything is working as expected.

> show ospf neighbor 


Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.122.22    1   FULL/BDR        0:00:34    172.31.99.2     inside
> show route ospf | begin ^Gateway

Gateway of last resort is 192.168.122.1 to network 0.0.0.0

O        172.31.45.0 255.255.255.0 [110/20] via 172.31.99.2, 00:41:15, inside


Application Visibility & Control (AVC)

Now onto the main event, AVC. I'll demo this out using FDM, but the concept is exactly the same in Cisco's full-blown Firepower Management Center. This is a feature I don't see implemented near often enough, and when I do see it deployed typically not in a way that maximizes its potential (looking at you again Derek). Let's start with a very basic objective and we'll work outwards from there. You've deployed this firewall, and you only want to allow the following egress traffic from your users.
  • Web browsing over HTTP and HTTPS (foreboding-music.mp3)
  • ICMP echo
  • DNS to OpenDNS
Everything else should be dropped. Additionally, you want to block URLs for online gaming sites. So you build out a couple ACEs in your policy, click deploy and call it a day. Tell me I'm wrong Derek. Let's first take a quick look at what our policy looks like, before talking about how AVC can help us.

Derek's super secure bulletproof ACP

Then we'll go to our client machine and make sure everything is working as expected.

DNS and ICMP work
HTTPS works

URL Filtering works

So that's it, right? Right?...Right? What's guaranteeing that the ports we've opened are going to used by the applications we intend? Of course, there are obvious things like SSLVPN and proxies to worry about - but then also less obvious concerns. There's an open-source, easily deployed, VPN client now that allows tunneling over ICMP and DNS. You read that right.

**Full disclosure, this was actually kind of a pain in the ass. While specifically blacklisting applications works perfectly well, that's not what I wanted to show off in this post. I wanted to show how we can not just open ports in our ACEs, but also specifically dictate which applications can use those ports. With respect to DNS and ICMP, that worked fine, but trying to write a rule that basically said 'allow tcp/80 and tcp/443 only for web browsing traffic' ended up being a huge undertaking that set me back on this post, and forced me to drop the FDM entirely in favor of a full-blown FMC. Cisco... this shouldn't be as hard as it was, fix it.**

There are a couple different ways to approach this problem, and normally what people do is opt to create a separate ACE that just blocks undesirable application types. That works fine, but I'd argue that implicitly dropping traffic is much more effective than explicitly blocking in this case. With our DNS and ICMP rules, this is easy from the FDM. Web traffic, on the other hand, requires a little more horsepower and granularity, so I'm going to forklift my configs into a Firepower Management Center. So let's tackle the easy tasks first. Within our access control entries for DNS and ICMP, I'm just going to drop into the applications tab and specify those applications respectfully. Additionally, I'll add a catchall rule that blocks w/reset so we don't have to wait for dropped traffic to timeout.

ICMP/DNS Matching Application

Now web browsing traffic, this is where things get a little tricky. Before we dive into how to build our HTTP/HTTPS rule to only allow Web Browsing traffic, let's look and see why you might want to do this. Eyes up Derek, no sleeping in class today. Remember our URL filter blocking access to online games? What happens to it if we tunnel all our traffic over an SSL VPN? On my test machine, I have both SoftEther and Window's native SSTP client setup, so let's get on our dodgy VPN and subvert security policies!

Connected to SSLVPN
Game On

This may seem like a silly example, but start thinking about how easily this could be exploited. Simply tunneling your traffic over port 443 suddenly bypasses DLP policies. Just opening the port to allow web browsing becomes a means of establishing a covert channel. This doesn't have to be over a VPN either, by default TOR will connect to relays over tcp/443 if it's default port (tcp/9001) is closed. So how to address this issue? The easy task is allowing the majority of expected applications for web browsing allowed, then we'll come back to the hard part. So to start, let's get some client-side applications added in (common browsers like Chrome and FireFox) as well as some well-known web-based applications (i.e. Google, Bing, Cisco, Facebook). The web-based applications become important because with HTTPS traffic Firepower can't determine the client-side application in use. To make things easier on ourselves, I'm going to allow any Web Application with a risk of Medium or Less. Now, logically, one would think you could select Risk "Very Low", "Low", and "Medium" and Web Application as a single filter. At the time of this writing, that does not seem to work. So instead we need three separate filters pairing Web Application with each Risk. 

Common Web Browsing

With this deployed, let's go back to our test machine. I'll try to connect to VPN first, then browse to a couple websites.

No Luck connecting to SSLVPN


Cisco page loads

AVC vs. Unclassified Web Sites

So if you've caught yourself thinking "Now we're done." You wouldn't be wrong, but user's experiences on the Internet would be mixed at best. The issue you'll often run into is, "what happens when a user browses to a site that isn't well-known?", that is, a site that Firepower doesn't have a defined Web Application for. Well to answer that, let's first see what happens when we browse to a little known, seldom visited site like networkknerd.com.

Congratulations! You've played yourself.



So now we're in a pickle. We could try creating web applications for every website Firepower doesn't have a canned definition for, but that seems like an administrative nightmare. So instead, I'm going to write a different sort of rule to get around this problem. Essentially, we need to write an application definition that can apply to browsing any HTTPS website (not already defined). To do this, I ran wireshark while trying to browse to my website, so I could poke around the TLS client/server messages. Doing so, I found a field in the client hello that I could match on.

We Got a Match

When we're writing our rules, we'll have the option to match on ASCII string or Hex. While I could just match on http/1.1, I decided to match the entire field with Hex 68 74 74 70 2f 31 2e 31. So, let's move over to our FMC, under Policies -> Application Detectors. Click on 'Create Custom Detector'. I'll leave Detector Type as 'Basic', and I'll Add to add an Application Protocol. From the Application Editor I'll create a protocol like this:

Custom Application Protocol

Click 'OK' to return to the Create A Custom Application Detector dialog. Set the Application Protocol to be "HTTPoverSSL", add a description then click "OK". Now we need to tell Firepower what pattern we'll be detecting to classify this traffic. Since I'm going with a field in the client hello field, I'll set the direction to be client, protocol tcp, and add my hex value from above. To test, we can actually upload a pcap file and click 'test' (script with gear icon to the left of the trash can). Got all that?! Derek?! Don't worry, accompanying screen shots below.

Custom Application Detector

Traffic Pattern to Match
Successful Test

Well that was easy.

Not really, but now we have a custom application detector to add to our ACP that should work as a catch all for all HTTPS browsing (in addition to the previous rules implemented based on application type/risk level). After adding that to my INSIDE-to-WEB rule, browsing works flawlessly and SSLVPN is still implicitly blocked. So, a couple more screen shots then I'm call this post from hell done.

Adding HTTPoverSSL to ACE

Web Browsing Working

SSTP Still Dead

Alright. Well... I need a drink.