CCIE Security: Troubleshooting (Ticket #1) – Solution

Since BGP is relying on OSPF for connectivity between peering interfaces (Loopback1), this seems like a natural place to start. The first thing we’ll notice on R1 and R2, is when running debug ip ospf packet, we’re only seeing traffic leave each respective router. We’re not actually seeing either router receive ospf traffic.

R1#debug ip ospf adj  

OSPF adjacency debugging is on
R1#
*Jul  3 16:14:56.959: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:06.512: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:15.996: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 16:16:34.653: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:44.567: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:53.574: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0

So let’s go over to the ASA and look at our logs to see if anything looks amiss.

ASAv1(config)# logging buffered 7
ASAv1(config)# logging enable

ASAv1(config)# end

ASAv1# show logging

ASAv1# show logging             
[…]
%ASA-3-106010: Deny inbound protocol 89 src INSIDE:10.0.0.2 dst OUTSIDE:224.0.0.5

[…]

Alright, that explains part of what’s going on. The ASA is dropping OSPF traffic from INSIDE host R2. This is because, even though our INSIDE interface has security-level 100, multicast traffic is blocked by default. The easiest solution here would be to create a new ACL with ‘permit ip any any’ and apply it ingress on our INSIDE interface. This shouldn’t break any of our rules as technically (multicast aside) INSIDE->OUTSIDE traffic was already allowing everything. Alternatively, would could permit ospf from 10.0.0.2 to [224.0.0.5, 224.0.0.6, 10.0.0.1] and ICMP between Loopbacks. 

ASAv1(config)# access-list INSIDE_in permit ip any any
ASAv1(config)# access-group INSIDE_in in interface INSIDE

ASAv1(config)# end

ASAv1# show access-list INSIDE_in
access-list INSIDE_in; 1 elements; name hash: 0x52aada44
access-list INSIDE_in line 1 extended permit ip any any (hitcnt=1) 0x4c917233
ASAv1# show conn
2 in use, 3 most used

OSPF OUTSIDE 10.0.0.1 INSIDE  224.0.0.5, idle 0:00:07, bytes 22780, flags 
OSPF OUTSIDE 224.0.0.5 INSIDE  10.0.0.2, idle 0:00:01, bytes 1088, flags  

That looks better, but looking at debug ip ospf packet, we’re not actually seeing much of a change from the routers’ perspective.

R1#debug ip ospf packet
OSPF packet debugging is on
R1#
*Jul  3 17:04:40.021: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:49.492: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:59.313: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 17:04:25.424: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:34.534: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:43.792: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#

Which brings us to issue #2, and this one is kind of dirty. Unless you have a keene eye looking through the config, or using capture with trace you might not catch this one, before I give it away, let’s look at a capture with trace output (especially since we can’t use packet tracer on a transparent firewall). *Note ‘clear conn’ is actually pretty important, otherwise our trace will only have fast-path information, and we won’t see the full packet flow.*

ASAv1# capture in interface INSIDE trace trace-count 1 match ospf any any
ASAv1# show cap
capture in type raw-data trace trace-count 1 interface INSIDE [Capturing – 0 bytes]
  match ospf any any  

!

ASAv1# clear conn
1 connection(s) deleted.

!

ASAv1# show cap in trace detail

2 packets captured

   1: 22:10:39.930249 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20491)
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ACCESS-LIST
Subtype: log 
Result: ALLOW
Config:
access-group INSIDE_IN in interface INSIDE
access-list INSIDE_IN extended permit ip any any
Additional Information:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 20, packet dispatched to next module

Result:
input-interface: INSIDE
input-status: up
input-line-status: up
Action: allow

   2: 22:10:49.253740 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20492)
2 packets shown

Now, that probably looks good at first glance… but take a closer look. A big giagantic hint? Look at the TTL of that OSPF hello, then walk through each phase in the trace.
*Pause for dramatic effect*
Yup. Phase 4, the global policy is set to decrement-ttl. Which means that OSPF packet, while allowed, is going to get dropped due to the ttl being 0. Now, in a perfect world, ‘show asp drop frame ttl-exceeded’ would show this too. However, at least on my ASAv, it does not. So let’s change the global policy.

ASAv1#  conf t
ASAv1(config)# policy-map global_policy
ASAv1(config-pmap)#  class class-default
ASAv1(config-pmap-c)# no   set connection decrement-ttl
ASAv1(config-pmap-c)# end
ASAv1# clear conn
2 connection(s) deleted.

Now let’s check the output on R1 and R2.


R1#
*Jul  5 22:23:12.245: %OSPF-5-ADJCHG: Process 1, Nbr 10.2.2.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R1#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:01:51, GigabitEthernet1

R2#
*Jul  5 22:23:12.217: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.1.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R2#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:02:17, GigabitEthernet1

Much better, OSPF is up but now we have a brand new error related to BGP.

R1#
*Jul  5 22:25:12.344: %TCP-6-BADAUTH: Invalid MD5 digest from 10.2.2.1(31914) to 10.1.1.1(179) tableid – 0


R2#
*Jul  5 22:25:29.900: %TCP-6-BADAUTH: Invalid MD5 digest from 10.1.1.1(43305) to 10.2.2.1(179) tableid – 0

Issue #3

The solution for this error in particular is going to be very version specific. The version of code used in the lab, we need to both allow TCP option 19 (MD5 signature) and  disable random sequence number for our BGP traffic. However in later versions of code, including 9.6(4) like my ASA, you only need to disable random sequence numbers as option 19 is allowed by default. That said, I’ll show you the solution for 9.6(1) that includes the TCP map allowing option 19.

ASAv1(config)# tcp-map TMAP
ASAv1(config-tcp-map)# tcp-options range 19 19 allow
ASAv1(config-tcp-map)# exit
ASAv1(config)# access-list BGPSPEAKERS extended permit tcp host 10.1.1.1 host 10.2.2.1 eq bgp
ASAv1(config)# access-list BGPSPEAKERS extended permit tcp host 10.2.2.1 host 10.1.1.1 eq bgp
ASAv1(config)# class-map BGPMD5
ASAv1(config-cmap)# match access-list BGPSPEAKERS
ASAv1(config-cmap)# exit
ASAv1(config)# policy-map global_policy
ASAv1(config-pmap)# class BGPMD5
ASAv1(config-pmap-c)# set connection random-sequence-number disable
ASAv1(config-pmap-c)# set connection advanced-options TMAP
ASAv1(config-pmap-c)# end
ASAv1# clear conn
4 connection(s) deleted.

After clearing connections, we see BGP comes up on R1 and R2.

R1#
*Jul  5 22:36:11.146: %BGP-5-ADJCHANGE: neighbor 10.2.2.1 Up 


R2#    
*Jul  5 22:36:11.118: %BGP-5-ADJCHANGE: neighbor 10.1.1.1 Up

Finally, we’ll do our verification that R2 can ping R1’s Loopbacks from both its own Loopbacks.


R2#ping 10.1.1.1 source lo1 repeat 2
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.2.2.1
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms
R2#ping 10.1.1.2 source lo1 repeat 2
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
Packet sent with a source address of 10.2.2.1
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms
R2#ping 10.1.1.2 source lo2 repeat 2
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.1.2, timeout is 2 seconds:
Packet sent with a source address of 10.2.2.2
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms
R2#ping 10.1.1.1 source lo2 repeat 2
Type escape sequence to abort.
Sending 2, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 10.2.2.2
!!
Success rate is 100 percent (2/2), round-trip min/avg/max = 2/2/2 ms