Sunday, December 29, 2013

My brief experience with Quad Sup VSS (VS-SUP720-10G)

This is a REALLY short post, about an amazing topic that with the assistance of Google, could easily eat up a couple days worth of reading. I just wanted to talk a little bit about the behavior of your 6500, running VS-SUP720-10G with Quad Sup VSS. The quick and dirty on VSS, for those who don't know, is it essentially stacks two independent 6500 chassis into a single logical unit. So the question that came up, when each of the two chassis have redundant supervisors... how the hell does that work?

Configuration wise, not much is different from VSS with only two sups in the mix. Aside from the fact you now have four 10G interfaces (two from each sup) in your EtherChannel for the VSL. Now the "weird" behavior comes in with what roles each sup takes, and what happens with failover. Essentially your sups break down into one of two distinctions. ICA or ICS, in-chassis active/in-chassis standby. Also note that on VS-SUP720, your ICS goes into RPR (cold standby) and you only get SSO between the two ICAs from each chassis.

Give that a moment to sink in lol. So that means if your active supervisor fails... the hot standby supervisor FROM THE OTHER CHASSIS, goes active. Alright, a little odd maybe. So if chassis A's sup fails, chassis B's sup will go active SSO. What happens to chassis A then? Or it's RPR sup for that matter? Well, bad news bears, that whole chassis reloads. It may sound bad, but if you build this right, you should have MCE (multichassis EtherChannel) built to everything off that 6500. So, theoretically the failure would be almost entirely unnoticed.

QoS Task

If you're looking for a task that will annoy you, this is it! Alright, currently you have a 10Mb connection to your upstream provider, you've been charged with implementing some basic QOS to prioritize voice traffic (marked DSCP EF). You're told voice should be given priority of 10%, and the QOS technique you're to use should allow for future growth with additional class based queues. No sweat right?

While you're out on vacation, one of your co-workers brings in a second WAN connection, and connects it to the same subnet as your primary WAN (10Mbps). This connection is only 512Kbps, better yet your boss wants the same QOS policy above to adapt in the event of a failure so that voice gets 50%. Like any good Engineer, you tell him it's impossible and this is a terrible solution. Now you're job's on the line, smooth move. 

Your mission, should you choose to keep your imaginary job:

- Create a QOS policy giving 10% to DSCP EF when the primary WAN link is up, and 50% when it's down.
- QOS policy needs to automatically adjust for the change in bandwidth in the event the primary WAN link goes down. Your solution should also include adjusting for the bandwidth change when the primary comes back online.
- Do not add any new IPs
- Do not create any additional interfaces
- Bonus avoid using the bandwidth statement


The video got cut short, but you'll see the solution =]

Sunday, December 15, 2013

MPLS Tools and Tricks - Part 1, show mpls forwarding-table (aka your best friend)

First off, I know it's been a while since I've posted anything. Between failing the CCIE (again), moving and changing jobs... I've been busy. Back off. That being said, I'm getting the itch again, and it's time to start studying, I don't care if it takes 20 tries over 3 different blueprints I'm going to get my number in R&S. So to get things rolling, I thought I'd start with my favorite topic - MPLS.

Let's get some lingo straight first, that way you impress all your friends with new acronyms.

LSR - Label Switch Router. This guy's life is label switching, LSRs make up your MPLS backbone.
LER - Label Edge Router. Your provider edge (PE), and as the name implies the edge of your MPLS network.
LSP - Label Switch Path. The MPLS path for a particular source and destination.

Now that we know who all the players are, lets take a look at our topology

You'll notice we're dealing with more "P" routers than usual, but having a few extra LSRs should help visual the MPLS process. First off, before setting up any VRFs, or doing any l2 vpns let's take a look at probably the single most important command for troubleshooting MPLS.  That's right kids, I'm talking about show mpls forwarding-table. As a sample we'll look at the LFIB (Label Forwarding Information Base) on P1.

P1#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     Pop Label     0             Et1/0
17     Pop Label   0             Et1/1
         Pop Label   0             Et1/0
18     Pop Label             0             Et1/2
19     Pop Label             0             Et1/1
20     Pop Label   0             Et1/2
21     Pop Label     0             Et1/2
         Pop Label     0             Et1/3
22     Pop Label     0             Et1/1
         Pop Label     0             Et1/3
23     Pop Label     0             Et1/1
         Pop Label     0             Et1/2
24     26            0             Et1/2
         26            0             Et1/3
25     Pop Label              0             Et1/3
26     Pop Label    0             Et1/3

Alright... what are looking at right?? I'll get to that, first let's examine why this command is so darn important, simple. By issuing it on our MPLS enabled routers, we can establish our LSP between two given endpoints. After all, MPLS is multiprotocol label switching, so when we're troubleshooting it we should always give the labels the majority of our attention. Now, what are we looking at? The first column is our local labels, generated by the router we're on. The second column, when all is working well, will have either an out going label received (in this case) via LDP or it will show "Pop Label". Without digging too deep, Pop Label comes from penultimate hop popping, really fun to say by the way. This is the process whereby the second to the LSR in an LSP will pop the label off an mpls packet, this is an efficiency mechanism. 

Now back to show mpls forwarding-table. Let's say there is a VPN built between PE1 and PE2, routing tables on the CEs look correct, import export statements look good while you're troubleshooting. However, CE1 and not ping CE2. So spoiler alert, I played proctor and jacked up P3 with some control plane policing, more on that later. Further more, since the LFIB is built off the FIB I adjusted the ospf cost on P4's link to PE2 so that P3 was the preferred path. VPNs are built between the PEs, and for a VPN to work there has to be a fully functional LSP between their peering address (loopbacks in our example). So starting on PE1, we'll verify we have labels to get to (PE2's loopback).

PE1#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
26     24      0             Et1/0
         26      0             Et1/1

So far so good. We have egress labels of 24 and 26 going to P1 and P2 respectfully. Next hop, I'm going to look at P1, but you could go to either.

P1#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
24     No Label    1523          Et1/2

Trouble in paradise, P1 shows that it's forwarding the packet out with no label towards P3, and has no other path. Let's see what P2 shows.

P2#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
26     No Label    126           Et1/3

Same thing, at this point we can jump onto P3 and start diagnosing. First up, look at the labels.

P3#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
26     No Label    1218          Et1/0

Verify it has any labels

P3#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label    0             Et1/2
         No Label    0             Et1/3
17     No Label            0             Et1/3
18     No Label            0             Et1/2
19     No Label    0             Et1/3
         No Label    0             Et1/1
20     No Label    0             Et1/2
         No Label    0             Et1/1
21     No Label  0             Et1/3
22     No Label  0             Et1/2
23     No Label    0             Et1/2
         No Label    0             Et1/3
24     No Label            0             Et1/1
25     No Label  0             Et1/0
26     No Label    0             Et1/0

Well that's not good! Why doesn't P3 have any labels?? I know I've told you the problem already, but humor me. So what next? We verify the router has any mpls enabled interfaces, then if we have any neighbors.

P3#show mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
Ethernet1/0          Yes (ldp)     No       No  No     Yes
Ethernet1/1          Yes (ldp)     No       No  No     Yes
Ethernet1/2          Yes (ldp)     No       No  No     Yes
Ethernet1/3          Yes (ldp)     No       No  No     Yes
P3#show mpls ldp neighbor


So from this we see that all the expected interfaces are in fact MPLS enabled and using LDP. However we have no LDP. Since LDP is our current environment's mechanism for exchanging labels, this would explain why we're having issues. This point requires you to know a little more about LDP, LDP sends hello messages out on udp port 646. However, the LDP session is actually built between the LDP router-ids (loopback interfaces for us, this is also the LDP/TDP default behavior) on TCP port 646. That being said, you need to check a couple things on R3

Can we ping our neighbors? Can we ping their LDP RIDs (loopbacks here) from our RID? We'll pick on R1 for these tests


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/28 ms
P3#ping source lo0

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with a source address of
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/21/60 ms

Alright that looks good, so now back our TCP and UDP ports... do we have any access-lists configured?

P3#sh ip access
Extended IP access list 100
    10 permit ospf any any (949 matches)
    20 permit tcp any any eq bgp
    30 permit tcp any eq bgp any
    40 permit udp any any eq 646 (314 matches)
Extended IP access list 101
    10 permit icmp any any (10 matches)
Extended IP access list 103
    10 permit ip any any (2229 matches)

We do! Great... now where are they applied??
P3#show run | inc 101
 match access-group 101
access-list 101 permit icmp any any

"match access-group" sounds like a class-map, let's take a look.

class-map match-all ICMP
 match access-group 101
class-map match-all ALL
 match access-group 103
class-map match-any ROUTING
 match access-group 100

P3#show run policy-map
Building configuration...

Current configuration : 141 bytes
policy-map CoPP
 class ROUTING
 class ICMP
   police rate 100 pps
     exceed-action drop
 class ALL
 class class-default

And finally, where is it applied?

P3#show policy-map interface  <-- Nothing here

P3#show policy-map control-plane <-- Nailed it!
 Control Plane

  Service-policy input: CoPP

    Class-map: ROUTING (match-any)
     ... out ommitted

So now we can modify CoPP to allow for tcp communication on 646. After doing that LDP will come back up, and CE1 will be able to ping CE2. Mystery solved.

Running Labs Again!

I'm back at it! I'd be lying if I didn't say I was considering quitting. CCIE is tough man, and failing it is soul crushing (especially failing it four times). However, the record's 19 attempts... so I figure if anything I can at least try to top that! I have a post talking about 'following the labels' that I'll be publishing immediately after this one... I'm not SUPER happy with it, but I hope it helps someone out there.

So here's an off the wall topic... window strategy on the lab. That's right, it's probably the most nontechnical thing you have to worry about after you sit down at your workstation. However, it's extremely important. Aside from practice, and strong knowledge you need two things to pass the lab - Speed and Accuracy. So it's actually pretty important that you lay your windows out in such a way you can quickly switch between devices and keep an eye out for console prompts. The general idea of how I lay my desktop out is identically to method I learned at an IE bootcamp. So I thought I'd share it out, I have all device tiled from the top left with notepad starting at the bottom of my last router. Why you ask? So I know, at a glance, exactly where my switches start.

It seems pretty basic I know, but it makes a HUGE difference in speed.

Wednesday, September 4, 2013

More MPLS!

So I can see that people like the MPLS labs, and while I'm down sizing and moving my physical lab I thought I'd crack out an MPLS lab out. This one is fun, I tried to add some extra complexity, while keeping it focused to R&S 4.0 blueprint topics.

Some key points before diving into the tasks.

- All routing is correct on Host 1, Host 2, and Server 1.
- Of the (3) devices just mentioned, only Server 1 is running any dynamic routing. This is by design.

*Insert cheesy scenario* You're a Network Engineer at a small to midsize Service Provider offering not only MPLS VPNs but also some managed service offerings. "Server 1" represents one of these managed service offers and should be accessible to both Host 1 and Host 2. Stop caring about what Server 1 is hosting, you're the Network guy, don't worry yourself with secrets of scary Systems people. After returning from a vacation you arrive at work with emails from the owners of Host 1 complaining they can not access Server 1. They also attach menacing pictures promising if the service isn't restored soon they'll cancel their contract and possibly slash your tires. Seems serious. Users of Host 2 do not complain of any issue, and fearing for your well being offer access to Host 2 for testing. Restore connectivity between Host 1 and Server 1,  you can test your solution with a telnet from Host 1 to Username Network and Password Knerd. Host 1 and Host 2 should not be able to connect to one another.

- Do not change any parameters on the shared Ethernet network between the P routers (Fast Ethernet 1/0 on all P routers).
- Do not change cost or any routing metrics.
- Do not disable any interfaces.
- Do not create any interfaces.

GNS3 Files


Menacing Picture


Wednesday, August 28, 2013

Still kicking!

I'm working on some spanning tree labs, which will require some real gear. I'll try to keep them to features only requiring a 3550, but I do have some PVLAN labs I want to work out. I've been taking some time off from studies to focus on "real" life. However... I still have the itch lol. I'll try to have the first STP lab out by this weekend with video solution shortly after.

Edit: Also my MPLS labs seem to get the most attention, hint taken. I'll work up some more magic.

Tuesday, July 30, 2013

BGP Ticket(s) - Multiple faults

Buckle up, it's BGP time. This ticket, or tickets features multiple fault points. I went a little overboard to be completely honest. So here we go! When all tickets are resolved, both R6 and R7 should have full readability to R1. You are not allowed to modify the configuration on R1.

*Note, recommended that you fix tickets in order.

GNS3 Files

Ticket #1

AS 5432 has a new customer, the data activation team has completed work but BGP isn't peering between R4 (AS 5432) and R6 (AS 6).

 - Do not remove any security features.
 - You have permission to access the CE device, R6, however you are only allowed to modify routing configurations that are directly related to R4/R6 peering. No global changes may be made.

Ticket #2

R3 is not able to peer to R1, a network admin has observed an odd BGP peer state on R3. R1 an edge router from an upstream ISP and is not accessible in this lab. Ensure that R3 is able to peer with R1.

 - NO static routes may be added to R1
 - Peering must be between Loopback interfaces
 - eBGP multihop may not be altered.

Ticket #3

R7 can not ping R1. You remember reading an email from the upstream ISP about security changes with regard to AS Path. However, that's all you can recall.

- Resolve this ticket without using any summarization or NAT.



Wednesday, July 24, 2013

Working on a Fun BGP Lab

I'm not dead and gone! I've just been extremely busy with work and studying. Things might calm down next week, and for the few that actually check this blog I have a fun BGP ticket coming. It's about 75% done now, I want to toss a couple curve balls in there though. Should be a good example of a (3) point ticket with two faults.

Thursday, July 11, 2013

Fun with MPLS - Part 2/'however many of these I feel like doing'

It's been a peaceful couple weeks at iHatethisjob INC., perhaps too peaceful... Come to think of it you haven't had ANY calls. Seeing as you're Xbox is broken, and you're all caught up on Downton Abbey (don't lie to yourself), you decide to investigate. First things first you try to ping the PBX over in Carmel from Delaware, to your horror pings are timing out. After logging into the Delaware MPLS router you see that Carmel's subnet shows as an Inter-Area route, this is not the way you left it.  After some troubleshooting you see that no traffic is reachable over the MPLS backbone, and people from the Carmel office have discovered cells phones. Peace and quiet gone you call up the SP to get to the bottom of this, before getting your first sentence out the solutions engineer confesses that one of the Security techs made some configuration changes and now nothing works. 

Your mission, fix the provider network (again) and restore connectivity between both sites. Insure that remote prefixes from both sites show as OSPF Intra-area routes. Also, don't be cheap, follow these restrictions:

- You may not add any new interfaces.
- You may not change any routing protocol boundaries.
- No static routing is allowed.
- You may not enable any additional routing protocols.

- When possible, avoid disabling features. 




Wednesday, July 10, 2013

No Dice

Well, it kills me to report that I've failed my third attempt. I knew at 9:30am my day was shot, I had not been able to complete either of the three pointers on the troubleshooting section of the lab. It's hard to explain how that felt... it was the first time I'd failed TSHOOT. Missing BGP and MPLS tickets, two subjects I considered myself very much expert in. Though I would not pass, I decided to stick it out and tackle the Configuration section... which I ultimately failed as well.

Suffice to say, it was not a good day for my ego, but that's part of it. A co-worker of mine put it best, 'the only attempt that matters is the one you pass'. Very true, but it didn't make it any easier to drive home knowing so many people had counted on me to pass. Knowing my boss's boss was expecting me to come back with a number. It's difficult, but I'm not done. I'll learn from my mistakes and come back in August for another round. In the mean time, I'll have another MPLS ticket out this week =].

Wednesday, July 3, 2013

Off to Raleigh, for my third (and hopefully final attempt)

No trouble ticket today, just nervous as hell and wanting to vent to the glorious internet. My lab @RTP is scheduled for the 8th, the Monday following this long weekend. I feel pretty good, but there's always the unknown... which is killing me. The amount of time and stress that comes with pursuing the CCIE is more than I think anyone can prepare you for. When I started prepping almost two years ago, I knew that having my IE was part the dream I had for myself. I wanted to be among the top tier Engineers in the world, and I knew that going after my number was the best way to get there.

Not because having your CCIE makes you the best, but because the work you have to put into preparing, the road you travel does. Over the past two years I've taken my skill set to a level I thought wasn't possible, hell the difference between my last attempt in February and now is noticeable to me. I know this is what I want to do, and I know that no matter how many times I fail... I'll come back for more. That's what really makes an IE an IE, being able to not only dedicate yourself to being the Engineer you can be but also being able to recover after defeat. To press on.

When you're preparing for the lab there's a lot of "opponents". You face off against your won ignorance in the beginning, once your knowledge starts catching up to where you need it you start facing more personal demons. I know I sound over dramatic, but for me I found that I had (hopefully that remains past tense) terrible time management and accuracy.  The combination of the two lead to endless embarrassing mistakes, because there's no such thing as partial credit on the IE. I spent these past few months leading up to my third attempt trying to isolate and exterminate these problems. Working on not only being fast, but working smart validation methods that help me check each section before moving on. That's something else that comes with a lot of prep time. GRADE YOUR OWN LABS! At least a few of them, because when you're running validation commands from your work book, you're not just seeing how well you did on that given lab... you're also gaining insight on how these beasts are graded. This is to your advantage, because you too should run the same validations.

So finally, I'm not an author lol. This post is terribly structured and likely full of grammatical errors. All I want to impart on my fellow geeks is that this path isn't for everyone, but if it's for you embrace it. You can spend a lot of time kicking yourself and feeling down when mock labs return poor scores, or when you fail the real lab. However I'd recommend that you don't. Advice I hope I don't have to follow July 8th, but true none the less. This lab is tough, likely one of the most difficult things you'll have to overcome in your career. Failure is part of it, but failing a lab attempt is failing at getting your number. It's nothing more than a mile stone in the long road to the top of the hill, and getting your digits. To all those after their number, best of luck, and with a bit of luck I'll have happy news the evening of the 8th!

Monday, June 24, 2013

Quick EIGRP Ticket

This is a really simple EIGRP lab, based off of a lab objective I ran into over this past weekend. There are a couple different ways to solve it, but for as simple as the topology is it makes for an interesting lab. As promised, I'll deliver on my MPLS labs, but I wanted to post this one as it was really quick to set up.

Here's the scenario, you have three routers, conveniently named R1 R2 and R3. All connected on a shared Ethernet segment. The objective is to have a full mesh peers, but achieve hub-and-spoke routing with R1 as the hub (i.e. R2 and R3 route all traffic through R1). When completed, your results should look similar to the following:

R2#show ip route eigrp is subnetted, 1 subnets
D [90/156160] via, 00:00:53, FastEthernet0/0 is subnetted, 1 subnets
D [90/158720] via, 00:00:28, FastEthernet0/0
D EX [170/2560005376] via, 00:05:43, FastEthernet0/0
D EX [170/2560005376] via, 00:05:43, FastEthernet0/0
D EX [170/2560005376] via, 00:05:43, FastEthernet0/0
D EX [170/2560005376] via, 00:05:43, FastEthernet0/0
R2#show ip eigrp neighbors
IP-EIGRP neighbors for process 200
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
1             Fa0/0             12 00:05:51   27   200  0  17
0             Fa0/0             11 00:07:33  137   822  0  19

R3#show ip route eigrp is subnetted, 1 subnets
D [90/156160] via, 00:02:40, FastEthernet0/0 is subnetted, 1 subnets
D [90/158720] via, 00:02:01, FastEthernet0/0
D EX [170/2560005376] via, 00:07:30, FastEthernet0/0
D EX [170/2560005376] via, 00:07:30, FastEthernet0/0
D EX [170/2560005376] via, 00:07:30, FastEthernet0/0
D EX [170/2560005376] via, 00:07:30, FastEthernet0/0
R3#show ip eigrp neighbors
IP-EIGRP neighbors for process 200
H   Address                 Interface       Hold Uptime   SRTT   RTO  Q  Seq
                                            (sec)         (ms)       Cnt Num
1             Fa0/0             11 00:07:46  186  1116  0  19
0             Fa0/0             10 00:07:46  183  1098  0  19

While meeting the following restrictions:

 - You may not change the IP Address on any interface.
 - You may not create any new interfaces.
 - You may not add any route-maps.
 - You can not modify the configuration on the switch.



Tuesday, June 18, 2013

Fun with MPLS - Part 1/'however many of these I feel like doing'

So you're an Engineer for a small company with offices in Delaware and Carmel, both locations are just awful but at least in Delaware you're near the water. Your SP provides both Layer 3 and Layer 2 MPLS VPN between both locations, you're not entirely sure why you have both but suffice to say the solutions engineer had a few extra drinks on you after that sale. Currently you have 768K per pvc, and you've noticed that you're not load sharing traffic between the two available connections. Weeks after discovering this the SP still has not resolved your issue, so you've offered to come work for them for a day to get it resolved. Sure it's a bit extreme, but if you get one more status call informing no progress has been made you might be looking at first degree murder and based on your research Delaware supports the death penalty.

With all that, your goal is to get Carmel and Delaware to load share traffic over their two available PVCs. You can test this via Delaware and Carmel's loopback address ( Here are your restrictions.

- You may not add any new interfaces
- You may not modify the configuration on either the LEC Frame switches, or on P1 or P2.
- No static routing is allowed.
- You may not enable any additional routing protocols.

GNS3 Files



Wednesday, June 5, 2013

R3's clock is out of sync!

You've just received a ticket from the NOC stating that R3's clock is out of sync. Please make sure R3 is receiving accurate time from it's NTP server. Also note that you may not disable any security features on any device, per company policy. Also note the solution of this ticket isn't based around clocks showing any particular time, but rather that NTP is just in sync.

GNS3 Files



Sorry about the quality, I'll be back in HD for the next one!

Sunday, June 2, 2013

Starting off with the basics

So, after a fun bout on Facebook I decided to take this troubleshooting ticket on the road! Here's the scenario. You have two routers directly connected via a Serial connection. R1's is assigned IP address and R2 is assigned Your mission, should you choose to accept it, restore reach-ability between these two devices within the restrictions below. The results of your solution should closely match that of the attached picture. 


  • You may not modify the IP Address or subnet mask of any interface
  • You may not create any new interfaces, or connect existing ones.
  • You may not enable any routing protocol.
  • Do not add any static routes.
  • Your results should closely match the attached picture.
  • Your solution should allow for running RIP over this connection in the future.