From Micro to Macro Management with Plexxi DSE

Where we are :

When working on today’s infrastructures, we’re typically weighted down with the fact that we are left to micromanage a lot of the systems within our infrastructures today. When listening to the Networking Field Day 7 recording of Nils Swart giving an overview of the Plexxi Data Services Engine again, it dawned on me. We’re shifting from micromanagers to macromanagers.

Right now we’re tasked with dealing with every little action that needs to be performed on our infrastructure, whether it be entering commands for carving off a LUN within a SAN, or configuring required resources for VMs within our compute infrastructures, or even configuring a VLAN on an interface. These are all forms of micromanagement. Excessive control and attention to detail. We’ve all worked under individuals who are micromanages, in meat space, and we’ve probably all hated it. But I want to speak more to what the opposite of micromanagement is. Macromanagement.

When looking at a system today, as engineers and architects operating under yesterdays paradigms of designing and engineering our systems, we’re constantly paying attention to Every. Little. Detail. Worrying about the flag that is set in whatever header that is used to signal something within our systems. And how to turn that flag on within a particular piece of infrastructure, whether it be issuing an API call, configuring the device via the CLI, or even ticking a check-box within a GUI. This becomes incredibly cumbersome and time consuming and leads to some of the most inefficient workflows that can cripple and eventually close the doors of today’s biggest and best businesses.

Tomorrow’s service offerings will, no doubt, revolve around speed and flexibility, more than they have in the past decade. How fast can we deploy our service, and at the same time, how quickly can we change it.

Where we are going :

There is already an industry trend going on within IT, where engineers and architects are starting to talk about the importance of having one foot in both the technical and the business and I whole heartedly agree with that. It’s vital that the individuals who are working to translate business policy into infrastructure configuration know how to speak both languages as they’ll then be able to move toward exactly what Plexxi’s Data Services Engine is doing for businesses today.

Within DSE, Plexxi is offering a message bus from toolkits and frameworks like Chef, into the the affinity component within the DSE. What this in essence does, is allow you to directly map your business structure and policy into roles within Chef which are then applied to the network through their Affinities within the DSE. No longer do you care about every little underlying detail of how the process is completed. All you do is describe, within the role, what the end product of the network should look like and allow the DSE and affinities to handle instantiating it all.

This leads back to the macromanagement comment I’d made earlier in the post. It allows you, as the Engineer / Architect / Developer / Whatever you’re called, to concentrate on translating the business semantics into policy semantics and allow the infrastructure to more accurately reflect the business as a whole. Thus leading to a tighter integration of the technology into the revenue streams of our businesses. We will work on identifying higher order business constraints, rules, definitions, etc., define them within a role, and allow the infrastructure to spontaneously move toward desired state, instead of having to worry about configuring every little bit within a device. No longer, on a micro level, do we care how the infrastructure has been configured. We can start to concentrate back on the why it is being configured that way.

As Albert Einstein(supposedly) put it, “If I had only one hour to save the world, I would spend fifty-five minutes defining the problem, and only five minutes finding the solution.”


Hung Process in IOS-XR

So, quick post to help anyone else who runs into this problem in IOS-XR land.

While attempting to issue commands related to the BGP process on the box and I was meet with no response from the box. I was able to break the process with the typical Ctrl + C process, and issue commands to other processes on the box, but the BGP process just refused to response.

Reviewing the logs, I was able to find some errors related to no response being received from the BGP process :

RP/0/RSP0/CPU0:Apr  9 22:55:42.109 : sysdb_shared_nc[382]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM request for 'oper/ip-bgp/gl/act/shared/vrf/default/afi/' from 'bgp_show' (jid 65855, node 0/RSP0/CPU0). No response from 'bgp' (jid 1047, node 0/RSP0/CPU0) within the timeout period (100 seconds)

You can see that there is a ‘no response from ‘bgp” string in this log message. The quick and easy way to take care of a hung process like this is to restart it by issuing the following command :

RP/0/RSP0/CPU0: router#process restart 1047 location 0/RSP0/CPU0

WARNING : Issuing this command will rock the BGP process, so plan accordingly. You may experience a brief outage so schedule it during a typical maintenance window.

One CLI to rule them all? It’s more than that.

Previous to the Tail-f Networking Field Day presentation, I had done some research on who Tail-f were and generally understood how their NCS product worked, but after sitting with Carl Moberg and his team and listening to what this device can offer, I was immediately drawn in. It would seem they’ve developed an appliance that can effectively, as Nick Buraglio wrote in his post here, the Rosetta Stone for all of the vendor NOSes that exist today.

Carl started the presentation with the phrase “hopefully we’ll show you something that will make you really upset, in a very good way”. And I think he did just that. He proceeded to tell us how Tail-f was founded and what they were up to for the first few years of their existence. There is a good chance that if you’ve used any current generation networking kit from the large names we all know and love, you’ve used Tail-f software and just haven’t been told you’re doing so. He then pointed out the pain point of configuring multiple OSes within an environment and handling the operational load of configuration management, service provisioning, revision control, etc.

It’s cumbersome to have to configure a mixed vendor environment between all of the syntax that most vendors pride themselves in maintaining. This has always been a pain point for operators and running a few mixed vendor networks in the past has always been a pain point for me. Especially jumping quickly between devices and having to switch gears into either OS. Even in the network I’m in today jumping between IOS-XR, IOS-XE, IOS, and now Junos, I find myself mixing up syntax all the time.

Enter NCS. Tail-f offers an appliance that can bring some simplicity to the idea of running multiple NOSes in today’s networks. But it can also do more than that. It allows you to model your services within the appliance and then deploy it throughout the network while also maintaining integrity of that service across your infrastructure through the transaction modeling done by the appliance when applying configurations to devices. This allows for change tracking and validation when rolling services out to network OSes that don’t offer any type of two phased commit process. Once a command is entered we all know there is no turning back on those platforms. This brings a certain level of sanity check to the process as well allowing the NCS to validate configuration before rolling it out to the device. Think about it like this, NCS knows about point A, the current state of the devices within your network and you describe to NCS point B, the service you want to instantiate, and NCS will handle getting you there along with making sure you don’t lose any limbs in the process.

Though NCS offers a lot of functionality through the use of their transaction process, I don’t think this is the biggest feature that NCS has to offer. The biggest thing that I have to say Tail-f offers with their NCS platform is the use of the YANG modeling language to model services within their product. YANG is a language that was written for use with modeling NETCONF. Since Tail-f utilized both NETCONF and YANG to build their service modeling platform on, along with other southbound protocols, I would say that they’re not your typical American Hustle(see what I did there Stephen?) tech company like we have here in the US, and I should add that Tail-f is a Swedish based company. Their service is built using completely open standards allowing possible future portability to other platforms, such as OpenDaylight (shameless plug). OpenDaylight’s entire Model Driven Service Abstraction Layer (MD-SAL) architecture will be operating on YANG modeling and what is built within NCS could possibly be ported into ODL. However, I’m starting to write some modules in YANG and I’m nowhere near expert in the language and still have a lot to learn, but the potential is definitely there.

The Tail-f product no doubt offers a ton of functionality in the way of faster operational application throughout the network, along with full fledged service modeling within their NCS solution. The ability to apply two phased commits is also quite the addition to the network devices today as we don’t have that functionality within the monolithic network OSes today. Being able to apply the two phased logic to something like Cisco’s IOS and IOS-XE bring a better version control along with operational flexibility to roll a change back should any problems present themselves. Along with all of this functionality is the fact that Tail-f has decided to build their solution on the use of YANG modeling. An open source modeling language developed for use with NETCONF, and that fits with the ability to possibly port service models through different products in the future. This would eliminate a HUGE operational burden to organizations that use lock-in products much like Infoblox’s NetMRI, etc.

Pay attention to Tail-f, I think you’ll see some pretty cool things come down the pipe, especially speaking with the team on their plans with ODL (shameless plug again). Pay attention to my blog as well as I saddle up my YANG Unicorn (YANGicorn?) and ride it around a while as I definitely see a future in spending the time to learn the intricacies of a language such as YANG.

Networking Field Day 7 – In Review

It’s been a week since I’d landed in San Jose and waited anxiously to meet the other delegates for Networking Field Day 7. The entire week was something like I’d never experienced and it definitely changed my perspective on not only the community, but my career as a whole. I can’t say this enough, but what Mr. Foskett has started and kept running for years now is something invaluable to the industry and I hope it keeps going to years to come.

When I’d landed in San Jose, I sent along a text to Stephen. He greeted me and promptly notified me that the other delegates had taken to a local bar inside of a bowling alley as the bar in the hotel hadn’t opened yet. I sat at the shuttle pickup for about 20 minutes waiting to be shuttled to the hotel, enjoying every minute of the 65 degrees and sun, and the fact that I’d just left 7 inches of another coating of snow at my house here in Connecticut. There is something inherently evil about the weather in Silicon Valley. I can see how it could become an addiction to live there. However, as the shuttle driver had pointed out to me, the hills are usually a lush green and with the current water shortage they were anything but that, more of an arid brown.

I’d finally arrived at the hotel and was promptly given the key to my room and I’d noticed the one and only Bob McCouch wasn’t far behind me, judging by his tweets. I send along a text and he was bout 10 minutes out. I hung my shirts and waited anxiously to see everyone. I’d meet a bunch from the previous Cisco Live but I hadn’t meet the other delegates whom I’ve now realized are a great set of people with whom I hope to keep a very active relationship with. Once I’d meet up with Bob and few other delegates we settled in and started talking tech right from the get go, while we waited for dinner. Awesome.

After our arrival dinner and some drinks we all settled in a bit early for the night to rest up for our first day of Networking Field Day!

The first day brought us three vendors exhibiting their wares :

The second day we was the first day which we were carted around the Valley to meet more vendors at their various facilities, I have to say this was my first trip to Silicon Valley and I was a bit in awe and caught myself, more than once, driving down the street and being amazed at seeing all of the big name companies with whom I’ve used their products.

The line up for the second day included :

After a very long day of presentations we were able to stop for some sushi for dinner where I introduced a good friend of mine Matt Oswalt to sake, both cold and warm. He didn’t seem to be thrilled at the flavor.

The third and last day of the event was a bit shorter than the others but we still have a great line up of vendors which included :

This post isn’t going to deep dive into any of the vendors or technologies, as it is more of a recap post than anything else and I’m still digesting the technologies and solutions and potential use cases for those technologies. But I wanted to give everyone a run down of the vendors I’ll be writing about over the coming weeks and let everyone know that there was one resounding message from this Networking Field Day and that was the networks and systems of today will definitely NOT be the networks and systems of tomorrow. The amount of open source involvement from the vendors and their commitment to the OpenDaylight Project has left quite the mark on me and where I see myself going within my career moving forward. And there will be PLENTY more of that coming down the pipe as I start to blaze my trail in that direction.

So, there you have it, the list of vendors who presented, the major overarching message that I’d received and what to expect from me in the coming weeks. Take the time to check out the links listed above for all of the Networking Field Day content and stay tuned for some of my posts regarding them!

The ‘Software Defined’ Policy Dilemma

Speaking with other individuals in the industry about what SDN is, specifically how we qualify and quantify it, has lead to me to quite a few conversations with people about how many people think of SDN as an NBAR / QoS on steroids solution instead of looking at it as higher level policy framework that can be applied across infrastructure. I, personally, can subscribe to the thought process there.

We got side tracked in the last Class C Block podcast I was invited to participate in, about what SDN is and how we can use it and I just recently listening to the latest Healthy Paranoia podcast where the discussion of how SDN could impact the security realm was had. And the key word that I kept hearing was policy. Policy that needs to be applied here, and policy that needs to be configured there.

I like the idea of defining a framework for our infrastructure in that we can mold the technology policies that we have within our organizations to the business. This is the fundamental reasons why IT exists. We’re here to transcribe business policy into technology solutions. That said sometimes there are very large gaps in translation from business policy into technology, and this brings me to my point.

If SDN, within the next 5 to 10 years, is going to yield us anything we need to make sure that we have not only engineers that can design, build, and deploy these next generation solutions, but they also need to understand the business side of the house a bit deeper. Right now we’re afforded the ability to understand the business logic on a lesser level as we’re living within a world of constraints defined, in majority, by the hardware and software we have deployed in house. To the point where businesses are willing to change their business policy to accommodate the limitations of their technology. The obvious constraint from the business side of the house that can be read into these new policies would be the financial impact. Being more network engineer than anything, the best I can align this with is having multiple upstream peers, all with different monetary costs and having the ability to intelligently shift flows in accordance with fiscal requirement. Right now its a lot of kludge edge policy that needs to be modified and addressed relatively regularly depending on demands within the network.

The more I think about it and the harder I examine the problems and pain points that organizations face and why so many of us are clamoring at the thought of having that panacea of flexibility that all of this hype promises to offer, we need ensure we really understand exactly what it is we’re trying to accomplish. From a business perspective, as well as technological. Otherwise, years from now we’ll look back at our next generation ‘software defined’ infrastructures and be facing the same problems we’ve been faced with over the previous decade. Except, this time we’ll be faced with unraveling a larger ball of yarn with all of the ‘code’, read pseudocode…ish…whatever you want to call it, that gets developed along the way.

Maybe I’m the only one that has seen this problem within organization but I still feel it necessary to at least bring this point to light.

Networking Field Day 7

I’ve had the awesome pleasure of being invited to attend Networking Field Day 7 as a delegate!

Taking a look at the list of vendors that will be presenting, we’ll be plenty busy with reviewing some of their new offerings, along with asking questions and providing some feedback.

I’ve said it before and I’m going to say it again, what Stephen has done with Tech Field Day is nothing short of astounding. Providing an unbiased platform for the community to get some face time on an intimate level with vendors and the ability to provide candid feedback and insight into their offerings and products is substantial, to say the least. It also provides the vendors with the ability to gain that insight into the community that drives a lot of the decisions that we, as engineers within the IT realm, make every day. Hats off to Stephen for doing such great work.

That said, I look forward to attending and will be considerably upping my blogging game moving forward. I can’t wait to get back into the community side of the industry, especially after immersing myself in my CCIE studies for the last 6 months. Let’s hope I pass on the 7th!

Follow along here at my blog, or at my many social media outlets, which you should be able to find within my blog. In fact I should go check that now…

Anyway, February 19th can’t come soon enough! Can’t wait!

CCIE Bootcamp Review

It’s been almost 3 months since I’ve sat for my CCIE Bootcamp with Marko Milivojevic and I wanted to take the time to write a review for the class to help others who are considering sitting for a Bootcamp before their CCIE Lab attempts.

First I want to give you a little background on the format I was using to study for my CCIE before I was able to sit it in on the bootcamp. I was using a particular vendor’s training solutions, including their workbooks, their training videos, and their rack tokens and all seemed to be going well. I was doing relatively well on the workbook material and content and I was able to finish tasks in reasonable amounts of time. Nothing that I was tracking with any accuracy, though. Just working through training videos and workbooks and doing my best to keep pace with the timeline that I had set forth for myself for my CCIE lab date. I’d read the blog posts about speed and accuracy and other requirements for being able to pass and I’d felt confident that if I’d stayed course, I’d have a healthy chance at passing. Maybe not the first time, but the second or third for sure. So, after passing my CCIE Written at Cisco Live last year, I’d started my usual routine, and targeted this February to sit for the real thing.

From June through November, I’d put in around 200-250 lab hours and worked a decent way through some workbooks, labs and videos as I’d said, and I was fairly confident when I flew out to attend the Bootcamp in RTP. On the flight I’d decided to read up on some BGP design as I was working through some design challenges for work. And I specifically remember reading up on conditional injection and some path manipulations methods. I landed, got my rental, and made it to my Dad’s house – who conveniently lives about 30 minutes from Cisco’s RTP campus. We went out to dinner that Sunday night, and caught up on our lives as we hadn’t seen each other in a few years.

The next day came and I have to say I was excited to see what the class had in store. I was up a bit early and out on my way to the class. We had the benefit of having the bootcamp on the RTP campus, so I was able to find out where I would be going to take the lab, and take that unknown away. Once I got to the campus, I was quickly able to find the building that the bootcamp was going to be in. I walked in and I was meet with about 10 other people who were there to sit for the bootcamp as well. We all got our name tags from the greeter and we waited patiently for Marko to arrive.

Once Marko arrived he greeted us with a smile and a hand shake and we found our way to the conference room that we would be working in for the next two weeks. There was some back and forth banter for about 20-30 minutes as we first arrived and then Marko dove right in. We did a round the room introduction of ourselves and we all learned a little about each other, helping break what would become a pretty thick slab of ice within the room and the people who were in it. We were also asked to address what we thought to be our weak points in our theory so Marko could help form the class to what he thought would benefit us all the most.

He was able to gear the class toward our concerns and help us to identify the gap between theoretical and experiential knowledge. And I can NOT stress this enough. It was a larger gap that existed than I’d thought. I vividly remember him putting an example on the board that consisted of 3 routers, none of his examples consisted of much more than this, other than getting into the BGP examples later on in the week, and after some questions from Marko and a lot of uneasy quiet in the room, we weren’t able to figure out the problem in any reasonable amount of time. A room full of 15 Network Engineers all striving to obtain one of the industry’s most recognized certifications for doing just that, network engineering, were just shown that we don’t know anything about a protocol we all work with on a, probably, regular basis. Now, how much of that was “I don’t want to look like an idiot in front of my peers” syndrome, I can only speak for myself and I say it was a lot. And I would’ve looked like an idiot most of the time if I had decided to speak up.

However, I have to stress, this was not Marko’s intention. He wasn’t trying to make us feel bad about how much we knew about a particular protocol, or if we knew how that protocol would’ve reacted in a specific scenario or topology. He wanted to show us what to strive to become. What it would take to be considered a CCIE. It was on that first day that I’d realized I wasn’t doing NEARLY enough in my studies and I needed to up my game drastically.

Fast forward to Friday and 4 more days of realizing how much I needed to increase my efforts, we wrapped up the week of theory. Within this week, I kept thinking to myself, “I don’t think I’ll ever be able to pass this exam”, on more than a few occasions. A few of us from class decided to get some dinner that night and I had a conversation with Marko. I remember telling him that of everything I’ve taken away from his class so far, I’d taken one thing to heart the most. I realized how much I had to rely on myself and no one and nothing else. When you’re in the heat of the moment its the trust you have to have in yourself to methodically walk yourself through a problem and figure out exactly what it is that needs to be done to fix it. There is no Google, there is no co-worker to collaborate with. You have yourself and some horribly organized and sometimes written Cisco documentation and that’s it.

We had the weekend to recover and started in on the labs Monday morning. All I have to say about those labs is that they are pure evil. Pure, calculated, evil. They are designed to push you to your absolute limits in terms of mental stamina, and when you think you’ve got it figured out, you don’t. Go back and try again. I thought to myself, and even text my wife a few times, “I don’t want to / can’t get this certification and I don’t know why I am wasting my time”. She would text me back and encourage me to stick with it and that everything would be OK. I even thought about leaving the class, that I was wasting my time and I didn’t want the stupid certification. What was to be a 2 hour troubleshooting section took me the better part of 12 hours and don’t even get me started on the configuration!

That said, the labs were tough, but they were fair. Nothing in the labs was anything we shouldn’t be expected to see. It was my lack of closing the gap between theoretical and experiential that was breaking me. Circling back to the reliance on myself, it was myself that was failing me. Nothing more. It was now that I was realizing that a lot of what I’d read online about time saving techniques for typing, etc were just a sham. After all of this was said and done, I took that one over-arching theme back to my studies outside of the class. I needed to up my personal game and investment.

Fast forward to today and I’m writing this post in my basement office at 0045 after a 3 hour QoS lab and I’m feeling much more confident with where I am at in terms of the lab. I hope to pass in a single attempt, but will be completely happy if it takes me multiple times. Reason being, in that time between the bootcamp and now, I’ve slowed down and tried to listen to what my peers has been telling me from the beginning. Obtaining your CCIE is much less about the number and far more about the journey.

To sum this up in a few words, go take the bootcamp with Marko. I regret nothing about the time I spent in his class and I believe it made me a better person, both personally and professionally. So much so, in fact, that if he offers an SP class when I decide to go for that, he will be the first instructor I turn to to invest my time and money into.

Thank you Marko for helping me take myself to that next level. The level required to really call yourself a CCIE.