Navigating AI Advancements in Network Engineering - C2C035 - podcast episode cover

Navigating AI Advancements in Network Engineering - C2C035

Jun 12, 202450 minEp. 35
--:--
--:--
Download Metacast podcast app
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more

Episode description

Send us a text

Discover the captivating journey of John Capobianco from the factory floor to the forefront of AI technology at Cisco on this episode of Cables2Clouds. John shares his inspiring path, detailing how his early fascination with technology and subsequent mastery of programming languages like Ansible and Python laid the groundwork for his current role in network automation and AI. Listen as he recounts his experience with early access to ChatGPT's API and discusses cutting-edge advancements in AI such as Retrieval-Augmented Generation (RAG) and the innovative Raptor approach.

Join us as we navigate the intricate world of AI integration within network operations and the ongoing debate between cloud and on-premises solutions. Using Cisco's AI Security Assistant as a real-world example, we highlight how AI is transforming complex IT tasks into more manageable processes. From prompt engineering to the unpredictable nature of AI outputs, we tackle the challenges and opportunities that come with adopting new technologies, drawing enlightening parallels to the tech shifts of the past.

Finally, we delve into the evolving role of security analysts in light of AI and automation, spotlighting Cisco's recent updates to the CCNA certification. Learn about the strategic importance of embedding AI knowledge early in an engineer's career and the safeguards necessary for handling sensitive data. We explore the implementation of Cisco's validated designs and the concept of a digital twin for networks, and share insights on fine-tuning AI models. Tune in to grasp how AI is poised to revolutionize network management, making operations more streamlined and elevating the role of IT professionals.

Purchase Chris and Tim's new book on AWS Cloud Networking: https://www.amazon.com/Certified-Advanced-Networking-Certification-certification/dp/1835080839/

Check out the Fortnightly Cloud Networking News
https://docs.google.com/document/d/1fkBWCGwXDUX9OfZ9_MvSVup8tJJzJeqrauaE6VPT2b0/

Visit our website and subscribe: https://www.cables2clouds.com/
Follow us on BlueSky: https://bsky.app/profile/cables2clouds.com
Follow us on YouTube: https://www.youtube.com/@cables2clouds/
Follow us on TikTok: https://www.tiktok.com/@cables2clouds
Merch Store: https://store.cables2clouds.com/
Join the Discord Study group: https://artofneteng.com/iaatj

Transcript

AI and Hybrid Networking Discussion

John Capobianco

Start like if you're inspired by chatting with a routing table like that might be challenging , but what you could do is , if you don't want to know the network automation piece , just take the text of a routing table , do show IP route , take the top 10 routes and just highlight them and then try to do a lang chain for RAG retrieval , augmented generation with a

text loader . It's the hello world sort of thing . I'm going to use a text loader . I'm going to use a text loader . I'm going to load 10 lines of text .

Tim McConnaughy

I'm going to see if I can transform it and embed it and chat with it . Welcome to the Cables to Clouds podcast , your one-stop shop for all things hybrid and multi-cloud networking . Now here are your hosts , tim , chris and Alex . Hello and welcome back to the Cables to Clouds podcast .

I am your host this week , tim at Juan Golbez on Twitter , and with me , as always , are my co-hosts Chris Miles at BGP Main on Twitter and Alex Perkins at Bumps in the Wire on I don't know what he's calling it these days . It's actually Twitter , but whatever he calls it is what he wants to call it .

Alex Perkins

We'll just call it Twix or something from now on .

Tim McConnaughy

Twix . There you go , Just to add in a little . Yes , I love it . We're using that when moving forward Twix , and with us tonight is a special guest , john Capobianco from Learning at Cisco , and he is here to talk to us about AI , and he has been , honestly , you've been doing this .

You've been doing this like , I won't say , since the beginning , but like pretty damn near the beginning , so why don't you go ahead and introduce yourself , john ? Well , thanks .

John Capobianco

Tim , and it's great to see Alex and Chris . This is a real pleasure for me . I actually reached out to the team and said I'm open , let's have a conversation about AI . So I'm with Cisco . I moved to Cisco security about four months ago from learning inserts and to focus as a technical leader on AI .

That's okay , Tim , that's okay you don't have to know that but anyway I have actually moved to a focused role on AI . I felt that all of 2023 , I was kind of like doing two jobs I did was trying to keep up with the bootcamps and really , you know , real quality and a hundred percent focus on that .

And then it was nights and weekends and lunch breaks and any time I could squeeze in to just try to figure out AI right . So I actually started with the first . You know you see the crazy numbers that in the first five days they gained 1 million users .

When they launched I was part of that early craze and then I was given early API access to chat , GPT 3.5 to kind of date . It would have been like March , February , 2023 , a couple months after launch when the API came out and then I was sort of off to the races and I just tried to not stay ahead of it but at least try to keep up with it .

But in terms of my background , I want to be very clear that , like I did go to school after let me go a step further back after being put on academic probation as a history major at Queen's University , which didn't click with me at all , and I had been using computers since I was like eight or 10 .

I had a BBS system before the internet in the 90s , right , it was actually like a top 10 Canadian BBS . Over time we added multiple phone lines and I was 10 . I was 11 when I was doing that , right . So technology always appealed to me .

So I went back to school , you know , after five years , at an aluminum factory in my city where my dad was a millwright and he was able to get me into the union and get me onto the production line of dealing with heavy gauge aluminum . And I did that for five years . But my last three years I was also going to school .

I went back to school after two years of that . I knew that long term it wasn't for me , but it was a good union paycheck and a good , steady job in a tough economy , Right ? So , I didn't want to give that up , but I also wanted to go back to school , so I made some arrangements to do both .

As a computer programmer I actually found my placement out of college was with a service desk they called it a help desk right for the Ministry of Health in Ontario , and we were doing things like building desktop images and ghosting , building up early primitive networks for the Ministry of Health as a student . But it didn't give me any experience as a programmer .

So I graduated with this programming degree C++ and Java and good coding experience , but I didn't do it . In my placement I did more IT versus IS type stuff , client-server network , that type of thing so I pursued that . I went to get certifications my A+ , my N+ , a raft of Microsoft certs and then a bunch of Cisco certs .

I've worked in the private sector for an insurance company for about eight years as a senior network engineer and then I spent 10 years as a senior network architect with the Parliament of Canada , the House of Commons . And you know what's funny ?

Just on my facebook I got a message that reminded me 10 years ago we first sent the , the first wireless packet over the parliament's secure network . That was a decade ago , right , and it's like , wow , it's time , why , right ? Um , now what led me to all this was automation .

To be honest with you guys , like if I didn't start with ansible in 2015 and then Python in 2019 , I would not have been ready for AI in 2022 or 2023 , right , and that's sort of my story , to kind of bring you up to speed . I've been trying to push the limits and try new things . You might've heard of things like RAG , retrieval , augmented Generation .

Now there's an even newer approach called Raptor that does a tree structure , and if you guys look at the tree structure , the first thing you're going to think of is , hey , that's a data center , it's got a core , it's got a spine layer and it's got a leaf layer in a non-blocking clove spine leaf topology .

It's so weird , like , as a network engineer , I was like , hey , I understand this , that's just a data center Anyway . And now there's a new thing that I'm really excited about is Raft , which takes the best of RAG and combines it with fine tuning .

So now we can use retrieval augmented generation like an open book exam to fine tune models with our domain specific knowledge . Now , why I really reached out and I'll lead this over to you guys .

I think the name of this podcast I've taken it extremely literally I said you guys , it's not a joke when I say we now have a choice of doing cables on-prem AI in your data center , or clouds , public or privately hosted AI , not just public clouds , private clouds as well .

And now it's up to us , as our decision makers , as our problem solvers , as our technical leaders , to lead our organizations down this right . Do we do a hybrid ? Do we do some in the cloud ? Do we do a private cloud ? Do we do it on-prem ? What does on-prem mean ? Do we have cooling and power , money , gpus , expertise ?

That's why I came to you guys , because of the title of the podcast is Cables to Clouds , and you guys , you know I feel like you lean towards clouds a little more , maybe on this show , but I think cables are still relevant , now more than ever . What do you guys think ? And thanks for having me .

Tim McConnaughy

Yeah , I guess I'll start and then I'll hand it over to Chris and Alex . So I mean , yeah , we . So this is a obviously this is a cloud networking focused podcast , but but not just cloud networking . Focus is very focused on hybrid networking , because we've all been saying since day one , the very first episode , we've been saying hybrid is the future Right .

So cables and clouds , really it's not one or the other , it's both . And it's interesting to hear that that also follows to , I guess , this AI thing . I think we might get a little head over skis though , because I would love to , and again , I'm going to hand it over to Chris and Alex here .

But the first thing I want to kind of dig into a little bit is you know , given your background , you know it's a pretty nonstandard .

I would say network engineers , most network engineers don't have the degree in programming and the programming you know experience and , like you said , the automation stuff that you and you were there like your book right , automate your Network right , like you were there again back then . So obviously you have a very non-standard upbringing here .

So anyway , I'm going to hand this over to Chris and Alex , but I would like to at some point get into , if we didn't do that , where we other network engineers get started . So here yeah .

Chris Miles

Yeah , just kind of echoing what Tim has said , I think I'm very excited to have this casual conversation about AI , because AI , I think obviously it's . Some would argue that it's not going to be the paradigm shift that we think it is . I think , john , you would probably argue that it is .

I think , if anyone's paid attention to you and what you're doing , it's very infectious and inspiring to get into this , because you've done a lot with AI specifically , but you've made it really captivating and made it look useful for people like us , people that are individual contributors within IT , things like that .

So I'm really excited about having a casual conversation about what that looks like going forward for people like

Opportunities and Challenges of AI

us . What do our opportunities look like in the cloud ? What do they look like on-prem ? Why would you even consider doing it on-prem ? Fine-tuning models , things like that . I'd really like to have that conversation .

Alex Perkins

Yeah , and so I mean I'm going to say some of the same things , but I like the way you phrase it right , like it's you can do both cables or clouds right . It's not like just one or the other .

And this this perfectly lines up with a lot of stuff that we've been talking about lately , which is kind of like AI has almost made the cloud discussion like get on steroids right , because all these enterprises are trying to make decisions . Are we going to bring it in house ? Are we going to ?

You know , there's all this repatriation talk , like there's all these things happening , and I think AI just turned that dial so far up , so quickly , that these conversations are happening on an even larger and faster scale . They're almost like kind of related .

Really interesting to have this angle and talk about something specific like AI and how that translates to the larger decisions that businesses have .

John Capobianco

Yeah , thank you . I really think that with our discussion here tonight , we actually will help . Maybe some decision makers make some of these decisions right . Like I think this is going to be an influential discussion in a relatively early part of AI's journey . I feel like this in itself is making it more mainstream and accessible .

Just having four network engineers who are very respected in their fields and have done this a very long time and have seen things come and go over the years that were paradigm shifts or were not . Chris , you made a good point about individual contributors .

Here's a real concrete example , right , and I'm not trying to sell anything here , but Cisco AI Security Assistant is coming right In the cloud and on-prem both flavors , right , we're going to handle both situations . And you literally say things like please add a policy to block Tim from LinkedIn , right , that's it .

And think of firewalls , right , this weird hybrid knowledge of you have to know networks , because firewalls have routes and interfaces and neighbors and configuration and policy sets and access control lists and stuff . So it's hard to find that . Like , we know the shortage in IT security , right ?

So think about having an actual assistant that is actually of quality and will answer things like how many duplicate firewall rules do I have ? Or do I have a policy blocking X ? Do I have an incident , right ? Things of those nature , right ?

So I think , just having prompt engineering as a skill , actually not being surprised when you see an assistant of some kind because Cisco is not going to be the only ones with assistants , microsoft has the co-pilots , and they're incredible . I don't know if you've played with co-pilots inside of VS Code or Office Suite or whatever , right .

Oh yeah , and it's going to spawn horizontally all across everywhere , right ? So I don't want people to be disarmed when suddenly there's a new . You know , I'll date myself . The mouse was new to me . I was like , why do I need a mouse ? I just have keyboard right like type commands and just drive the computer . What , what the heck is this thing for , right ?

So solitaire came out with windows 3.1 . I don't know if you guys know this is a bit of trivia that I throw out there . Solitaire was written partially to help people learn how to use a mouse .

Yeah , right click , left click , drag and drop , double click things well , yeah , absolutely even mind sweeper think of mind sweeper clicking and learning how to move and pointer and move the pointer around I it sounds . It sounds so trivial and infantile , right , but that's sort of where we are with .

Ai is in this infantile stage , but you have to learn how to prompt it , how to extract information , how to ask the right question and phrase it right . Forget about the API . Can you just interface with the thing as a beginner to get going right ?

Alex Perkins

Well , and that's . You know , that's kind of how it should be right , like any really advanced technology . I'm not going to say the quote where you know indistinguishable from magic .

Tim McConnaughy

Come on , Arthur C Clarke is rolling . I love it . That's my favorite quote .

Alex Perkins

But , like , the technology should be invisible , right , like we shouldn't have to deal with the intricacies of developing the models , right , we shouldn't learn how to interact with them . But I just like that it's getting so integrated into everything that it's almost any product you use . Like we talk about this all the time .

There's all kinds of businesses that are trying to find use cases and fit it into things , and it's just a new kind of normal almost .

Chris Miles

I think the difficult thing , especially with AI and generative AI , is , I mean , at least in my experience , to date technology has been mostly focused on being predictable . You know , if you want to put in certain criteria , you want a certain outcome every time .

And now we're branching off into this section where , like , the outcome is actually going to be a variable every time and it's always going to change . You know , I mean , it could change day to day , it could change based on the inputs , it could change based on the model , and it's really day to day .

It could change based on the inputs , it could change based on the model , and it's really hard to predict . So I think that's where some of the fear you know the FUD fear , uncertainty and doubt , all that kind of comes in right .

John Capobianco

At least , that's where I'm at yeah , it's not immutable right .

Chris Miles

No , yeah you're right .

John Capobianco

You're not dealing with an SQL database or , like you know , Ansible . Playbooks were awesome because they were immutable and you could run it a million times and get the exact same result every time you ran it , right ? But this is not that . This is an entirely different beast , right yeah ?

Tim McConnaughy

Yeah , I mean . The other thing is , you know , obviously there are , and this is I don't want to . I know we've talked about several times and I think it's going to continue to be a very , very scary part of this process .

Exploring AI Integration in Cisco Networking

But you mentioned , like the Cisco Secure AI . You know , some people heard what you said and thought , wow , that's amazing . And some people probably heard what you said and thought , holy crap , I'm going to be out of a job , right ?

John Capobianco

Yep , Well , I'm hoping it's an augmentation right and that it can elevate security analysts in particular to do more important things than root out duplicate firewall rules , which I don't think is a good idea .

Tim McConnaughy

It isn't . That's exactly the type of stuff that should be automated away . That should be Right .

John Capobianco

Well , what I think is interesting you had mentioned the CCNA to me , tim , in the notes . Yeah , that's right that AI has actually replaced and I actually I did some research here and I have the screen up . I'm not going to read this , I'm going to read it so I don't get it wrong .

So in the CCNA , the 200301 , 6.4 used to be compare traditional campus device management with Cisco DNA Center enabled device management . 6.4 is now explain AI , generative and predictive and machine learning in network operations .

So I talked to Nick Russo to throw a name out there about this and some other people what we were thinking about it internally and how deep is this going to go right ? How much AI machine learning do I need to learn ? Is this just specifically because of the DNA replacement ?

Is this specific to assurance in Catalyst Center and that predictive AI system within that ecosystem ? Because it's replacing DNA Center ? So I don't know . Like I don't have any insider information guys , I really don't . I had to look it up like everyone else .

Yeah , and I think it maybe is an assurance type questions , but maybe it is about generative and predictive and machine learning . I really don't know , but to see it on the new CCNA , it's like seeing when Ansible was added or the programmability and automation became 10% for the exam .

Tim McConnaughy

I remember that , yeah .

John Capobianco

Yeah , so you know , I think Cisco exams are a bit of a trendsetter , or at least a baseline Right , At least that 6.4 within that chunk of the exam . You have to know what six questions about AI . Maybe that translates to what yeah , Six or seven questions out of the out of the exam . You have to know what six questions about AI .

Maybe that translates to what Six or seven questions out of the stack .

Tim McConnaughy

Yeah , something like that , Absolutely . I mean , and I remember when we were doing the read , you know , we were redoing the whole thing and we were adding in the programmability piece . I remember we had a big old workshop and talking about how much of the exam should be programmability .

And we got into a big old workshop talking about how much of the exam should be probability and we got into a big argument about should we actually support this is when I worked at Cisco , by the way . We were talking about should we support Ansible over Puppet or Chef or whatever any of the other ones , and there was a lot of internal discussion about that .

So I imagine , for this to have made it to the blueprint , there probably was another round of discussion about exactly what we're talking about . How deep should we go ? Should we just confine it to assurance ?

Because , Chris , I know you and I just both did the Rev Up to Research for Cisco Learning and 16 credits or whatever , and that was focused on Catalyst Center assurance and the AI piece that they've added to it .

John Capobianco

So well , I think that's a good study place to start . If you could go find that program about assurance right to help augment your CCNA studies . You know they're not going to put these questions on there without a source of truth . To study from questions on there without a source of truth to study from .

I know the books are going to take another cycle to refresh and update , but that might be a good place to start to study for this . That section of that CCNA is the assurance module .

Chris Miles

Right , yeah , and just I mean just to play devil's advocate here . I think you know , obviously adding something like AI to an introductory level training course is probably going to lead , you know , people getting into the industry , to AI products or you know kind of .

I think it kind of showcases how Cisco is going to lean into the AI paradigm shift that we're calling it here , right . So I think that I think that's a good thing . I think there's obviously a monetary value that can be extracted from , you know , telling people about , about this this early on in their studies .

But at the same time , that's also telling that they're putting it in at the beginning , right , it's not something that's going to come later for seasoned people that are wondering how to operate large-scale networks right , they're teaching this from the very beginning . So I think that's a telling aspect of it .

John Capobianco

Yeah , I think that's a great point that it's actually starting at the CCNA level and not something that they introduced at NP or IE or something right . I think that is telling that Cisco seems to be going in on AI , at least in this regard , right ?

The other thing I've noticed is that there's actually validated designs now that Cisco has actually validated designs , now that Cisco has . So I know , when I was designing , you know , an architect in a primarily Cisco shop , we , you know those CVDs were invaluable .

We built , we based everything we did off of the validated design , mainly because we could point back at Cisco and say we followed your validated design .

Tim McConnaughy

That's right . That's why everybody does it if they can right Unless it's a brownfield right , but if you're building greenfield , the cvds are are gold right for for multiple reasons , that being one of them right .

John Capobianco

Well , what I think is neat ? Um , I I don't . I can't take any credit for this . I'm not part of the team that built this . Don't don't misunderstand me , but I I do remember having conversations at cisco live or impact or or just colleagues that wouldn't it be great if we had an AI FlexPod type system , ai in a can where you buy the rack of stuff .

It's got NetApp or whoever for storage , we've got UCS for compute , we've got the fiber interconnect or the fabric interconnect module and we have Silicon One at the top of the rack . Plug in as many GPUs as you want and that looks like . That looks like what the valid validated design has come up with and at scale .

So you need x number of these pods to train x number of parameters and then for its model , right . Um . So I think that in network engineers on prem this is my own I think that it's inevitable that someone's going to march down a project manager right with a goofy smile on their face going we have to . We bought some Cisco UCS or some GPUs or something .

We've got like a three-week project . You've been tagged as the network engineer to kind of connect all this stuff Like that's coming right and there's new protocols , even rocky , right , from all this stuff like that's coming right .

And there's new protocols , even rocky , yeah , from rdma over converged ethernet that we're going to have to pick up and learn and understand because we didn't spend 40 000 or 50 000 per gpu to not connect them directly together using this protocol , right , or infiniband or something else . Right , it doesn't have to be rocky yeah , we had a .

Chris Miles

We had a pretty extensive conversation with , uh peter jones these , these kind of purpose-built networks just for AI model training and it's it's going to be a need that I mean people are . You know , some some of the early adopters are already putting in for this stuff now , but it it .

I imagine we will see some more kind of off the shelf-shelf offerings that allow for these temporary model training exercises and then they can be probably ripped out right , because you don't want to be paying for that level of infrastructure over the course of time .

But I mean , that being said , obviously you've talked about some of the stuff that you're really finding exciting in the world of either doing AI on-prem versus doing AI in the cloud . Can we kind of go back to that ? Let's kind of set a base layer there , like when would someone be wanting to do it in the cloud versus on-prem ?

What are kind of the decision factors that go into making that ?

John Capobianco

Sure . So there is like no friction to use AI in the cloud , right ? You sign up for ChatGPT or Anthropic Cloud from AWS , or you know BARD or Omni or whatever the Google version is called Gemini . Gemini , excuse me .

Chris Miles

What's the ?

John Capobianco

So , and then there is paid tiers . Certain things are free and certain things are not , but let's just assume we're all just going to pay the fee to make it frictionless , right ? And it really is that simple .

Once you've March March , one month it was about a hundred bucks $20 on embeddings , which is programmatic , that helps me make vectors and do programmatic stuff , and then $80 in just chat , gpt LLM calls to GPT four from my Python code locally , right ?

So I've got local code and in my Lang chain , which is an open source framework that lets me develop Python scripts to do AI stuff , there's like a line that says LLM equals and that's my large language model .

For the rest of my code chat , gpt 4.5 or , you know , turbo or whatever and I just pay whatever the fee is per token to do that development , and sometimes it's worth it because one that's the best answer . You're going to get GPT-4 or Claude from Anthropic , which are both paid in the cloud , are lightning fast and give you the highest quality answers .

There's also a lot less friction , meaning I don't need GPUs , right , it's not just training data that needs GPUs . If you're fine tuning or if I want to run a model locally , I need a GPU , more or less , to do that with any sort of performance .

So I don't have to worry about any of that cost of upgrading my home computer and getting a nice graphics card and all that stuff and then figuring out how to download the open source server , how to download open source models , and then figuring out how to download the open source server , how to download open source models and then how to chat with them .

All of us on this call may be able to do that quite easily . I would argue quite easily , but that's not for everybody . I'm not going to say that it is so . The cloud is like Netflix or anything else . It's just a convenient you can shoot your own home movie right . Or you can watch Lord of the Rings on Netflix , right .

It really is that sort of thing and what you're into doing . So the other thing about the cloud is that there's certain capabilities that haven't become other than theory easily enough used at home .

So , like things like image generation , right , you can use MidJourney in the Discord server , which I would call the cloud , or Dally 3 on the web to do text to image generation . That's a little shaky for free and locally . That's just not there . Text to speech You're going to do text to speech in the cloud .

It's just the absolute best and it's worth the token cost , right ? Geez , I'm trying to think of other benefits of the cloud . You know , if you can afford the innovation , I would suggest it's worth it , right , if money is absolutely no object , or if you've got $100 or $50 a month , you can invest it and develop code against the cloud quite easily .

There's even free . You get like $12 with GPT and tokens for free when you sign up . Claude gives you $8 or five bucks worth as a starter kit , you know . So the cloud's pretty easy actually for AI to get going . The other thing that I just did and this is gonna blow you away I use the GPT fine tuning capability .

So what you can do is , if you create a data set of your own and my data set was 250 lines , that's it 250 rows with things like what is my default route , and then I used RAG and a routing table as JSON with PyATS to answer those questions . So that was my data set question answer 250 of them .

I uploaded that data set to the fine-tuning model and in about eight minutes I could ask the model that I fine-tuned what is my default route , and it would get it right . I could say , if I was a packet going to this IP , what interface would I use , and it would get it right . Now that is instantly and it's fine-tuned .

It's baked into the knowledge of this new model that I made and I don't need RAG or any of these other things that I've been talking about to help augment the answer . It knows the answer . And then what was really neat was that model that I created . I'm actually able to call it via my code now .

So in Postman or something I can do , a post against the chat completions API specifying my custom model that I fine-tuned and chat with it . So it's just the cloud . Ai capabilities are just they feel futuristic .

It feels like there's time dilation between how fast AI is moving and how fast the rest of the world's moving , like it really feels like it's just moving exponentially faster than day-to-day life . I don't know if you guys have noticed this .

Every day you're bombarded with a new model , and then another new model , and then some new crazy deep fake technology , and then right , it's just on and on and on right .

Tim McConnaughy

We've noticed I mean our news articles , our news shows have sometimes been completely taken over by AI , because of how quickly the stuff is moving right so yeah , we've definitely noticed .

Fine-Tuned AI Models and Model Distribution

Alex Perkins

Real quick . Going back to the model you were talking about , that custom-tuned model , because this is something we talk about a lot . Is that a model just for you , like can other people access ? That Is that kind of like a walled garden , like how does that aspect of it work ?

John Capobianco

That's a wonderful question and I wish I knew . I don't want to get this wrong , because I would love to be able to give you a link and let you chat with the model that I made . I haven't found an easy way to actually download or get it out of the chat GPT ecosystem , and I'm not sure if that's part of the deal that you pay them .

Oh , by the way , it cost me 87 cents to fine tune those 250 lines of code . Just to give you a rough sense of cost . It was less than a dollar for me to make the model you know , and about three bucks in rag , so I don't wanna underestimate that . It did cost me some money to generate the data set that I'm talking about .

Once I had a data set it was , like you know , 87 cents to tune the model . So any of your own hobbies guys , you could make a data set of questions and answers about whatever and fine-tune a model with that information , but I would . So what I then tried to do was recreate that with cables with the theme of the show right .

So I I tried to do it locally with the c3 model from Microsoft . Now , this is a small language model , not a large language model . It's only 2.2 gigs in size , locally 2.2 gigs in size and it's like 3.8 billion parameters , but it's beating 13 billion parameter models because of Microsoft's secret sauce .

And I think the secret sauce is that they're training it on textbooks , not internet data , so that garbage in , garbage out . Theory is that even though it's much less data that they're training it with , it's much higher quality because they're textbooks right .

Alex Perkins

So that model .

John Capobianco

I can run some code and Microsoft literally goes here's the code to fine tune this model . Go and make new models . I follow their recipe with that routing table data and fine tune it locally . So then I could publish that on Hugging Face . Now , if anyone listening hasn't heard of Hugging Face , it's the equivalent of GitHub .

Now I hate referencing a reference that people might not know . Now , github is an open source repository in the web that lets you store open source code . So I have a ton of Python scripts on my GitHub .

Well , hugging Face , what I understand is I could make that model available to you by publishing it to Hugging Face and then you could download it and chat with the routing table , so to speak , and to do it locally . It doesn't cost me the 87 cents or the three or four dollars to generate the data set , because I'm doing it locally and privately .

The other thing is I don't know that I would fine tune a public use , a public service like that , with a production set of firewall rules or routing .

Tim McConnaughy

That's the sort of question I was going to ask .

John Capobianco

Actually , that's sort of the paradigm of where clouds fall a little short . Right , At least public clouds . That's sort of the paradigm of where clouds fall a little short . Right , At least public clouds . Right , Because they're going to learn from your data . We've all heard the Samsung story . Right when ? Some core developer leaks and stuff to JetGPT .

The model learned it .

Tim McConnaughy

It's gone forever . You'll never get it out , right ? No , yeah , I think that's what Alex was kind of getting at . If you fine-tune a I think that was Alex was kind of getting at was , if you fine tune a model , that data is God knows where . What happened to that data once you put it in there ? Right , right ?

John Capobianco

Well , that routing table was a definite sandbox in the cloud .

Alex Perkins

I got to cover my ass here .

John Capobianco

That was just a that was just a , a ephemeral , definite sandbox routing table .

Tim McConnaughy

Okay , yeah , I know you're're . You're not leaking proprietary cisco code or anything . I didn't get that . No , I am not but , it's terrifying that you not you , but like that could be the case , right people trying to yeah , the collective , we , if you will , could be trying to fine-tune and get better data , because I see that .

You see , the value of fine-tuning is , of course , so that you get the right answers back and so that your model will make the right inferences on your questions . So the value is there and that's why people want to do it .

Same thing with , like copilot and all that where like use copilot to tweak your code and make your code better , but what you're doing is you're canning that model data , right , right .

John Capobianco

So actually it just another thing off the top of my head because it fits in here OutShift , which is Cisco's kind of like R&D kind of wing . They have a product called Motific . I don't know if it's fully out , but it's called Motific and it's really neat .

It's a cloud service that will show you it's like an observability dashboard for what's going on with your organization's AI , but it has things like PII scrubbers . So if someone drops a serial number or a SIN number or a credit card number accidentally , somehow this sort of will intercept that before it reaches the AI and scrub that PII .

Tim McConnaughy

It's a proxy , so it's like an AI proxy before you actually get it to the model .

John Capobianco

Yeah , and how much it costs and how many tokens and what AIs are being used and you can have whitelist and blacklist of which models are allowed and services are allowed . That sort of thing Interesting . That's really cool .

Chris Miles

One thing that kind of came to mind for me when you're talking about something like maybe even like a private AI model , or I mean I guess in this example I should probably reference some kind of public model If you're doing something like managing the state of a network or looking at the state of a network or security aspect , things like that .

One thing that is common among networks is there's typically a lot of churn . Like you know , route , advertisements are propagated , things are changing on a regular basis . Are you constantly like having to feed that churn into the model to retrain it , or is it more kind of the model I don't even know if I'm using the right terminology here the model can make .

API calls to determine the state of the network at the time , or how does that work ?

John Capobianco

So you may be able to do a bit of a hybrid thing where . So when you pre-train the model , once the training's done it sort of seals up and that's sort of the state of the model , right churn

Exploring Dynamic Routing Tables and AI

I . I agree that a routing table it's like my hello world thing . I just wanted to see , if I asked an llm what my default route was , if it could tell me yeah , and then these next hop sort of things .

But in practice that right , routers , routing tables are so dynamic , that might not be great , but if we took a baseline of , you know , say , a healthy state without a lot of churn , if we could capture that in the snapshot , then that base model could be used to say look , this is a healthy , stable state . This is the routing table .

I'm going to fine tune the model so it understands that that's my healthy , stable state . As the churn happens , we could then write code that says maybe do RAG retrieval , augmented generation , to get the new state and then do a comparison against the model . Right , here's my new state , here's my healthy state . That you know about Is everything okay ?

What has changed ? Are any routes flapping , that sort of thing ? Right Now , that would be pretty complicated to do , but you could , you know , one-time , train a model with like a baseline and then do the ongoing thing with not a non fine tuned approach . Right , you would be doing dynamic calls . Now .

I think something that might be more interesting is the actual PDF . That's that that Cisco validated design I've been talking about . That is a fairly static document .

Maybe twice a year it gets updated , or something right , or once a month or something , you could in theory take a PDF , as opposed to say , or once a month or something , you could in theory take a PDF , as opposed to say JSON data that's dynamic , or a policy set or something else that is fairly static and fine-tune that , as opposed to using something that

is super dynamic . Right , network state data might not be the best . Now a config , however right unless your configs are constantly churning , or a 24-hour view of your config that you refine , tune at the end of the day , and now this model sort of has a sense of your config over time .

That that , I think , would be a more valid use case maybe than state right .

Tim McConnaughy

This is kind of giving me yeah , sorry , go ahead , chris .

Chris Miles

I was going to say almost like using AI as a , as a linter , so to say right , yeah , yeah , yeah .

John Capobianco

Well , show logging is just one of the better examples . Right and now . That wouldn't be for fine tuning but for , say , inference . Right , Go get the show logs . And who wants to go through pages of that and figure something out . Well , take all that text and dump it into a vector store and then ask your questions that you're thinking about . Were there any ?

Was there any interface face activity between 1 and 2 am right or whatever . Your right , that's right , and let the ai dig through all that noise . Yeah , and come back and go actually , yes , gi 43 went up and down between that window of time or whatever , right ? So you had asked earlier about that .

Individual contributor practical uses right if you could write . Write something that's approved by your organization , either with their private LLM or whatever you can do to make sure it's secure and safe , right , logs might not be something you want a public , open cloud to learn about . Who knows what's in those logs right .

But if you had a closed-fence , private cloud or on-prem implementation , feed it those logs right , right , or your sem data or whatever , um , and start chatting with it , right ?

Tim McConnaughy

this has kind of given me , uh , digital twin vibes too , like the same way that , like a company like forward does , where you can , you could go train your model on your network , whether you know , on the the steady state of your network . And then I started asking it , if I pull this cable , like what's , what's going to happen , right ?

John Capobianco

That would be really neat . Right and and or neighbors or paths through the network or traces or whatever Right .

Tim McConnaughy

Yeah , so that that would be a really cool implementation .

I don't know if anybody's doing it or if it's gotten there or whatnot , but that's this idea that , like you said , of training a model on what your network looks like in steady state and giving it all of that data , and then you know , hopefully you could , you could infer then , like hey , that being the case , if I remove this route , like how will I get

from A to B ? Or something like that .

John Capobianco

Right , now I do have a toy that you can play with . It's out there . I've got two implementation . One is one is for less than a hundred packets in a PCAP , let's say , and one is for anything that is actually a substantial PCAP , you know , into the kilobytes or hundreds of packets . So one uses RAG and one uses Raptor .

It's just they're different approaches for how much data is in the , in the source , right ? So you literally upload a PCAP and can chat with it Now . So I had PDFs working and JSON working and then I was like , hang on , I think with Tshark at the Python command line I can actually flip a PCAP into JSON data and in Langchain there's a JSON loader .

So could I take a PCAP and load it into a Langchain and then chat with the PCAP . And then it turned out to be it actually worked right . So now imagine the Wireshark . I know filters and I love Wireshark , but is there a better way ? Did we just upload the packet , capture and go ? Are there any drop packets in this PCAP ?

What packets have been retransmitted in this PCAP ? Is this secure information in this PCAP , whatever you could think of , right In natural language , right ? So that was pretty exciting . That is pretty cool to actually . You know , upload a PCAP and talk to the thing , right ?

Alex Perkins

Yeah , absolutely Quick question on this RAG thing , because this is the first I've heard about this , but it sounds super interesting . Do you have , like I guess , like pre I don't know how to say this like a predetermined RAG that you use to like test against , Like what it ? Does it work like that ?

Or is it just specifically , like you add in something that you wanted to kind of reach out to ?

John Capobianco

Basically find something you want to chat with . So there's like a six step process with RAG via Langchain that I found and it's very repeatable . Now , the thing about Langchain is , think of a bicycle chain . Right , one link in the chain doesn't mean you need to replace the whole rest of the chain .

And what I'm getting at is I can write code that say lets me chat with a PDF , okay , and I there's a PDF loader . So we need a loader , then we transform it , we break it into small chunks , then we do embeddings , which is floating point numbers . We store them in a vector store , a database , and then we can chat with them using semantic searches .

So when I ask you the question , what is my default route ? What's actually happening is it's looking into the vector store and trying to find the closest matching floating point numbers to retrieve the zero , zero , zero , slash , zero route .

So what's neat is once I had pdf loader working , uh , you can just swap out the pdf loader for a youtube loader so you guys could actually chat with your own transcripts from your youtube videos . Wow , and the whole rest of that chain , the splitter , the chroma or the , the vector store , the retreat , all that .

You don't change it , you don't touch it , you swap out your one loader for another loader and your data basically . Right . Or let's say I want to play with a different vector store . I don't want to try chroma , I want to use pine cone in the cloud for my vector store . I don't change anything else in the chain , I just swap out the vector store code .

So it's very easy to get going once you have an early success . But there's GitHub loaders so you could actually chat with someone's open source GitHub repository with natural language Help me understand this code Right and the GitHub loader will load the read me and all the code and everything . And that's pretty nuts , it's really nuts . And then the JSON loader .

Well , every API , rest API gives us JSON back . So when I say a routing table , please don't limit yourself to that . That is any JSON . You want to chat with your AWS API , whatever API of choice , you can chat with it through this land chain system . Right , it's so remarkable . Guys .

I'm excited I can see you Googling , alex , and kind of looking into some of this stuff . Hit me up after and I'll dump you a bunch of links to get you going .

Alex Perkins

Absolutely . Because , my mind is just spinning with ideas of what you could throw in there . That's crazy .

Introduction to AI and Network Automation

John Capobianco

I'm being a little facetious , but I haven't read a PDF in months . Right , Like I have my own little utility that I just oh , there's a PDF I want to read . No , I'm going to upload it to my line chain and I'm going to chat with it . Give me a summary of this PDF . Give me the highlights and the important information .

I'm interested in their sustainability initiatives .

Tim McConnaughy

Could you find that right If it's an annual report or something , right , yeah , so actually this segues real quick and I have to segue because we're going to run out of time otherwise .

So , for people to get started , and we'll include notes in the show , notes and everything , especially network engineers , people who didn't start with programming , backgrounds and all of that how do we get started ? How do we start doing this in a meaningful way that network engineers can get value out of it ?

John Capobianco

Yeah . So I would really strongly look into Langchain and they have a YouTube channel and I you know I don't gain anything by promoting them . It's the framework I found . There's also like Lama Index is another framework for meta . There's LLM Studio is another tool . There's so many tools .

I think it's just subscribing to the right information if you're interested in this , finding the right people on Twix and whatever , and LinkedIn . There's a lot of people talking about this .

Tim McConnaughy

I love it . I love it . I started something , so add them to your list of things to pay attention to .

John Capobianco

The other thing is like start , like if you're inspired by chatting with a routing table , like that might be challenging but what you could do is , if you don't want to know the network automation piece , just take the text of a routing table , do show IP route , take the top 10 routes and just highlight them and then try to do a lang chain for RAG retrieval ,

augmented generation , with a text loader . It's the hello world sort of thing . I'm going to use a text loader . 's the hello world sort of thing . I'm going to use a text loader . I'm going to load 10 lines of text . I'm going to see if I can transform it and embed it and chat with it .

Um , I do have like a ton of stuff on my youtube to get going . Um , there is other . You know , if , even just if you start using chat gpt , just just like , try to integrate it into your daily life .

If you've got the next time you want to go to Google or Wikipedia or Stack Overflow , if those are your natural places to go , just try to start augmenting that with another tab in your browser .

That's on ChatGPT , and I think over time you're going to really start to graduate more towards using that and that's going to help you sharpen your prompt engineering skills and stuff I was about to say the same exact thing .

Chris Miles

I've been doing that yeah , I was gonna say . I think the thing is you're gonna have to . Just , you got to train yourself .

Right , you got to train yourself , you got to fix your muscle memory , to start using this instead of your normal retrieval tools , which you know I do it for some things , but you're the way you're talking , I'm like I'm thinking damn , there's 10 use cases I can think of this week where I should have tried to do this instead of going to .

Google or going to whatever , but yeah this is great stuff .

Tim McConnaughy

I'll be honest , I'm just using it . I'm mostly using it to learn , to do Japanese stuff to text , to test myself and do quizzes and explain stuff . But the point is , as I've been doing , that I've gotten better at prompt engineering . To get that , I started using the Risen .

There's so many frameworks out there for prompt engineering , but I like the Risen framework . That's what I've been using . That's great .

John Capobianco

Well , guys , thanks for having me , and I hope this inspires some of your listeners to start their journey with AI . I know it comes on the heels of some crash and burn fads like NFTs and blockchains . I don't believe this is that . I don't think so . I think this is going to be ubiquitous . So I think this is going to be ubiquitous .

I think that it's going to be in everywhere , from the gas station to your dentist's office to Home Depot to work , everywhere we go . I think AI is going to be a part of our society . Now , right ? So you know , that's just my own take .

I might be wrong and I hope I'm not , but I'll leave you with one thing Anyone scared of this I'm an optimist and a utopian . I believe in a utopian outcome . So , the first antibiotic in 60 years 60 years , our first antibiotic was created with artificial intelligence's help , and it's for MRSA , the medically resistant staff . Okay , this is .

That alone is going to save hundreds of thousands of lives , right ? Yeah , anyway , good to have me , guys . Thanks again . I really appreciate it .

Tim McConnaughy

Yeah , no , hey , that's awesome . Yeah , we are going to wrap up . I love your closing thought there , john . That's a really inspiring and good thought , because I know a lot of people are either afraid of AI or just dismissive of it , and it's good to be excited about it , I think .

And whether you're excited or not about it , you're going to have to get used to it . So we all are , especially in tech . So what do you guys think , chris Alex ?

Chris Miles

I think it's great . I think we're going to have to put some good stuff from John into the show notes .

So if you are listening and , like I said , you're interested how to get started , what to follow things that we're going to put some stuff into the show notes , because I think that's the important part is what you honed in on , john , is what to listen to , what to follow that's going to be relevant to you , because if you're just following AI as a trending

topic on whatever these platforms are , you're going to get tons and tons of bullshit that mean nothing to you . Yeah , you gotta , you gotta fine tune it a bit .

Alex Perkins

So I'm really eager to dig into that . It's been awesome . Yeah , I just my head is just still spinning . I just there's so many things I can think of for for all this stuff that you've talked about , john . So I'm I'm very excited . I'm going to be diving into a lot of this and going down a very deep rabbit hole for the for the next like week or so .

John Capobianco

I will say for you or anyone who happens to come across this if you get stuck , if you need help , if you've hit some error , if you need some inspiration , just send me a note . You know I will get to it . I'm pretty busy but I don't mind , right I ? I am very open , I'm a dot communist . I want everyone to have this code .

I want everyone to have access to artificial intelligence . I think it's yeah , you know , I've written my my MPs in Canada to say I think we should socialize AI and have a Canadaai for everyone in the country to have access to the thing . Right , that's how far I'm going . I think it is a human right , almost , to have access to this .

Tim McConnaughy

Right , very cool .

John Capobianco

All right guys , Thanks again .

Tim McConnaughy

Yeah , absolutely Okay . So if you guys , if all the listeners liked what they heard , please follow us on Twix , follow us on TikToks .

John Capobianco

On the TikToks TwixTalks I was trying to figure out how to work that in there . Twixtalks .

Tim McConnaughy

Follow us on YouTube , you know follow us home , especially alex , who loves to have people . All right , and we'll see you in another episode of cables to clouds . Hi everyone , it's tim and this has been the cables to clouds podcast . Thanks for tuning in today .

If you enjoyed our show , please subscribe to us in your favorite podcast catcher , as well as subscribe and turn on notifications for our YouTube channel to be notified of all our new episodes . Follow us on socials at Cables2Clouds . You can also visit our website for all the show notes at Cables2Cloudscom . Thanks again for listening and see you next time .

Transcript source: Provided by creator in RSS feed: download file
For the best experience, listen in Metacast app for iOS or Android