Steve Orrin on the Importance of Hardware in AI Development

On this episode of data driven Frank and Andy interview stephen Oren, the CTO of Intel Federal yes. Intel, the computer chip company. Because if you want to train your AI models in a reasonable amount of time, you need better hardware. Well, it turns out that intel has developed new CPU instructions to accelerate AI workloads FPGAs allow for faster development in custom applications with specific needs. Speaking of intel, you have to check out an upcoming intel and Red Hat webinar

link in the show notes. Tell them Bailey sent you. Now on with the show. Hello and welcome to Data Driven, the podcast where we explore the emergent fields of data science, data engineering, and of course, artificial intelligence. As with me, I always have Andy Leonard, my most favorite data engineer in the world. And today we have a special guest, Steve Oren, who is the federal CTO of intel. Yes, that's right, intel, the chip company. And although they do a lot more stuff

now. So welcome to the show, Steve. Thank you and glad to be here. Frank and Andy cool. So one of the things that I think people have not realized, people think that AI is a software story, right? Primarily. But quickly, once you get into it, everyone goes gaga for things like Chat GPT or well, no one's really gone gaga for Barred just yet. We're going to give that a few more time for the paint to dry on that.

But quickly, I think when people start becoming builders of AI tools, the number one restriction, aside from kind of what your data engineering pipeline looks like, is how quick you can train these models. And obviously, I'm pretty sure intel has a thing or two to say about hardware. Absolutely. And as you've as you've

alluded to AI, and all the things that make up AI rely heavily on the infrastructure that you're training you're inferencing. But even before you get to the fun stuff, how do you do the data curation? How do you suck in the data? The ingestion get the large multi node data sets that these large language models are trained against. There's a lot of hardware and infrastructure that has to make

that happen. And then when you get to the important phase with how do you train those in a timely fashion, hardware is the answer. And what we're seeing in a lot of these spaces, especially we start looking at things like large language models and transformers as well as looking at other approaches that are coming out, is that not only does the hardware matter, but the type of hardware matters. If you think about it, it's not a one size

fits all. It's a heterogeneous architecture to make sure you have the right hardware for your workload. One great example. So large language models in graph analytics requires not just heavy duty hardware but the right memory architecture to keep those nodes in place while you're training. And what you find is that often doesn't fit well. Intel just a classic GPU only kind of mode, which is what the classic AIS leveraged, just the sheer number

of cores that you would have in a GPU. And so what we're seeing is optimizing the hardware for the kind of workload is the answer to getting timely training. And especially when you start doing more. That sort of iterative. And feedback training, it's not a one and done, it's an ongoing process. So you need that to be quick enough and powerful enough and robust enough to handle those

workloads. And then the other side where hardware really starts to matter is on the inferencing, you want to be able to ask the question and get a response fairly quickly, if not near real time. If you're in a car and it's autonomous driving, you want it real time. You want to know that's a tree and not a shadow. If you're talking about online and doing some fun stuff with chat GBT, you still don't want to wait 20 minutes for your response. And so inferencing matters, training matters, and

the kind of hardware and infrastructure that support it. And that's why intel and our ecosystem are looking at providing a heterogeneous set of architectures. So our classic CPU, so the Xeon and the server and CPU and the client core, but also FPGA based logic AI accelerators like our Habana chips in the cloud and our targeted edge AI chips like Movidius for video processing and the like. But then really, besides the hardware, it's that software infrastructure layer. How do you

optimize your code? Because most AI developers are not hardware experts, nor do I want them necessarily to be. So a lot of it is about building out those abstraction layers that optimize your code, that's doing your hugging face or whatever, to take full advantage of the hardware underneath you, without you having to know what hardware is underneath you so that you can provision your workload where it needs to go and not have to

worry about the hardware infrastructure. And that's part of our overall strategy. And working with the broader ecosystem, the open source community, the commercial providers, and the software frameworks to give them the tools to get the best performance out of their AI and their data science, right? And I think you hit the nail on the head. I think we're at an inflection point. Not so much in engineering,

right, but more in the perception, right? Because whenever you think, oh, we have a large workload we got to do, let's throw some GPU at it, right? And it's a little more nuanced than that. I think people are finding out that you need more than just a bunch of GPU. And I was on a call and I want to get your thoughts on this, because he said something very similar to what you said. You ever have these moments when you're on a call and somebody smart says something, you're like, I don't know

about that, right? And it's kind of like what they did in World War Z and where there was like the 10th Man Rule, where no matter how ridiculous it sounds at first, you kind of want to investigate it. And that's why I was glad when your name popped up in the feed because I'm like, yeah, I want to talk to

you about this. Because he was basically saying that GPU usage is overrated and that where the real advantage is going to be is going to be in software acceleration and on CPU kind of optimization too, which sounds a lot like what you said. And when I first heard that, my first thought was, I don't know about that, but this guy's plugged in. He's a big shot at Red Hat, right? He's plugged in, he knows a lot. And I

was like, I didn't want to just dismiss that. Like, if my cousin said that, I'd be like, yeah, okay, but if this guy says it, whether or not he's right, maybe yet to be determined, but the fact that he believes it means that there's a trail there to follow. So I've been kind of poking around at stuff. Tell me about that. It sounds like there's some weight behind that opinion. So Frankie, you hit it on the head there. It's not that GPUs aren't important, it's just GPUs

aren't the only and best solution for all aspects of AI. And there are certain vendors that want, again, for a variety of reasons, want GPU to be the foundation for all of your AI activities. Like if you're a GPU based hardware company. Exactly makes sense. But

when you actually go look at the benchmarks across multiple and here's the key thing, across multiple AI types. So different algorithmic models as well as the flow, so there's different stages. So the inference versus training, ingestion and curation versus the training, versus the feedback training, what you'll find is that GPUs will rock for certain things and they are important for certain things,

both from that vendor as well as from a variety of other vendors. GPUs do play a key role, but when you look at the breadth of AI activities and the benchmarks associated, you actually find that a lot of really good work just happens on standard commercial off the shelf CPU. And actually most of the inferencing, I mean, we're talking in the 70% to 80% of inferencing happens best on CPU and areas like large language model and graph analytic based

approaches. The numbers really show very clearly that it's not a core bound problem, it's a memory bound problem. And so having efficient in and out of memory, which is what you get from a CPU or an accelerator with ample memory on board, is actually much more powerful for training those types of data sets because the GPU you're dealing with that latency across the bus. And that actually starts to matter when you're talking about billions or trillions of node graph

analytics. So I wouldn't say that GPUs are a dying breed. That is absolutely not the case. And there's going to be a huge market for GPUs or GPU like functionality. I want to be careful about that because you don't have to have a discrete card. The reality is you can have GPU capabilities embedded in your processor. We've already seen from intel and from other

architectures. The real interesting thing is making sure that whatever your workload is can be optimized, like your friend said, optimized through software to that hardware. So that if you are running a large language model, that you're actually running it on the right hardware, and that the hardware and your software know how to work together to give you the best performance if you're working

on. I'm seeing a lot of really cool things right now around graph based approaches in the memory intensive side of that and the switching back and forth between that. Those latencies can really come to bear when you're talking about cross bus

kind of communication. So having high amount of memory available directly to the CPU to be able to do those training, keep all that data in flight so you can train, is going to be one of the key differentiators of how you can take those large angle models, apply them to more than just writing cool essays by Shakespeare. I think what we're going to see is things like chat, GPT, and that whole category of transformer based approaches applied to just about everything, not just chat, but prediction

approaches. And it's really about getting it the training sets to become smart on those very vertical domains. That's going to be a resource intensive process and it's not going to be throwing a bunch of GPU or it's going to be a lot of cloud scaling and it's going to be a lot of memory intensive activities. And like your friend highlighted, the software is going to really matter, that it's taking full advantage

of the hardware to get you those performance report. Well, this reminds me a lot of just patterns I've seen over the decades of being in computing as a hobbyist and then a profession is you see a lot of things come into the fore as being very monolithic, and then people realize, wait, that's really a team effort. And I think about it as a baseball team, right? You don't want to put the pitcher, the person who's skilled at pitching in center field, can they

perform there? Well, gosh, yeah, but you're wasting them, right? They are tuned their whole body, their desires, their motivations. They love being pitchers. So put that person on the pitchers mound and you see this happen. And it's in all sorts of places. We saw it, frank and I have seen it over the years when the unicorns were the big deal, the data science unicorns who could do data engineering and everything that we've kind of broken out now into other fields.

And we're seeing it now in the hardware and in the distribution of the separation of concerns and the distribution of concerns, getting every component to do what it's best at. And along with that, and I'll shut up after this, is this whole idea that it's moving so fast that the hardware that's going to perform the task first sometimes isn't even identified yet because some new approach popped into the equation. If

somebody tested something and went, this is great. Now whether I run it and you just see that and it's on a scale now where it used to be measured in years and moved to months, it's now weeks and sometimes days. It's just amazing how fast this is going. And not that long ago, people were predicting an AI intel. Right. I think Dolly kind of and the whole generative artwork

stuff, I think kind of like, wait a minute, there's something here. Then Dolly came out and then OpenAI did the one two punch of here's Dolly a couple of months later, here's Chachi BT. Now you're just seeing like it's on fire. Like it's not just AI summer, it's an AI heat wave. Yeah, exactly. It is. It's a full El Nino. I like that. That's the quotable, for sure.

I think one of the things I think people realized is, and a lot of the thinking was that AI winter was coming because we're hitting processor or hardware kind of upper barriers. And I think we're finding out, I think much to what you said is that it's not just about throw this many GPUs at it. It's right. The entire story, the entire bus matters. Right. So the shortstop matters using the

baseball analogy. Right. The outfielders. Right. You can't really win a lot of baseball games if not everybody on the team is playing at their best. Absolutely. And just to take that metaphor all the way, the turf matters, too. The infrastructure that you're running those specialists on, you're going to play better in different fields. That's true. That's a good point. I love that you took the metaphor to the next level. That's awesome.

I think you mentioned whether it was in the virtual green room or here something called habanero. And I know you're not talking about just cooking. Right. Spicy habana. Yes, habana. I'm sorry. I had food on my mind, as is often. What is habana? Because I've heard whispers of it. I know we're recording this middle of May. There's going to be some announcements at the Red Hat Summit. Well, they'll probably already happen by the time this goes live, but what is

it? So Havana is an architecture, an AI

accelerator, and it's a specialty chips specifically designed for accelerating AI. And it's actually two chips. And the reason it's two chips is that you want, again, going back to what we were talking about, you want the right hardware for the AI workload. So you want to be able to have the right hardware to opt optimized for training flows and a separate set of hardware for cloud scale and hyperscale inferencing workloads. And so that's actually what Habana is. It's a two

chip strategy. So habana gowdy which is out available. V two is available. V One has been out for some time. If you go to the Amazon cloud, you can get it today. It's also available in data centers, and a lot of universities have them in their high performance computing environments. And it's geared to doing that sort of scale, large data set training that you would find whether it be in a cloud kind of environment, a chat GPT level of analytic, or in the case of high performance computing.

Whether you're doing climate modeling or flow dynamics, those kind of big training model sets that you want to be able to do at scale. And what's nice about it is that like your cloud scale, it scales with your architecture. So it allows you to be able to scale up your training based on the compute needs with an AI accelerator specifically tuned to that. The other chip, the Goya chip, is an inferencing

chip. So it's again tuned for that inference. But the reason, again, this is for high end cloud scale hyperscale or things like high speed training, where you want to be able to do large amount of inference in as near or close to real time as possible against really complex kind of data flows that you're trying to do the analysis of. And again, looking at the right hardware, we wanted to make sure to not just meet what we call the sort

of the normal scale. So the kind of things you would interact with when you're going to do fraud detection, but you also want to be able to handle really large scale inferencing because you're dealing with ingestion of multi data sets across multiple different domains and having to be able to do that inferencing in a streaming kind of mode. And that's really where the Goya chip shines, is an inferencing platform that can scale

with the cloud. And that's really the Habana strategy is about giving you the hyperscalers and high performance computing, the equivalent of an AI custom chips. And that's really where Habana sits. And then when you look at sort of the majority of what most people will leverage in a cloud or on prem, what we've been

doing there is adding new instructions to the CPU. So VNNI was the first really big one in AVX 512, which really accelerates the math that you're doing behind inferencing and training and give you those instructions. That software, whether it be Intel's OpenVINO software or TensorFlow or other frameworks can take advantage of that math to use hardware offload to accelerate the math that you're doing in your training and your inferencing workloads for most of your normal

kind of AI. A lot of the AI we deal with, not the high performance computing style. And so you get the balance. And again, it goes back to what we talked about in the beginning, the right compute for the right AI. We've also introduced data center graphics because again, there are workloads that absolutely make sense for a GPU besides fun gaming. And that's really where you'll see GPU shine on, those kind of specialty workloads that take full advantage. And a lot of the deep learning object

recognition ones work well on GPUs. They actually work well on other kind of platforms as well. And one of the things we're seeing in the Edge is a shift towards more customized approaches, whether that be using an FPGA as sort of a hardware platform that you can code in your algorithms to do inline inferencing, do feedback loop training. And you see this a lot of times in the image processing, video

processing side, also in the signals processing. So whether it's five G and being able to do signal quality testing or signal acquisition and being able to do RF signal analysis, FPGAs actually really shine for that kind of workload. Where you want to put in your custom algorithm that you're going to actually test against or use as part of your conditioning. And then we get to the idea of what we call an ASIC. And that's where you know your workload, you

know you're going to be doing this kind of inference. You can actually code that into a custom chip that will do just audio AI inferencing or do certain aspects of video coded. And this way you get the most performance in a low swap. And that's the idea here is you want to be able to handle everything from the pointy end of the spear, the Edge sensor and give it the ability to do AI as opposed to waiting for it to send the data to the cloud and get a

decision. You want to be able to give it something, but it also has to operate at the size, weight and power that you'd expect from an Edge sensor. You obviously don't have a data center power system for your car, for your drone, or for your camera on the streetlight. Right. That would be a very heavy to fly that drone. That's okay. I'm curious how you kind of manage what I'm just going to make up words here, but like an innovation chain,

I'm thinking about like supply chain management. And I know I've got experience in electronics engineering, and I know some of how much it takes to go into mind you my work was decades old, but this whole idea of getting ahead of the curve or at least being able to predict where the curve is going and how steep and when. That sounds like a huge challenge for figuring out what will be needed next. So what you're talking

about is how does a company that's building out both the hardware and the infrastructure, stay ahead of, like you said, the week to week turnaround in the AI world. Part of that is having a diverse team of specialists. So the Intel Labs, which is our team that looks five to ten years out, is over 1000 people who full time looking at process node technology, security, AI data science. They're across multiple domains and within each domain we have specialists in different areas.

One of the really I'll give you a great example. Before Chat GPT blew up, I had two different of my AI specialists, one on the government side and one on the performance side. Start talking to me about this thing called Transformer. Like, oh, there's this really cool thing that we're seeing here, it's called a Transformer. And I'm like, okay, that's interesting, and tell me more. And they explain

sort of how it worked. And then fast forward, six months later, Chat chips BT shows up and I'm like, I know what that is because that has the word Transformer. I've seen this. And again, it's about giving your people the ability to go out and look. I think one of the advantages of being at intel, and it's really why I've been here so long, is everyone knows intel inside. But there's something to that. Our chips are inside the edge. Clients are inside the financial services, healthcare,

manufacturing, oil and gas. They're in the government system, they're in the cloud, we're in the network. Which means we see workloads both current and coming from all those different domains. So in some respects we're on the cutting edge because we're seeing what people do because they come to us, say, hey, I've got this software, I want to optimize on your hardware. What does it do? Well, it does blah, blah blah blah. I'm like, okay, let's help you. And then eventually that becomes open AI.

That's the kind of thing because ultimately every startup, every big company wants to get the most out of their software and our teams. And one of the things people don't realize is intel has over 19,000 software engineers and a large majority of those do you know, they really divide up into three areas sort of research and pathfinding, ecosystem enabling, and then software development for compilers, software services, software tools. That ecosystem enabling team

is a very robust team, it's been around for a very long time. Whose job is to make Microsoft Windows rock on intel, make Oracle rock on intel, make red hat rock on intel, make open source. We have over 1000 open source software developers whose full time job is committing

to open source. We're actually one of the largest committers to open source community and a lot of what they do is build the optimized version of those Linux kernel libraries or to that AI model running on intel and give it away and open source it. We've created whole model zoos optimized for the variety of intel architecture because we know if you can run it best on intel, you will run

it, and that consumes resources. We like that. But ultimately it gives us they call them bell cows, if you will. We're seeing those bell cows of what's coming next because they come to us and they say, hey, help us. And very few see us as competition because we're not going to go build the Chat GPT. We're not going to build a new operating system or a new sort of predictive maintenance solution. We're going to give you the architecture for you to run it

best. And even our OEM, whether you buy from Dell or HP or from Lenovo, we don't care. You're buying intel hardware inside. And so let's help you take the best advantage of those platforms. And that's really been the approach from intel, is we want everyone's software to work. And even with the GPU vendors, they still run on a CPU platform. And so we want to make sure that that code runs best. So that, again, you're driving the overall consumption. We raise the bar for everybody. We

raise the bar for everybody. Nice. Yeah. I think there's a lot to unpack there. Right. And I think one of the things you brought out, which is something that people don't, I don't think people have widely realized yet that Edge is probably going to be the next frontier in just computing. Right. Obviously the last ten years have all been about cloud. Right. But I think we're swifting as companies kind of take a look at the bills and realize that lift and shift was not a

financially great decision. Right. Whether or not cloud is a good thing or not, I think it always goes back to those two words that every consultant and every It person always says it depends. Whereas previously it was last ten years was oh, definitely was the two words. But I think now we're realizing it depends. And I think one of the drivers for this are things like autonomous systems

or drones or self driving cars, right. No matter how good 5G is, and I can tell you I know all the dead spots in the DC area, but if you're driving along at 60 miles an hour, 100 miles, 100 km/hour for our friends overseas, and like you said, is that a tree? Is that a shadow? Is that a person? Is that a grandma? Right. You don't want to wait on the latency to come back. You want the inference or the decision to

be made on device. So you're really bumping up against the speed of light and you're talking nanoseconds, not milliseconds. Right. What do you see? Because you mentioned you want there to be sensors, but obviously these things have to be relatively low power. I guess in a car it doesn't matter as much, but certainly on a drone that matters. What sorts of challenges does intel see in that regard in terms of you want the most performance, but you want the most

energy efficiency. That seems like two opposing forces. You would think that way, but if you

look at Moore's Law and you look at what's really behind that, it's about reducing the size. And really that means the power and increasing the performance, increasing the amount of transistors. And that's really been what's driving compute all along, is how do we get to lower power per density. Now, where it becomes interesting is in the cloud. It's a cost measure. It's about getting more for your dollar in a car or in a

drone or even in a factory floor. It's about being able to operate closer to where the decision needs to be made without having to, again, to have to power it and have that immense cost. Or in the case of a drone, the weight of the battery pack and so forth. So lower swap actually enables those edge use cases. And again, one of the things that people realize is that Edge can mean different things to different people. You talk to the cloud providers and Edge is just

a couple of racks closer out of the cloud. On Prem, you look at Azure Stack or Snowball or these kind of approaches. It's really about pushing pieces of the cloud closer to the edge through like the core or they called it the fog back in the day. You look at the edge and you take a look at a Tesla, it's like a driving data center. There's compute capabilities in there. A plane is a flying data center. Your drones are getting to be more

computing. And when you move from a discrete mode to a logical mode, and I've seen these already, where you have a drone who actually has one processor but multiple containers, so actually running multiple functions that could be thought of as different applications on different nodes, but now they've all been collapsed with either virtualization

or container. So you can have navigation being one, you can be doing object detection and mapping with another, and then be able to do sort of other kinds of sensing like temperature or barometer and things like that and doing analysis in real time. One of the best examples that we demonstrated at our last year's Fed summit was a set of drones out

mapping a region. They were going about their business, but they had a policy that if somebody walked into a specific area of interest, let's say in front of an embassy or in front of Lloyd or too long, that one of the drones would be retasked and go over and investigate and do facial recognition. All the things you want to do to make sure, hey, is this person up to no good? And it didn't require a reprogramming of a drone. It didn't require a special drone that was just the investigator. It

would basically retask itself with a new. Mission in real time and go investigate. And when the person left that zone, it go back to its day job of mapping the environment. That's just sort of the tip of that simple prototype to show that even a very small autonomous system and these were like sort of my mini drones here, is capable of the compute necessary to do multimission kind of use cases. So the edge absolutely is that new frontier. And it's again similar to the cloud. When you say cloud,

everyone thinks, oh, public cloud, really? Cloud is all those architectures all the way down to the edge. It's the way we develop those cloud native apps that can flow back and forth. So from a cloud provider, it's moving more of their cloud infrastructure closer to the edge. And what the edge, folks, whether it be the actual device or sensor manufacturers are looking at, is bringing some of those cloud capabilities to their device to operate

independently. And there's a reason for that is that, number one, latency, like you mentioned, Frank, but also the cost of shipping all that data. No one wants to ship Raw 4K video feeds to the cloud just to be able to tell me, is that a tree? You want to be able to send the results that I saw a tree here with the longitudinal latitude, which is a small data packet, and let the sensor do the AI, do the inference

at the edge. Right. And then you have the case where you're talking about planes or vehicles, right? Like the whole time it's tracking, did the wheel fall off? Did the wheel fall off? Did the wheel fall off? Right, but at one point when you get to your destination, the wheel either fell off or it didn't. Right. So you collapse that entire thing to one integer level or really not even an

integer. Like a bit. Right, a bit. And then if the wheel does fall off, I'm sure there's plenty of other stuff you can pick up too, but hopefully nobody gets hurt. But I mean, ultimately you're right. The problem with data is so much that there's value, but there's a certain amount of we've gotten to the point where just because we can, we've done it. Right. Yeah, sure. Bring up that 4K. If I'm a salesperson for one of those cloud providers. Yeah, man, bring in all that 4K data you want,

we'll take it all. We'll be happy to charge you for it too. Right, but I think as we get to the point where there might just be too much data, I think people organizations are going to start thinking like, where can we scale back on the storage? Because we don't really need it unless there's some kind of regulatory reason for it. Now, one thing I want to double click on, because this is a fascinating conversation, we'd love to have you back

on the show at some point. What's the deal with FPGA because you mentioned that and this was a huge deal. So a couple of things that are interesting is that I first heard about Transformers at the Microsoft has this internal data science conference MLADS, and they first talked about Transformers. I went into the talk and ten minutes, my head went boom, right? I didn't quite follow it. Somebody later on in the day in the reception area was kind enough to explain it, how it

works. And one of the other things that came out of that conference was talking about the importance of FPGAs and what they're going to be like in the future. Now, again, I'm a data scientist. I really don't focus on hardware so much until when I need to buy new hardware, like a new desktop or laptop. What are FPGAs? And I remember hearing a lot about them and then they kind of went dark for a while and then now they're kind of coming

back into vogue. Can you talk to us about, one, what they are and then two where you see they're going? Sure. So Ed and FPGA are a field

programmable gate array. They've been around for forever. I mean, computer science engineers going back, electrical engineers going back to the 80s played with FPGA. They were very early FPGA, but basically they're programmable hardware. That's really the way to think about it. You think about a CPU or an Ace or any chip it's laid down with its transistors, and the flow of those transit is fixed. CPU can do multiple software flows, but the instruction flow is the instruction

flow. What makes FPGAs interesting is that you can create new RTL, new layouts of flows, what they call netlist of those instructions going across those transistors each time. You can go in and customize it after. So the manufacturing builds you a clean slate of a bunch of think about a bunch of rows, and then you program them to your specific need at a hardware style abstraction layer. So it gives you a much faster capability because you're now really writing in hardware. It's a lot more

complex of a coding. It's not like doing Python, but what you get is a very optimized piece of hardware for your specific use case. And what's nice about that is one of the great examples is in signals conditioning. When you're doing like 5G research or testing signal amplitudes and things like that, as you put in your algorithm actually into hardware, you go out

and test it. It works sort of here. I need to tweak it well, instead of going and spinning a new piece of hardware, you just upload new code and you go right in. So it's a much faster time of development for doing those custom things. What people have found when we start looking at sort of AI use cases and machine learning and pattern matching is that FPGA really lend themselves well to be able to create different kinds of architectural approaches to how

you process that data flow. If you think about a GPU or CPU or even an ASIC, it's a fixed data flow. It's good for the things it was designed for. What FPGA allows you to do is to customize your flows based on what the data is or based on what your algorithm are. And so a lot of the FPGA work they were seeing in AI is people coding their AI algorithms or the machine learning algorithms right into hardware and then deploying it. And so it allows you to be able to deploy

your thing quicker and you get pretty good performance. It's not as good as say, as a custom ASIC for your algorithm. And it's not as scalable really as like a software abstraction on running on a cloud set of CPUs. But for a lot of these training and inferencing use cases, one of the areas where it shines is in the whole area of neuromorphic processing. So a whole part of the AI machine learning space is modeling after brain activity or how our

brains process. It's a whole field. FPGAs are actually well designed for those kind of algorithms that X 86 and other CPU style Arctic just aren't yet. And that's why FPGAs really shine in those environments, because you can create these linear sort of permutation flows that you find in neuromorphic algorithms. You just code those into the path for the

FPGA. They're really good. You'll see, FPGAs are very often used in cellular and RF communications that are really good at those sort of channelizer and signal optimization and be able to do those kind of algorithms that you do on RF and Comps, again, really good for those kind of workflows. And so why we see the resurgence of FPGAs, although they've never gone away, you find them everywhere. Open up your big screen flat screen TV, you'll find a couple of

FPGA in there. Where they're shining is because it allows you to do some rapid prototyping on AI. And because we're seeing now FPGAs come to the cloud. So you go to Azure has an FPGA cloud. You can now deploy those algorithms at cloud scale, or you can deploy an FPGA into your edge sensor and be able to do that real time, sort of. Let's go try this inferencing model. Oh, we're

going to change the inferencing model. Let's go do that one. And where this becomes really interesting in those low slop environments is a modern FPGA is reprogrammable in milliseconds, which means you can go from one program to another by just pushing a firmware, if you will, update. And now you go from a 5G communications system to LTE or to a six G without actually going and swapping out the hardware. That's wild. That's wild. Yeah, it's exciting times. So with that, the updatable part of it,

how do you secure that? Because I can easily see that being like particularly you work in the in the federal space, right? Like security is top of mind in that work. It should be top of mind everywhere, but in the near term it's top of mind, at least in the federal spaces. FPGA sounds like awesome, but it also sounds like that just seems dangerous in a lot of ways. You can reprogram it in milliseconds.

There's got to be some kind of security story there. Oh absolutely. And Fpjs have actually in many cases led as far as the kind of security mechanisms built into the hardware for that very reason. At its core, at the core level, it's the same kind of approach you do

for verifying your firmware on your system. It's signed by hardware so that basically you're verifying your load and if you're going to do an update, you're going to verify a signature against a hardware rooted key so that you make sure that only legitimate folks can do the update and that it's only be able to be done

by someone who's got the permission. From a cryptographic perspective, what we find in the current FPGA that are out in the market is that they've built in a whole suite of security capabilities. Things like Puff Provably, unclonable functions, which is basically a hardware root key that is really secure as that hardware route of trust, signing in cryptography functions, anti tamper functions to make sure someone can't go pop open the lid or put in a jumper and try to try to change

the code. So those kind of mechanisms have been in place for a long time because FPGAs have been used in such critical places. We find them in radar stations, we find them in systems and so they've been building security in for a very long time. And it's part of the workflow that when you build your code you're going to take advantage of these implicit, let's call them IP blocks that do security for your RTL, for your code that you're putting

in place. The other important thing is that the way that the code works is once you lay it out, once you translate your software into that layout, the layout is you can't just sort of go and reverse engineer back. And so it's really a very powerful mechanism as opposed to say firmware. When you're it's software.

If you think about the BIOS update, it's software that you're loading just deeper in your platform and if anyone wants to go inspect, you'll find there's a lot of software in the hardware that you don't realize is actually software. The same kind of security mechanism we did there. You verify it against a hardware of trust, you make sure it's signed before you run it and then you apply cryptography to make sure that it can't be changed or it's

integrity protected. You find those same capabilities built into the hardware of an FPGA and the software development tools, the dialogue, the cordis and so forth have the mechanisms to take advantage. So again, programmers don't have to be security gurus. They basically say, I'm going to push this, and it's auto going to take advantage of those features. It's good because programmers historically are very bad security people. I say that. It says, yeah,

it's its own specialty. And yeah, you can't be good at everything these days. There's too much. So I'm going to echo what Frank said earlier. Steve, we got to have you back. I really appreciate you being here. We could talk and geek out on hardware stuff forever, but we want to pivot and go to our questions and if that's okay, we want to start with unless Frank, unless you had anything else you wanted to do before. Let me

rephrase. No. In the virtual green room, you talked about some things that are going on and kind of operationally and wow, we didn't even get there. I mean, I think the important thing I took from this conversation is that one, GPUs, they are important, but they're not the whole story. And two, at the end of the day, chat GPT, any of these magical looking AI models, magical seeming, right. They're all mass, right? Yeah. And being beneath the math are electrons

bouncing around inside these microscopic chips. And there's all sorts of things you could do to tweak and improve that, even if it's like a billionth of a second, right? A billionth of a second times a billion adds up. And that adds up in terms of whether you're driving a car or you're flying a plane or you're a company like AWS or Microsoft, where, hey, if I save one compute second per transaction, I do trillions of those a day. And that's real

money. Exactly. And that's the thing that blew my mind. But yeah, let's switch because we could geek out for hours. Because this is very true. Yeah. Amazing. It really is. So how did you find your way into not so much data, but it how did you find your way into data? Did you find it or did it find you or hardware specifically? So ring. It's a really good

question and going back to the very beginning, actually, I started out in the molecular biology bioresearch side of the camp, going all the way back. I was going to be a research biologist and probably still be there today, except for a couple of key life events early in the early ninety s, I was a hacker as a kid. I loved seeing how things fell apart and how to code and break code and things like that. But in the late 80s, there really wasn't a career other than a COBOL programmer, which

wasn't an exciting career at the time. So I went the bio route, which was my, the love. And right after I graduated and was going to start med school, I had a year off and someone had some money, wanted to do a startupy thing and they knew I was a hacker and say, hey, why don't you help me get this thing running? And I'm thinking, well, med school is expensive. This would be a good way to help

pay for it. And so I started my first company in 95 and after three months just fell in love with everything that was going on. It was the exciting time to be in the internet. Got to apply some of my security hacker background in an interesting way and had some really good mentors. People like Bruce Schneier, the writer of Applied Cryptography sort of took Zebru Schneider. Zebrus Schneider was one of my mentors and took me under his wing.

And like I say, I sucked his brain dry as best as I could. But really it just sort of got the opportunity to get on the ground floor right before Netscape went public. So really early days on a startup in the email encryption space and then one thing led to another and I just felt this was what I was going to do. And for the next sort of several years, I did multiple security startups throughout the then in 2005 got acquired by intel. I like to joke, I'm still trying to figure out

how I ended up here for 18 years. But I think what intel has provided me and provides a lot of our folks is the ability to sort of innovate in an environment where a, you've got a big company behind you helping you do that. But one of the best reasons why I think intel has been fun for me, my most successful startup, we had 500 of Fortune Thousand companies using our product. The first project I worked on in intel went to 40 million PCs. So the impact is just

unbelievable. Now from the data side again, at the end of the day, like you mentioned earlier, underneath the data, underneath the machine learning, underneath the AI, and even before we were talking about AI was machine learning and advanced pattern matching. There's electrons moving around it's running on hardware. And so a lot of what my job has been before I came to the federal team was looking for ways to innovate or take advantage of new use cases in software, to

take advantage of hardware in interesting ways. And so we call that pathfinding. So you think about our labs or thinking about the next generation hardware five to ten years out, I ran the team, the security pathfinding team that was looking at the two to five year horizon. I knew this was the hardware platform that was going to be there next year. What would be some interesting things I could do with it to either advance security or

increase security, that was my area domain. And so things like antimalware technologies, cloud security, before they knew how to spell cloud. We called it virtualization security first and things like that. Web security, that was the fluffy stuff. That was Steve's world while the hardware engineers are figuring out low level cryptography and hardware roots of trust. And we sort of worked in tandem to innovate. And so as things like data science started to take off, it was like,

this is a key area both from a security and perspective. How do I secure that data? How do I secure the algorithms? How do I use that? I mean, one of the really cool things is being able to use machine learning and AI and apply it to the cyber problem. And when you start doing things like that, you immediately run to, well, we've got too much data flowing in. I mean, the classic example is streaming

analytics on network at network speed. Well, how do you do deep packet inspection at gigabit or higher speeds without losing data? That's a big problem. That's where hardware can help save you, that you just can't do in software. And then when I transitioned to the federal team and took over and drove our federal technology practice, you really opened the door to all the different use cases. And one of the things I like about the federal

government is that it's a macrocosm of all verticals. You want to talk finance, you've got IRS and CMS, some of the largest processing of financial data. You want to talk healthcare, the VA is the largest provider of healthcare, the largest insurer in the world. You want to talk logistics, DoD logistics is huge. So you sort of look at it, every kind of use case you'll find in government. So it's really a good way of looking at all the different verticals. And they

all have unique or interesting data problems. There's some commonality. And one of the things I really like about the federal government is that you get that commonality across the divisions. They all are having trouble doing data ingestion. That is just fundamental. It doesn't matter if you're the federal government or Citibank or startup in Silicon Valley. Data ingestion is hard and doing it at scale and being able to then do something once you've got the data. And I like to use the analogy

of an iceberg. So AI, Chat, GPU, all these are the tip of the iceberg. That's the cool, sexy stuff you can do, the hard work, the data curation, data wrangling is all the work that has to be done before you ever get there. And that's data ingestion, it's labeling, it's curation, it's data set management, it's all that stuff. And then layer in things like removing bias or dealing with bias and securing and integrity, protecting your

data. Like all those things have to happen before you ever start having the fun math that happens towards the end of that curve. That's where you find that coming out. Everyone is challenged with those things, and I think that's where the excitement is today. No, you definitely hear in your voice, sorry, Andy. Yeah, definitely. No, it's okay. We refer to that as kind of a joke that's been going on for seven years now. We say, first you get the data,

and that's 90% of the work. We know that and your iceberg analogy fits that, Frank. We need a shirt that has a picture of an iceberg against us. First you get the data under the I like that. I'm definitely going to do that. We launched a magazine, actually, yesterday as we record this, and the cartoon segment is called First You Get the Data. And it kind of like cringy things that you'll hear about data, and one of them was like, yeah, first we get the data. My favorite was how

to prep and clean the data. And they were like, oh, no, our data is already in the normalized database. We don't need to clean it or prep it. It's already ready. Like, oh, boy. You need you need a picture of someone throwing data into a washing machine. That's a good shirt. We could do that. Yeah,

no, that's cool. And I think you bring up something that I think, folks, we don't know our exact age demographic. We have a rough idea, but if there's anyone, let's say, under the age of 30, right in the car with the parents or they're listening, it's hard to imagine the time because we're about the same age. I think you're a little older. If this was not seen as a good career path, like, coding was not the whole learn to code movement is a modern

phenomenon. I started my college career to be a chemical engineer because I had to convince my parents that software engineering was a viable career path. And my mom, God rest her souls, was like, I don't want my baby to be one of those weird people in the basement. Right? And then my dad, God rest his soul, was like because when they came to visit me, I had a Sunday print out of the New York Times, which of course had the job section, which was

at one point like a book. Right. And look at all these jobs for computer programming. This is a thing. And my dad looked through it, and he saw all the starting salaries, and it was like seven or eight pages of near six figure salaries in the early 90s, which was a lot of money back then, right? Yeah. Like, looking through, like, on Wall Street stuff. And he's like, I'm sold. And it's like and my mom was like, no.

That is literally, like, my experience as well. When I told my parents that I was going to not go to the research biology route and do the MD PhD, I was going to go into the security thing. They wanted to do an intervention. They thought something was wrong. About two years. In 96, after I'd done the start, for about a year and a half, there was an article in the New York Times, Paul Cotcher, had done the timing attacks against RSA, and it

was front page news. And when you read down the first blurb, it says, 22 year old bio student from Stanford cracks RSA encryption. So I cut that out and faxed it to my parents because they have an email yet and said, look, another bio student doing security. It can happen. Right? That's funny. One of the best web developers I ever worked with, his degree was in biology

as well. And I think there's something to be said about understanding natural systems, and I think there's some pattern matching gifts that go along with that. I know my friend was that way as well. And Frank, when your mom said she didn't want you to be one of those weirdos in the basement that flew through my head, but I maintained discipline was too late. And I could say the same for me as well. Too late.

In her defense, my mom stayed with us in a house that my wife also works in technology too. She had an entire suite in our basement of our house, which was not windows, walk out yard, everything. It worked out well. Sometimes your parents my mother encouraged it without realizing. She allowed me to buy the haze modem and connect it to our phone. And I did get disciplined when I had that $1,000 phone bill from dialing into BBS's overnight. But they should have seen it coming. Yeah,

my mom freaked out when I wanted a modem. She's like, no, absolutely not. And my dad was like, yeah, you probably should stay out of trouble. It's easy to stay out of trouble. Then. I think I was lucky that my parents didn't know what a modem was, so I didn't know what they were getting me. Right. This is awesome. But I want to jump to question too sure. And ask, what's your favorite part of your current gig? Favorite part of my good gig? I think honestly, I thrive on being challenged,

on trying to solve big hairy problems. I think that's what has always excited me is present to me with something that isn't being done well today and trying to figure out how to do it. And I think one of the things that I love about my job is meeting with government customers who have big hairy problems and looking at a variety

of technologies. And I think what makes my role somewhat unique at intel, so we have like a CTO for memory and a CTO for various architectures is my role is pan intel so I can look across FPGAs server parts, networking, and sort of see that collective of where do the bits can

come together to solve big hairy problems. And that's really, I find keeps me very excited is that every day I could be talking about an IoT problem today with an edge sensor, and they're talking about petabytes of data being processed in the cloud tomorrow. It's looking across the technology domains and again, coming from a background of cybersecurity, which again looking at various different domains from a security perspective, but then adding to that AI, high performance computing,

it's a technology playground, right? And the federal government, when I first joined Microsoft, I was in the public sector, part of doing basically technology developer evangelism for the federal government. And a lot of my commercial sector colleagues were like, wow, it must be really boring there. I might be like, you know, we see things that you don't see and what it is, is like there's interesting work going on, but the folks doing interesting work for many reasons do not want

a lot of attention. Indeed. So you see some things that like, wow, see, I hadn't really thought of that type moments. Well, decades ago I spent just a little bit of time in a really odd shaped building up that way. Just a touch of time. So I can have five it did. So I can go yes and amen everything you both have shared about. So now we have three. Complete the sentences. When I'm not working, I enjoy blank.

Spending time with my kids. I have two small children and they keep me young and full of fun and keep me trying to stay in shape to keep up with them. Very cool. Both Frank and I have children as well. Frank has the younger kids. I'm probably the old guy in this conversation now that I think about it. But number two, complete this sentences. I think the coolest thing in technology today is blank. One thing that is a tough question,

I would have to say. So the two things that I think are really cool. Number one, again, not the chat GPT, but what the future will do with that capability is one area. And then again, because I'm a security geek at heart, post quantum crypto is going to be fun. Figuring out the next generation of algorithms and how robust they'll be once quantum computing comes online. I think that's an exciting area of math that is going to

spurn a lot of mathematic. Academia is excited because it's a renewed interest in that space and the algorithms are really interesting. The lattice space structures are fun area of math to look at. Nice. Interesting. The third and final, complete the sentence. I look forward to the day when I can use technology to blank. So I'm going to give you two answers. I look forward to the day when I can draw something on a whiteboard and it turns into code. That's one thing I'm looking forward

to. Oh, nice. I can totally and that's not that far off. It's not, I think a little bit of sort of the

image to text, image to code. I think building box, you have to be able to read my horrible handwriting. That's going to take an AI in its own right. But I would love a day. When I can start draw my design like I like to do I'm a whiteboard kind of guy, and then have it create a prototype. I think that's one thing

I'm looking forward to. And then I think the other thing is I'm looking forward to the day when augmented reality becomes reality, where it's not just a cool toy, but where we actually see it integrated into our daily lives. And I'm not talking to glasses and all that. I'm talking about having the digital world and our physical world actually start to make sense instead of it being a throwaway toy and I think we're seeing pockets of it, but I think that the future is going to hold a lot

more of that immersive experience that we only see in movies today. I think those are the two things from a technology perspective, I'm looking forward to. Although I have to say, if I can get that, the code from the whiteboard is going to make me a lot more efficient. No, that's true. And it's funny because things that once seemed impossible are now possible and even mundane. So I remember when I was a kid, there was a story, there was like a story we

read about a kid who wrote a built a homework machine, right? And this was like first or second grade and a bunch of us kids were like, yeah, how do we do this? We got to make one of those. Now you look at Chat GPT, obviously we abandoned the effort because it just wasn't possible at the time. But you look at how kids are using Chat GPU today, that machine exists not in the way or the shape or form we could have imagined, but

it's definitely here. So to have that whiteboard to code thing, it's totally within sight. Whether it'll be within reach, only time will tell. Probably a few weeks. If there are VCs out there listening, this is an idea to invest in, for sure. I would love to see especially for you, Steve. I'd love to see whiteboard two FPGA code. That'd be even better. We're just combining ideas. There you go. I know that would make some of my engineers happy. There you go. Really

cool stuff. So we ask all of our guests to share something different about yourself. But we caution everyone to be fair that remember, we're trying to keep our clean rating at itunes, so please keep that in mind. So something different about me. Well, I guess one thing we've already talked about that I have a bio background, but the other thing I like to do is I play tournament poker. I am an avid poker player when not in COVID Lockdowns and things like

that. I played in the World Series back in 2013. Really? That's something I like to do as a past. It's a different use of my skills, of sort of social engineering, if you will. And I like the tournament play because it's sort of a long game. Right? Well, I have a stack of money and I'd love to learn more about is that the joke? All you need is you're always welcome to my table. I'm lying about the

money. My wife is actually a pretty good poker player, and when she was pregnant with our second, she's short and she would carry a stool with her because she would have to set up and her feet didn't reach the floor. And I think I gave her like $100 in seed money and said, go knock yourself out. And she came back like she was spending money. I think she turned that into something like two grand before she had to quit and go have Emma. I would love to see you, because I don't think she's

your level by any stretch, but she did okay. We should have a data driven poker tournament. We should. There we go. That's an idea, Frank. The other time we had an idea of somebody on the live stream said we should do like an ATV race or something because we always go off track. That's kind of the joke. Very true. But no, that's cool. Audible is a sponsor of data driven can you recommend a good book? Ideally audiobook if you do, audiobooks if not. Sure. Absolutely. Actually, I just

finished one that I think would be perfect sort of summation of this. So Chips is an excellent book. You think it's talking about today, but it gives you the history of how we got here. And even one of the things I thought was really interesting is some of the decisions that were made early on from the policy, the government policies that we've seen and how it affects where we are today. Fascinating reading. So, yes, absolutely.

Chips wars, it's available on Audible because I literally just finished reading listening to it on Audible. So that would definitely be a book I would recommend. Cool. I watched a show called Halt and Catch

Fire a few years ago when it was at, and it was similar. It was in that vein of when things were developing and trying basically the laptop development story. And of course it was fiction, but I know enough about it to know there were some true parallels in there. So this would be very appealing to me. I'm going to get it. I hadn't heard of it. Thank you for recommending and our listeners can go to thedatadedrivenbook.com I didn't test it today, Frank.

Some days it's moody, but if you go there, it should redirect you to Audible. And if you decide you get a free book on us. And if you decide later to sign up, then it buys Frank a cup of coffee. So when you do that, we get a little bit out of it. It's a great way to support the show and we really appreciate it. Awesome. And where can people find out more about you and what the federal team at intel is doing. So find out more about me, go to my LinkedIn page. That's S-O-R-R-I-N on

LinkedIn. And then to find out more of what intel is doing in public sector, just go to Intel.com public sector and it will redirect you to our Government Solutions page. It covers everything from AI data science to Cybersecurity to Edge, with lots of white papers. Use cases podcasts with folks like myself and others that are recording content on how intel is helping our ecosystem. So definitely come check us out. Awesome. And with that, I'll let Bailey finish the show. Now that was some

show. Is it me or are the shows getting better? It could be my bias that leads me to say that, but I figured I would ask to get more input. After all, what's an AI without good input and a feedback loop? Speaking of feedback, have you checked out Data Driven magazine yet? We are looking for writers for the Autumn 2023 issue.