Hey, Carl and Richard here with your twenty twenty four NDC schedule. We'll be at as many NDC conferences as possible this year, and you should consider attending no matter what. The Copenhagen Developers Festival happens August twenty sixth through the thirtieth. Tickets at Cphdevfest dot com. NDC Porto is happening October fourteenth through the eighteenth. The early discount ends June fourteenth. Tickets at Ndcporto dot com. We'll see you there, we hope. Hey, gis what it's dot
net rocks. I'm Carl Franklin and amateur campbell fishwes Lele is here with us and we're excited to talk to him. How are you doing, my friend, seen any hotters this year? Oh? Yeah, well, the otters are everywhere right. It's been summer up on the coast, so it's hard to be on happy up here. But we've got the kayak out, cleaned them up, did some repairs, all the things you need to do, and then you get to go, you know, meet the otters where they
are. How's that? Do they taste good? Well? They think no, they don't. They smell and they smell terrible. But now, don't eat otter. That's that's not a good eating carnivores in general, not a good idea. That's a good idea. But no, they've got to stink all their own. Man Like, it's not hard to find an otter den you'll smell it first. Wow. Yeah, okay, I have no such things to report. But summer into your place means barbecue, right like the
girls are running. I just bought two new twelve hundred watt subway first for the band. Well, he's that kind of guy. You guys have been playing up a tornado these days. I mean I only see it from your calendar. I could seep seeing every every Friday, every Saturday, blocked out through the summer. One of the guys that saw us at Ocean Beach just sent me like fifteen videos that he took of us playing wow out there.
I don't know what to do with them. They're little clips. You know, you put them on TikTok, dude, that's what the hip kids do. I guess. Yeah, while it still exists, yeah, before while it's look, it's not going away. It's just a question of who's going to own it. Yeah, maybe it's the reality. All right, Well, I have something really funny and very cool from our friend Simon Crop for better no framework, roll the music. Alright, man, what do you
got? I love Simon Simon's and Assi and he comes up with really great stuff. Every once in a while he crosses the line into comedy, which is also always helpful. But he's got this tool called waffle generator for the big waffles. Well, in a way produces text which on first glance looks like real ponderous pros replete with cliches. So this is like you know your lorem ipsem if you just want text to fill up the screen instead of using
Latin random Latin? Is this lo ifs in the large language model era? Is that what this is? Oh? No, yeah, so this is great. Look at the example content on the Githovery boat. This stuff, I might say, the aesthetic of economico social disposition quote in this regard the underlying surrealism of the take home message should not divert attention from the esthetic of economic economic social disposition Humphrey Yokomoto in the Journal of the Total Entative Item.
And then there's some on any rational basis A particular factor such as the functional baseline, the analogy of object, the strategic requirements, or the principle overriding programming, provides an interesting insight into the complementary functional derivation. This trend man dissipate due to the measurable proficiency complete and utter nonsense, But it sounds good. This is LLM degeneration, you know, toombs for a purpose. It's
so awesome. So it's hilarious. Uh yeah, so he basically get this. He has a blazer wrapper around it, of course, that's what it's called blazing waffles. Blazing waffles. Oh my god, it's, oh, simon, so awesome, brilliantly awesome. If you need to make text. Yeah, he's also got an extension for bogus to use waffle generator, which is great. Okay, so good. He must be stopped though. Sometimes I got to ask him about this. What I'm usually the one he says
stuff like that, he must be stopped. What do you think of that? Vishwaz Yeah, it's great, it's good. You know, whenever every time we get something that's both a functional tool that we're going to use and makes us laugh, it's like the perfect better no framework. It's a good day. It's a good day, every day, no question. All right, So who's talking to us? Richard grabbed a coma Hoff Shows nineteen oh
four, the when we shot back at build with Mark Brown. We were talking about COSMOSDB and its role in large language models like chat, GPT and others, you know the role of AI and Cosmos DUB in general, and our friend Brent vander Mead had this comment. He said, oh yeah, thanks for the post traumatic event. At minute thirty two, Carl talks about trying to put together training on artificial intelligence and be patient for the tools coming out. Oh yeah, And Carl says, you know, this stuff is
changing so fast, this is not the time. We just got to wait, wait it out and see what the major tools shake out like. And I said, it only gets easier. Carl said, you could imagine going into training class like right before Windows came out, and someone teaching you to write your own windowing system. YEP, exactly what I was thinking. And Brent, that was my point. That's why I said we shouldn't be doing
this, just hang on. Yeah, and Brent comes back with I immediately was thrown back into setting up a three t your team foundations on premises server cluster. If only the rapid changes were occurring back then, and how people were willing to suffer setting up Jenkins pipelines for automation into AWS. Oh man, am I happy with aks Helm terraform as your DevOps and get hub actions.
Thanks for the great show. I mean it's a great example. Think about how many shows we did talking about continuous integration back in the day, sort of building your own tool set every Now this bespoke pipeline and today it's set of services right, you turn them on right and you've got it set up even as I AD as infrastructure as code, so you're able to just really run a script. Is like players my pipeline, let's go. We set up CI for many people in many companies back in the day. Yeah,
it was a whole business. You know, you're right the folks that would specialize in just getting that right for you so you didn't have to figure it out. I pretty sure you got a copy of music coe by, but we'll hook you up with something cool. Give me a ring at Richard Aguapp dot com. Thanks you so much for your comment and a copy of music Cobe is on its way to you, and if you'd like a copy of music, co buy I read a comment on the website at donnat Rocks
dot com or on the facebooks. We publish every show there, and if you comment there and I read on the show, we'll send your copy of music go by very good. And you know you can always follow us on ex Twitter as I call it. But we've been there for many years. But the cool kids are hanging out. I'm masked on I'm at Carl Franklin at tech Hub dot social, and I'm Ridge Campbell at mass it. I'm dot social. Send us a two Rudy two two Rudy two dy fresh and
fruity and uh weird. The last time vishwas Lele was on. We had such a great conversation and I was under the impression at no. I said to him, you were on just like a month or two ago. It was nine months ago. Yeah, yeah, I don't feel nine months older. But obviously I am been pretty crazy. And first show is all the way back in two thousand and seven. I like he's been Oh yeah, he's an og pretty sure we owe my sub sandwich at this point or something.
Yeah, so his title has changed. He is now co founder in CEO at pwin dot Ai. He's also a noted industry speaker and author, and the regional director for Washington DC. Welcome back, vishwas, thank you both. Glad to be back a new job. Dude, what are you doing. You've been AIS CTO for ten fifteen years? Yeah, almost sixteen years. I've worked at AIS for almost thirty years, So quite a big change here since the dast But it does seem to me like you're not alone
either. Some really great tech people I know are jumping into AI startups like it's the hip thing to do right now. It's such a massive opportunity. Yeah, so, Richard, I was not going to be started on this idea. I was not envisioning. If you asked me like nine months ago,
what was your plan. The plan was whenever a new technology like Azure, open ai or open ai comes along, We've always had this culture of let's can we find a few people jump into it, try to understand it, try to build something so that we can understand it better and take it to our customers. I mean, that's what we're planning to do. And with Azure open Ai, we went through a bunch of use cases and then said, hey, can we work on a use case which is can help
our internal folks. We have a pretty sizable proposal team, so it can we build something that can help you. So we build something, and then we took it out to other people and one thing led to another and here we are so really good because you were making things that people wanted. It's like this is actually a product and we should do more of it. Yeah. So what we did, Richard, was first we build an m VP for our internal teams and got some feedback from them, some good, some
bad, all of it. And then we said, okay, let's do this. Can we before we build any more functionality into this, can we go to eight or ten companies? Ended up going to a dozen or so companies and said, would you allow us to generate some content for you and then we'll talk about this later. But this is a co pilot for proposal writers. So we went to a bunch of companies and said, can we generate the first draft of a proposal for you? Maybe eight or ten companies.
It's going to be called the Aesthetic of Economico Social Disposition. You're going to love it, so sorry, that's great, dash waffle waffle false. So we we said, okay, we took it to eight or ten companies and they got some feedback. And what I learned from it, Richard, to your point, A lot of people are jumping in, and as they should. This is definitely a disruptive technology. Can I can feel it from early on in the share Point days you remember two thousand and seven show Getting
into share Point, or remember the cloud show early days. This seems like much much bigger than all of those things combined. And there's this idea that you hear technology that can reason on your behalf. Forget the content generation pieces right, forget that can summarize text for you, but the fact that it can reason. Given some things, it can reason and give you an output back. Think about it as applicability all across what we do in it.
But we took one use case and maybe in hindsight, if I think about it, you can do a lot bit generative AI. But the generative AI models are hard to sort of control their good But how do you get them to focus in and generate one twenty thirty forty page document and that has an ROI associated with that because you know come, sure are time consuming to build?
Yes, they are time consisting to build. So I think, in hindsight, picking one use case which aligned well with the superpower of generative AI, which is to generate content, but then to solve a problem where people were spending struggling writing this content and then generating the first draft. And it is important to note that it is a co pilot very much using the term copilot. Yeah, so you're still the pilot. It's your fault. It's
still your fault. Yes, we're just getting you to this first draft. And later on in the show, I can talk about as a CTO, as an architect, I have supported proposal writing for at least two decades, right, and I can talk about all of the trouble and trial and tabulation supporting that, and then how that product helps with all of those challenges that the organizations face. So let us go off. I mean, I'm all about the summarization only existing content, like I get that from a large language
model. But you talked about reasoning. Yes, where is the reasoning needed and how does the tool do? Yeah, so the reasoning is needed because let's just literature Defied's okay, I'll take a step back and sort of describe the problem and then we can talk about how we solved it. Right. So, I remember nine months ago as you were ending that show. At that time M three sixty five Copilot had not come out or it was a yeah, it was being talked about. I think it available in a limited
review or something like that. And Richard, I remember vividly that you asked a question that if you're generating the content based on your past responses, right, and let's say you've submitted twenty thirty forty RFP submissions in the past, you stored them into a SharePoint site. Perhaps now you're M three sixty if I have co pilot has now scanned those documents, and you're now able to
go and say, hey, generate me an answer to this question. And you at that point said, why would I not use M three sixty five co pilot for that? Right? Why do you need this special purpose co pilot to do what N three sixty five copilot seems to be doing out of
the box That that was a question proposed to do. Yeah, nobody really knew at that point, right, And the answer to that question is that M three sixty five Copilot is a very good general purpose chat You give it tons and tons of content, could be any content in your share point one drive teams, what have you. And now you can use the graph connector to pull in content from anywhere else you want to putting data from Salesforce, you can do that. But the fact is that it is a general purpose
chat bot where the content is broken up. And let's say there's some sort of a chunking strategy. You chunk it by the page numbers, or you chunked it by the discussion, or you chunk it by sections. You chunk it some way. You store it in a vector database, and there are lots and lots of them available. But then you are asking a question that Hey, based on the information you have, can you answer this question right?
And it goes about answering that question. I keep in mind that chat bought including chat gpt our System one chat butt And what do I mean by system one. I'm really talking about the Daniel famous book from Daniel Canman System one system I'm exactly I'm talking about, right, and it is can you elaborate on that visual lot of it? Yeah, So there's a famous book by by Nobel Laureate Daniel Canman called Thinking Fast and Slow. Highly recommend folks
read it very good. Book really talked about two things. And this book came out many many years ago, well before CHATGBT was a thing. But in this book they talk about system one and system to system one. Think of that as the way your brain works that allows you to avoid a ditch. Right, You don't have to think too much, You instinctively take a reaction. System two is when you're thinking about reptile brain exactly. So chat
GPT and other chatbots today are system one. Right, you can go up to chat GPT and say, I'm going to ask you a question, but this is a really detailed question. I want you to think about it for thirty minutes and then only respond. There's no way to do that. It starts responding in three seconds or whatever the latency is. And so when we talk about a special purpose chatbot in this case, in the case of proposals, you really have to think about the proposal domain. Think about the important
entities in the proposal domain. As you look through past documents, you need to collate them in a certain entities which are very domain specific. And then generating a response to a question. There are some best practices that are available out there in terms of how do you write persuasive content right and how do you reason about So these questions can be long form questions, open ended questions in RFPs, like we have this problem about modernization and this system needs to
be modernized, and these are the challenges we are running into. It could be anything, right, I'm just giving you an IT example, could be a healthcare example, could be something else. So these are questions that you have to respond to. Their very broad based questions. But then there are best practices in terms of how you write persuasive text in responding to that question. And that's where the reasoning comes in. Where our our domain specific copilot
understands some of these best practices. When it ingest the content, it automatically organizes them based on the entities things like that I can I can make an analogy here if you go to chat GPT to ask it for a problem and how to configure this middleware or whatever. Right, you're you're taking a chance, whether it's going to assume that you're using startup CS or program cs right a dot net six and before version of asp net core, or you know,
or the new, new or modern stuff. And if it understood this, there would be another layer before it started spitting this stuff out that said, oh, we have this contact that says it should know we were using dot net eight and we're using the latest version of see sharp, and we want to you know, frame the answer in that context. Right. So that's setting up a system prompt and making sure that chat, GPT or any language model has the right context so that it can give you an answer that
is very much aligned with your background and your needs. I'm also talking about KYLA. I'm also talking about a scenario where in some cases you're trying to answer a question. So let's just take that t T example, and I think that'll appeal to the listeners here as well. So there's a question about somebody, some government agency or some enterprise issued in RFP with a question about modernization of some data platform, right, and they talked about, hey,
this platform needs to be modernized. How do you go about have you done this kind of work before? How do you go about solving this problem? Again, I'm over simplifying this, but let's say that's starf right. So in this case, we would have to go find out if you have done this kind of work as an organization, right, and then we will have
to search through your past RFP responses, your core competencies. Maybe you've written some white papers, you've written blogs, you have transcriptions of podcasts and things like that. We go look through that, and that's a common RAG pattern that I'm talking about that people are familiar with. But we have to go find a couple of examples, or maybe three examples of where we have done this kind of data modernization. So you get that text which says, well,
these three examples match somewhat the requirement of the RFP. But then the reasoning comes in, how good a match is this? Right? Should we use this as an example? If it is not a good match, should we be changing our prompt and looking for a different kind of an answer. So all of that is happening dynamically in run time, and we are not asking our users to write any of the proms, right. We are essentially generating all of the prompts for them based on the best practices that I talked
about. Right, And there's an interesting inverted ratio that we have learned about. So if we're generating a twenty page document, in many cases we end up generating two hundred pages. That's the inverted ratio of proms, which includes all of the prompts and the context that you're feeding into the language model. I mean, that's not that weird when it comes to a good search string. Even like when you focus in on a specific thing, you need to
know you have to be very descriptive to get the answer. That's right. That's a good analogy. Yeah, I mean, I think it's useful for folks t als to understand. You're in Washington, DC, like you guys are in the business of writing RFPs for government, which tend to be lengthy and have very specific rules, like it's not a trivial thing, it is not a real thing. There is a multi hundred page document called the Federal
Acquisition Regulation which talks about all of the details of an RFP. What would go in section M, what would go in section L, Section C, things like that, so the rules around that. And I mean, I think a language mode would be good at given this rule set and this goal and these capabilities, write me the paragraphs right, and what we are finding is as good as these language models are. Just like human beings, if
you tell them twelve things to do. Yeah, they will do the first second, and then we will do the eleventh and twelfth and forget everything everything in between. That's so, is that interesting? It's very interesting, But it does sound like the in building this RFP generator, you literally want to go section to section and write a separate prompt for each one. That's effectively.
Again, because the rule set's so consistent, you should be able to generate almost all of this, right, right, So let me actually take you back into the dane in life of a technical architect or an engineer like me who is supporting RFPs, and then we'll come back to this point. So I'll try to be very quick here. So let's say an RFP arrives. It's a fifty sixty or one hundred page PDF which is montrivial to read. Somebody has to read it. This is the request for you to propose
to them. This is the request. Your proposal will be larger, although in some cases they want you to write it. They have very strict page limits into how much you can write, right, don't know. They can send you as many pages as they want, but you can only send them. You can only and then you know, all the tricks about you know, creating really dense diagrams to sort of save on words comes into play.
But let's just say you've got a fifty page complicated RFP. Now some human being has to read that and understand what is happening in this RFP, and then let's say just spend a day or two to figure out. And then the organize is a meeting where they invite different people. There's a proposal manager, there's a technical sme there is account team. Right, there is no point in responding to an RFP if you don't know anything about that customer.
That is a general rule. That's not just in government. That's a general rule. Right. Why waste you know, there's no price for the second and third and fourth place based on your response, So you have to read that. You organize a meeting and they present what is requested in the RFP, and then somebody like me, an engineer, would join that meeting and say, yes, we can do this. This is SharePoint related, this is records management. Oh, we will use this tool and we will architect
these pieces. And somebody will say fine, you go off, and the other architects go off and start writing the first draft. Now I don't want to write anything because I'm already busy with my day job. And they give me two weeks. I go off and start writing, and turns out that I don't know about other my colleagues. I'm not a good writer. So two weeks go by, and then we're expected to show back, come back to this meeting with what we have written. And that's when the yelling begins,
because you know they don't like what we have written. Everybody has written it differently, and you know, of course I am over emphasizing on all of the projects that I've worked on, and the same is true with my colleagues, and there's a rich set of experienced library, but I'm only focused on the ones I've worked on, right, because that's what I know best. So now two weeks have gone by, and really a draft is there, which is like poor quality draft, which reads like ten people have written
it. And now you've only two more weeks left to the final submission. Right. And it is often said that proposals are one either at the beginning or then. What they mean is either at the beginning, you understand that you're going to win, you put the right team in place, or at the end, if you have enough time left, you want to take a step back and think about what should be a creative pricing strategy. Who else would be bidding this right? How can I ghost my competition? If you
do those things, your chances of winning goes up. But the sad reality of proposal writing is you spend most of the time in the middle, chasing
the first draft, chasing the second draft. So the value proposition for the tool, and especially designed for people like me, you have taken all of your past performances, stored them in a library, and then when the RFP arrives, our engine, peven engine, will take a look at the RFP, parse it down into what is exactly being asked, and then rather than having to write ten pages of pros and return it, you're really dealing at a higher level. You're providing hints to our model. We call it flight
plan and very creative. Right if you're calling ourselves copilot, a flight planner seems to be logical. So there, so you set up a flight plan, which is really and this is Richard and cal This is where I think this is a very key concept in my mind. Again, looking back, it seems trivial, but this is one of the concepts that we landed on earlier on that has served as well. So if you think of calling yourself a co pilot, which means we are not here to replace the proposal writers.
You know, these people have gained experience and skill or many many years. We are here to augment them. And how do they work with the model without becoming a prompt experts themselves or without becoming experts in a RAG pattern. What we ask them to do is we ask them to fill out a flight plan, essentially tell us in great how should we be responding. So these are the solutions we want to incorporate. Here are the win themes.
We think we are differentiated because we are the best Microsoft partner in the world. What are the pain points? If you really know the customer, you should know their motivation for issuing this RFP, So you provide that vision in the form of a flight plan, and then our engine will take that flight plan and then generate these prompts dynamically as we talked about, and then generate a twenty thirty forty page response. And why twenty thirty forty pages, because
that's what's been asked, And that was the other innovation. If you go to any other co pilot, you'll end up getting one or two pages. Even chat GPT, if we ask a complex question, it will say, hey, this is a complex question. Let me just tell you how you can get to this answer. In our case, we can't tell our users how you can get to an answer. They want to see twenty pages. So we have two. Like I was saying earlier, these language models,
as good they are, they tend to forget things in the middle. So it's our job to take two requirements, give it to the language model, get them to generate something, and evaluate it at the end of that and see if they really answered the question. If not, send it back again, send it back again. Now you have section one, you do that for section two, section three instead of you do that repeatedly. And that's why people ask me, why is it taking you guys four or five hours
to generate a twenty page document? And when I ask a question of chat GPT, it gives me an answer in seconds. Why is it taking you four or five hours? Well, the answer is that we're having to iterate over the content that's being developed, and at every step of the way we are evaluating it for the sake of completeness. And by the way, we also do hallucination checks as the content is being generated, right, you have
to without that human to check things out. Customers can't just expect you to plug stuff in and copy and paste out of a chat bot, you know, right, right, So I agree hallucination checks up. That's why very important. And you know, going back to my example, Carl, the RFP is asking for companies who have done a five thousand or fifty fifty thousand notes sorry, migration to the cloud. And that's what the and now the
text that was generated says you have done five thousand note migration. Yeah, we can't find a reference to a five thousand note migration in your entire past history. Say, hey, you may want to go double check this. Well, how effective has it been in generating proposals? So well, you should ask our customers that. But one, well sounds like they're happy but happy with it. Yeah, I think I'll lot more work needs to be
done, obviously. Language models building, Richard, Building a production grade application with language models is hard because they have their own personality. When Microsoft goes from four to four turbo to four omni. There's some personality aspect of these language models can change, and we have to account for that, especially if you're generating a multi layer prompt, you have to account for that. So take some additional time. But so far we have had customers who said that
we've accelerated their timelines, given them more time. We said, you reached out to a dozen customers, were they all pretty happy with what they got? So I didn't complete the story. We built an MVP. We went to a dozen customers and said, we just want you to try our product. We didn't even have a UI at that point, right, And because what's the point of the UI if the content is not of the quality where
it is helping you with saving time and improving the outcomes. Right, So we went to them, got some really good feedback, added more capabilities, got the UI, and of course our UI is SharePoint because it turns out that eighty to ninety percent of proposal writers live in share points. So why should why should we build something outside an environment that they're already Why blow against the wind? Yeah? So then, Richard, what we did was back
in February or March. This has been a very fast moving thing lest in nine months, we said, okay, we're going to offer this in a commercial manner as a paid service, and we went to many of those original data customers. Many of them signed up, so we were at this point running several production customers who are paying customers of ours. And I'm really proud
to say that Microsoft's own proposal Center of Excellence. Microsoft has this group called the Global Proposal Center of Excellence. We went through an evaluation with them and they just onboarded as wow the tool. Wow, congratulations, congrats. Yeah, that's great, and gentlemen, we should pause for one moment for these very important messages. Now we're back. It's not that Rocks. I'm Richard Campbell. That's Carl Franklin. Hey here with our friend vishchwas Lela now the
lead on this new Pewin company. And obviously you know your MVP has gone
well, you've done some initial testing. You know. I'll throw my sissedmin hat on because I've talked to a lot of CTO CIO types who are looking at these technologies and they keep being told by Microsoft, you should have your quote data estate in order, like what I'm afraid of with the generalized tools like this, is that your data state is not in order, and so data is going to be pulled into the model and then subsequently revealed in prompts
that everyone's going to be very unhappy about. And there is no it's not like a sign pop out of a server somewhere when your data state is in order. You don't know, it's like your system should be secure. It's just as impossible. Like, one of the things that appeals to me about your approach to this is you're encapsulating the data set and arguably you're taking that responsibility saying where the folks who will make sure that the data that appears in
the model is appropriate. I mean, I'm wondering. I'm just wondering. How much of work is that for you? Yeah, that's a huge amount of work. If I had to look back, we've spent thousands of hours data science hours in building the multi layer prompt engine. I would say an equal amount of time in worrying about ingestion of the data. And now, fortunately for us, this is why a domain specific copilot made sense, because
you know you're limiting the data set, right. Some of the things that you're talking about is you might have hundreds of thousands of documents in SharePoint and stored somewhere on one drive, and that one document which nobody is found until now. You can be sure that M three sixty five copilet would find it. And somebody asks the question, that question will get answered inappropriately because that
document was available, but nobody knew about that document. In our case, it helped that we limit the kind of data that goes into the engine. Right And as I mentioned, your past RFP submissions, your white papers, and things like SPARS, which is a government tissued document which says you've worked
on these technologies well structured. So we limit ourselves to those documents. We do things like many of these proposals include resumes of people, and we exclude those resumes because now you're experienced from a previous employer might pollute your current employer's data, so we exclude that. So as part of the ingestion Richard, we break the documents down into logical sections. We use the language models for some classification. Even so we don't have human taggers to say this is a
management plan, this is an executive somebody. We're using language models to do some tagging of the documents themselves, and I'm seeing this now inside of perview Microsoft's and again this IT conversation where it's getting really good at calling out PII. Even before you got around a tagging. It knows the shape of identifiable
information. So starting with a limited data set helped us. And you know that may not be applicable to other people here, and frankly, that's the challenge that Microsoft had with m TH sixty five copilot, because you have everything, and of course they've added capabilities like hey, don't include these documents in my index or leave this library out right. So obviously those governance controls have come into play. In our case, we started a small data set,
but even then it was really important to parse it. I'll give you an example. You might partner with a company when you're applying for a certain piece of work. That company may end up teaming with a competitor reviewers in a subsequent RFP. Now if you've taken their content included into your knowledge base, chances are that that content will show up, so you have to be careful about that in terms of how you're using maybe partner's data. So all of
those things, Richard apply and we're breaking that content down again. I talked about entities and then we use those for our content generation. I got a question, did you, and you may have said this early on, I didn't hear it, but we're using Microsoft copilot Studio to create your're We're not losing max of copallet studio. Have you used it, I've used it,
But this we started our development well before copiallet Studio. And secondly, our response engine at this point is so complicated that we needed to go directly build things in Python and asure machine learning notebooks and things like that. Gotcha. I mean what I like about this is you clearly brought your own meta understanding of RFP writing to the tool set that's right to then make into a product that works well for individual companies against their data. But you know, there's
three pieces here. There is the language model as it stands. There is the understanding of the rules of writing RFPs effectively and applying those. And then there is the data set that is the companies for what their capabilities are and how they want to respond to RFPs like those. You need all of those things. It's not there's no magic wand here. Smart people had to work
really hard to make this work. But when it works, the fact that you're going to be able to respond to more RFPs and to write better responses like this is money in the bank if done why if done right? And I should add one other thing which I should have mentioned earlier. I believe that in order to develop any effective AI solution, you need to have two good mL engineering. Of course that's a given, but then you have to
bring domain expertise. And you alluded to that that we brought some domain expertise, yes, but we are not experts in proposal writing right. So what we did Richard early on is we went to a company called Shipley Shipley and you may not be familiar with them if you're not in the proposal space. But Steven Shipley fifty years ago wrote the book on proposal writing right, and really an influential book in terms of how should you be thinking about your RFP
responses? And that translated into a company. People can go to Shipley wins dot com fifty year old company and their job it is to teach classes about better proposal writing, teach nuances of writing compliant proposals and when you need a surge capacity, they make their consultants available. Wow, okay, yeah, so they literally the business around it. That's a business around it. So we went to them and we said, you know, we are a tech
company. We of course write our own proposals, but we are nowhere near the subject matter expertise and the name that you have, and we would like to partner you to be co creators of this tool for us. Right and Shipley had never partnered with a company, and I was clear that I'm not just looking for a referral partnership that hey recommend us. I want you to be joining the team so that you're constantly helping us improve the prompts. You're
bringing that domain expertise. And they went through their own evaluation, Richard, for a period of time, they looked at other tools, and very proud to say that they decided to join in as an exclusive partner, as a conciliation partner. Right and having them on our engineering scrums and saying, generate this content in this manner, because what is very interesting about proposal writing is if you don't write it in a certain manner right there, you are restating
the problem. You're saying you've done these things in the past, You're reinforcing the point. There's a certain format, and proposal writers tend to be very personniketty about that, right, if you don't follow that format wrong or all that's are off right, Yeah, and to the three of us, if you read one page of highly curated proposal writing ready stuff and then some other stuff, you would say, hey, reasonable people can argue about it.
Oh yeah, your sentence is here, your reference is here. But they don't like that, right, They want the format to be exactly, So how do you write content that is persuasive, that follows the structure. And having a domain expert like them, so so fortunate to have them. They've taught people the process, the methodology, the best practice is I'm betting they're testing like that. They're evaluating the output. Absolutely did a show on run
as with Lyn Langett. Now that logo we call the hard part of machine learning. It's like test test, test, Like you've got to evaluate these outputs, right, and it takes experts to even know what's good and what's bad. It takes experts and you cannot use just chat GiB to say which section is better, right because it requires experience, skill, all of that. So they're helping us with testing, they're helping us write this. So
that was an important partnership. And that's maybe one tip to the list us out there that in order for an YI solution to be effective, you can't just have five spart engineers go to any domain and say I have a solution for your problem because I know as are mL and Python better than you. Well, that does not matter. Now you'd know my problem set. Well, there's plenty of smart developers that have built quality software that was wrong because
they didn't have the domain experts involved to make the right thing. This is the same roles. You sit a different place in some respects. You're not qualified to evaluate this output. You need an expert to evaluate this out.
So having that, I was just finishing that other point that you succinctly made, which is it's really important to have a domain expertise, And I was just adding that our expertise was not enough alone and we needed to partner with the best in the business to do that, and we're fortunate to have them. I mean, I've got to think there's somebody who's made their career to writing great RFPs. This is still pretty threatening like that, at least but
admittedly you presented the right way. It's like it's only going to give you a draft. We're going to need your help right every time, at least to still be read. Yeah, Richard's that's a good point. And at some level we are all concerned about the socizeful impact of generative AI and all of the things that we do. But the way I presented two important points. One is, remember the notion of flight plan that I talked about.
An expert proposal writer can quickly fill out the flight plan because think about that, you're setting the vision, or you're setting the intellectual momentum for you to win. Without a vision, nobody is winning any RFPs, right, And this is where the proposal directors or proposal writers come in and they fill out the flight plan. And we have been talking to them about how their job
changes with generative AI, where they are operating at the object layer. We talk about them, We talk to them about objects, and then we are dealing at the work words level. Right, Yeah, they're changing the objects and attributes and methods maybe, but then we are generating all the code for them. In other words, that we are generating all the words for them. So that's one, and it is a model driven architecture in the sense that you change the flight plan, you look at the content and you feel
like, hey, this came out pretty flat. Let me go back and change the flight plan and regenerate and regenerate and regenerate. Right, So it's a model driven architecture and the better they get it with that setup, they can make these quick changes. That's one. The second important point is we are not claiming authorship. Right. I get this request from some people that hey, can we use your tool? Our knowledge repository is pretty sparse.
We have not done this type of a work. Can you help us generate good content? And well, this is not cheating as a service. You can just push up button and you get a proposal response. Right. If you don't, if you don't have good past performances, if you don't have good background, if you don't have creative architects who can come up with good creative solutions to the problem, chances are that what we generate is going to
look very flat to you. So we're not claiming authorship. Those things have to be in place for an organization to be able to use over tool. So do you do you ask your customers. When you give a proposal, do you mind if we use this as and feed it back into the machine, so as as training data. So cal one important thing to note is we are not training the models. And this is a question that I get
asked often. We're not training the model at all, because it's not like your data is going to be trained and then and then use for someone else. But you do you do use those as? Yes, we use them that we use them in a rag manner. Where right? And so I was just about to say this is retrieve all mana generation. This model is
the model, I would say, you still have to ask permission. Yeah, So, so just to be clear, if somebody acquires our tool, they go through an onboarding stage where we look at the documents and we tell them some best practices and then we ingest those documents. Of course, that's their asset and that data belongs to them. And whenever engine runs, it dynamically makes a call into this database to look for right pieces of information.
So of course we are using their data, but then we dynamically call into this engine. If there is no results to be returned because they have not done this kind of work, then don't have any white papers and that technology, then it's going to return pretty flat results and we can't infuse that content back into language Models and c rate the content appropriate. Okay, soa vishwas what's on your to do list? What's next? Well, the next is
to continue to improve the product. That's what I'm focused on for the foreseeable future. Take advantage of all of the amazing enhancements that are coming to the Language Model space. What I love about this is this allows me to tap into all of the Azure work that I've done in the last ten years. Because ultimately Language Model is just one service. There are thirty eight other Azure services that are powering this solution. So a lot of things that we've learned
over the last ten years with the community are coming into play. So continue to see how far we can take this, how we can support our customers better. So that's what next for me. Fantastic. Hey, it's always great to talk to you. We always learned something, we always get a new perspective on AI and when we talk to you, and thanks, thank you, all right, and we'll talk to you the next time at dot
NetBox. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot com. Visit our website at d O T N E T R O c k S dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September two thousand and two. And make sure you check out our spot
sponsors. They keep us in business. Now, go write some code. See you next time. You got jad middle vans down there that means hard than my taxes are