How'd you like to listen to dot NetRocks with no ads? Easy? Become a patron for just five dollars a month. You get access to a private RSS feed where all the shows have no ads. Twenty dollars a month, we'll get you that and a special dot NetRocks patron mug. Sign up now at Patreon dot dot NetRocks dot com. Welcome back to dot net rocks. I'm Carl Franklin and I'm Richard Campbell, and we are in our respective places on opposite sides of the North American continent, absolutely as we usually are.
But why don't know you caught us in the middle of three or four conferences here listener. Yeah, well, in the last few shows and get in time shifting being hilarious. We're all shows. We did it in Porto, and some awesome conversations. Really neat folks, folks who never talked to before. I always tend to when we're going to do in person shows, I try to go after guests we've never had on before. Right, Yeah,
new stuff, all good, I would point out, I am. You know, it's getting towards the end of the year, so you know what I'm working on the end of the year, geek out. Yeah, yeah, I know it's close to the end of the year when I have to open up the script my script files and start pulling stuff forward for space and for energy, and I can't wait. There's been so much, it's so much, so much good stuff. They're both they're both going to be I'm
just committing now. They're going to be long. I mean, you could just do a whole one on the JWT right, oh, JWST and the crisis in cosmology, like the current proposal they're floating around. Maybe we're long about the length of the age of the universe. Maybe it's twice as long as we thought. It's twenty six billion years instead of thirteen billion, wow, which just is stunning. I mean, I love the fact that this is why that instrument went up there, right, it was to test all
of the assumptions we've been making and it has delivered. Like do we talk to Amber Strong about that, like this was one of the site and she's just like, Okay, there are things we're finding that cannot be there. There's something we don't know and it's a big something. And Vishwaz is looking at us like did I just parachute in here from another planet, land on the right show anyway, don't get me started. Oh wait, you did, I did. I'm sorry you late. I can't wait. I can't
wait, I can't wait. So uh yeah, well let's get started with better no framework awesome? All right. Well I've talked about this show I've done with Brian McKay, the a iBOT Show before we had them on. Yes, yeah, and we have just started getting into code and using c sharp with the API to do you know, the chaining, you know,
get the results of one prompt chain it into another. But we attempted to do one last week and by last week, I mean you know a few weeks ago, I think it was the twenty fifth of October, we recorded with a guest and for some reason Riverside FM, which is what we record with, decided to completely trash the are not finish uploading the screen share on top of that, so we're going to record it again. That's that's the thing. So the episode eleven, watch for that because it's going to be
amazing. We're basically trying to break the or push GPT four V image recognition to its limits. Oh wow, that's what we're trying to do. But in the course of doing that, Brian actually got some results and we're going to rerecord it. So I don't know what's going to happen the second time of the first time. First time he got results that he didn't expect, and he tried these same prompts with the same images and got completely different responses.
And it turned out either somebody's watching or it's just learning, Like, no, that wasn't a good response, Like there might be something else looking at the responses and tuning the algorithm, like in a matter of twenty four hours. Well, there's also another possibility, which is that generate of AI models have a variability that allow for a range of answers. Yeah, and I kind of got into it with him and his guest as well. You know, the whole idea of do these large language models reason, And my
stance right now is no, they don't. And you know, when presented with the facts, this is what they do. They take this, they take this. How is that not reasoning? My position is well, because it gives you inconsistent answers. Well, more partly it's just a statistical model, right, Like, there's no reason to be had. It's not weighing
any factors at all. Gives a statistical result, right and and you know, but if you want to just look at it empirically, when you ask somebody what's one plus one and on Tuesday they say two and on Thursday they say nine, Well you asked the same question over and over again, it'll give you different answers like yeah, that's it. Yeah, and so there therefore you can't count on it. It's not reliable and it's not what you think it is. You don't ask You don't ask a sarcastic parrot for facts.
That's not what it's meant for. I'm sure Vishwas has lots of things to say on this, but anyway, tune into the Aibot show dot com. Uh. Starting at about episode nine, I think is when we started using code. But in episode eleven is where we start really pushing the boundaries with GPT four v Love it, don't learn it, Love it Cool.
Boo's Talkers Today, Richard grabbed a comment of a show eighteen twenty nine when we did just earlier this year with our friend Vishwaz when we were talking about fusion development, which apparently is a term that's now fallen out of favor. But this is about the idea that the power platform presents an awesome way to build heetrically need as clients against cloud back ends, and that there's a role for traditional net developers in there and building some of those back end services and
even building client side components. And that was the idea of Fusion, that we can all work together. I don't know how else caught on with folks, but not that hip. But we did get a comment off that show from Jim, who I believes are regular He said, every time I hear someone talk about how easy it is to build with power apps for a power user, I shake my head and I have to think to myself that we've
lost something. In the past twenty years. I went from it support dude with no real software development training to a software development leader, in large part because of visual Basic for applications, which made it easy to learn to program. We didn't need a solution that put hard to maintain Excel formulas into web app. We had a problem that needed to get solved and get back quickly
to developing more apps. We need tools that get us to the equivalent of the late nineties and early two thousand desktops development experience, not things that try and mimic access. I mean, it's one click to create a project that I could start dropping UI elements into and then writing clean cut with a minimum of ceremony, not hiding logic behind textbox dates out of textboxes. And I'm going to turn off my rant before it turns into a blog post now.
I mean, Jim, I would say you're misremembering some stuff too, like first thing that I came to appreciate. I said this the other day on another show, where I said, I think wind Forms is the exception. It's the only product that ever had a great designer. Every other attempt to make a great designer look at you, front Page or dream Weaver, much less WPF. Every other attempt to make a great designer has fallen on hard times. Only wind Forms had a great designer. There is a reason for
that. The reason is wind Forms is a pixel based grid, and all of these other things are dynamic in don't use yeah, and that's why can't You can't have a visual designer when you're not pixel based. Well, and the downside to the pixel based is then you go to higher risk monitors and oops, or you want to work at mobile oops, or you want to scale. Yeah, all of those problems are introduced, and so that's why we have this one draggy, droppy designer that people fell in love with back
in the day. Yeah, but the idea that that vb ever built clean code of any kind and with minimum ceremony, Like it was good at getting stuff done, but boy it was. You had to practice carefully to make maintainable projects. Yes you did. It's a bit of a nostalgic effect to think all things were so much simpler them. Well they were because we had VGA displays, one type of machine to deal with, and you know, it more or less work that you were writing code on the same kind of
computer you were going to run it on. Like life was simpler in that respect. You know. I had a flashback to the nineties when I had my little kids and we used to play DVD games right on the computers of the day. Dug out some of those DVDs, you know, like played them, brought them up, and I was thinking, oh, this is going to be great. You know, we get to do these cartoony mathey things, right. Yeah, it's like the size of a postage stamp on
my screen. Your screens just a little bit higher resolution now, yeah, and you blow it up and it does not look good, you know. For the for the history of dot net, I have all of those great slides of the conferences from two thousand, like the original pds A tech ed when dot net was first announced, and like Mixo six and so forth. You know, I had to get all of those graphics remade because they were like three twenty two eighty yeah, six hundred like back then, screens weren't
the same resolution, and they they looked terrible. Tretty tramble on eighty ten eighty. So I literally sent copies of them to designers, said remake these as ten eighty for me. So I have the only ones. I have the only good looking mix O six graphic, Like, it's me. I made it, it's mine. That's very cool. Anyway, Jim, thank you so much for your comment and a copy of music Coobu. It's on
its way to you. And if you'd like a copy of Musiccobeu, I write a comment on the website at dot net rocks dot com or on the facebooks we publish every show there and if you comment there and I read it on the show, we'll send you copy of us to go by. And you know, you can follow us on Twitter if you like to, or X or whatever the hell it is today. It's going to be Z tomorrow, I'm sure. But yeah, we're still there, but the cool kids
are hanging out. I'm mastadon from from my perspective anyway, I'm at Carl Franklin at tech hubs Social, and I'm Rich Campbell at Macedon dot social. Yeah, send us a two. Maybe you'll get a copy of music to go buy. There you go. It's another way that you can leave a comment. And that brings us to our good friend Vishwas. Vishwa Leiley has been on the show some thirteen or fourteen times. He is an Azure an AI expert. His bio is long, but I won't be labored that here.
Uh he's a real great guy as well. Welcome back, Vishwas. Thank you what you've been up to, what's doing well? Doing the Like everybody else, got started with chagpt open ai, Azure open Ei. Specifically for asure developers like me. It was an easy sort of way to get into generative AI I've been working on some aim al stuff previously, but generative AI, of course has changed everything that we knew before. Now you're in watching BC I think. I think a lot of your customers or governments like,
aren't they pretty careful about adopting new technologies. I'm always impressed that you're in these things so early on. Yeah. So, yes, Richard, that's true. But I do work with government and commercial customers both. You know, the the idea is that can we can we try out some solutions in the commercial world bring those some of those solutions to the government community. So that's that's been the pattern. But you'll be surprised the government customers.
And you know, Microsoft has Azure Open the Eye service that was fed ramped, right, so it has available in a certain secure setting. That was an announcement that came a few months ago. So there are civilian agencies who have lots of needs for automation of processes and processing of documents, and they're definitely looking at generative AI models to be able to expedite some of that work.
Well, I know the government is currently sharpening its clause to to put some restrictions on AI even sort of be fuddled about this whole thing from the very beginning, and I'm not so sure how well they understand it. But clearly there's some legislation needs to happen, especially around copyright protection and that kind of stuff licensing. But what are your thoughts on that. Have you heard
anything being in DC about what they've planned to do? Well, you know, like you, there's been news that White House is preparing for a major announcement. I don't know if it's a good thing or a bad thing for You're right, I think we do need some guidance there, but I always worry about what kind of guidance and will impede the amazing amount of overall. I mean, they are negatives and there are positives, like everything else.
I do think that this is just an amazing development that we will look back five years from now and say, oh, something new started in twenty twenty three. So I'm hoping that the legislation that they're coming up with, I'm hoping with the industry leaders is a responsible step but does not curtail the progress and the amazing possibilities that we see from this technology. Yeah, when I think about the logical roles, it's like it would be good to have some
privacy legislation. You've been needing that for ages, you know, large language models notwithstanding. But to me, the one area that you want from a public perspective care about is source of data full stop. If you're going to put make an LLM available to the public, even for a fee, you have to tell the where you get your data from. That's an easy one, and it's and you know, there's nothing profound there. It's not a
secret. Presumably, it certainly deals with it starts to tackle the cocky right issue, but it's just like, hey, you know, you want to compete, make distinctions between different lms, talk about source of data, right, So, Richard, I think source of data. And there's been a debate about you know, what do open source models mean? Right? Is it just publishing the weights of your model, Does it make it open source? Or does announcing to the world what sources that your models were trained on.
Yeah, that's the definition of open source. So you're absolutely right. These companies have to at some point tell us what are the source of data. But that's different from citations. Once the response is being generated at a sentence level, that's that's a very hard problem to do to spid well. And that's what you really need, right, I mean, nobody wants a you know, fifty terrorbyte list of sources, general sources that it's been trained on. But what I want to know is when it tells me something,
where did you get that fact? You know? But as you said, it's very hard because it's a statistic model that predicts the next word that's going to come. How does it know it? Arguably, it's not pulling from any one source. It's the points of the data that it processed against. Right, Yes, that is true. We can sort of break my mind, break the generative way technology into two parts. The ability to comprehend what you're saying. You know, so it's throwing it a prompt and it is
trying to comprehend and trying to give you an answer. And the second part is the general knowledge that it has captured through the training process. Right. If you think of I think of generative Way in two terms, right from and I think we're going to talk get into this more. I look at it from the perspective of how can generate your way I make me a better developer? How can I solve everyday problems? That are there in the businesses
in it. How can I solve that? So from my perspective, I trend to break, like I said, into two parts, the part about comprehending and sort of listening and reasoning, and then part about general knowledge. I tend to leave the general knowledge part out if I can. When I'm asking for a prompt or asking for a response, I'm going to provide you a context and you should only reply based on the information that I am providing you commonly referred to as the RAG pattern, and that is very important to
me. So Richard, your question about sources and citation is very important. But in the kind of solutions that I'm building, I am providing it a context and saying I'm going to give you these ten pages of information and you need to process that, and that way, the hallucination and the accuracy is much high to begin with, because I'm controlling the source. I agree, and I think that code in general, you know, code generation or help
with coding is where AI. You know, this large language model absolutely shines. Sure, and nobody's going to complain that, oh, that algorithm was lifted out of my open source project. Well it probably wasn't it was probably put together just by statistics, but but I've found it. That's that's where it really shines. And that's not where people are complaining. People are complaining
because it's taking their art, artwork and appropriating it. You know, artists, like all the artists are crying about this, right, you know, and this is why people don't trust it. But as you know, anyone who's writing complex documents that are in or code that are in a specific as
you said, context, right, that's where it really really shines. And before I stop talking here, the other things that I like about it are the new chat GPT being web interface, so instead of using the data that it's trained on, you can do essentially what is a web search, but it goes out, finds multiple sources, puts things together, and if it finds the answer, it sort of gives you a nice concise answer instead of
having to scrape it out of different sites. And also if it can't find an answer, it'll say, you know what, I searched all over the web and I couldn't find that thing that you're looking for. Maybe it doesn't exist, and maybe here's some documentation that you can go look at and see if you can figure it out yourself. Yeah, that integration because all of these language models have a knowledge cutoff date December twenty twenty one or some date
like that. So if you ask a question that goes beyond that, of course, having this big integration where you can do go do a web search, you could do that. I guess with ch AGPT too, with some plug ins, but beings sort of gives you that. What I've found is it generally works, but there are there are situations where the citations are not complete, and that merging of real time look up versus what the model has
been trained on does not work as seamlessly. You're right as you were copect, and especially for things like c sharp like you're better off using the default model that it was trained on to to do some code figure you know, right, a routine that does X, then you are using the bang stuff. The other thing I like about it now is I can say, here's a u r L to a PDF, digest this, and then tell me
why, you know, my fridge doesn't work. That's right, and it will actually do a good job of diagnosing, And you can take pictures and upload it and it'll understand what it's looking at in the picture most of the time. This is what we're finding out anyway, cal that's a very good
example. Give it a fifty or one hundred page document and let it read it, and then have a conversation with it, and say, because you know this, this fifty page document presumably is talking about different problems with the fridge. You don't want to go through that. You can say, Okay, now that you've read it, I'm going to tell you two symptoms that I'm seeing with the problem that I'm having, and can you narrow it down? And then having a conversation with that. That's just amazing. It is
amazing. That's aig. It speaks to the strength of this model. It is a good summarizer to be able to ream through a lot of information. And plus you can validate you know it got it from those hundred pages. So once it gives a statement like you should do this, you can take that phrase and search in that document and probably find exactly the reference you need to say, yeah, the document really does say this. I found I did this once in my amplifier and it actually gave me the answer and said
on page fifty three in this section here's what it says. So it actually told me where in the document the answer lied. And this was a real experiment that I did. And I had a real problem with a piece of gear, and I literally told her to read the PDF of the manual and it in like five seconds gave me the answer the challenge that you know, let's say if your one hundred page PDF document, either it can read it
completely, that's one thing. But you know what if you have thousands of fifty page documents, Yeah, then you're trying to sort of come up with an answer, and you're eventually going to run into a limit. Even though GBT four gives you thirty to two K models, which is what about two
hundred and fifty pages or so what if you had more pages? And then now you can't feed that into the model, right, and now you have to do some sort of a search, a vector search of some sort, get the right content in and pass that as a context you prompt you said the V word vector databases. I find these baffle and I'm not so sure that you know, Joe web user is going to be able to figure them out. Can you briefly explain what they are? And sure, sure,
sure, and I shouldn't have used the words without explaining it first. No, of course you should. How are we going to ask you questions? I mean, yeah, what's our role here? No? True? So let's take a step back. So you started out with a great example of you know, having this great fifty page PDF about your amplifier and it just you. You had the model read that completely, and you were able to do that because that fifty pages fit into the size of that that the model
supports, which is thirty two. Right. What if you had thousands of fifty page documents and you're trying to build an answer. You can't feed all of those to the models, at least not now. Maybe GPT five GPT six would allow us to do that, who knows. So in the absence of that, what you have to do is you have to go do a
search against those thousand documents. And the problem is that if you did some keyboard search, as we've been used to doing, right, it is not that keyboard search is really not conveying the intent of what you're trying to search. You might search for some words and you might get good hits, no question about it. But what if you're trying to just describe the problem. You're just saying, my amplifier works this way in certain conditions, but I'm
noticing these problems. I can guarantee you there will be no keyboard hits when you write a statement like that. But in the you're describing the problem. And what if you were able to take this two word two lines worth the description and find a paragraph of texts that closely resembles this text. And that's
what a vector database is. What we are trying to do, is we're trying to take every piece of text in that large corpus of documents and we are assigning some statistical values to it so that we can place certain sentences in a certain manner. So think of every word having a certain number of attributes, and then words that are closely aligned will have similar number of attributes.
Things like that. So now you take that sentence of your description of the problem, you create an embedding out of it, and you know, you can use any of the models to create an embedding, which is you're getting a mathematical representation of those two sentences. And then you are and this going back to high school math, you're doing a coscience search of this mathematical representation
against the mathematical representation of the entire corpus. Maybe you get two or three hits which generally talk about this problem, and then you bring that text back and only send that text to the model because now you've compressed that information. So that's if you're able to do that, then you can get answers to the question even if you have one thousand documents. And I'll talk about later on the product or application that I'm working on, which basically uses the pattern
that I just talked about. I mean, we don't have enough time for you to explain this, but in general, how does one go about creating a vector database from a tome of documents? Right? So you know, let's just take the simple example of you those two lines of description of your problem. Right, you can go to any number of embedding models that are available. You can say, hey, please give me a vector representation of these three lines of text, and it literally gives you back a vector representation
of those three lines of text. So, because these models have been trained, like I said, you play this prediction game what comes next? What comes next? You play this prediction game, and by playing this game, millions and billions of times you have assigned a certain attribute to a certain word in a certain phrase of that sentence. And then by placing these values in a multi dimensional space, you can say these words together mean something, These
words together mean something. And you know, we all remember the twenty fourteen word to wac paper that Google came out with, you know, twenty fourteen, and we were so amazed when it could do simple things like king minus man plus woman equals what oh queen, And we were so happy that you know, these models have now some semantic understanding of English language. Right that
was twenty fourteen when the word to wac paper came out. And of course, you know, we've given the complexity of the models, the transformer model, the availability of GPUs. Now we can do so much more. We were embedding words literally, now we're embedding paragraphs and pages essentially, so that to your listeners, essentially you have taken your corpus of text and you've created a mathematical representation that tells a model. The model is not sentient, it
does not know what it means. But because of mathematical representation, can take the problem description and say there are these four paragraphs out of these thousand documents that seem to be talking about something similar, and you go and pull those paragraphs, and then you go to your prompt and say, I'm going to give you four paragraphs worth of relevant information about my amplifier, and I'm going to give you some symptoms. Your job is to take that symptom and try
to come up with a plausible answer for the problem that I'm having. That's that's essentially what's happened. I get that with the paragraphs that you generate for the prompt. But how do you know if I have thousands and thousands of pages of documents, is there like literally an app that I can just drag and drop those into and proof its a vector database or is there a lot of manual work on the part of the of the user. That's a great
question. So Microsoft sixty sixty five Copilot is going to come out. The release data is November first or something like that, if I'm read correctly. So let's just use that as an example to answer your question. Card. Sure, so Microsoft is sixty five copilot would be Now you have inside Word, PowerPoint, Excel, whatever you will be able to ask. You can go into Word and say, hey, can you please generate me an invoice based on the last five invoices that I sent out? Okay, you'd be
able to do something like this. How do you be able to do that? Well, maybe you have thousands of invoices stored on your one drive somewhere, and when you start training the semantic index, which is you know, which is a component of the Microsoft sixty sixty five copilot, what they would do is they would go through your one drive, open up every document, and most commonly this is what people are doing. Open up every document at the page break, Take the one page so or some fixed number of words
or tokens. Read that from your document, create an embedding, store it. Take the next page, create an embedding using the AI model that I talked about. It could be any text model, and store it. And that's how Microsoft internally is creating the semantic index. Of course, cognitive search now has vector database built into it. Right, Cognitive search used to be
text search, now has vector database built into it. So now you've got through this whole process of ingestion, which can take lots of hours, yes, because you could have thousands of documents. But now you've taken your entire corpus and broken it down into a set of embeddings. So I take it then this isn't something that Joe programmer would do with a bunch of data, like you need some sort of powerful tool on the back end to do this
for you. This is something I'm time to understand your question. Maybe this is the time to sort of talk about the application I'm building and see if I answered that question. And I'll come back right all right, but hold that thought. We'll do that right after this break for these very important messages, and we're back. It's starting in Rocks and Carl Franklin. That's Richie Campbell, and that's vishwas Lele and he was just about to start telling us
about this amazing application that he's working on. Please do go ahead, sir, thank you. So I don't know amazing, but there's an application that we've been trying to build the last six months that has taken up a lot of my time. So we talked about the Microsoft THY sixty five copilot. We are building a similar capability but to help proposal developers. So imagine you're getting a fifty one hundred page RFP or RFI or RFQ or something like that,
and that's a very time consuming and expensive process. Right, wouldn't it be nice if you were able to take all your past performances so you're responding to these kinds of RFPs over years prior. Can you take those documents and create some sort of a vector database and then use that to generate a draft
of a future RFP. So to going back to your question, Carl, as a developer, what we had to do was to write those ingestion routines where we take those past proposal responses that you may have submitted to the government or some agency or a state or local or what have you, and write that ingestion routine where we take that document split it up. Now, there is a lot of work that is happening which says, should you be breaking
the document on the page boundary? Is that enough? Because think about it, if you just break it on a page boundary, and maybe the important idea or concept is transcending that page boundary, right, break it up by topics, right, break it up by topics, But then how do you break it up by topics table of contents. That's a good point, but sometimes you know, most in our case, at least the PDF documents that are usually make up these responses. The table of contents are not that great
and PDF is not a great possible format in any case. Right, So you have to use in some cases language models to be able to even passe them out and to break the document down into some sort of a section in some classification format so that you can use it for embedding. So you store, maybe not do it on page boundaries, maybe do it on some sort
of a lexical boundary. But then you also think about what if the table of contents has some sort of inheritance built into it, is it important for you to capture that inheritance because that might come in handy as you're trying to respond to a question. Right, So those are all important things to do.
So going back to your question, Yes, as a developer, you have to think about where your embedding boundaries lie because when it comes time to generate content, the more precise context that you can provide to the language model, the better response are going to get. And the more precise context you can get by thinking deeply about how you're going to break your corpus into the
right set of embeddings. Isn't Microsoft Copilt going to do a lot of this for us, Like, this is a solve problem if you just are all in on them three sixty five and pay whatever per month they're going to ask for. That's a great question. And let me tell you the difference is so Microsoft Phrisks have Copilot is going to be a general purpose answering engine, right, so you can ask it and say, hey, please draft me the text for an invoice based on the last five invoices, and I'm sure
it'll do a great job. You've seen demos of that. It also has access to the graph, so you knows which people you're collaborating with, and it can take into account that. But if you have a complex writing problem where the questions are not single line questions, right, the RFP questions are. Sometimes the questions themselves are maybe a page long or half a page long. Right, how do you sort of create a prompt? So in case
of Microsoft, there are sixty five you're essentially writing a prompt. You're saying, hey, genade mean invoice which is based on the past five invoices. Right, So much context in that statement that needs to be known that you have in mind. Right, in our case, the context is enormous, right, Richard, Because if you're trying to respond to a complex question,
you have to infuse a lot of attributes into that. Right. You have to think about what are the pain points that the buyer is expressing in this description? What are the then themes? How am I the one that they should be selecting and not somebody else? Is the buyer cost conscious? Have we done this work? Like? You know, it's great to say that we are the greatest company it can do agile development, But want it be nice if you can provide some proof points that we have done this and we've
had success. So I've just given you three examples. There are many such attributes, So when you are crafting a prompt, you need to take these considerations into account. So in case of sixty five copilot, you are writing the prompt, but in our situation you're writing a prompt that is far more complex. So we are trying to do some prompt engineering for on behalf of the user where they can tell us at a high level what they want to do with their response, and for us, it is very much about about
That's why it's a copilot. It's not a pilot or an autopilot. You're leveraging the experience of experienced proposal writers. The experienced proposal writers who spent ten twenty thirty years writing this, they know exactly how you should be responding. We are simply augmenting them and providing the first draft. Right. All of the innovativeness of our engine will not help you win any proposals. Our engine will help you do sort of the mundane steps quickly so that you can spend
more time adding innovation to your response. So you know, if you if you have to generate this prompt, we think that every user is not going to be able to generate those levels of prompts. And I'll tell you why, because as good as GBT is, if you give a certain size prompt, it starts forgetting some instructions that you give in the middle, right, so you have to repeat it. So we have to have sort of layers
of prompts that go in. And this is why you know. The team that I'm working with has data scientists in them, and people ask me sometimes, hey, generative AI is pre trained, do I need a data scientist? Well, you need a data scientist if you're if you're crafting this sophisticated
prompts on behalf of the user by taking some input from them. So, Richard, in that manner, we think that this is a very specialized example of what we are trying to use the language model, and this is sort of the difference with the M sixty five copilot, and there are a few other differences for us. Hallucination is very very important. Don't please don't go out and say that you did a five thousand SharePoint node migration when you've only
done fifty or five hundred, right, that we'll be a problem. Right, So we do explicitly accuracy verification, and there are many techniques that your listeners may be aware of that. You know, you take the generator to response, you try to extract entities from that and lage language models. Right, So any definitive statements in there, yes, yes, and language models
are great or extracting entities. And then you take those entities and then you do a search against your knowledge repository and say, do I see a five hundred node migration ever mentioned here? And if it is not, then let's
just flag it and let a human proposal writer go check this. We're doing those kinds of things peer, if you I'll just just extend this by saying that I don't know if if your listeners have seen this study from MIT, where there are many studies out there which said, okay, let's just try
to measure the productivity improvements or something like chat depety. Right, So what they did was they took four hundred people college degrees good writers, broke them into groups of into two groups two hundred each, gave them a writing assignment, and then people who were evaluating them had no idea whether they were using some tools or not. So they gave them on writing assignment where nobody was
allowed to use any tools. Say, got the baseline. Then they give them another writing assignment, but this time half the group was only allowed to use chat deputy. The other group was not allowed to use chat jpety. And now once again the evaluators had no idea who was doing what. And if you look at the results, pretty amazing results. Right. The group that was using chat GPT completed their work in thirty or thirty three percent less time. And I can point you to the study. So my numbers may
be off. But more importantly, more importantly, the people who are using chat GPT scored a thirty three or thirty five percent better grades than people who are not using Okay, so a third better grade in a third less time, third less time. I mean that's amazing. And then what was it fact checked? Because that's my problem. You might be able to split up
beautiful text, but is it really right? Yes, and human evaluators evaluated that, right, It was just like you know, people testing that that's always a case, right, And we know that human evaluators make mistakes too as well well, and we don't know exactly what they were trying to write, like as soon as you get away from non factual writing, like just exploratory fiction based writing and stuff, it's not the same set of problems right,
right, And you know what is interesting is that you know, remember I said that they were all given a task where nobody was allowed to use in external tools. So that was to establish a baseline, right. What is interesting in the study is that writer folks who did not have great writing ability, right, even they benefited from when they were put in a pool where they could use chatgputy. They benefited from. Their grades went up significantly.
I mean I would argue they would benefit more, they would manafit more right. But you know, there's very very interesting thing that about that study that I'd like to bring back to your question, Richard. It was very interesting in the study that there's something called, and I'm going to use a fancy term that I did not know until I read this m study called the human machine complementarity. Okay. So what they did was they evaluated the grades
at two levels. So the team that was remembered the team that was using cheat gpty. It took the output of chat gpt and graded it, and then they took the output where humans did some prompt engineering on top of chat GPT and generated an output and evaluated that too, okay, and they found that there was not much difference, right. So what is interesting is,
so where is this human machine complementarity coming in? So people looked at the chat gepety output, made a few tweaks, and then they submitted it.
So, tying it back to your question, Richard, my theory is that the reason there was not much different from the chat gept and the human machine output was people were not thinking carefully about the bronze and actually writing detailed prons, right, And we think that by creating these layers of prompt for them you can get a benefit on top of so everybody is going to get that third percent benefit. That's going to be at some point table stakes, right,
was it? Steve Snowski I was reading a post from him recently said every textbox is going to be LLLM enabled soon and it's going to be like you start writing and it will just start giving you some stuff. So no, everybody is going to benefit from thirty percent or twenty eight percent or whatever that is, right, So how do you go beyond that? Well you go beyond that through sophisticated prompt engineering. So that's been my journey to build
this tool. Take your past responses, and I have to say this. People ask me this all the time. This is not what we are building. Is not cheating as a service, right, I tell people, if you don't, if you don't have good past responses, if you don't have good architects who can come up with a good solution, if you don't have creative solutions, then what we produce will be pretty you know, poor and
not something that you want. But if you have good past performances that you've done this SharePoint migration work and you can come up with a you have architects who can come up with a solution, then this is going to benefit. So really, from a responsibilii standpoint, it is about taking your knowledge and then generating that quickly and then accurately checking it that you're not hallucinating and saying you did five thousand not server migration, right. I love it as a
service, chees. But also you know the old the old comside comment of gigo right of garbaging garbage out. Yes, like we still have data quality problems like this comes down to the quality the data you load the set with and how well you load it. But it is an automation tool, and all automations stuff from the same problem, which is they're only going to amplify what you tell it to amplify. And if you tell it to amplify your
stupidity, it's going to amplify your stupidity. You're you're absolutely liter Richard my sort of in my experience in six months working on this project, it's clear that you spend a lot of time doing data wrangling. You talked about garbage and garbage out. Do you worry about what data is getting in? So so be prepared to do those kinds of things. Even though these models are great, But I would say one thing, these models are far more accepting
of the mistakes. So I was just doing this example recently where you know, we had to calculate some accuracy or some data and we went out to three or four external services to go validate this data. And each of the services returned the data in a different format, right, one return in a
certain Jason structure. Somebody gave us XML back, and previously I would have to write a lot of data wrangling code to sort of pars out that data, get this and then compare it sort of very format it's yeah, to get them into very imperative code, very much imperative code that I had to write, yeah, right, and then immediately throw away like you're only going to do throw it because it's going to And in case of language models, you can say, hey, I'm you know, I'm interested in validating this
source information that I have. I went out and made a call to four external data sources, and they're going to give us some data back. And this data may not be exactly what you want, might be wrapped in some XML or JSON. Please try to make sense of the data and it actually tries to extract entities whether it is XML or JSON right and does the comparison for you. So so I found that writing that data wrangling code was much
quicker than previously as possible. So there are these examples. So he was able to discriminate a tag from content it will not it will yeah, so as is the case with the imperative program. So I do feel that as a software developer, as we think about these problems, and Carl you talked about, you know, being a coding assistant is important, and I tend to use most of the times, I tend to use chat jupety for my coding stuff because it is you know, of course I can use the get
ub copilot and get ub copilot. You know, the benefit of course is that you know ID you're based in the ide right there as you're coding. It is showing. But for me, even more important thing is I go to chat gepet and say I'm trying to write this routine and this is generally my idea about this routine. I can describe it in plain English. It comes back with some code and maybe I take that code, I paste it in there and then I'll test it and maybe'll give me an error, or
maybe the logic is not quite accurate. I can go back and have a conversation that, hey, you gave me some code, but it is not handling this edge condition. Oh that's let me handle that edge condition for you. Go modify these three lines of code from here and then bring it back to my ID and change it. So in this manner, it's an iterative development model. So that is a significant timesaver. I had this exact experience
you're talking about, without going too far into it, Kelly. My wife and I like to do word search puzzles, right, and we were getting it in the paper once a week, and we're like, you know, maybe I had to go get a book, and the books were like terrible. So I actually wrote a word search using chatch ept to help me do it, and it was completely iterative, and I probably should publish the conversation
that I had to get the code. But it started just by putting them all in a line, you know, and I'm like, can you mix them up? Some backwards some diagonal some backwards diagonals? Like sure, So we just went through this reiterative thing and then it actually was great. I mean I used, first of all, I use a GPTAPI to search for words around the topic of X camping right. And I tried, first of all, I tried all these other libraries that try to do semantic you know,
matching and searching and stuff, and they just fell flat. But you know, the GPTAPI was like, no problem. It just produced these words and I could tell it maximum word length, minimum word length, how many words to come back with, you know, don't use dashes or apostrophes like. It was really really an easy and it was just a single sentence of English that I had to perfect in this API as a query. So yeah, I know exactly what you're saying. The iterative process of using GPT chat
GPT in particular to come up with some solution is really really powerful. I knew the model is able to reflect sometimes and say hey, I asked you to write a poem and which I let's say, I want I wanted to make sure that this is does not rhyme, and you give me an output which is rhyming, and say hey, this does not make sense, and it will say, hey, sorry, I apologize, I don't I met your requirement. It's very polite. It's polite, but Richard, I do
want to go back to you said it is a stochastic parrot. Right, I would challenge that a little bit and I put a put a video out. I'll send you the link that is not based on my work. That's based on some great research that was done something called the Othello GPT and your
listeners may know Othello as a game called reversy. It's an eight by board, right, So somebody did this, this awesome research, and I have an eight or ten minute video that I talk about it, and I especially if you want to reference that, because you know, we always think, hey, this is a stochastic parrot. You know, it just comes up
with something based on what it has heard. And then you start typing and you start talking or you give a response, and it says, oh, these eight words, the ninth word must be this, and the tenth word must be this. So what the creators of the Athology GPD paper did was they said, Okay, let's constrain the problem as any good engineer would do.
Rather than having a fifty thousand vocabulary of English words and then having billions of parameter models, let's just use the same technology, use a transform on network. But let's constrain the problem down to the game of Othallo. So Othello is a by board and you know, for limited number of moves, as you can imagine. Right, let's just take the sequences. Let's just go to the World Championship Othallo website and let's just download all the game scripts.
And now your vocabulary is not fifty thousand words, it's just the limited number of moves that you can make in that game. And now train the model. Train a language model based on the scripts of Othello game. Okay, And now there's a technique in in neural networks that has been around for some time called the probes where you can go to a certain layer of this neural network and say, what have you learned? What up until now?
What is your understanding? And what the creators of this paper did was, so let's say two players are playing this game and then you let's say you're they are halfway through the game or at whatever level they are, And what they wanted to know was, does the model really develop an emerging representation of
the board. It's not just predicting the next move, right, And to prove that point, what they did was what if I'm going to suddenly change the board positioning, I can move these these these moves around or the pieces around, and does it now change the prediction based on that? So two things I said, apply approbe figure out the emerging representation of that othello game.
So now it is predicting based on that. Now go in and manually change the position of these pieces and now see if it predicts something accurately. So what this paper tries to do is try to say that, you know, it's not just a stochastic prediction. In training this over and over and over again, it has developed an emerging representation of the world, and can we just try and apply this to CHATGPT In some ways it is developing an
emerging representation of the world. Now there are people who absolutely disagree, Ian Lacun being one of them. But there are data scientists who who think that these models are learning something more than at giving them credit for. So that's a debate. But you know, I'll point to that. TELLOGPT if you want to read the paper, I found the link and I'll included in the
show notes. Yes on your post and yeah it's in I mean, yeah, I'm you know me I follow in firmly in the Angkoon space of this is there are no emergency property emergent property here except the ones that we project upon them. It's software, but you can't argue with its ability to summarize, And so is this a kind of summarization, right? And a good summary also represents an emerging understanding. Right? But keep in mind, you know, the model has no notion that you're playing a game. It has
no notion of who's winning, who's losing. Right, are there any of those things are important? Right? And that's there in nice part of the problem like this is such the opposite of common sense that you could possibly imagine. Right. But at the same time, the eight by eight board is complex enough that it cannot memory the moves. You can't keep it in your head. You can not keep it in your head. So there is some
emerging representation the rules of the game. Some strategy is emerging that you know, anybody who's played othello knows that you know you want to take the coroner positions quickly to block your opponent. It is developing those emerging strategies. Nobody said you should do this, So there's something to be said, I found that paper so fascinating that I had to read for your listeners. I had
to read it five times to understand that paper. These papers are complex to read, and then I tried to synthesize the meaning as I understood it in a ten or fifteen minute video. Looks like you found a link to that. Yeah, it's twenty minutes, but okay, it was complicated, so
it was a little longer. But I appreciate your thinking on this, and it's great to see you building stuff here because it does give us all encouragement that there is space for some useful products if you put the proper parameters around it, like the capiats are important, covets are important, and think about it in this manner. Right again, going back to I'm always looking for how can I build application faster? So think about the three applications type that
are emerging. Right of course, three sixty five copilot, you have productivity tools they are adding lllms to. That makes sense, right, But then maybe you have some sort of a line of business application or complex claims processing system that most of your listeners can relate to. There's one piece of functionality
that we've never been able to you know, imperatively code. Maybe that functionality can be we can embed LLMS into your existing line of business applicant, right right, we you know, we get some large discounts because if we keep track of all of the invoices and certain products from different vendors, if we can somehow put them together, maybe we can come up with a way and negotiate a contract with somebody. Right, very hard problem to do. We
have never been able to add that to over invoice them. Maybe now we can inject LLLMS to do that. So that's the other type of application. And then of course the third type is a native you know, problems that we have not been able to solve without generative AI, and now people are coming up with applications and new applications come out every day. No, and I appreciate all of that sort of truth that as we narrow in on what these tools are reliable for, it is a new class of automation and it
will take certain workloads off the table that just we're bloody hard. I think the easy win is the raising of the average Like you talked about the writing side, that yeah, LMS will mean that your ordinary writer is a better writer, one would argue, the very best will still be better than any tool, but most of us aren't the very best, but far more interesting
will be this true. We couldn't have solved this any other way, Right, you were mentioning a fellow and immediately go to the deep mind stuff that that went after go. And while that's interesting, deep mind's also tackling protein folding and is now producing results that we're going to take years to validate because
they've tackled such a complicated problem. Like at this point they're like, well, it's now come back with some protein for whole routines we've never seen before, and until we prove them, we don't know if they're right or not, and it's actually hard for us to even prove it. Is that's interesting is the IBM Watson Chess program and AI. Well, it's a decision tree
model built in the nineties. Like it's way more primitive and compremitive. But if you look at what it's just, it predates the deep learning models that we use in generated AI, but it uses just pure brute force to go through those trees in every possible combination. Yeah, well, it has a
triage because you can get too deep. But one of the variations on Alpha GO they actually use the same training model to train against chess, and its slaughtered the best digital chess players out there, and it learned no time at all. Like it just if there was an indication that for this kind of
problem space, it was wicked good at it. And also that, unlike the Decision model was taught against historical games, that Alpha my model literally was just given the rules and played itself, right, which is what they ended up with GO. Originally, they with the original version of the Go they trained against classical games and then made a much more efficient version that just trained
against itself. So, I mean the space is where these genera AI models make sense, none of which is fact based, especially especially when it comes about extracting facts from the Internet. You know, that place that really lacks
a lot of facts. Yeah, so you know, it's as long as you use a tool appropriately and you don't fall into the trap of trusting it as an authority, right any more than you trust your calculator being authority calcuators, Well, at least the calculator comes up with the same answer every time.
Oh, the input it's are the same, for sure, don't discount the user error issue with calculators, right, I mean it, you know you have to follow the same sequence of keys, but yeah, you get Well, how many times we run into companies where they based their actions on a on a spreadsheet that had bad math in it. You know it's easy to do. Uh. The hour flies by with you visual, I know, kind like I want to keep going, but I could easily go another
hour. Oh well, thank you for this. It's I'm really grateful for the work you've done. Yeah, me too, And and especially I really appreciate your business oriented mind of using AI where and that's something that I'm I'm just now starting to, you know, wrap my mind around. How how are real businesses going to use this? I mean you mentioned the Azure thing, but also Google Bard now can search Drive and Gmail and everything else.
Gmail search was always really good, but all these business tools are coming out, and therein lies the opportunity I think. Yeah, So thanks again, Vishwas and dear listener, we will talk to you again next time on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at
pwop dot com. Visit our website at d O T N E t R O c as dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September two thousand and two. And make sure you check out our sponsors. They keep us in business. Now go write some code. See you next time. You got Jack Middle Vans