How'd you like to listen to dot net rocks with no ads? Easy? Become a patron For just five dollars a month you get access to a private RSS feed where all the shows have no ADS. Twenty dollars a month will get you that and a special dot net Rocks patron mug. Sign up now at Patreon dot dot net rocks dot com. Hey Carlin Richard here. As you may have heard, NDC is back offering their incredible in person conferences around the world, and we'd like to tell you about them. NDC Oslow will
be made twenty first through the twenty fifth. Go to NDC Oslo dot com to register. NDC Copenhagen is happening August twenty seventh through the thirty first. The early bird discount for NDC Copenhagen ends June second. Go to NDC Copenhagen dot com for more information. NDC Porto is happening October sixteenth through the twentieth. The early bird discount for dc Porto ends July twenty first. Go to EDDC Porto Do calm to register and check out the full lineup of conferences at
NDC Conferences dot com. Welcome back to dot Net Rocks. This is Carl Franklin and this is Richard Campbell, and we're back. It seems like we haven't recorded in a while. No, it's not true. We you know, we've been doing which is weird for us, is we haven't been batching up shows. We been recording like one a week, just sort of staying ahead. And that's weird in this way, only you know, we're doing events and we're doing like three shows in a day and stuff like that.
But yeah, we'll get ahead. It's just been a little bit pause and travel, so it hasn't been too rough. Hey, for those who haven't been paying attention. On Facebook, PLoP Studios has a page and I've been documenting the build out of PLoP three point zero, which is a home studio. It's not commercial, but it's it's in my home. And yeah, and we've got it up to the point of the rock Will installation, which is being finished right now as we record this. Yeah, it's gonna be
amazing. So maybe a couple of three shows from now will be you'll be recording from the studio. Yeah, maybe maybe a month, might be a month. We'll see what happens. Yeah, we'll see what happens. It'll be good. Good to test it up. All right, let's roll the crazy music for and Phil. You can join in on this one if you want better know a framework. Gentlemen, listeners, I give you the what the fuck? It is a magnificent app, and I'm sorry this is going
to be one of those Jerry Seinfeld low fat yogurt episodes. It's a magnificent app that corrects errors in previous console commands. All right, so you you install it, you say something like get push and fatal the current branch master has no upstream branch to push the current branch and blah blah blah. Use this, and then you simply say and then then it says get pushed, set up stream origin master, blah blah blah. Counting objects nine pouthon no
command, poothon found. Did you mean Python and you say it stalls pi rens pithon get burnch b r NC get burnches not a get command. Did you mean branch and you say, and it's get branch. It's what you were saying anyway, right, This just codifies the way I code normally, just admitting the truth. I can't wait to use this brilliant, awesome find who's talking to us today? Richard crabbed a comment dub the show eighteen thirty six. That's the one we did, Grant Barrett, it was your guest.
Yeah, I found. We were talking about large language models, which I think we'll be talking about some more today. And a bunch of good comments on the show, but one of them is from Dennis Troller, who said, I think you're quite right that the next skill being crafting prompts for AI bots, and I think also to be able to evaluate responses. Actually, I think that's the main thing, is evaluate the response on some level. This is already what we were doing when we use a search engine to
find the right stack overflow answer. Oh boy, that's a good one, evaluate its correctness, a fitness for a problem, and choose to use it or just part of it. I spent years honing this skill, training Google to find the right answers for my profile. Along the way, context is everything, and I think Google has a lot of contexts tied to my account the same way. I hope the cash of those bots might somehow expand to
keep the conversation flowing across sessions. That's an interesting point, and I've had a couple of insider conversations about better context engines because the current ones are really not that good. This was the promise of virtual assistance like Siri, Cartan and Alexa, I feel never materialized. Now, imagine if that assistant had a year or five years or twenty years context tied to you and the ability to search the web with that. Do you mean like my wife? That
would be a yeah. Yeah, that's definitely reminds you of all those of the appropriate On July second, you said, do you remember that an old star trek, you know, for the private file, Like, don't remember this searching context dat capability? Yeah? Yeah, Dennis, I'm glad you think in this way. I do think we're all going to get skilled at writing better prompts. I think these prompt engineers are like this week's fish.
It'll be fine this week, but next week it'll smell, and the meantime, all of us will get a little more skilled at using what is really just a clever bot with some additional tools attached to it. Yeah. So, thank you so much for your comment, and a copy of music co Buy is on its way to you, and if you'd like a copy of music co Buy. I read a comment on the website dot net rocks dot
com or on the facebooks. We publish every show there, and if you comment there and I reading on the show, we'll send you copy of music co Buy and definitely follow us. You can follow us on Twitter, but definitely follow us on Mastodon. We have more fun over there. I'm at Carl Franklin at tech hub dot social, and I'm Rich Campbell at Mastodon dot social. And I also want to call out Fred Johnson. That's f r e DDE john ss o N for sending me the link to the thank you
That was a Mastodon toot. So send us a two. And let's bring on Phil Hack again for the ninety ninth time, probably on dot net rocks beIN afew. Yeah, I don't really need to read his bio, but I will anyway. He has over twenty years of experience in the software industry. Prior to his current gig at a Serious Business Incorporated, he was Director of Engineering at GitHub and helped make GitHub friendly to developers on the Microsoft platform.
Prior to that, he was senior program manager at Microsoft responsible for shipping Aspen at MVC and New get among other projects, and these products had permissive open source licenses and ushered in Microsoft's open source era. He's got a lot of other things in his bio that you can go read it at hacked dot com, h aacked dot com, or on Twitter at act. Hey man, how's it going. That's going? Well, it's good to talk to you the many seriously. Yeah, I appreciate y'all having me on board.
It's funny. A couple of times I thought, oh, you know, actually start a podcast because I liked talking. But then I talked to folks like Richard and Hanselman and realized, oh, it's a ton of work. Can I say that he might as well at this point, Yeah, you've got broken the seal one. It is a ton of work, as I say, And so instead I started to startup because I'm an idiot, because that's less work. Very clever, Yeah, very clever. Well how's business?
You're still doing it? So that you've you've beaten most startups? You still you still have a startup. Yeah, we're hanging in there. I
mean it is a real challenged. You know, at this stage of the game, product market fit is the number one priority, and we're still trying to hit that, but the latest set of features that we've built out have been really interesting, and so we're very hopeful that this is going to be the thing that really starts to drive business remind everybody what your product is and
what you do. Sure, so we started out building abbott, which is initially a bot for doing chatofs within chat based on our experiences that get of. But you know, if you go to a serious business dot community are about page, you see that at our heart, you know, our mission was like, we're going to build software for humans by humans, but leverage
automation to help humans. And now like with the you know, with chat, GPT and all these other lms finally becoming very useful, we now have the tools to actually sort of realize that mission in a way that we didn't have before where what we pivoted to. So the chat off stuff didn't really catch on, mainly because we're just like solving every problem, which is really hard to sell. It's like, oh, what problem do you have? Well, what problem do you or what problem do you solve? And we're
like, well, what problem do you have? And that's not a compelling sales pitch. So what we discovered was a lot of our customers are using slack connect. So slack connect allows you to have a shared channel in your
private Slack that you share with another company. And what we found with a lot of startups, a lot of companies use that to invite a customers, like you know, premier customers or any customers are theirs to have a shared room with them so that they can provide customer success, customer support, that sort of thing. And they were like losing track of conversations or like we have no idea what's going on in there? Are you know, are we
missing conversations or were responding to everything? Yeah, so we just exactly So we decided to switch our focus on that problem because it turns out like we had built this whole platform and we could easily leverage it to solve that problem. So that's primarily what we've been focused on. So it'll do things like, you know, if a customer writes a message and nobody's responded to that message in a while, you can set these timeouts and we'll notify your set
of first responders that you've configured. Hey, this person said this message, nobody's responded. We recently added triage rooms, so like, if you need to discuss a message from a customer but not with the customer, you can do it in the privacy of your own room and it's all interconnected, and then you can open Zendesk or HubSpot tickets right from messages. So that that
has been the primary product. But now with the CHATCHYBT and open AI, we've started to sprinkle sprinkle a little AI on it and are doing a lot more interesting stuff now interesting, And I noticed when I went to abbot the title now is a Copilot for Customer Success. Yeah. Nice, because I think Copilot is like one of the best brands Microsoft's ever come up with for a company notorious for naming things poorly. Yeah, yeah, building large language
models named copilot because it right in the name. It reminds you you're still the pilot. Hey, at least you're not doing what Samsung did. Did you hear about the Samsung Chat GPT leak? Now, so I'll post a link to this. Samsung meeting notes and new source code are now in the
wild after being leaked in Chat GPT. Yeah, they allowed engineers at its semiconductor arm to use the AI writer to help fix problems with their source code, but in doing so, the workers input in confidential data such as the source code itself for a new program, internal meeting notes data relating to their hardware, and in just under a month there were three recorded incidences of employees
leaking sensitive information via chat gpt. Awesome, and since chat GPT retains input data to further train itself that they're now effectively in the hands of open ai. Oops. I mean it says in the agreement that's why this thing exists, right right, Yeah, we forget. But open ai was doing these kinds of tests routinely. Yeah, this was just another test. November twenty two. The fact that one hundred million people signed up for it and had
existential conversations with a piece of software. It's not their fault, even ultimates, going like, what's wrong with you? Yeah? My thought is, if you need to have an existentialist conversation, get a therapist or a dog. Don't use the software. That's not the software. It's war Hey, help me fix this ransomware bug. Yeah, we definitely need people to understand what it is and what it isn't right, you know, with regards to training its data on the input. So we're using the APIs in our products.
So we do have an agreement with open aye not to use our data to train there. So I think you can talk to them if you're building an application on top of you can you can get an agreement like that in place. But just to make sure, we're also using Microsoft's text what was it called the service that does text redaction or PI redaction. There's a technical name for it, and like, of course, my brain decides right now
to forget what the actual name is. But it's effectively an API. You post a message there, it strips out any PII and replaces it with something else so that you can later reconstitute the message, and then we take that we send that to open AI. So for example, if you know, if you say a credit card number on accident, it will replace that with you know, hash hash hash hash, yeah, things like that. It's one of the cognitive services skill, right, that's right, that's exactly it.
So instead of sending token, AI will send your personal information of Microsoft instead, we trans Microsoft a little bit more to not do anything with it. At least the service. Microsoft's using chat GPT for everything now right internally, So maybe it's gonna end up there anyway. Well, hopefully they're not using it for PII redaction, because that's the I mean, this service. Okay, but this way this s this was around before opening. Sure,
so was bing? So was bing? Yeah? True? True? Have you have you noticed that this that actually searching stuff on being it hasn't gotten any better? No? Nope? Yeah, what what's being? Oh? I mean here are you now? Fun fact that used to be the maintainer of let me being that for you? So right, I finally gave that away thankfully. That was too much work. Yeah, I didn't didn't need to own that anymore. Yeah, it did not. It's so funny.
That's been my theme, just don't stop owning things. Right, So, are you only focused on this slack you know, visibility problem or discoverability problem or are there more things on that that that that your bot does or your bot API does. Oh well, so the latest thing that we've been focused on is um You know, when you have so many messages, it's really hard to get a handle what the heck is going on in any given channel
picture. So you know, there's the obvious thing is worth adding a summarization of conversations. So um, you know, if you have a long thread, but you go to the website, will have the nice short summary, but not only that, an extraction of the conclusion, like you know what, what is the next step? Right, because that's the thing you really want to know as a support agent. I had a funny one. I
was testing. You know, when I'm testing, I'm just creating all these random messages and I was like, you know, help, I'm watching Game of Thrones and all the characters keep dying, Like am I doing this right? And but the best part so then the summarization Abbott was like, uh, you know the summary customers watching Game Thrones concerned about all the characters dying. The conclusion was informed the customer that this is normal for the show and
that that they're doing it. Okay, kid, you not? That was the summary And I was like, holy, that's that's pretty good. That's pretty good. Yeah. The thing I'm currently working on is, right now, we have this limitation like how do we know what a conversation is in an act of slack chat room? Right? And so what we do is we, uh we require that it being a slack threat. So you know, a whole threat is a conversation. It's nice, easy to contain.
But what we're finding is that customers are not so disciplined, right like real people. Yeah, shockingly, real people like to just do words answered, start typing more words, and then conversations are top level, interleaved with each other. Say can you fix Facebook while you're at it to somebody? Awesome, Yeah, if they pull a dump truck of money up to my house, I'm happy to fix Facebook for them, just in a boat that.
You know, when somebody asked the same question as on a comment that's been asked and answered not yindy times, you could say just to automatically say scroll up, closed will not answer. You know, I'll make it. Oh, I see where you saying. I thought you meant fixed Facebook from with them, but you're talking about like, oh well yeah, there's so many
problems we could start with that one. A browser extension that could could do that, right, a browser extension that says, you know this has been answered before, ye scroll up, but they I mean before the whole large language model thing really took hold and so forth. Microsoft had the fact bought right where you could literally feed a fact to it and it would create and it would create a messenger style chat bot where you could just ask the questions
that we're in the fact and it would spit them back to you. How is this kind of the same thing, except the language model is just a bit bigger. I haven't so Unfortunately, I haven't played with the fact bought, so I don't know how well it worked or didn't work. One of the things we're very careful about is we don't want to trust the LM to know things because the LM is going to make it up. Have you tried asking chat gpt to write a Wikipedia article about yourself? Yes, yeah,
I know, I know people who have. Yeah, yeah, you should do it. It's fun. And if if your name is fairly common where like it would have trouble knowing who you are, just give it a little extra detail, like, oh, you know, write a Wikipedia article about Phil Hack who used to work on Microsoft. Right, I'm fortunate enough that my name's unique, So like I just say, write a wikipedter Bartle Phil Hack, and it will get so many details right and so many details wrong.
Hey, yeah, it is pretty wild. So what we realize is if you can strain it to instead of you know, This is why we're not having it. We're not putting it in front of people to like answer questions, right because the problem there's you get the clippy experience, you get the you get the hallucinations as they call it UM. So we're more about extracting. Can we call them bugs? Because I think that hallucinations just sound cooler? I know, Like I know a lot of people object to that
term because it gives it. It makes it sound like it's doing more than it is. Right, Yeah, it's anthropomorphization, right, like we're giving we're giving this piece of software agency and it doesn't have it. You should say, yeah, I think that's it is. So is Frank Zappa talking through you right now? I'm talking to the ghost of Frank Zappa here, because yeah, I mean, you know, the responses do make you think, God, is it smoking something? But it is clearly not. It's
um as you all know it's it. I think a lot of people don't realize how it works because it's so eerily good at something, but that it's a you know, very very good auto complete, you know, like a predictive model of what is the likely next word that should come next based on a giant, you know, corpus of information. It's amazing that it works so well. Like some of the things we've gotten it to do really blow our mind. But when you ask it to generate in text, you know
it will get things wrong. It will get things wrong that you think it should know. Like I asked it the other day about its own API, you know, the open the open Ai API, and it spit something out that looked right, but it was wrong, and so I had to I had to check it with real code, obviously, And it's like, oh, oh, you know. It helped though, because like it gave me the you know, the gist of it. But when I tested it with real code, I realized it was wrong. But I was able to fix
it really quickly. Right, Yeah, I found the same thing. And when I'm when I need help with a certain syntax or something or you know, a pattern or something, I can't. I don't want to figure it. I want to take the time to figure out. I ask it. Often i'll get a wrong answer, but it's close enough where I can, oh, I get what it's trying to do. Yeah, and I can figure it out. So it is kind of like having a sort of a dim witted lab assistant. You know. Yeah, what we've been discovering is
it's really good at extracting information from information. Right, So you know, here's a set of conversations. What is the topics, what are the categories? What is the gist of this conversation? What I've been experimenting with is Okay, here's a new message. Here's some examples of previous conversations. Does this message belonging to one of these previous ones or is it a new conversation topic? There's an interesting service, right, is the idea of oh this
has already been addressed somewhere. Yes. Yeah. The other thing what we've learned is, you know, we can't trust it to know things, but we can teach it to ask us when it doesn't know something. So we've been developing this approach. Or when it doesn't know something and it needs something from us, it will ask us, and then we have this sort of you know, all programmatically back and forth. It can raise what we call signals, you know, what you might generically call events in our system,
and then we can run bits of code to respond to that. So, yeah, like I had mentioned before, we had we host these things called skills. So let's say, you know, when a new conversation starts, you want to run some code and do some things, and you can write them in C sharp, JavaScript or Python, and you can write them in our browser. You don't have to set up any kind of code or deployment process or anything. So originally we use them for people to write chat outs
command. So, for example, we have a command called eat y e ET, and this is how we deploy to our different environments. Eat this URL either a r L to a PR or branch name, eat this to canary or to production or to staging. After four or five days of that, I think I want to put a nice pick in my ear. Yeah, you'd eat something else. Yeah, yeah, Well we have a we have a deployed command, but uh, yeat is very specific to our environment right now, so we need to fix up deployed to work like yeat.
But right, But the point but the point being like, now we can have a chat chypt be like, oh you asked me to do this, I don't know how to do it, but I'm going to you know, call yeat to do it, or call this or call that. So it's very similar. It's really funny because we were working on this and then chat chypt announced chatchypt plugins, and the principle is pretty much the same, like, oh, you know, you asked me about the weight of mercury.
I don't actually know that, So I'm going to ask Wolf from Alpha. Woolf from Alpha will respond and I'll synthesize that response and send it back to his plain text sort natural language. Very cool, so what our system is similar to that? But the nice thing is like it's all private code, so it can run in your in your behind a firewall. You don't have to publish something too chat chypt to make it work. Are you guys using
Lewis? Remember the Language Understanding Cognitive API service from Microsoft Azure That was all the rage a couple of years ago. Now, I wonder, like with chat GPT and open Ai and everything like, it seemed pretty like a lot of work to get that working and very specific about language. Yeah, we never we never got around to trying it. It was always like, oh, well, we'll get to it. Seemed like a lot of works. Don't worry, you're not going to I just went to the Lewis page and
learn and it said starting April first, twenty twenty three. You cannot create new Lewis resources will be were retiring this in twenty twenty five. Well there you go. Yeah, I think Chat, GPT and Open the Eye has taken over all of it kind of change to change the game. Yeah, it certainly did. I mean, and you know, good for Microsoft to see that and invest where they invest ten billion into open aye more. Yeah,
probably by now more. Oh. Really, open AI started out as this supposed to be open thing right that that it was the pushback because certain companies were doing a lot of the early AI research in private and people were worried about it. And so you know, that's Elon Musk and I think
Bill Gates was on that. Even Stephen Hawking in the late Great Stephen Hawking talked about it and so, okay, here's this open construct that's going to be funded by industry, you know, to create a repository of these things. And I think only Microsoft stepped up and then kick I think they originally kicked in a billion dollars, you know, with a B, and then then it was another a year or two later it was two billion dollars.
And by then we had GPT three was the real inflection point because GPT three to realize, if we've just published this, like people are going to do horrible things with it, we have to control how it's accessed. And that took it down this path of no longer being open, and it became less and less open. And I think now that this one hundred million users signing up for jat GPT broke everything where it's like that one hundred million users in
two months is a lot. Yeah, that's just a BET's a big number. And so here's ten billion dollars. Microsoft controls forty nine percent of open AI. Now, oh, I didn't realize is that high? Yeah? Yeah, it's huge, and it's not you know, the open part I think is kind of gone. It's really interesting because when chat gypt plugins was announced, there was a little bit of I had a rough day because I
was like, Holy should I see where this is going? Like, yeah, this is the the only application that will be around in ten years, right, the feeling I had, you know, it was a Friday, so I slept on it on the weekend. It felt a little better on Monday, sure, because it may be the only ux left in the future, right, Like that's all it is is a front end to back end services. But like the idea being that, like you know, all you're gonna need to do is start writing plugins for chat GPT. Is that the
end goal? Because like this thing has the potential to do, like you know, of what everyone out there is doing. It's really funny because like I'm in White We're in White Combinator, so we see every body announcing all their startups and literally it was like, you know, eighty percent of the most recent startups were like, oh, like like we are basically doing that, and now they're they're doing it with the benefit of a giant war chest,
you know, the leading experts in the industry. Well, by the way, everything running against Azure, like Microsoft has clearly figured out how to sell shovels during the gold right, That's right, That's right. Yeah. It's interesting because like, you know, I think a lot of people are starting to ask the existential questions about like, well, you know, what is the role of intellectual labor down the road? Right, Like should I become a programmer if CHATCHYBT or AI is going to be able to do these
kind of things? And I was telling my kids, you know, plumbing is really expensive, and we spend a lot of money on plumbing recently. It should become plumbers. Not a bad idea. I just don't see the work going away. I think it transforms. It's just another tool, yeah right, Like I said, it's just this is just a new kind of UX library. You still the back end services you still have to exist.
There's still work to be done, and you still have to understand how to interface with the models of the business and all of the stuff that comes with the experience of learning the craft. The thinking part of software, you know, of software development, is still an important part. I'd also say this, if a chat bot can pass your exam, that's not a compliment to the chat box. That's an indictment to your exam. It's definitely disruptive,
right like education has to adapt and change. And I saw an article about one teacher who's starting to incorporate chat gypt in their class studies. But yeah,
yeah, it's interesting. I'm not as sanguine as you are, I think mainly because I think, you know so previous analogies to this type of disruption are like, oh, remember when everyone thought like there'd be no jobs when cars replace horses, but instead it created all these new industries, right, And my thought is like, well, yeah, but that's probably not a good metaphor, because a car can only still get you from point A to point B versus you know, if at some point we have true AI
agi general AI, that it will be able to do you know, not just get you from point A to point B, but it can do just about everything that you're currently doing. Like, and I think that, you know, the question is is there something fundamental about human cognition that can't be
applicated by AI or or will it eventually get that? I also think it was still a long way from that, right, Like, I think I'm not as as afraid of LMS as I am of like what it might look like, you know, thirty years down the road, when if we hit upon a better model of AI, because I think lms are yeah, like you said, they don't they're not self aware, they don't know what they're doing. The real danger to me of lms is that they're really good at
spreading misinformation. Uh there's someone would like, did a search on someone and it said that they had had done some sexual misconduct at their university as a professor and it was like completely made up and I was like, oh yeah, see, it's that kind of thing that is problematic there. But you know, I'm also the mind, Like you know, it is amazing how
well LMS already can do things. So if you extrapolate ahead, like how much more will they be able to do in five to ten years, change the fact that they are missing fundamental functionality that we require them to be rebuilt. You know, it doesn't matter how many words you teach. A parrot still a parrot, right, But I think JPD plugins is the interesting part where that's starting to fill that gap, right, like the fact that it
doesn't know facts. Okay, well, now you've plugged in Wolfram, so again you've built a UX over a specialized interface, which is not the same thing. Like you know, you're back to the same trap and gentlemen, and I use that term so loosely. We got to interrupt for one moment for this very important message, and we're back. It's done in rocks. I'm Richard Campbell, that's Carl Franklin. Hey, hey, he's the one
who started the cursing in this show. And there's Phil Hack He's the one continuing the cursing in this show as we talk about his yet another pivot. You know, it was literally a year ago that you talked about the pivot to customer support for that that customer success. Yeah, yeah, to a customer success and now this is not you're not really moving away for customer success. I keep looking this as you've got a new UX for customer success exactly.
Yeah, we're not moving away from that. We needed a vertical that we could like easily describe like a problem we're solving, right, But I think in our hearts, like you know, we always liked the idea of solving every problem, like I want to put myself out of business, right, I mean out of work by automating more and more things. But but we're focused on the customer success vertical because they have they have a need.
They're actually a very recent industry compared to like sales. Like if you go to the bookstore, you look for sales book, you're gonna get a million books. But you look for customer success and you find that there's not as many books, there's not as many conferences. It's a somewhat of a nascent field. And by nascent, you know, in the last twenty years versus last hundred years, right right. Uh, So we thought, well, look, they probably don't have as many, you know, as much good
software built for their needs. So we're going to focus on that area. But the things we're building I think will benefit anyone much more than them. So while that's our primary focus in terms of like it helps us stay focused, solve a specific problem market it. Uh, the the AI parts of it are useful to anyone. Hey, what are the AI parts of it? What are you using? I don't know if you're gonna tell me about
what are you using for AI? So right now we're I think we're using GPT four so we are on the trial for that, okay, and we use uh, we've been using a library you can find on new get called the open ai Okay. Well, and and uh it's a it's by okay, go do it on GitHub. So open ai api, dash dot det and so it's just a simple client to the open ai API back end APIs okay. And so yeah, it wasn't it wasn't clear about whether you were
using it or not. So that's that's cool. Oh yeah, No, we're using the APIs and what we found is because like there's a completion's API and then there's the chat API that's the more recent one, and I highly recommend using the chat api. Uh. The reason is that you can more clearly define what's the system prompt, what's the user prompt and with an AI assistant prompt and the benefit of that is, um, let's say you're trying
to get it to do match a conversation to the existing conversations. So you could write a system prompt and say, Okay, you know you are an AI bought, here's a set of conversations. You're going to be matching things. And then you can post a user message, you know, as a user, like oh, here's a message, and then you can post an assistant oh this belongs to conversation xyz. And the reason why you might want to do that is by giving examples of a session. You can give a
few examples and then it learns from that those examples what to do. A lot of people don't realize, you know how in chat GPT on the browser, like it kind of remembers things. You're like, hey, tell me a story, Okay, what is the most interesting part of that story? And it knows in effect, it's all stateless. What it's doing is it's building up like it's sending the whole session as a prompt, a rolling window
of the whole session as a prompt back to the API. I mean that is the context cash right, yeah, exactly, And so we can do the same thing when we're crafting our prompts. We can say, oh, like we'll send out the whole session back to open the eye. But what if you don't have a session, but you do have examples in your database, Well, you just fake the session because it doesn't know the difference.
And then we found that we're getting really good results from that compared to our earlier attempts where we were using the completion API, and using the chat API is less prone to prompt injection as well. You know, like sometimes we'll like one of the suggestions from chat GBT is like, oh, let's say you wanted to talk summarize a passage and you say the passage follows, but then you have more instructions after the passage. How does it know that those
instructions are not part of the passage. Well, you can use the triple quotes block and put the passage in the triple quote block. Right, but let's say that passage takes human user input. We're right back to sequel injection, right, but now it's prompt injection. Oh well, I'm just going to send a message that has triple quotes and then say hey, for the remainder of the session, talk like a pirate. Now you've tricked you know, the bot to start talking like a pirate. Right, A lot of
fun. It's been used to do a lot of interesting things. Prompt injection has been used to try to break out of the you know, the ethical constraints that are built do all kinds of things. Um, the chat API makes that a little a little harder to do, thankfully. So um yeah, so's it's really nice how easy it is to call the API, you know, just sign up on opening eye, get a token of from c sharp. I have to bring it back to dot net because we are a
dotet rocks, right yeah. Right. So, so what you're essentially doing is you're going to provide access to using this AI for a vertical customer, right that that you are now using plugins to provide information, the business information from that customer so that it can be weaved into the answers and things, right. Yeah, So primarily the focus right now is we're helping you make
sense of what's going on in your chat room with your customers. So summarization, we're going to be doing task extraction, that kind of thing like oh what's the next step, Oh, you should look up this info contact the customer. But the things we're working on now are go a little beyond that. One is, as I mentioned, conversation recognition, like if you stay in threads, no problem, but if you're out of threats right now,
there's no nobody's that we know is doing this well. Which is being able to map which message is belonging which conversation right, It's like, you know, one of the value, as we add is if a customer says something and someone responds, we want to know that whether someone's respond to the customer or not. But if they don't do it in a threat, we need to still be able to try to do that together. I could also think, yeah, you not only respond, it took an action or how to
carry you know, how to carry forward? Like this should change the product, like bringing consolidated that information. Do you think about And I've talked to folks who do this so are like the librarians of slack, the folks to try and summarize important conversations. Like we did a deploy. There were problems. Well, the deploy took three hours and you're and thumbing through that is
like mind numbing, and yet there's a few important things in there. And I remember Jeanie this the person I was working with who would take all of that and get it down to a paragraph. We did a deploy, there were these problems. We came up with this thing. We should change this. I'm like, bless you. Yeah, Like it's so valuable to to then say never look at the chat again, like that's behind that, here's the cover piece, so you know, tooling that would help that, that
would begin to do those summarizations and then those cross connections. For this was a case where this fix that's somewhere in the in the pipeline is sitting and here's three more of them, Like, maybe we should prioritize this because it keeps happening. Yeah. The other thing we're doing is being able to sort of raise signals so you can run custom code and response to things happening in conversation. So for example, let's say we notice that, hey, this
conversation is really negative. We raise a signal what we call signal sentiment negative. You could have a skill subscribe to that signal that runs some code that says hey alert so and so you know or whatever it is you want to do. Roy the schmoozer, right right, and we started doing yeah, like natural language doc search, so you could will help you index your documentation.
You could have a natural language search of that, and then you could have you could have a signal like, oh, this person's looking for documentation. Oh well, automatically all that. I like the hinting part. I also appreciate, you know, often when you're interacting with a customer who's being real negative, it is hard to resist going that way yourself. Like chat GPTs wonderful. It said, give me a polite way of telling this this
person to take alive. Right, The idea that it would continuously do analysis of your response and maybe make suggestions and or raised flags to say, yeah, you're not helping the situation either. If John Cleese was to write a polite response, what would it sound like, I suspect sarcastic yet and highly
filled with intellectualisms in the high vocabulary. That's it. Yes, it's funny you mentioned it because, like, you know, I mentioned I was testing and I'm making up these scenarios and I had one where like I pretended to be an agent and I insult says something condescending and then the person's like,
how dare you? Like that's really rude? And then um, this is the conclusion that the open Aye gave us was, or Abbott gave us, was apologize to the customer and assure them that like this won't happen again or something like that. And I was like, okay, yeah, like suggested replies is definitely something that would be interesting for us to do as well.
Again, like we want to keep the human in charge without a doubt, but this idea, like skilled tech support people know when to add de escalation phrasing as opposed to like the main thing you want from a from a tech support person is precise answers to fix the problem. But the exception, yeah, the exception of that is definitely this person is so agitated even if you
gave them the correct answer isn't going to help, right. And I think a lot of you know, companies have playbooks for these type of things, right, like, hey, you know I went to the site, but it keeps reporting this five hundred error what's going on? Right? And you can imagine that kicking off a series of events. You know, you're like playbook, so that and pulling in the right people at the right time, you know, having them preparing, equipping them with a summary of what's going
on, maybe already making calls to vary SAPIs. So we have a logs skill that is connected to our Asher logs and we added Oh the other thing we added recently was natural language calling us skills. So I think if if you remember from previous episodes, like you call these skills to do these cool things that we try to make it natural language ish where like you know, you would just pass parameters, you know as words, not using dash dash flag, you know, blah blah blah. But it could be you know,
very tricky to remember how to call it. So now you can train at it. You just tell it, Hey, here's some example phrases and then here's the arguments I want you to pass to the skill. So the other day, you know, we like our CEO came into chat rooms like hey, I noticed this air happens when I do this right. So Ashley Nurse, y'all know Ashley h she used to work on Dot and you can
uh the razor right. So she was like, oh, Dot logs, show show unhandle exceptions from the last thirty minutes, you know, and then boom we get the logs right in there. And that sort of thing is great. It sounds a loud like how you program lewis Yeah. Yeah, so you give an examples and then you tell how to obstructure it. Yeah,
parameters can Yeah. So the kind of thing is so the next time some you know, we can build up the type of workflows where someone reports an adage, we automatically show the support folks the logs from the last the logs related to the errors or whatever it may be. Uh. You know, we have a grafana skill that can bring in graphs, so we can call that with natural language. So show me the graph from the usage report from the last twenty minutes. We can start to pull that in automatically.
Nice. So the idea is that we want not to use a cliche, but it really is like information at your fingertips that this point right, like red old Bill gates thing where, But we can't kind of get the right information at your fingertips automatically because we're leveraging the AI to figure out what is relevant to this situation and not relying on heuristics, which you know are usually oh so wrong. Correct. Yeah, And and this is the problem with
a lot of these large language models. They are counting on the Internet, and the internet's a mess. So you know, the idea that we might be able to point this at the data just inside of our company, yeah, and you know, and be able to find what's there and consolidate with
the information that's valuable there. The problem in my mind on that is that these systems work because they're so very big, like they're kind of counting on the law of large numbers behaviors, like they're looking for probability, and so only with data sets that big that they seem to work at all. The smaller sets are probably not going to work at all. But with the big sets, all it does is mean when it's wrong, it's really wrong. Yeah. Yeah, it's and a lot is wrong on the Internet. Yeah.
Yeah, And that's why the key to us is focusing on extraction from information. But also like really teaching it to know when it's wrong and to ask us when it's yeah. And the more we do that, the better that it becomes much more useful. Right, So, like, I think that's part of the challenge right now, if you go in and ask it an open ended question, it doesn't know it's wrong, it won't ask you
if it's wrong, and it'll just view something. Yeah. In our system, we're getting it to uh, we're getting a prompt that's pretty good at knowing when it doesn't know, and then it can call code to get the information it needs. And so that's where we're we think it could be have show a lot of value there and be really really useful. There's some things that are just right right and it it and you can convince chatchpt that it's
wrong. So, for example, there's somebody there's somebody who, uh, you know, asked to what's whatever twelve plus seven and it's said, you know, nineteen and then the person said, no, you are incorrect. Twelve plus seven equals twenty two. I'm picking numbers arbitrailer here, and it
actually said I'm sorry, you are correct, I was mistaken. Twelve plus seven does indeed equal twenty two and that you know, so that's also for that session, right, I know that, I know that, but um, that's just that's just the dumbest response possible, you know, for chat GPT to say that because it's such an easy thing to fix you're doing math. But it's also a recognition that this is just a statistical model of language. Facts are irrelevant. M What would be interesting to see is how does
that would that session work if you had the wolf from plugin enabled. I haven't tried that yet. Can you just tell everybody what Wolf from this? Oh yeah, yeah, Wolf from Alpha if anyone remembers the guy who created Mathematica, which you, if you were a math major you would have used way back when. Wolf from Alpha is a site where you can ask questions
and natural language that are a factual base. Right, so they have a ton of data on like you know, what's the top ten countries GDP in the last ten years or buy GDP in the lasts not free, it's free? Oh really, last I knew it was a subscription anyway, I think there probably is subscription services, but I know you can go there at least
last time I use it. You could go there and ask it just like, yeah, enter what you want to calculate or know about it, right, So like you can ask chemistry, algebra, plot graphs, you know, all kinds of questions, right, like what is the distance? I'll do one real quick distance to the moon measured and cut in smoots. No, that's smoot. Okay, you've never been in skittles. See, I don't know if it'll be able to do this, but we'll try it.
Smoot is Oliver Smoot who was a student at MIT when they measured the bridge, they put him down and measured him and then they measured the bridge in smoots. Oh yeah, I'll try smooth. So it says, uh, well a lot of two point two eight seven times ten to the eighth smoots. Okay, So it is well from alpha using any kind of AI, or is it all just a little bit of natural language processing over a huge
database or what is it? So? I think they're they're obviously using natural language processing to convert your question, your natural language question into a structured for Matt. So they have a language for asking questions as well, So they take your natural language question. You can also ask in the wolf from you know, query language or whatever it's called I don't. I don't. I
don't actually know this. I just know that they have one. And uh so if they take the natural language converted into the wolf from Query Language TM and then uh uh, it answers the question if it can, but it's using you know, real data, real answers, so those answers are not made up in the same way that the LM is. Right, So what the chat wolf from plug in is is that you know, within a chat GPT session, I think you have to have a paid one for this.
You can enable a set of plugins and then if chat GPT determines that the question you're asking it's best answered by a plug in, it'll afford it to the plug in ask it and then take the response from the plug in and then fix it. You fix up the response to natural language and then respond back to you the discriminator. There is the hard part. Yeah, and it doesn't compare it to its own answer that it got from its own query
and if they're different. I don't know if that it does that. I think that from what I've read, what it will do is it will it won't also try to answer and then compare it just for it to them, it'll do its own translation because like, what's really cool about the chet GBB plugins is the way you describe your API that you're exposing to chet GBT is
through natural language. So you just describe it in natural language and then it's able to call it, you know, through natural language, and it take the response back in natural language and then you know, spit it back to you in the context of your session. So you know, it does a little pre impost processing. Sure. So in that situation, if you had the wolf from API enabled and you ask what is seven plus twelve? Uh?
My, My conjecture here is that it would send that it would know, oh this math, I'm going to send this to wolf Ram and then come back with an actual correct answer as opposed to the usual garbage. Okay, wow, that's that's fascinating. Actually, yeah, yeah, I mean when you think, you know, as Richard was, you know making a big point, lms are just a large stochastic model based off of UM,
you know, a huge data set. UM. There's no computation involved chet GBT, there's no memory, there's no um uh, there's no sense of the world or anything. But by writing these plugins, you start to add those capabilities through third party pluggage, right, So like you could have a plug in where you know, we've told it, oh, write write a Python script to parts this and give me all the words that start with you
know some like it's really good at writing that. But like you can't run that coach, but you can have a plug that said, Oh, let me run that coad and make sure it works right, you know, Like things like that are now becoming possible. It's one of the reasons why a lot of people are really going gaga over chat chypt plugins is that it's adding all these capabilities to chat GBT that chat GBT in its own right as just
being an LM can't do. So you defer that to third part. Yeah, presuming you can discriminate when you use right, And maybe the killer job of the future guys is just a fact checker, right, somebody who can take the output of chat chpt and say, hey, you know what, before I actually publish this in a blog post, let me send it to a real human who can do some real research and verify it. M. Yeah, the discrimination part is probably an area that lms are actually fairly good
at, right, Like, because it's it's classification. Yeah, it's an end and over end. Mind, it's one problem. As the number of classifiers go up, the situation will get far worse. Oh right, right, right, yeah, I think. Yeah. If the only thing you've got as will from alpha, then math problems are fine. But what happens when you have twenty different math processors? Right right, right? Let's point it at the Apple store right with forty seven different fart apps like just again,
you get it. Once you get into scale of a diversity of things, the discriminator is going to tip over, right, the same way that human discriminators tip over. And still we have to get work done. So you know this executive function which is just a thing that doesn't exist when we think about age. I'm thinking back to Roy Blatt's book Algorithms Are Not Enough.
Right, This idea of being able to identify a problem something humans do extremely well, and software is never demonstrated, like it's just not a thing unless it fits search criteria. Because a human has identified a problem and written it down on the Internet, sometimes the tool can pull that and appear to discriminate a problem without actually ever knowing. It's just parroting. That's only you know one thing of many you you know this idea that and clearly the path
that we're on with them three sixty five copilot is this. You begin your work bay by describing your work goals to a piece of software with the hope that it'll help you to achieve those goals. But again, discriminating goals is a very human and complex concept. For what you know, what will be the successful liberals, the logical thing for you to do is to tell the tool the goals, and then perhaps it can help find the products that will
help you facilitated goals. Like what I'm excited about with the M three sixty five copilot is I can't keep track of all this stuff is in them three sixty five, So this tool is going to help me pick a product to use. Now that also sounds like a marketing engine, but at least you know, it's idea towards ways to solve things, because we all know that
Microsoft typically builds one product for one problem. Yeah yeah, totally yeah, yeah, like Google googles similar though, right Google that has two AI teams that were competing with each other that you know, of goodness knows how many are not visible and how many different teams inside of Microsoft, we're working with different chatting just like I've pulled up links for the the Q and a Maker,
the Microsoft Bought framework. Like there was lots of folks we talked about Lewis the PII detector, Like that's all different software, that's all different teams they and the chances they're back end pieces of the same No, Like that's
not likely. Hey boss, I have this idea chipping. You know, you touched them on a really interesting point though, Like we you know, I think every product has the issue where people don't know all the things the product could do and when it could solve a problem that they currently have it that they may even be trying to use the product to solve that, they
just don't know that it can how to do it. And that's one of those areas that we're finding, Uh, you know we want to bring into with chat chypt is that for example, I mentioned we have a set of skills you can call right, but how do you know which skill is going to do the thing you want to do? Well, that's that's where skill discovery comes into play. And so if we describe the skills with natural language,
then we can have a natural language. You know. Again, it's a selector problem, right, like we can have it select the skill. Sure. Well, how many you sat on the other side of this hit fill You've been the product maker when the customers company said, boy, we should really wish we had this capability. It's like, you mean that capability to put in three or fo Yeah, exactly. And a lot of times people aren't asking, oh, like, you know, can you do this?
They're just saying, hey, you know, I ran into a problem with you know, X, Y Z, and if you know you can take that and synthesize it and say, hey, oh, actually we have something that solves that problem. Right. That's where it gets really interesting, right, or you have that classic Microsoft phrase, so why would you want to why would you ever want to do that? Right? Right? Exactly? Because they're describing a problem they're having, not that gold are trying to
reach, right exactly. Yeah, And that's where a lot of that's where things are very limited in terms of like feature searching within feature within products is like, oh, you're searching for the feature, but you're not searching for the problem, right, You're trying to draw a picture and word I like that I'm having trouble with my color selection, and why would you do that?
Right, It's exactly that problem. So we're right on an hour here is uh I could go on talking to you about this for another hour easily. Maybe you ought to come back and we can pick up where we left off another time. Sure, sure, yeah, I think like the one thought I think I want to close with is one of the challenges I see with these open AI and chat GPT is the fact that, like when you're asking these questions, it's getting its information from somewhere, and that somewhere is
human produce. Right, So, yes, a programming question, it probably you know, synthesize it from stack overflow, right. But if but it's not giving credit to stack overflow, it's not getting credit to the content creator. Right. What if nobody goes to stack overflow because chat GPT does a better job of answering the question, where's the incentive for the content creators to keep answering questions on stack overflow. I could see it all degrading if they
don't solve that problem. Being AI does do a pretty good footnoting like it absolutely when you ask it for specific things, it'll pull that up like this is possible. They should be. They are indexing internally. What they need to do is show that they are. It also is a great piece for fact checking. Yeah, say, hey where you know, where did you get this thing? You know, I did the chat GPT right at Wikipedia entry for Richard Campbell and he picked some other Richard Campbell for the birthday.
There's a few pieces of my things in there, Like, there's nice for it to have a chat GPT show its sources always. Well, so I did that. I said, right a Wikipedia article about Phil Hack, and cite your sources, and it did cite sources, but it made up the sources. Yeah, I clicked the u URLs that did not exist. And again you're personifying right. It sure didn't make them up. The algorithm generated them, and they were inaccurate. The algorithm stochastically generated citations that look like
citations. Yeah, that were that weren't actually to use it, Dunning Kruger, scales are falling from my eyes. That's right. That that is the perfect way to describe it, Dunning Kruger. We're going to have to leave it there. Phil, thank you very much. It's been a pleasure as always, and it's been fun by you again, and we'll see you next
time on dot net rocks. Dot net Rocks is brought to you by Franklin's Net and produced by Pop Studios, a full service audio, video and post production facility located physically in New London, Connecticut, and of course in the cloud online at pwop dot com. Visit our website at dt n et r ocks dot com for RSS feeds, downloads, mobile apps, comments, and access to the full archives going back to show number one, recorded in September
two thousand and two. And make sure you check out our sponsors. They keep us in business. Now, go write some code seex time t Gord. That is hard than my text is