As software engineers, only a fraction of your time is spent coding. A lot of your time is spent thinking, how should I lay out the interfaces? How do I work with the other software within the company? How do I make sure I've got really clear requirements for my code? Like, all of these things are the really big problems that still humans have to wrestle with, and I'm not seeing LLMS taking that away from us anytime
soon, at least for now. Hey everyone, my name is Henry Surya Virawan and you're listening to the Tech Lead Journal Podcast, the show where I'll be bringing you the greatest technical leaders, practitioners and thought leaders in the industry to discuss about their journey, ideas and practices that we all can learn and apply to build a highly performing technical team and to make an impact in your personal work. So let's dive into our journal.
Hello Doctor Dan, Doctor Leo, Good to see you in Technically Journal Podcast. Welcome to the show. Now, thank you for having us. We're excited to be here. So I always love to start my conversation by asking my guests to actually share a little bit more about your career. If you can maybe mention your highlights or turning points that we all can learn from, that will be great. Thanks, Henry. I'd be happy to. I think the beginning might not be too surprising.
So I started in university in computer science and I did my undergrad degree in computer science. And then I started a grad program. So I started a master's program and I was doing something called formal methods. So like normally when people want to get confidence in their programs, they run them with a bunch of test cases. Like you know, as many test cases as they can come up with. But formal methods is different. Formal methods, you try to prove mathematically that the program
is correct. And I was pretty interested in that, although I could kind of tell as I was working on it that it was just extremely difficult for me. And I don't mean to say, oh, if you're, you know, if you're not getting something right away, you should just give up. Like that's not what I mean. But I I mean, there was some extremely impressive people in this field and I was happy to be part of it. But I also just sort of realistically and realized that I wasn't going to be able to
make a huge impact in that area. But I was still having a great time with it and I guess that's all that mattered to me. But then my supervisor one day happens not to be able to teach. I think he wasn't feeling well. So he called me sort of last minute and said, can you cover my class For me?
It was a compiler's class and I was worried because I hadn't taught a class before, but I, you know, I gave it a shot and that was the turning point for me, going from this kind of abstract research to education research. So I taught that lecture and then it was over for me. I was just how can I start teaching more and studying education. So it was a complete shift at
that point. I think to this day my supervisor would probably say that like, his biggest mistake was not getting himself to class that day because we were planning on working together further in in that area and it
didn't happen. And then maybe in 2010 or 2011 or so, another big career change happened, which is I met Leo at a conference, we were at both at an education conference and just sort of met up there and immediately just had a lot in common, you know, not just about our research but just hobbies and sports and video games and I think just like worldview. And so we connected immediately and just we've worked on dozens of papers since then and our our book most recently.
And it's a career highlight. It's a real honor to be working with him on in so many capacities. So I I have a bit of a non traditional career path in that I did my undergrad in computer science. I actually switched into computer science as a major and then I did 4 years as an officer in the United States Navy. I was a navigator reguided, missile destroyer. And so a lot of my lessons about leadership and building teams and ethics actually come from my time there.
After I finished my time in the Navy, I went back for a PhD in computer architecture, did lots of processor design, very similar to kind of how Dan's describing formal methods. I enjoyed the work, but it was really when I started teaching and being for the classroom that I got the most excited. Right towards the end of my PhD. It started shifting into to computer science education research.
And it started really with a colleague, Simon, kind of introducing me to Dan. And as Dan kind of pointed out, it was just incredibly fortuitous me because I didn't know the methods of computing education at the time. Dan was in an education PhD and it was really through the two of us defining our research direction together that we did tons of really productive work. And I'm super appreciative to
Dan for all we've done. I mean, Dan's been in terms of the kind of big stuff we've done in our careers. We've investigated how effective this pedagogy called peer instruction is in computer science classes and done most the main research on that topic, at least up to like 2017 or so. And then other people kind of took over the research.
From there, we built this assessment of how people learn basic data structures that's actually validated and is used by the community, called the Basic Data Structures Inventory. The two of us use machine learning to predict which students are likely to succeed and fail, and we were being pretty accurate about predicting student success very early in
the quarter. And just in terms of our book, that happened because I happened to hear about some of these generative AI tools like ChatGPT and GitHub Copilot. And I tried them out, and I immediately was worried about our introductory programming courses. And I thought, everybody's going to be panicking. It did end up happening like that. Everybody was panicking, trying to figure out what to do. And it's like I had gone on a call with Leo and I I said, Leo,
there are these tools. We have to look at these. Maybe we should write a book. Somebody has to. And it didn't take Leo very long, maybe 15 minutes or so of playing with the tools himself before he agreed. He was like, yeah, I mean, somebody's got to do this for the community, just to give some direction, some perspective on
what was happening. And then Dan, as Dan said, he said, hey, I I started playing with these LLM things and they are terrifying for trying to teach programming because they're solving essentially all the tasks we used to give them. And so we're going to need to change how we teach interactive programming. And as Dan said, it took me about probably 15 minutes playing with this thing before I went, Oh my gosh, we're in serious trouble.
And we then sat down and started figuring out how are we going to build a class that would adjust to the fact these amazing tools are available. And then as we kept going with that, we said, well, wait a second, we need a book to help structure the class. And that's when Dan kind of suckered me into writing a book. And then we, we had a lot of fun writing in after that. So it was great. Thank you for sharing your story. I think one thing that I really picked up is like, how did you
start writing this book? Right, Learn AI assisted Python programming. I think you just thought about it in 15 minutes. That's really cool. So today we're gonna talk a little bit more about what you've done in terms of doing the research and also from your experience playing around with this LLMAI assistant. And also maybe the impact to people learning about programming. Hey, thank you for being part of
the tech regional community. This show wouldn't be the same without your ears and you are the reason this show exists. If you're loving TLJ and want to see it keep on growing. Consider becoming a patron at techledjournal dot dev patron or buying me a coffee at techledjournal dot dev coffee. Every little bit helps field the research, editing, and sleepless nights that go into making this show the best it can be. Thanks for being the best listeners any podcast could ask
for. And now let's get back to our episode. So let's start probably in the beginning to just level set our understanding what is actually AI Assistant. You know some people heard about Copilot, but maybe if you can maybe describe what is AI Assistant? So I guess I would define an AI assistant as a piece of software that helps you get work done
more efficiently. But the key to it is the way you communicate with it. So typically when people think about using computers, they think you need to be very rigid in how you communicate with a computer. And that's what we would say when we teach programming courses, right? Every symbol matters, every space matters. Every key press that you make is important. And that's the language of computing, right? It's very precise. What an AI system allows you to do is communicate in English or
any natural language. And there are way more people in the world to know natural languages compared to programming languages. And so the hope and the goal is that people be able to use their own language and have the computer translate that, because computers still can't run or do anything with these languages, like English for example. The only one I know. So I keep talking about it, but still it has to be translated into something a computer can
work with. That's what we're trying to automate with these AI assistants. It's like making it so that people can communicate in their language and have it automatically translated over to lower level stuff that the computer understands. And so I would expand and just say that what's great about these AI assistants is it's almost a step in the natural progression of making programming and interacting with
computers easier for humans. And so we've seen this evolution from having to write assembly code or actually before that you have to push buttons on a machine to make it do things. And then writing assembly code was a huge improvement of the stored program computer. And then we moved to the point that we could start writing in higher level languages that were more English readable than assembly, compile down to assembly and actually then the assembly ran.
And then over time, we've developed more and more advanced languages that become easier to express our goals in the language available to us. Now, what's unclear to us is whether or not these LMS are going to be just the next language in which we interact. Right now you can interact with them and get working code fairly often, but it's not always correct, and so it's not quite the same as a compiler which is deterministically going to be correct.
But it does seem like the next step in a really nice evolution of making it easier and more accessible to write software, right? So I think that's a very interesting thing. I've not played around with all these copilot tools a lot, I mean in my day-to-day role because of the nature of my job. But one thing that I think very interesting when I heard developers using it right, it's like it seems to improve their productivity.
So just like what you said, right, it could be the next evolution of how we write software. And I think in your lecturing role as well, I think it will be different now that you teach programming to new students, maybe a little bit, you know, how do you find the difference now that there is this LLM AI assistant and students now learning programming, does that become much easier or does it actually make it harder? Is there any kind of a stock
difference that you can tell? Oh geez, this is kind of a long answer, so I'll give it a go. So we've been experimenting with how do you teach new learners how to program in the presence of LLMS? And I'll be upfront that we don't have all the answers yet by any means. I think that's going to be probably a decade before we actually really know from the research what the best way is to
teach students. Now, these new tools are available, but I can say a few things, which is that students very quickly recognize how powerful the tools are. I actually downloaded in my class, the very first class, and they all were. I got a gasp from the crowd. They couldn't believe that I was just basically writing the code for them. And so the question then is what are we teaching the students? What changes here? And I think there's still a lot to teach students.
It's just the scoping has shifted and what the skills are shift. We also decide what our goal is. Are we training the next software engineers or are we training someone who works in business or data science or accounting to be able to write software that does something useful for them. And I think those are actually slightly different audiences in terms of what we want to teach.
What we've done is we've kind of realized that the skills that you need to interact with an LLM are actually fundamental programming and software development skills. So the general workflow in working with an LLM is you give it some, you have some desired function that you want to write, You know you want to accomplish some small tasks, and you then describe that task. And then the LLM is going to generate code for you. Now the code it's going to
generate may not be right. It may not even be close to kind of addressing what you want. If you read through it, you can quickly recognize this isn't what I want. And then you can pull up basically a window of saying, are there other solutions that are good for me? And so already students needed to know how to read code, understand what it's doing, potentially be able to trace code, and be able to pick from multiple code examples which
one's going to work. And then the next step for them, once they've kind of picked one they think works, it's the right test because you can't trust the LM. And this is actually a point that I think is a really encouraging piece for new students is testing has been a point where new students struggle. They tend to write code and by definition assume it's right, which is basically the opposite of what we all do as software
engineers, right? You write the code and you gather tons of evidence for it being correct before you have any faith in it. And so the students, I think because it's coming from a machine that they know that makes mistakes, they're actually more willing to test. And so that's a research question. We haven't. I don't have the data to support that yet, but I have suspicions that they're more willing to test when it's coming from a tool that they know can make
mistakes. So then you test the code. Writing good tests is super important. And saying that we haven't taught as well as we should have candidly in the past. And then once they've tested it, now they know that that piece of code is working. However the one catches, sometimes the code doesn't work and sometimes you can't get the LLM to give you the exact right answer. And so there's still this last
step of being able to debug. And so you have to teach explicitly how do you modify code that's slightly buggy to do what you want. So we're still teaching debugging skills, and that also is a fundamental skill, right? So, so far your listeners are probably saying, well, what's
different here? But there is a shift onto reading and modifying and testing code away from looking at a blank screen and writing code from scratch, which is what we used to do the most of. Yeah, and Leo, maybe I I could just say this may not be obvious to people who have been programming for a while, but the syntax of a programming language is something that takes some students weeks and some students just don't get past it. It's extremely stressful for
students, and also artificial, right? Like the only reason we need all this syntax. And you know, maybe for this nice little bit of discussion, people can think about some horrendous syntax, like the way that like C function pointers are defined or something like. The only reason it's like that is because the compiler has to be able to unambiguously understand what your goal is. And if you're getting stuck on syntax, some of us do program
for the sake of programming. I I get that a lot of people probably listening to this podcast just love programming, present company included. But there are many people who just want to do something with the code, right? For example, many of us may not care how our appliances work. Like if my microwave, it makes my popcorn for me, I'm happy. And that's how some people are with code, right? They don't need to know what exactly each piece of syntax is
doing. What's super exciting for me and Leo about LLMS is I think for the first time we see a future where people who don't know how to program could be afforded some of the same benefits as people who do. Thanks for the explanation right of how it changes the dynamics. Now for people to learn programming languages.
I think the skill set that you mentioned, you know, like still people needs to be able to read the code, needs to be able to test it, need to be able to debug it and maybe also need to be able to express the task that they want to solve, right? Because LLM cannot just solve everything in one go. You will probably need to play around, do a little bit back and forth before you come to a perfect solution so to speak.
But one thing I think I want to highlight about this tool is that there's a risk of not getting it right the first time, right? I mean, even if you ask the same questions, it might spit out different answers, right? Maybe a little bit of the underlying why, why the tool doesn't seem to be deterministic. That's the first thing. And maybe a little bit about LLM how it works for people to understand.
So I really appreciate you pointing out the you can't just give a large task to an LLM in one go. And that's actually a super important skill that we're now teaching in classes that candidly we did not in the past. So what we used to do in the past, and this is common across pretty much all of computing education and I bet many of your listeners had this in their classes is students are given
essentially just a function. The function's basically perfectly described because you need to be able to auto grade it and so every possible case is covered in that description, right. And then they have to just fill in the code for that function. Now, LLMS do that incredibly well. And so the shift now is, if I give a fairly vague task, like a large project to work through, how do they break apart that large project into smaller tasks that the LLM can then help them solve?
Now this for your audience and for all of us as software engineers, this is problem decomposition. This is probably one of the most important skills we learn as software engineers, and it used to be that we didn't teach that to new students learning how to program until much later in their careers, and now it's actually front and centre. Incredibly important to learn in your first programming class because that's how you have to interact with these LLMS.
And so in my class this last fall, I had students doing things that they would have never been able to do in a previous CS1 class. I gave projects like find a data set on Cagle, ask a question of the data, and then write the software to answer that question. That's way beyond the scope of what we'd ever asked in ACS One, especially given the scope of what many of my students did, where they often did really nice visualizations. They pulled in really interesting data sets from the
domains that they cared about. Some of them had interactive programs where they were you could interact with it and ask, I want to see the relationship between there's a stroke data set and you could say I want to see the relationship between age and stroke, and it would actually plot age against stroke in really clever ways. These are things again, you've never seen ACS one. It's really open-ended and the students are doing all the problem decomposition on their
own. And so we're really excited about teaching this skill. And if I'm real reflective, I'm a bit disappointed that we as a community stopped prioritizing that so early in the careers. It really should be a first word priority. You mentioned about problem decomposition, right? I think that's really important skills for any programmers with or without AI assistant, right?
It's like to be able to break down problems or even requirements into small tasks, into modules, design classes and things like that and decompose that such that we can make a good software right? That is maintainable rather than just one big function that does everything in one go. So I think problem decomposition definitely is very important and I want to come back to the question earlier about LLM because it's non deterministic so far.
Maybe if you can explain why is that non deterministic, How does it work underlying so that people actually understand that? I mean, it's not going to replace programmers in one day, right? Yeah, Thanks, Henry. It's a really important point and it's also counterintuitive for a lot of computing people because, you know, we all joke about shutting your computer off, turning it back on if the thing you're trying to do doesn't work.
But I think many of us go under the assumption in our day-to-day computing lives that computers are deterministic. If you do something, you're going to get the same response. I mean, they're always examples of this and race conditions and stuff, but overall, that's how we feel about computing. If you write a program and you run it, you're going to expect that if you run it a second time, the same thing is going to
happen. And if it doesn't, you probably start thinking about, oh I I maybe I have a memory allocation bug or some sort of like transient behavior problem in my program. But Henry, as you mentioned, LLMS are inherently nondeterministic. So you ask for some code and you get the code, and then you ask a second time and you'll get different code and you ask a third time and you'll get different code. And this is first of all kind of challenging as an instructor.
We have reported recently that it makes it very difficult to plan sometimes because typically, you know, our lectures are sort of scripted in some ways where we need to demonstrate specific things in our lecture. And it's very hard to do that when you don't know what the LLM is going to respond in real time in class. So there is that. But on the other hand there's also a benefit, believe it or not, of being non deterministic. And that is because these things can make mistakes.
Imagine how frustrating it would be if you asked it for some code and they gave you some code and it was wrong and then you were like, well OK, now what do I do? Do I try it again. And in this case you don't want it to be deterministic, right? Because then you're just going to get the same wrong code every time. So the fact that it is non deterministic means that you have a chance even if the most
probable response is wrong. Maybe you can ask again, or look at maybe the top five or top ten and maybe you could pick out the correct code from that list. And this is a skill, right? This is a skill that our students or learners did not need before, but they do now, right?
So because the first response may not be correct, students have to know how to go through the list of potential solutions and figure out which ones are perhaps not correct immediately, but which ones are worth further testing, further consideration. And so for that reason, Leo and I are very careful to continue to teach the programming language.
So it's true, our book and Leo's students are working with LMS throughout the course, but at the same time, we are still teaching the students Python, because right now Python is the language that we teach in the introductory computing courses, and it's still a important part of the loop. And so we're not yet at the point where you can give English or whatever natural language instructions to the LLM and get
back your language. You're still getting back Python code, and so learners still need to understand and work with Python, not at the syntax level. Like we said, we're spending less time on the low level syntax details, but they still need to understand Python, And one of the reasons is so that when this non determinism is happening, they can look at and evaluate a bunch of different solutions for which ones may be
correct. So I think the interactions that you mentioned, right, asking back and forth, you know if you got the first solution probably not quite right, you ask again and you ask again back and forth until you find the right solution. I think this comes to the term prompting, right? So I think many people would have heard now prompt engineering is kind of like a new job even. So tell us about this prompting, right. I think it's a skill set.
It's a new skill set that everyone needs to learn in order to get the best out of AI system. Maybe a little bit about prompting, Like what do you feel about this new skill set? So prompt engineering is just the task of writing a prompt in such a way that the LLM will give you back a good response. And what is tricky there when we're teaching people who don't know how to program to start, is the LMS do very well if you describe problems in a technical
language. So if you say I want this function to find the largest value in this list, right? Like that. I'm using terms that we know as computer scientists, right? I want to make sure I describe it as a list if it's in Python or if an array if it's in Java or things like that, right? And I'm saying I want to find the maximum value. So I'm specifically saying
exactly the behavior I want. I may even describe it even better to say something like I write a function that returns the largest value in the parameter list. Now I really specify the behavior and the LM does a much better job of that. The problem is, I'm using keywords that you have to teach learners, and so there is still this task of teaching them how we would speak about these functions. And I think, and by no means an
expert on how LLMS work. However, they are reading from large code bases and they're learning from those large code bases. And so in a sense, what we're trying to get them to do is just give that function header that a human would have written to describe their function. And if you can generate one that's very close to behavior you want, the model is going to find something very similar in its train set and then generate code that's going to be paired
with that. The other thing you do with prompts is there's a whole bunch of ways in which you can basically tell the LM to behave in a particular way, and Dan has a lot more experience with that, so I'll let him take it from there. Yeah, we had a good time near the end of our book, sort of going into these other prompt interaction patterns, people have started to catalogue the different ways of interacting with the LLM.
It kind of reminds me of the object oriented patterns, you know Henry, like Observer pattern and Model View, Controller pattern and Visitor pattern and all these patterns that people have identified and they documented over the past several decades. And people are starting to do that now with LLMS. And I probably could have kept going and going about it in our textbook, but I managed to control myself and I only, I think I only talked about a couple, but they're very interesting.
And so, for example, one of them is what if you don't know what information the LM needs to perform your task? And this links back to what Leo just said. You have to be very precise sometimes in your natural language, hopefully not as precise as you do with programming. But like Leo said, you would still need to know a lot of the terminology that you might not know. And so one thing you can do is you can use this flipped interaction pattern where you
ask the LM to ask you questions. I think the example we have in the book is you want a function to validate a password and you might not know how to ask for such a function. And So what you can do is you can ask the LLM to ask you for all the information and needs and once it's done asking you, it will write the function. So one of its first questions to you might be OK what are the parameters? And you might be like, well I don't know what parameters are.
And then it will tell you what the parameters are. And hopefully then you can be able to answer that question. I mean, there is a risk here of you going down a rabbit hole that you don't understand. And so we need to balance these prompts patterns against teaching the fundamentals of programming. But there's another pattern that I find kind of interesting, which is called the persona pattern. And a lot of educators are using this to good effects right now where you can ask the AI to act
like a specific kind of person. And So what educators right now are doing is they're using the persona pattern and they're saying, OK, LLMURA CS1 instructor. And that conveys a lot of information like don't use advanced programming concepts that have not been taught yet. Or the persona pattern could be like you are a student in an introductory computer science course, things like that. So that you try to scope the types of responses that you get.
You try to change the types of responses from the default ones, because there are a lot of situations where the default responses might include code that students have not seen before, might use types of code that you don't want them to see yet. It's like not in the scope of the course, and it's kind of amazing to me how much leverage you can get by just telling the LLM how to behave in the
upcoming interactions. So it's definitely an ongoing area of research and I'm definitely of listening carefully to what's going on there. Right. Very interesting to hear some of the patterns right. I've read your book as well, so I find it also very very interesting. So for people who might have applied AI assistant like with ChatGPT or bot, right, They would have seen this pattern as well. Some people also share if you wanna do something, here are the catalogs of prompts that you can
use to solve the problem, right. I think same thing applies for programming. So I mean flip interaction pattern, persona pattern, those are definitely interesting. So it's not just one way we ask question and the AI assistant will just give the solution. You can also be creative and sometimes use it differently, right? So I think Thanks for mentioning the patterns. I would love to see more patterns in the future. So I think we'll leave to those creative people to come up with
the patterns, right. So maybe if you can share from your experience so far using the tool, you know, cracking the code right? Are there any techniques that probably is a little bit less utilized for now, but for people to try it so that they can actually see the true power of LLM in their day-to-day workflow?
It's a good question Henry. So I I think a lot of people are used to, if they've played with these tools at all, they're kind of used to asking chat GDT or GitHub copilot for code and that's a great use case. Something else Leo and I have learned is you can also use these tools to ask for libraries or modules that you might be able to use to make your task
easier. So the chapter we have in our book is called Automating Tedious Tasks and it's amazing to me how many libraries in Python are available to help you. If you didn't know about these libraries like I'll just pick a random example. We have an example in the chapter about automating the tedious tasks where you've got two huge directories of images and the back story in the book is maybe they came from
different phones. So like your partner that has a bunch of pictures on their phone and you've a bunch of pictures on your phone and they're duplicates because you've been sending them back and forth. I think everybody listening kind of knows what kind of mess you can get yourself into. And so the idea is we want to remove the duplicate pictures. And this I think sounds like a super daunting task until you realize that if you ask copilot or ChatGPT, hey, here's a task I
want to perform. Is there a Python module or library that I can use? It will come back and tell you about the libraries that are available that might help you. For example, something that tells you if two pictures are the same picture, like all the pixels are the same. And then you can ask copilot for clarifications. You can say is this module built in? Is it something I have to install?
Are there other alternatives? And actually I want to throw it over to Leo for a second, because in your class at the end of 2023, I think you managed in one lecture to do an example of adding up all the word counts in like a ton of documents. And this is something I'm assuming you would not have done in a normal CS1. I guess is, you know, like, is that accurate? Absolutely.
I I I think it's really hard for students to work with new libraries, and so the LLM having that conversation is really clean. It gives you nice examples. And so I meant for this to be a whole lecture. And what happened was that I ran along from the previous lecture and we just had that quick conversation with the students. But I'm asking Copilot, what's a good library for me to find out how many words are in a Word document? It gave me a great answer. It gave me some star code to
work with. And within I think it was about 15 minutes I actually spent with my students and we we'd solved the problem. And that's just way beyond the scope of what we normally teach. NC is when I completely agree with you again. Yeah. And again, it gets at that higher level of abstraction, right? Like a lot of listeners have probably had to dig into API docs reading about how functions
are called. Oh, this thing takes 5 parameters, the first one is a pointer to a pointer to a stream, and the second one is blah blah blah blah blah and all you want to do is just use this thing and the sample code is out there and these LMS could access it. And I just found the example really cool. Leo, that something we would not have even attempted in a previous introductory computing course.
We wouldn't have attempted it because we would have wasted too much time learning or showing students how to use the library, how to call these functions correctly, and now we can just do it. And I think it speaks to the resilience of the students right when they learn this way, when they're working with a whole bunch of different libraries, because they're working with a whole bunch of different samples from the LLMI think they become more resilient than we can give
them to small little code snippets that perform specific tasks and specific domains. Jumping from domain domain I think helps them a lot in building kind of a robust understanding. Right. So I think another important thing about AI Assistant, right, I think Leo mentioned it a little bit. It is trained from a large code base. So if you're using Copilot, I think maybe most likely it is trained from GitHub repositories, right. I think there's a race about copyright.
There's a race about just bluntly copying, you know from those repositories. Any kind of race that you have seen so far from the introduction of these tools. Henry, that's a fantastic point about the ethics of using these tools, and I think there's a few directions we could take this discussion. And the first is the ownership of that code base that they used to learn from.
I don't think we as a society have figured out first how we should view this ethically and 2nd, how should we view this legally? We are obviously building tools that can help empower people. And so in some light we would say this is a good ethical thing. But we do have to ask how are the tools built and who aren't benefiting from their code potentially being taken or things like this? So that's kind of the first concern. The second concern would be a
copyright. Are the LMS commonly parroting code which might be under someone's ownership? Is it hard to assess, particularly for the kind of small pieces of code that LMS tend to be able to generate Well? But occasionally, I mean, I've seen it in my interaction with a copilot.
Occasionally it will generate an author name like in in its recommendations to me, and then clearly like, I don't know if that's the author or if it's just doing next word prediction and it happened to say author and then predicted some words after that. But it does give you some doubts about where this came from and whether or not we have ownership over it. And so we said fairly early on in our book, this hasn't been resolved legally yet.
Feel free to use these tools kind of for your own use, but if you were to go try to build a company off the software that you're writing, you should be a little careful until these laws get resolved. And the third piece I'd say in terms of the ethics is models. And we've seen this across artificial intelligence. Models reflect biases within society. And so if you ask for a list of names, it will probably give you a list of Caucasian male names
just on first try. And you have to ask a question, why would it do that? Like why is that its default? And it's obviously learning from a code base that probably has those more representative, but that's not a good sign for students who are coming in not from those groups.
What I think is important to do, since I don't think, again, we've worked out these issues as a society, is to bring in the readers for a book and bring in the students in our class into this conversation and say these are the ethical concerns of these models and have that a direct conversation about it and be frank about what we know and what we don't know.
I think the fear is if we aren't, if we kind of pretend these models don't exist and we try not to let the students use the models and they go on to use them on their own, they're going to run into these issues. And so it's better for us to teach them up upfront than to just leave them the blind on it. It also just sort of seems just to keep going off what Leo just
said. It also seems a little to me as a teacher, it seems a little dishonest to not show students these tools as soon as we do. But then what Leo said comes into the picture, right then. We cannot pretend that these issues don't exist. But there are people who try, you know, to pretend these tools don't exist and ban them so the students can't use them in their courses. And I totally get why it's a very upsetting thing that has
happened. I don't mean upsetting in terms of, you know, like making me sad. I mean in terms of, like, upsetting the status quo of how courses are taught. And it's very tempting to just try to pretend these tools away. But the tools are out there and our students are going to be using them. And I think more importantly, they're going to be using them when they get their next Co-op position or their next industry
job. Or at the very least, they're going to be asked about these at future companies and asked about their opinions of these tools. And I just have to super agree with Leo on this. We need to be teaching these ethical concerns. We may not have solutions to them, but I don't think a solution is to try to scare students away from using these tools or somehow trying to prevent them from using these tools, because it's never going
to happen. And I think it's more useful if we teach the tools along with the concerns that we have. Like, I think it goes without saying, but we have a lot of work to do, right? Like, the issues that Leo just mentioned are not small. There's a reason that they're at the beginning of our book and not at the end. Right? Like, these are not like a oh, by the way, you know, these things are going to reproduce like cultural norms. Like, still, this is a big deal,
right? We can't just say it at this. Oh, this is like, look at Appendix A for the problems. This is not an Appendix A stuff. This is like chapter one stuff. So Leo, you know in your course you talk about these early on, but I don't think that that means we can't use these tools. I think actually it makes it more likely that our students will use these tools appropriately.
I think one of the worst things we can do is introduce students to these tools and then not help them understand what the costs are. Because I even think once students understand what's going on, they'll be on the lookout for this and they won't just accept whatever the LLM tells them as the correct answer, right. So we're trying to balance the fact that they're out there and students are going to be using these tools with also training students to understand the deficits.
And who knows, our students might be the ones who end up in positions where they can make these kinds of improvements. Like you know, students are potentially a couple of years away from graduating and being able to inform how these tools are deployed and how these tools are used. So I definitely think that this is a very important part and a new component of an introductory computer science course.
So thank you for highlighting this potential risk of using AII think it's not just for coding or programming, right? I think it's a bigger conversation, responsible use of AI, copyright and for example ownership as well bias. I think all these is like it's a new thing, right? So people are trying to grasp some countries also try to come up with the guidelines, right? But I think you are right, maybe banning it all together might not be the wise idea.
We have to adapt with this tool. And I think we all as a user of this AI assistant, right? At the end of the day, when you use the code and apply it to your system, right, it is also your responsibility to actually make sure that the thing that you apply is correct, right? Because it might potentially affect other people's lives as well. So I think one related question about using this tool right in our day-to-day life is definitely people are afraid of
being replaced. Many people think that, oh, we don't need so many developers anymore. You can probably cut down the number of people that we have in the companies, right? The potential is there for people to think that we may not need so many, you know, developers anymore. What's your take about this? I know it's probably hard to know the actual impact, but what's your take about some people being afraid of OK, AI is going to take over the world and, you know, replace so many
people. OK, I think if any of your listeners spend a little time with Copilot, their fears will be quickly taken away. So, I mean, these tools are fantastic. They do great things, but they make mistakes. You realize very quickly there are still essential skills that are required to use them properly. And so I don't think we as programmers are going to go away and that that's kind of the first take away. The 2nd is of our jobs as
software engineers. And you know this far better than I do. But as software engineers, only a fraction of your time is spent coding. A lot of your time is spent thinking, how should I lay out the interfaces, how do I work with the other software within the company? How do I make sure I've got really clear requirements for my code?
Like, all of these things are the really big problems that still humans have to wrestle with, and I'm not seeing LLMS taking that away from us anytime soon, at least for now. And Leo, I guess just to add, I think if you look back at computing evolution, I wonder if people have had the same discussion when Visual Basic came out, you know, where you could drag and drop components onto a form. I wonder if back in 1995 people were saying, Oh, well, that's it. We have like rapid application
development. I heard that term. It was called RAD, I think, and I I wasn't around really. I was like a kid having a good time. But I think probably people back then were saying the same sorts of things, right? Like, oh, look at this, We can develop these applications by dragging and dropping.
And I think these advances, I don't know if they lead to more or less jobs, but I think it's likely that it's going to be a steady state and perhaps will be more productive with what we're able to do. Just to reiterate what Leo said, I don't think that they make jobs in programming go away. I should also add, and Leo, I wonder what your opinion is on this. Most of what we've been talking about in reading has been for introductory programming.
I don't know if we know what the impact will be on industry level projects. We know people are using these tools in industry and we know they're more productive, but I don't know if we know whether there are more or fewer jobs or if there will be in the future. I just I have a feeling that the result will be that the existing programmers are just going to be more efficient. I agree.
I suspect there's going to be AI think there'll be a shift to like with all the other major advances in technology. When Python came out, we didn't say, oh OK, well, we need fewer people to write code. It was a oh geez, we can write larger software or more quickly do data analysis or now deal with the influx of big data. Like we've just adjusted and done bigger and better things as the technology got better. And so naively, I think that's the case.
But I do think there's gonna be a bunch of research on this topic. I mean the next 10 years, probably in the software engineering community. Yeah, maybe one few things that I pick from the industry point of view, right? I mean there are maybe people saying that it improves their productivity, maybe like 3040%. Maybe the gap from junior and senior might be lesser now because the juniors might be able to take on more advanced
and complicated problems. But I agree with Leo that writing code is not the only job for software developers, right? So they still need to understand requirements. And we know in the industry a lot of times requirements are vague or not well specified, right? So I think it's the software developer's job to actually translate that into a good design, proper design. And also don't forget about evolving the code right, writing it in such a maintainable way, writing it in such a way it can
scale. I think Those things I still haven't heard that the LLMS can do for us. For example, you tell them build me a few microservices that can interact with these kind of APIs. I think that will be too much task for LLMS to solve, but maybe one day it would Happy to see that future.
But for now, I think it might take as well that we have to be able to live with it, leverage with it to improve the productivity so that we can move on to solve bigger and bigger problems just like Leo said, right? And I'd add that it's not great at writing efficient code. So if you're you say no, this is this is an inefficient algorithm, Could you use dynamic programming to solve this? At least in my experience it hasn't done very well.
And then I did try. I teach a really specialized class on writing high performance software that's architecture aware, so like knowing about caches and like super high efficient code, extracting cache locality, things like that. And it did terrible. Like I asked it to write like a blocked matrix, matrix multiply and it could not do that in any way. So I think there's still for the advanced code. There's a lot of room for us as software engineers to be
developing those ourselves. Yeah. I guess what I find kind of interesting about the discourse right now is because it's so new that people want to be able to make these claims like LMS or crap, right? Or LLMS are amazing. And, you know, it's very early, so people are going to make these kinds of claims right now. But I guess I'm more interested in what happens when the dust
settles. And I think all polarizing opinions right now, I don't think any of them are going to end up being what actually happens, right? Like, is every software engineer going to be fired? No. Are we going to have a different number of software engineers? Probably, right. Like it's to some extent. I don't know if it's going to be more or less, but I think there are many, many opinions right now. Leo, what's that statement you have about overestimating the effects of technology?
Oh, there's this famous quote. Let me see. Yeah, track it down. But there's a famous quote which is like we tend to overestimate the effects of technology in the short term and underestimate it in the long term. Yeah. And so perhaps that's what's happening and that's this kind of stuff that grabs headlines too, right? So I mean, we're still in the throes of this thing where it's very difficult right now to separate pipe from what's actually happening.
I guess I look forward to maybe getting to the point where we have more research backing, because until that it's fun and everything, but it's just people talking about the things that we don't really know the answer until the research gets done. And that it's Amara's Law is the name of it. So Amara's Law is the we tend to underestimate the effects in the long run, but overestimate in the short term.
Yep. So I think, yeah, one thing clear for sure, right, if you rely too much on LLM, I think still we are not there yet, right? So I think in your book you also mentioned it is not an expert, it is actually trained from existing code bases, right. So for example, if you want to solve a new problem, maybe quantum computing, let's say, it may not be able to even give you a proper solution, right. So let's not forget about that.
I think we still need to use our judgment as well as a human to actually apply what LLM is suggesting to us into our software. So maybe one last point I would like to ask. Since you are also part of the university teaching students, right? You mentioned about equity opportunity, probably is last time for people to learn about programming computer science,
right? There are only limited number of people now with this introduction of AI system, potentially more people will be able to get into computer science and learn about programming. Maybe about the syntax problem will soon becomes lesser of a challenge. So what is your take on this? Creating a more equitable kind of a society for people to learn computer science? Yeah. Thanks, Henry. So this is something that Leo and I have been thinking a lot about and we're excited by the
possibilities here. But we don't want to say anything too early because again, we don't know what's going to end up happening. But just to summarize for everybody, the deal is that people who already have prior programming experience they it's unsurprising, but they tend to do better in introductory CS courses.
So if they had more opportunities in high school, for example for you know, their parents had access to some maybe computing or courses or they directed the students into this field, then they tend to perform better. And I wouldn't necessarily have a problem with this except that these opportunities are not evenly distributed and so they're made more accessible to dominant groups.
And so then this gap in prior experience leads to a gap across different types of students, which is obviously not OK. It's what we're hoping, and the research is ongoing or hoping is that because there's a reduced emphasis on syntax using LMS, we're hoping that prior experience, while it will certainly still exist, the gaps will still exist. Perhaps the gaps in prior experience will not lead to the gaps in outcomes that we've been seeing in introductory CS
courses. So again, there are a lot of caveats here. One of them, for example, is that maybe the students with privilege are going to be using LMS earlier than other students, and then they'll have prior experience using LLMS too, and that may conveyed advantage just like a prior programming experience does right now. I guess our hope stems from the fact that learning syntax is so difficult and it's such a barrier for so many students.
And then maybe these LLM skills, maybe the gap can be made smaller more quickly. I want to ask Leo to jump in here too, because this is a question that's definitely worthy of multiple discussion points. Oh, absolutely. I I think you've summarized the issue really well. I think there's a couple of other reasons for optimism and I am being very cautiously
optimistic. As Dan points out, we have to do the research, but I mentioned earlier the kind of status quo of how we assess students in computer science classes and it's solving these really small functions that aren't particularly exciting to be quite frank.
And there's been a whole bunch of work within the community that has shown that students from demographic groups that are currently under representing computing tend to want to see that their work is going to help society, It's going to be for the societal good and they want to see that the computing can
serve that good. I think when we move to LLMS you end up, unless you want to do these outdated assignments that they LM solve for you, you have to move to these kind of open-ended large projects, which is what we were using in our class. And then they can pick the domain that matters to them and then it can be something that's
meaningful to them personally. And I think if you can do that, I think we're going to bring in a broader audience of people who are interested in competing because they see how it matters for them as people. So that'd be the first reason for optimism. And then the second one is 1, where I'm also kind of cautiously optimistic.
And as there's been a whole bunch of research and already started by members of our community, that's really interesting in terms of how can we turn these AI assistants into tutors, essentially intelligent tutoring systems. How could we help through prompt engineering, through really careful crafting of the introductory prompts.
How can we make these make it so when the students struggling, they don't have to wait till the next office hours of instructor, they can just have a quick conversation and they're going to get mostly correct answers. Which is how with LLMS you got to get correct answers, you're going to get encouraging answers ones, they'll encourage them to keep trying. How can we get them the help they need, when they need it is if there's a gap in terms of how much support different groups
need. Making sure everyone has lots of support will help everyone. That will help disadvantage groups more. Yeah, and Leo, It's not impossible that this happens. Like in case people are skeptical out there. Leo and I, of course are disinterested as well. Slash skeptical because we're scientists, but there is precedent for something good to happen here. And Leo specifically, I'm thinking about the way that we teach our classes.
So for example, using student discussion in classes through something called Peer instruction seems to be able to reduce this gap. Yeah, it seems that this techniques like active learning disproportionately help students who are underprivileged, and so it helps everyone kind of boat raises all waters, but the folks who are struggling are raised more when you see a larger impact for those struggling groups. Yeah. And and that's because the new supports are there, right?
Like other students, in the case of peer instruction, it's I think perhaps partially a community aspect. So now they have more students who can help them kind of catch up. And so this is the hope right now. So people are already referring to LL Ms. as like one-on-one tutors. And I'm not willing to go there quite yet. But I think that's the dream, right? Like Leo said, the dream is that they can reduce the time delay between having a question and getting an answer.
Because if we can reduce that to 0, like just imagine that any question a student has could be answered immediately. That bodes well for students to catch up, right? As a lot of the times I think the limiting factor is just resources, right? Like I only have office hours once a week for example. So if a student gets stuck before, maybe they have to wait for me to get them unstuck and maybe they can get unstuck sooner with LLMS and then catch up. So again, this is just kind of
the hope right now. Maybe in a few years we can revisit this and say yes, we were right or no we were not. But for now, it's definitely something we're interested in, I. Think that brings up a really good point of kind of comparison groups, which is where I've kind of shifted my thinking about how are we comparing. And so I'll give you kind of three examples here.
One is, you'll hear folks say we can't change what we're teaching in our introductory courses right now because students are learning the fundamentals and they they'll start kind of hammering on how great the currency is. One class is, but the evidence is indirectly students finishing interview programming class. The majority of them can't find the average of positive numbers in a list. That's like a super easy task for computer scientists, like
for software engineers. And the majority of students can't do that at the end of ACS one. So we we need to make sure that we're very clear about what we're comparing against. What's happening now isn't
successful for everyone. The second reason this is what you made me think of for the tutors, was we've done a whole bunch of research out of my lab, finding that both students and tutors have significant incentives to essentially just give away the answer and just fix the problem for the student right there. Basically act as human debuggers without actually teaching the process.
And so I think when we imagine that human tutor interacting with a student, we imagine the great teacher like Dan, like sitting down and going back to like step one and diagnosing the problem and giving them the right instruction, the right time to address their misconceptions. When the reality is, it's mostly students giving this kind of tutoring help and they're maybe not giving the best instruction.
And so we have to be honest with ourselves about what are these AI assistants being compared against, and then we can actually do a fair comparison, right? I think it's like what you said, right? We can be cautiously optimistic about this kind of equitable future, right. So I think really looking forward for more chances, more opportunities for people. It's been a great conversation so far, right? So I think we will have a lot of more topics if we don't cut it
short for now, right? I have one last question before we wrap up, which normally I ask for all my guests. I call this 3 technical leadership wisdom. You can think of it just like advice as well for people to learn from you. Maybe if you can share your version of three technical leadership wisdom. Henry I I love that question and if you don't mind, I've got a slightly long answer for my first one, and it's I had a really close colleague who is just a fantastic cyclist.
His name is Allen Snavely here at UC San Diego, and he was part of a race and he was a fantastic cyclist and they were in the second pack. Even if all of cycling knows that you're in packs and the first pack is up ahead of them, they can't catch them. But at one point along the race, like the front pack seems to go the wrong direction and Allen pretty quickly realizes, wait, that's actually not the direction to the finish, What are they doing?
And so he steers the second pack towards the finish and it's the one race that he ever got to win because the main back went off in the wrong direction. And whenever he tells that story, I I always get a kick out of it. But it reminds me from a leadership perspective that it's important to be good. It's important to be fast, it's important to be able to be productive, but it's just as important or even more important to know where we're going.
And so I spend a lot of time with my group and with my lab and making sure we have a vision for where we're going. It's that we are going the right direction. Thanks Leo. That's a powerful one, especially for researchers like us to remember. I have another one I think that relates to research too, which is actually even more important now I think for people who are not researchers as well because of the LLM discourse right now.
And that is always test assumptions or always be aware of assumptions that people are making. And I I bring this up specifically now because I think we're at the beginning of this in a flood of research and commentary that's going to come out about LLMS. I mean, obviously this applies to everything, right? Always, you know, take the time to understand where the writer is coming from or where your own assumptions are coming from.
But especially now I just want to caution that people are going to be making sweeping statements about LLMS and Leo. And I read a lot of research around LLMS, and often, you know, if you're a busy researcher, busy professor, you can get some summary of the paper by reading the abstract. Sort of great practice. But if you're very busy, you can get a sense of what the paper's doing. I don't think this necessarily works for LLM papers.
There's so many assumptions that are baked in to the experiments that people are doing right now, We can't even agree on the right skills that we want students to have when they're working with LLMS anymore. And so I think we're seeing a lot of papers that like there are the headlines, like LLMS suck or LLMS are amazing or whatever.
But I think we need to dig beneath the headlines to see exactly what's going on, especially in a new area like LLMS where there are so many assumptions that have not even been written down yet that people might be making. That's a brilliant point, Dan. And we see with the new papers coming in, they're coming in very quickly. And because we were trying in such a race to get the research done, it's really important for us to go to the methods and
actually read the paper fully. And I know, I know you're fantastic at that. And so I I really hope all the other practitioners, all the people teaching programming, do the same thing. Spend their time making sure they understand the studies that have been done. Yeah. And it's not even that anybody who's involved is being deceptive. I think everybody's being super honest about what's happening. But the assumptions, I think, are so new that we're not even
necessarily writing them down. Like if we're not being careful enough, we may be making assumptions about LLMS, like, So for example, I could just think in my head, OK, students still must know syntax. And maybe that's true, Maybe it's not true, but it might be so obvious to me one way or the other that I just might not even take it into account in my research. And this is one of the most dangerous things for researchers, right, Leo?
It's like an assumption that is apparently super obvious that you don't even question it, or even worse, you don't even write it down. And I think we're. As a community, we're at risk of doing this right now because everything is moving so quickly. Exactly. We've been studying how to teach programming for the last 40 years, and so we've so many assumptions built in about that. I think just even the assumption of what is the angle, like, is syntax an end goal of a intro programming class?
We don't know. Like, I think there's going to be a whole bunch of discussion about that. Yeah. Or like, does it make sense to compare what students learn with LLMS against what they learn without LLMS? Like, what do you compare? Right. What's important? Like, I don't think we know the answers to these questions. So I guess I'm asking more questions than I'm answering, which I don't think was what Henry wanted for this section.
Yeah. Yeah. And Leo, you have one more I think you wanted to share. Oh yeah, absolutely. So the last piece, and this is going to be me, honestly kind of just taking from the great wisdom that Henry's already shared previously with some of his guests. And that's I believe everything is done with people. Like if Dan and I work fantastically together, I love working with my lab. And so I believe very fervently in the notion of empowered
teams. And the message from Marty Kagan really resonates with me. I first heard from Monty Hammond Tree at Microsoft and it's just a really powerful message of you want to make sure your teams are empowered to be able to do the work they want to do and solve important problems.
And I think Dan and I have both seen this as PhD advisors in empowering our PhD students to find their own path is probably one of the best things we get to do as faculty of watching them, not really knowing what they want to study initially and us really being close to them on every project they do to six years later, five or six years later when they are now essentially running their own research program and we're just
giving them occasional advice. And so I I think for the tech leaders out there who've been listening to Marty Kagan's message of empowered teams, I think it applies more than just software engineering teams. I think you empower all the people who work with you and you, you end up in a better place. Yeah, it's like tutoring, really. It's like one-on-one work is the really the most powerful work you can do. I'll take my classes of 300 or 400 or whatever.
I'll you know, I'll do my best. But you can't match a small team just empowered to do great work. So I totally agree. So yeah, I think for people who want to learn more about this AI assistant from your book, is there any resources or a place where they can find you online? Yes.
If people are interested in getting our book and trying to learn how to write software with the AI assistant, they can just look for our book on Amazon. It's freely available in all countries and we candidly are very open to feedback. This is a very, very new space. As readers work through it, we would appreciate the emails or the comments on LinkedIn that would let us know how they're appreciating the book and what we could do better for a second edition.
Thanks for organizing this for us, Henry. Thank you for listening to this episode and for staying right until the end. If you highly enjoyed it, I would appreciate if you share it with your friends and colleagues who you think would also benefit from listening to this episode. And if you're new to the podcast, make sure to subscribe and leave me your valuable review and feedback. It helps me a lot in order to grow this podcast better.
You can also find the full show notes of this conversation on the episode page at techlitjournal dot dev website, including the full transcript, interesting quotes, and links to the resources mentioned from the conversation. And lastly, make sure to subscribe to the show's mailing list on techlitjournal dot dev to get notified for any future episodes. Stay tuned for the next Techly Journal episode, and until then, goodbye.