#222 - Closing the Knowledge Gap in Your Legacy Code with AI - Omer Rosenbaum

The greatest risk with the eye is that if you don't understand what it does, then you really sabotage your own understanding and learning journey. I don't think it makes sense for someone in 2025 to work even the same as they worked in 2024. The danger will be even worse, right, if some of these agents hallucinate and they all hallucinate with each other. At some point it's going to be hard for a human to understand what's going on, and that's where I think they'll have to

stop and think. Will AI actually improve our critical thinking or actually reduce our critical thinking? They do think that people will get burned a lot because they will believe what AI sent them and then they'll learn in the hard way that they shouldn't have. They should resist the urge to send out anything you don't completely understand. Before the people were scared or reluctant to write because they would say OK, no one ever read it. Now AI read is going to read it.

AI is not lazy. AI is going to read it right. AI is a super powerful tool and especially in explaining things to humans and also aligning humans to ask questions in actual language. It's amazing. But we shouldn't expect AI to understand the code. No, AI can look in the code and understand that the way didn't do 5 things that are not implemented because you tried them sometime and you know that it's not going to work I. Think there's some danger here about hallucination.

How much context should you give to AI and? When you provide it with complex code bases, AI makes mistakes. It has some very fundamental limitations and that's where you need other techniques like. Hello everyone, welcome back to another new episode of the Techni Journal podcast. Today I have with me the CTO and Co founder of swim, Swim dot IO right, Omar Rosenbaum. He's here with me today in the

show. So I think we plan to have a lot, a lot of discussions about, you know, learning research, AI, of course, and also other technical leadership wisdom that Omar will share today with us. So, Omar, looking forward for our conversation today. Me too, Henry. Thank you for having me, it's great to be here.

Yeah. Omair, I'd like to invite you maybe share something about yourself first. Any career turning points that you think we can learn from you? Of course. So I started my career and I had an opportunity to be a part of a special technological unit and actually started with a very special training that kind of changed my life. It was 13 weeks throughout which I learned more than I had in all of my years before that this

professionally speaking. And it was really mind blowing, like how much you can learn in such a short amount of time and also the different methodologies. And it really got me into getting interested in teaching or in learning, which also affected a lot of the things I did later. Then I went to the university where I took all kinds of courses in variety of topics including chemistry, math, linguistics, psychology and

others. And while I was a student that we also taught a lot, wrote a few books and started a few training programs, one in Singapore by the way. So yeah. So I got to teach in different places and again think about teaching methodology as well as research methodology. And in 2019 I Co founded SWIM and ever since then I've been the CTO and the Co founder here. Well thank you for sharing your interesting journey. I didn't know that you kind of like started a job first before

you go into university. What do you think is an advantage of starting the job first before you get into uni? Yeah. So first of all, you don't have to start a job per SE. It's just that you basically have a few years of hands, hands on experience, right, which is more similar to a job. And then when you go to the university, either you choose to focus a you study computer science, right? But you come with understanding what this looks like an application in the real world.

And you want, you want to learn about the science or about specific aspects of it like algorithms and so on. And you can also choose to learn other things and broaden your horizons while you already have a lot of knowledge and a firm basis based on the hands on experience you got. So I think it gives the whole learning experience a very different atmosphere. And you can approach it not as like, this is my way to get a job because I already have my skills, right? I'm here to learn.

I'm here to deepen my knowledge to understand things better. I think that's really interesting. OMSO what I think as well, right? If you started kind of like a job first before you go into uni, you kind of like have a good understanding of what kind of roles that you like or don't like. And probably the hands on experience will be also quite relatable, right, when you study. Because sometimes I imagine if we go to uni, we can't even relate to some of the subjects, right?

So we don't even know it's going to be useful or not. And I think a lot of challenges in parts of the world as juniors, right, is actually to understand whether what I'm learning now is going to be usable or relatable with my job

or not. And especially these days, you know, the crazies about AI, right? And there's a lot of fear for juniors about, you know, getting a job because some people think AI could replace the need for any kind of juniors. So maybe let's start our discussion by discussing about this challenge, right? So what do you think juniors should do in the age of AI these days? About getting a good job or

about learning new skills? So first of all, I think juniors need to acknowledge that it's a challenging time for juniors. First of all, because it's unclear. I mean, we can postulate about the impact of AI on junior developers. Different people have different opinions. We don't know them for a fact, right? So clearly it's more challenging. And I think, I mean, as a junior, you can't impact the fact that now AI is changing the

world, right? What you can do is acknowledge that and try to find the opportunities. I think on one hand it's really challenging to justify the need for a junior developer for some tasks. If you think of a junior developer as someone who will do the small tasks right while gradually learning and becoming better, then now you could let some coding assistants do that for you, right? On the other hand, I don't think juniors are going to be extinct from the world, right?

Everyone starts as a junior and we eventually evolved to being more senior and more experienced. Everyone go through that. And I think for junior developers, the important thing is to understand what's important to learn and what is not. For example, I think learning with AI and how to use AI effectively is important and it's one of the advantages they

can get, right? Like if you see, I don't know our, whenever we see our grandparents operate the computer, we feel the difference, right? Even when you see like a 10 year old, he's computer native, right? They started with a computer, they know what it's like. So junior developers can actually start with AI and feel even more comfortable working with AI tools. Perhaps on the other hand, they should be super careful about not understanding things.

I think what differentiates experienced, high qualified and effective engineers or researchers from the rest is the fact they understand deeply how things work. And the greatest risk with the eye is that it seems to do, you know, you're asking it to do something, you get some code back, it kind of does what you wanted it to do. And if you don't understand what it does or you don't understand what the alternative is, then you really sabotage your own understanding and learning journey.

And as a junior, that's the most important thing, right? You need to learn and improve everyday. And also you might introduce critical bugs without knowing that. So I think the urge to just ship something, let's say I got a task, OK, I sent it to one of the AI coding assistants, or I use the completion tool or whatever. I got a result. I can just issue the PR. I think of juniors. You should resist the urge to send out anything you don't

completely understand. On the other hand, there is an amazing opportunity here before I I on something they could get stuck and just they need a senior developer to help them. That's it. Which might be a waste of time of the senior developer. Maybe I don't feel so comfortable as a junior developer to approach a senior to help me and admit that I'm blocked, right? And AI can actually unblock you from lots of things.

It might be a generic thing about, you know, as it's junior developers who might not know how to run some tests, which is something that generic that would take you a long time to learn. But if you ask Claude or GPT or whatever, you'd get an answer really quickly, right?

And also if you select a specific part of the code that is hard for you to understand and you ask AI to walk you through it step by step, it might be an easier, a faster way to learn than the non AI way of having to understand yourself. Or again, if you're stuck asking for help from someone. So it's like having someone with you. I think on in a sense the dream of every junior developer to have a senior developer who doesn't get tired of your questions. Right?

You just need to realize, not really a senior developer. They don't really know the code base. They don't really understand all the pros and cons, right? They're a limited senior developer with some strength and some weaknesses, and your goal is to make sure you understand what's going on. That's the main thing I think of the junior developer you have to do. I think you brought up very good points, right, specifically like

your term. You know I'm going to use it like AI native generation, right. So because when they go into the workforce these days, right, they can use AI. It is simply available, right? And I think looking back at my time back then, right, we probably the best that we had our books, right? Maybe Google Stack Overflow was probably also just starting, right? So like now you have AI natively

that you can use straight away. So I think this gives a lot of opportunity for sure for anyone to upscale and learn about something. And especially if the workplace also allows AI usage, you know, like AI coding assistance, you can also get up to speak to the code base, asking about specific parts of the code that you don't understand without bugging the so-called the senior developers, right?

And I think you brought up a good points about, you know, understanding your code base, even though it's generated by AI before you actually submit or even push it to an environment of production, right? So tell us maybe about this workflow, right? So for example, a junior, you know, specifically has a task and it has to run specific, I don't know, like business logic or something like that, but AI gives them like a code that they don't understand.

What specifically should a junior developer do? As a junior developer, I mean, right, let's start from what you shouldn't do, right? So I gave a specific coding assistant a small task and I wanted to see what it does. And I wanted it to help me save me time. And I told you, OK, write a test, implement the task and keep iterating until it passes the test, right? And then it says, OK, great, everything passes, you can work with it now. Then I look at the code, right?

And I see that it has an if for the specific case of the test and it does something to pass the test specifically, right? So it has a string of like the mock data that I used. So if that returned this OK, the test test great. Right now it's an extreme example, but it happened to me right now, right? I mean, very recently and a junior developer might say, OK, I have it, the test pass and I

will ship it, right? This would create a really bad impression of course, but also think of a less clear example where if you actually understand the code, you understand that it actually solved a very specific case and not the broader scope. Or maybe there are some other considerations, security considerations. Maybe there is something in there which puts a vulnerability, maybe performance considerations like it works, but it wouldn't work on scale.

And in case this code should operating scale, right? And performance is important, you must know that. So I think of the junior developer, you can take the task, explain it. By the way, the fact that AI makes you explaining natural language what the task is, is great. Because whenever I got to teach people how to program, the first thing I told them is explain to yourself. I wouldn't say natural language, right? Just explain to yourself what you're supposed to do and what the steps are.

Now if you write this to AI, it's actually a good practice. And then when you get back a response, you need to go over everything and understand everything and make sure it makes sense in the projects of the task. At which point, if you don't understand something, you can use AI. Ask it. Why did you do this? Is there another way? What are the pros and cons? Right, Keep checking that, but never ship anything you don't completely understand.

I think it would be very challenging for people now. It would require a lot of discipline not to ship something you don't completely understand. Because they're just like, why would I even care, right? People say like, AI will write code, then AI will fix the code. At least for now. It's very far from the truth. And again, for a junior, you need to understand so you can grow. You won't grow by delegating all of your tasks to AI. Yeah, I think that's a very good reminder, I would say, right?

Because sometimes I think even seniors using AI, right, if they think, oh, this code looks OK just by looking at a glance, right? Sometimes a particular bug could just appear and before you realize it makes a, you know, like a production issue or something like that, right? So I think no matter whether you're senior, junior, right, always look at the generated code by AI, right?

And make sure that you understand, again, like the key point that you emphasize is like understand exactly what the code is doing and maybe also the design, right? What certain aspects suggested by the AI, I think that's a very good thing. And I think this brings me to

the next question that I'd like to ask, right? But simply just by understanding and, and maybe asking back, right, or maybe being curious about why certain things are suggested that way, it's actually like a critical thinking kind of a capability, right? And there's a recent research saying that, you know, using AIA lot will actually impact your critical thinking ability. Maybe it will reduce it or even make you less critical. So maybe in your point, what's your view about this?

Will AI actually improve our critical thinking or actually reduce our critical thinking? I think that it's hard to tell, right? I'm not a prophet. I do think there is a risk there, but I do think that people will get burned a lot because they will believe what AI sent them, like the responses they get, and then they'll learn in the hard way that they shouldn't have. So it might make them more willing to check everything they

get as a response. And so I'll be optimistic and say it would help our critical thinking if some junior developers are listening. I would advise you to do that elaborately. Make sure you want to improve on your critical thinking rather than get burned and then iterate from there. Yeah. I think like any kind of technological advancement, right, it makes your life easier, right? And by making life easier, sometimes we get lazy, so to speak, right? So don't forget like you have to

be critical, right? That's the first thing, understanding fundamentals. And I think the other aspect

that is really important, especially for junior, is actually understanding the domain knowledge, right? Maybe it's the business aspects of it, it may be, I don't know, other aspects of the code base that not necessarily just technological. So maybe in your view, what's your take about juniors, you know, upscaling themselves in

domain knowledge? Because yeah, sometimes you can ask AI maybe if it's like generic domain knowledge, but there are specific things that are in organization that probably is not easy for AI to suggest something. Of course. So I think we're raising 2 areas where juniors can grow here. 1 is the technological area where it will also be with you when you switch jobs or switch business domains, right? And it's the fundamental of being an engineer, I think. So it's a must.

But you're also raising another very valid point to AI excels as creating demos from scratch, right? Because it's something generic. It doesn't XLS much. It's getting better. We get other tools that enrich it with context, but AI never knows your company really. It doesn't understand the broader business logic. It wasn't in that meeting where the product manager met with the client, understood the needs, and then on the meeting whether the product manager told you what they need, right.

So I think one of the responsibilities of every engineer, not just junior engineers, would be to communicate to AI and to the code you're writing yourself, the specific business logic, business rules, constraints, all of that context that is unique to your organization. And I think writing it, by the way, is crucial and something we've been neglecting as humanity for various reason.

But I think with AI it can actually be easier because you explain to AI why you did something and then you can hopefully preserve that knowledge alongside the code.

Yeah, so I think writing I've been hearing a lot of times, right, in so many different episodes, like writing skills, very crucial, right. Leaders especially, you can't just lead by talking to people. You have to write more, right? And writing here also means like some kind of knowledge base, right? And I know swim dot IO is, you know, kind of like dealing with knowledge problem, right.

So tell us, how can we actually learn better by, you know, having a knowledge base within the company or the practice itself, right? Having more writing more documentations within the organization. Right. So I think if you have an organization that values written knowledge and specifically documentation, you got lots of benefits before the era of Gen. AI and coding assistance. You would say it's crucial for developers and the communication between them, right?

I think one of the extremist cases I've experienced myself was a crucial real time system that two separate teams worked on. And there was a queue of messages where one team assumed that one was the top priority and 10 was the lowest priority. And the other team assumed exactly the opposite. And they didn't understand why they get random sequences of messages and it's not according to priority, right? So this is just an extreme clear

example of miscommunication. But in general, when you write code and it accumulates over time, what you lose is the business logic context and why you did lots of things. There are things you can't deduce from code. No way I can look in the code and understand that the way didn't do 5 things that are not implemented.

They're not there because you tried them sometime and you know that it's not going to work because of this and that and they don't know that it's a request by a specific client. So when you have such unique knowledge, it's critical to capture it and preserve it. And I think now given that we provide the context to AI coding assistants to help us with the coding tasks, it even more clear that you have clear value from

writing. Before the people were scared or reluctant to write because they would say, OK, no one ever read it. Now AI read is going to read it. AI is not lazy. AI is going to read it right. Your coding assistant will read your docs and use them. And then one of the biggest challenges is actually keeping that knowledge up to date with the code as it evolves.

And this is actually one of the things we solved first in SWIM, even before the era of Gen. AI as it is today when it was commonly used, we started by allowing developers to write documentation and make sure automatically that it's kept up to date with the code as the code evolves. With AI, it's even more crucial because AI can look at a piece of documentation, not know that it's outdated, and rely on it when generating code or other docs or tests and so on.

So this is one of the pillars for having a knowledge base that is comprehensive, describes everything you need to know, especially things that you cannot deduce from the code, and caps up to date as your code evolves and then reachable to both humans you might need it and AI assistance and agents. Wow, I think I left when you mentioned that before we were not so sure anyone would read the documentation, right?

So sometimes we are lazy to write some documentation because we think nobody is gonna read it. But now I think you make a fairpoint, right? The AI will be the first audience, especially if the AI tool has access to your documentations, right?

So I think you have dealt with this kind of challenge, knowledge sharing and also understanding documentations with your company, right? So tell us how maybe some use cases of how AI actually improves this kind of knowledge sharing and also documentation. Because all we know, I mean, we are laymen people, we think, we just feel to AI. AI will summarize, AI will tell you what to do. I think there's some danger here about hallucination.

You know the context, right? How much context should you give to AI? Do you actually also share everything within the organization to AI? So tell us a little bit more about this, because these are the nuances that I think some of us might not understand. I agree. And I think the difference lies in the small details sometimes because when you give AI code piece in isolation and you ask AI please explain what's happening here, it will do an amazing job most of the time.

However, when you talk about knowledge that you might need and you think what kind of knowledge is it? So one type of knowledge is understanding some code in isolation, right? But there are other things. The things that are not in the code cannot be deduced from the code, right? Obviously.

So again, some business logic that is not clearly translated into the code, requirements that stem from some regulation or client requirements, things you ended up not implementing for some reasons, all of that is not something you can deduce from just reading the code. And I think AI is great at writing things in a way that humans can understand, right? So if you provide it with very clear context and all the contact need, then you say, now create, write it in a way that's

easy to understand. Perhaps translate it to another language because not all of us are native English speakers and not everyone wants their documentation or explanations to be in English. Translate that. This is amazing for AI, but we also need to understand and acknowledge the limitations of AI. 1 is not assuming it could understand things that are not in the code, but another is that also code is not always easily understood even by AI.

Just for given a few examples here, if you give AI one function and it's written in a way that is very clear, the flow is linear, or the function name is clear, it's documented, Of course, it's easy to understand, right? You take the same function, you rename it, you remove all the comments, and you change the variable names to be something a bit less clear, which might sound like an exercise in bad coding, but I've seen enough code bases to know that it

happened in many companies. OK, then all of a sudden AI is starting to make some mistakes, one understanding it. And then you get to complex flows. And when you have flows, you have lots of cases that compilers know to take into account and AI doesn't. For example, you have ambiguity resolution. So let's say you have a call to a function called find and you have the find of, I don't know, no JS find. But you also have three different find functions in your

code base and you call find. And if AI picks the wrong implementation of find, it can get the whole flow wrong right? So ambiguity resolution is 1 case. And also if the code involves looking at resources that are external like reading from database that it might not have access to. So in Jordan, I think AI is a super super super powerful tool and especially in explaining things to humans and also aligning humans to ask questions in actual language. It's amazing.

But we shouldn't expect AI to understand the code. It doesn't really understand the code. And when you provide it with complex code bases that are convoluted with mixed conventions, specific domain knowledge that it hasn't been trained on, of course, sometimes even misleading comments that are not up to date with the code variable names that sometimes are cryptic and sometimes are even confusing and misleading. AI makes mistakes.

And I've tested it thoroughly, OK, here at SWIM, we also create documentation automatically from code bases. And one of the first hypothesis was, OK, let's just use an LLM and let's try that. And we tried a lot, OK? And it's made us understand it's an amazing tool, but it has some very fundamental limitations. And that's where you need other techniques like static code analysis and other things.

You can do that. When you join with AI, you get clear coherent documents or other forms or written knowledge that explain in a way that is useful for humans to understand.

Yeah, I think so for some people who have used AI coding assistance a lot, right? Especially working with like a bigger code base, very complex, you know, written by so many developers. I think that's also one thing, right? Because you can see the amount of inconsistency or like not so coherent kind of a code from one as one module to the others, right?

And variable naming as well as some people like to use a certain terms, the others use other terms and they can be duplicates but mean different things. So I think that there's really a big challenge here if you just rely on LLM, right? Because LLM will just take it words by words. And I think you mentioned a very good point about combining it with, you know, maybe like static code analysis or other kind of, I don't know, like compiler's ability or something like that.

Because computers are also good at that, right? Not AI, right? Maybe tell us how do you actually combine these results at like for example any specific study or maybe a customer case that you have solved using these

kind of techniques? I think the most extreme cases we had in that regard and it's also our focus now is actually legacy code bases and more specifically mainframe legacy code bases with COBOL, which is a language that I had never run into before I started working on this problem for those client. And since then we doubled down on COBOL, right. But it's a language where most of the code is not available online or there is almost no real Kobo code available on

online. The code that you have on GitHub doesn't look like the code that companies run. And it stems from multiple reasons, but I think the most important one is that when GitHub launched, no one was writing Kobo in an organization that wanted to publish their code, right? Let's say you start a new project today and you work even not with an AI assistant.

You start with lots of libraries, you start with Python, you have your libraries for Python And frameworks, You start with JavaScript, whatever, right? You have lots of frameworks. And those frameworks are built on open source. When people developed big code bakers in Cobo in the 70s, in the 80s, they didn't have libraries. They had to re imagine everything themselves. So every organization looked

very, very, very different. So LLMS don't have access to real world Kobo code and LLMS don't have access to the specific code of your organization. And in Kobo you have cryptic variable names all the time and the structure is different than other code languages. So it was I think the most extreme we saw. And we clients try to send some code to an LLM and ask, OK, what does he do? You get super generic and confusing and wrong results. But of the times.

And there what we did was we wrote a COBOL parser that actually takes the code, parses it syntactically, and connects the dots together in a way that makes sense for an LLM. So in the end, for example, we have a variable and we want to understand what it does. We take the variable name. The variable name can be reused in lots of different places in the code base. So we find only the occurrences that are related to this occurrence of the variable.

And then we send all of that context to an LLM and we ask for a summary about this variable. So when the LLM gets the right context and only the right context, it does a great job at explaining it in natural language. If you just throw the old code base at it, it starts combining different variables who have the same name. It happens a lot in Koble and you know, I'm not blaming it. It's really hard to parse all that code and understand what really belongs together and what doesn't.

So what we do is first analyze the code base statically. We build our own internal representation of the code, how things relate, what function calls other functions, what variables are used, where hierarchy of say a flow. And then we slowly build the knowledge by sending small bits to an LLM to explain. After we do a lot of work on

cleaning everything. And this is an example I think of combining static analysis or code that analyzes code in a deterministic way, not something probabilistic that an LLM produces right? And then using an LLM for what it does best, which is taking text or specific parts of code and explaining it in natural, coherent language.

At the last phase, we also asked the LLM to generate parts of the documents we show to the user, because again, the LM is great at formulating that in coherent English. That is clear. That explains the story after it already has all the context that we build bit by bit. Wow. I think it's a very novel approach, so to speak, right? And especially you dealt with the most extreme code base available, I think COBOL, I think I also didn't have experience with COBOL, right?

I can only imagine like the difficulty dealing with such legacy code base. And I think you brought up like a very good realization for people because I'm sure a lot of tech leaders or senior executives think, OK, now we have AI, any kind of code base it can understand and explain to us. So we probably don't need to be concerned so much about losing the ability of understanding the code base. And we can even probably hire some maybe less good developers and just use AI to fix all the

problems we have. I think maybe for simple cases, maybe more up to date libraries and programming languages, you can do that. But if you look back, we have some, so many legacy systems, right, written so many years ago, where with the people also leaving the business, knowledge probably is also changing a lot, right? So I think this is 1 task that probably AI would not be able to do. And maybe combining different kind of approach like static code analysis would make AI

works much better. So maybe in terms of knowledge

base, right, with the ability of AI these days, any kind of medium size or maybe large organizations, what will be your advice of doing some practices or maybe cultural things that actually can fit into AI and help you know, the knowledge sharing aspect or knowledge base aspect becomes much more effective and maybe even like a multiplier effect within the organization, right? Any kind of practice and cultural things that you can share?

Yeah. So I think the first thing is to acknowledge that it's really, really important and for all the reasons we said before, right? Especially now AI will read the knowledge base, right? So you want to invest in it and it's going to be a multiplier effect, as you said, Henry, I think specifically you want to create a culture as a leader that values people who capture knowledge. Some organizations used to look at these people as wasting their

time. We're doing the easy task, but just explain what the reason instead of creating. I don't think it's a viable argument anymore. I didn't think it was a viable argument back then, but let's say it's arguable. I think now it's not a viable argument at all. It's clear that if you document what's happening, it will help AI accomplish anything

afterwards. The other thing is that you should put effort into finding the tools that will help you with the task of writing this comprehensive knowledge base documents with keeping it up to date as the code evolves, because otherwise you just have misleading and wrong information. And 3rd, to be able to find that information again, both humans and AI should be able to find the information they need when they need it.

So you should invest in those tools to help you with that task and also create a culture that values the knowledge creation, preservation and sharing. Well, I like the the emphasis you put into like valuing people who actually do the so-called. I would say it's a hard job actually to actually capture knowledge, distillate, summarize it for other people to understand. It's actually not an easy job.

I would say it's maybe becoming more valuable now because you can fit it into AI as a context and everyone can benefit just by one simple writing, right? And maybe it can be reused multiple times. So I think the other challenge

about, you know, knowledge based documentation and all that, keeping it up to date, right? Any documentation that you have within an organization, I'm sure most of them are still not up to date, maybe even wrong when you read it again. So tell us maybe some good practice that we can do to actually make it up to date. Right. So I think there are two kind of ways to approach it.

And we at Swim spent a lot of time working on this specific problem, so very emotionally attached to it I would say. I would say that nowadays there are basically 2 approaches. 1 is regenerate the documentation every time you generate documentation automatically just regenerate it all the time. If you regenerate now, it will match the codes state right now.

I think it makes sense. In some cases, for example for API documentation, it could make sense, but what you're going to lose if you regenerate every time is additional context that is not there in the code, and you need a way to preserve that, because that could be the most important piece of knowledge

that is written there. So another approach is to somehow track the changes made to the code that is referenced in specific documents and then update those parts of the documents based on the changes and maybe ask for a human intervention in case the code change drastically, for example. So this is actually something we provide with SWIM.

When you create a document with SWIM, we track the changes made to the code you relied on in the document and we either automatically update the document or if the change is drastic, we tell you as a human, please decide what you want to do from now. If you want to reselect this part of the code, if you want to rewrite it, maybe this part is no longer relevant, maybe you need to add some unique

information. But the goal here is to understand that a lot of the unique knowledge that only developers have in their minds is what you need to work so hard to preserve. Therefore you can just rely on AI generating documents. Yeah.

So I think looking in the past, right, when I have difficulties finding knowledge, right, maybe sometimes the knowledge is there, but we just don't know where to find, right? I think that's the one thing and keeping it up to date, right? Because sometimes I find this piece of documentation, I read it, well, if we assume it's correct, but it's wrong, right? It's also quite dangerous, right? And I think we have also multiple tools within the organization, which is in like silo, right?

So for example, some information maybe in our tracking, ticketing, tracking system, right? Maybe let's say Nigeria, some Confluence, some in Slack, some in e-mail. How do you actually build these kind of linkages, references and again like probing people to actually, hey, these parts of the knowledge base is not up to date. I think this is like a real world challenge if we can solve it right.

Actually, AI has a great promise in the sense of accumulating all of this information from across the organization, right from, say, Jira, Slack, documentation tools and others. And I think it's suddenly possible to just ask a question and get a response from various resources. The key here is to understand that some of these resources are more historic references than provide actual up to date information, which is also sometimes valuable, right?

Like the Jira ticket can tell you what a product manager wanted you to accomplish at some point, right? At least most of the times it won't tell you what's actually happening right now, right? But you have a code documentation software that actually explains what happens in the code and keeps it up to date, then you can relate on it.

So I think for AI coding assistance or AI tools that help you find information from across your organization, they should always explain what resources they're using to formulate the responses and perhaps mark some of those messages or snippets of knowledge as how likely they are to be up to date. Yeah. So I think it's very challenging right in the 1st place accumulating.

So I think what provided that we can give AI the tools capability to maybe like, I don't know like crawl our knowledge base and you know, get the contacts and all that.

I think we we all know one danger of AILLM is actually the hallucination part, right? You mentioned about providing references and all that, but assuming that we have like large knowledge base, how do you actually ensure that it is not

hallucinating? Because sometimes the hallucination could happen in a very small part of the summary that it generates, Especially now these days, we have like a deep research tool where it can do on its own in hours, whatever that is and provide you a summary. But the always the challenge is like, how do you Fact Check it, right? How do you know which part is hallucinating? Or maybe they provide statistics. How do you know it's actually

correct statistics? So do you have any experience in, you know, doing this fact checking and preventing hallucination to actually make your decisions wrong? Yeah. So we worked a lot on it when working on SWIM, when we generate documents to make sure they reflect the accurate state of the code. And for that we do lots of things.

But I think the most interesting part in terms of what the end user can get from it is that when we generate the documents, you see for everything we write what we relied on to provide that information. We show you that it's grounded in this part of the code or that part of the document. And you can validate it yourself.

And in addition to, of course, making sure we do everything we can to avoid hallucinations by all kinds of techniques that are available to eliminate or at least decrease hallucinations. It can always happen, right? There is some hallucination. And as the end user, you can never tell if the LM made a mistake or maybe the other providers made a mistake.

So I think you should trust tools that show you what they relied on. And if you want to incorporate a tool into your organization that gives you answers based on questions, based on your own knowledge base, you should enforce the fact that they give you resources or citations to everything they output. Yeah, I think it's pretty dangerous if you do not have the citations, you know the

references, right? And even if these days you do have the citation, sometimes I from my experience, right, they give you the citation, but sometimes the summary itself can still hallucinate a little bit. So I think that's very interesting experience as well. Yeah, I agree.

The the other thing these days, people are crazy talking about, you know, this agentic capability of AI and also maybe the MCP protocol, right? Maybe tell us the next evolution. Do you think you can see using AI? You know, with all these cool things, and especially in the context of documentation and all the sharing, is there anything that you can see up and coming? Well, I think MCP will be a game changer in the sense that you will see lots of information being fed all the time.

And I think the IT will help create a flying wheel effect where when you put the effort into generating valuable documents or a knowledge base, then all of the AI assistant would be able to reach it, find the relevant information and make use of it. And then we would get some mind

blowing things, right? You can have an AI assistant that analyzes your Jira tickets and provides a summary and all of a sudden it knows what's happening in the code base and it gets the broader contact from a document that was written partially by AI and partially by a human. And I think that's what we want to get, right. But to get there, we need to make sure we create these explicit knowledge fragments alongside the way and also that

we rely on them. So we guide say the AI coding assistant to rely on specific resources. Yeah, I think that providing all these bits of information, again, coming back to what you said, right, the value of writing or you know, capturing the knowledge I think will become a key. And then the next part is actually to expose that, right? Maybe in a genetic manner, right, using this MCP protocol, right?

I'm actually really excited about this MCP capability, especially when doing coding, right? You can communicate with different tools and ask it to do certain things just by natural language. That can be really super powerful, right? And I think I can be certain that once we see more and more agentic capability may be provided by different companies and tools, we can see this multiplier effect.

Although the danger, it would be even worse, right? Because if let's say some of these agents hallucinate and they all hallucinate with each other, like we probably lose track of what kind of things they used to deduce the decision, right? Any take on this from you? Now, just this week I saw a friend of mine posted that he used Claude and I think it was cursor and cursor ran RM minus RF and deleted lots of his valuable information by mistake,

of course. And you know, it's, it's like a funny example, but those things will happen. So I think if we go back to the beginning of our discussion about junior developers taking code that they don't fully understand and committing it to the code base, right, If you have an agent doing that, you have to somehow constraint it and validate the output that it generates. And I think of the next step this week with an announcement

of A to a like agent to agent. So we're talking about new protocols for agents communicating with other agents. And at some point, it's going to be hard for a human to understand what's going on. And that's where I think we'll have to stop and think, right, like what's actually happening here? Where should we have the human in the loop? And we're not. I think it's going to be really exciting times in that sense.

Yeah, I think stop and think will be a point in time where we all realize, OK, we probably hallucinate ourselves thinking, yeah, we'll solve a lot of problems. So yeah, probably one day we will have to build guardrails, you know, constraints such that AI won't lead us to like a dangerous, right. So I think these days people

talk about AI. I'm sure every team, every organization also want to integrate AI somehow, build capability, you know, build something on top of AI model LLM, whatever that is. And for that, they need to do some kind of research, right? Some companies have their capabilities, but most of the companies, they don't have this

knowledge and capability, right? And doing research is partly something that, you know, some organization find it challenging, maybe finding the time, finding the resource. And you brought up a good point before our discussion thing here, but you know, as a product company or maybe it's like a business organization, if you want to do research, what's the best way to approach it, right? Maybe you can share a little bit so that people who want to build capability by doing research can

do it more effectively. Sure. So I think first of all, we need to kind of define what research is. Engineering organizations are usually called R&D, right? So it's research and development and the research piece comes first. But I think in most teams, there is no pure research. And that's fine, right? Usually have the research as in like problem solving and you need to find the best way to do something and you need to learn alongside the way.

That's fine. That's all fine, but it's part of development, right? And I think where I draw the line is if you know the task is achievable and you know the approach, the right approach to get there, then it's development. It's research. When you have a task, you're not sure if it's possible or you're sure it's possible, but you really don't know how to get there because there are so many different options and it's unclear. That's where it's research.

So for example, if I have to, it could be even a really hard development task, of course, right? Let's say I would want to implement, I don't know, VS Code from scratch, right? There are lots of things I don't know I would need to learn along the way. I would need to design the architecture. I'll have to work hard on it, right?

But it's all development. I know it's I know what the output looks like and I know that it would involve a lot of engineering, whereas research and say I have a Cobalt code based and I need to generate useful documents. I'm not even sure what documents are useful at first, right? I need to learn that and then I need to find different ways. Should I go with generative AI all the way? Should I go with static analysis? Combine them where? And that's more for research.

So I think when you want to work on a product and research is something that is blocking, for example, I don't know if it's possible to achieve this, right? And the the difference between development in the research is time estimation in development. I mean, it's notoriously hard to give real time estimations. But usually when developers say this will take me a week, it won't take a year, right? Usually like there is something between the estimate and how long it takes.

With research, you can sometimes just don't know, right? Like, I don't know, maybe it will take me two days because there is an easy win and maybe I'll get into a wall which would be much harder to pass. So I think it's something to acknowledge. And when you have a research that is guided by a product, it means you have some problem you want to solve and you need to do a few things.

One is to lay the entire flow from beginning to end, even though you can't solve all the intermediate stages. For example, in the example, if I take a couple repository and I need to generate useful documents automatically, the first thing I would do is take a couple repository to play with and generate the documents by hand manually for myself and get feedback on them. Are these documents really valuable? Is this where I'm heading to? Right. Once I know that's what I want,

say OK, what do I need to do? So I need to say parse the couple repository and I need to find say a few components, for example. OK, I don't know how to find the right components. So for now, I'll wrap that in a box and I'll keep going to the next step. Now that I know what components there are, how do I document the component? And I actually write that on a white board with boxes and I keep them closed. I don't want to open the boxes now.

I want to make sure I can understand what the process would be and then before I open a box, because the most the thing that makes what you would want to do is open an interesting box, peek inside, try to solve it right? But it might be irrelevant. So you need to first make sure you can achieve everything. If everything works, you say, I assume all the boxes work. Will this work? Yes. OK, why don't I know now there is this specific box. I'm not sure it's possible.

I don't know if an LLM can read a couple program and describe it. OK. And then the key thing, and this is I think the crucial thing when managing research is to deliberately pause and think about the different directions together, because every researcher will do the research themselves, right? Let's say you give someone a task of understanding what a

program does. OK, they will read the code and try to understand what's happening, but you can help them by stopping and thinking what their best technique is. When I LED a cybersecurity course, we taught reverse engineering and one of the exercises we would give what we would give as part of this course, right? So the students would reverse engineer more and more applications and they, they would get a game. And the question was, what are the rules of the game?

And after an hour, we would stop them and we would show how to approach it correctly, which is you open the game, you click on help explain, and you have a textual description of the instructions, right? And the lesson learned is you don't always have to reverse engineering by reading through the code, right? And what we wanted to teach them is that before you jump into one way of solving the problem, stop and consider different solutions.

So what I usually do when I work with people on research tasks is draw it as kind of a tree, like, OK, we're here. How can we solve it? We have option 1-2 and three. OK, we don't have a time estimate because we don't know what we'll find out. If the LLM can just read an entire code base and give us great documentation, OK, we're done here. OK, let's give it a day and see what happens. And I usually call it time to

leave. Like for how long are we going to work on this before we stop and re discuss what we found out and whether we should keep pursuing this specific direction or change to another direction of the research? And one of the most important things is to make everyone stop and think about the various ways to approach a problem. Sometimes the easy solution is there, but you need to think about it. Sometimes it's just clicking help and you have the solution. You don't have to read through

the code, right? And they have many, many, many examples for that. Also not from courses, right? From real life where people, you know, in retrospect, they say, oh, right, we should have done this, right. And when you work on a product, you don't have all the time in the world to just do research. You have to make sure you can provide a product to a user in a timely manner. So to summarize, I think 1 crucial thing is to understand

the time estimates is hard. What you can do is give it time to leave. How long am I willing to spend on it before I stop and re evaluate? The second thing is to make sure we get the end to end process from the input to the end output that the user sees or that the other product takes into account as an input, and so on.

And the third part is pausing and thinking together about the different ways to approach a research task, because again, what characterizes research tasks is that it's unclear how to make progress. So we need to stop and think together on how to approach this. Things so many good Nuggets I would say, right, because doing research by itself is kind of like unpredictable, right So like you mentioned, right, you don't know the time estimation

required, the effort required. Sometimes it could be easy, right if let's say some you know, one day you find oh, there's a library that you can use, but most of the times you know like you don't have the skills you have to gather a lot of knowledge, maybe ask expertise and things like that. So I think you have given some good things. I would just call out a few things that I could remember, right? So the first is try to workflows the direction that you're going into, right?

Because sometimes we can go into rabbit hole easily, especially playing with technologies, right? So techies, we all love playing with technologies. So we keep digging and digging, but maybe we go to the wrong direction. The time to live, I think it's also very crucial, right? You can't spend all the time just doing research that goes nowhere. And I think many people find it difficult to juggle or even

like, for example, justify the value of doing research. And we all know these days, especially again, like bringing up the point of the age of AI, right? If you don't do research about capability of AI that could help your business, maybe you will lose out in the next, I don't know, the time span is can be really short these days, right?

So how can maybe business leaders or maybe executives have this in mind, right, to spend some time doing research even though it's difficult, unpredictable, maybe cannot justify the profits and the revenue coming out of the research, Maybe can you give us some examples here, Right, so I think. On a personal level, say if you're a CTOI think one of your responsibilities is to know how technology can empower your business, and it is also

internally right? So as a very recent example, you're a CTO, you keep yourself informed with what's happening. You know that there are great AI coding assistants that can help people become more effective. You go to your engineering teams and you introduce them to that tool and you help them adjust or adopt new tools. Some people will always be a bit wary of trying new things, right?

And I think as leaders, one of our responsibilities is to show them, look how easy it is, look how useful it could be. So this is like on a personal level and how to incorporate new tools or methodologies or techniques to the organization. So another great way to drive change is by giving talks. So get your company team grouped together and give a live demo of using the cool new tool, for example. So that's more about incorporating new methodologies,

techniques, and tools. When we talk about deep research, I think it's not for every organization. I'm not going to say that every organization needs a research person or team, right? But if you do, you need technology takes time and a different mindset. You can't expect a research team to operate the same way as a development team with a clear timeline for every milestone.

So you can adjust people there, assign them to research tasks, Make sure that the value for the product is clear and easy to get, relatively easy at least, and get those people to understand what research means. Make them be professionals at managing research, at assigning time to leave to different directions and having brainstorms about what the best way to approach a specific issue is. If you don't have this expertise, it's fine. Consult with others who do. There are people who are

experienced researchers. They work differently from people who are not experienced researchers. It's the same with engineers and anything else, right? But research is a skill. It's a skill that people can improve at. It's a skill you can learn and you can get help from others if you don't have the experience. Well, I think that those are really, really great. Advice, right? I particularly like about the aspect of changing your mindset, right?

Because people think doing research is straightforward, right? You do research, you get something out of it, and you can use this straight away. Especially people think that, OK, now with AI you can even get more intelligence, right? You can speed up the research or whatever that is, right? But I think especially doing something that you are not capable of in terms of capability within the organization, it's something tricky, right? You can't sometimes justify the

effort. So hopefully people today learn a lot of things you know about knowledge based research, using AI for documentations and all that. So Amir, as we reach the end of our conversation, I have one

last question that I'd like to ask you, which I asked to all my guests. I call this the tree technical leadership is them if you can think of it just like an advice to us, what advice do you want to give to us today? OK, so I. Think the first one would be put your time and effort into your people. I mean they make all the difference right? And it means talking with them about how they are and what can help them and making sure you have them grow in their position. Finding them is hard.

Leading people is sometimes hard, but I think it's also the most rewarding part of the leader's job. The second thing is, in this era, you must be open minded to try new things. I don't think it makes sense for someone in 2025 to work even the same as they worked in 2024. And it sounds almost childish, right? I used to make fun of people saying things like that, but nowadays it doesn't make sense. Things change really fast and in order not to stay behind, you have to be on top of it.

So you have to be open minded and keep yourself informed. And the third thing, in case it's viable for your organization, put the time into research because it can open up new directions and it can help you in ways that you haven't dreamt of before. Sometimes spending two days on a research task can make you change your decisions completely, so I think allocate the time in case it's relevant for your business case. Yeah, specifically about being

open. Mind, I myself also quite concerned, you know, with some of my habits and skills that I learned in the past, right? Whether that it can still be relevant, especially the pace of changes these days is so rapid, right? Every day you would probably hear, oh, there's a new way of doing things. There's new tool that can help you do what being open minded and willing to try and willing to be challenged.

I think it's also another thing that I feel sometimes as a senior, right, we think we know the problem well, we can solve it by heart, but sometimes there are new ways of doing things these days. So thank you so much for sharing those wisdom. So if people want to connect with you or you know, asking you more things, is there a place where they can find you online? Sure.

So you can reach out to me via. E-mail it's Omer Omer at swim that's SWIM m.io also on LinkedIn, though I don't really use social media as much so if you send me a message and I don't get back, I apologize but probably didn't see it. I do answer to emails and I'll be happy to stay in touch. Thank you again, Omar, for spending the time today. I. Think we all learn a lot about using AI, building knowledge base and doing research as what

you advised just now. So thank you again, my pleasure. Thank you for having me, Henry.