Hello, everybody. Welcome back to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ben Popper, Director of Content here at Stack Overflow. Joined by my co-host, editor of our blog, Ryan Donovan. Ryan, your voice is a little scratchy today. It's a little rough today, yeah. But it's on the mend. I'm on the upswing.
Okay. Apologies to our listeners. Ryan's beautiful baritone is not quite with us, but he's on the mend. So rest assured, things will get better from here. So Ryan, I think both you and I have a lot of experience. discussing thinking about and in some ways creating documentation you were a technical writer for a long time you wrote api docs and ever since i got to stack overflow i've been thinking about these ideas because
Stack Overflow for Teams is kind of a way of doing documentation inside of a company. One of the first pieces I ever wrote for the company was called Documentation Must Die. And it was about just like, there's a better way to do it naturally in a question and answer format when a question arises, not asking somebody to write an article out of context.
And at that time, Joel Spolsky was still with the company and I got to interview him for that. And he was sort of the way he put it was like, imagine you're building a big, a big building, a big apartment complex. The better the blueprints are that you leave behind, the easier it will be for somebody to fix the plumbing in the future or address a structural issue. And the worse the blueprints are, the harder it's going to be. So think about documentation that way.
But there was a great piece that was in our newsletter last week. Cassidy Williams put it in there. And it was all about documentation sort of as a form of engineering. elevating our respect for this craft and doing it in ways that align with some of the workflow and the tools of engineering so that it can be kind of a...
integral part of just creating a company with a healthy code base and a great developer experience. So I want to bring on Fabrizio Benedetti, the author of the piece and a documentation engineer.
to discuss some of his thoughts on the topic. Fabricio, welcome to Stack Overflow Podcast. Hi there. Thank you for having me here. So folks who are listening, tell them a little about how did you get into the world of software and development and what led you to this kind of unique title that you have today? Well, I majored in psychology something like 20 years ago, but I was always drawn to tech and software. And I landed my first job as a content editor for a software download portal.
That was 15 years ago. And since then, I've been like a writer in tech, in the sense of handling different types of content deliverables, coming up with content strategies. And as of late, I've been gravitating more and more towards developer documentation because it really brings together two things that I love. One is programming and the other is writing. So I get to do both things.
as a documentation engineer. I haven't seen this job title that much, though it's increasingly common. But yeah, really what I do is technical writing with a stronger technical accent. Yeah, I did a lot of developer docs as well. It's been interesting to see the evolution of documentation as like, you know, you put it in a help file, include it with a product, you know, done.
where you're putting Markdown files in every Git repo with every service. So it's part of the actual code. What do you think is driving that trend? Oh, you mean like docs as code in the sense of having the documentation together with the code? Yeah. Yeah, I think it's a very practical, it's a sociological thing, really. the whole Docs as Code movement, or as an idea if you want, was born out of the need of getting really the developers to participate more in the documentation process.
So instead of asking developers to enter a CMS or using tools that we're in love with their liking, what happens is that technical writers or documentation engineers, as we fancy ourselves sometimes, we are getting into the repos where the code is living and we are kind of growing the documentation out of that code together with the code. And we are using the same tools as developers are using.
the same methodology, and we are also asking them to apply the same quality checks and have the same care they dedicate to code when they craft documentation. In that sense, technical writers, we are always... There are always few. I haven't seen a company where there are too many technical writers. So we end up acting as editors-in-chief of the documentation. And yeah, as I say, my post.
Documentation is infrastructure, first and foremost, because it's what enables you to use the product without being the product. you know like a server or any other infrastructure so you mentioned right like can you apply some of the same
quality checks to documentation as you can to code. I guess the question would be, how do you write, you know, unit tests for documentation? Like what, you know, what, what is the way to determine if the documentation is good? You know, if the analogy is to writing code. So there's many things you can do there. The simplest, of course, is if you're authoring docs using Markdown or any other markup language, you can limit the Markdown itself for things like...
broken links, formatting, maybe you're using heading levels in the wrong way. That's the basics, the very minimum. Then there are tools out there called prose linters, which is fascinating. It's a great idea, really. But essentially, you can link the pros of your documentation as if you were linking the code of your application. You define these rules. For example, I'm a big fan of the Veil linter.
It's open source. You define these rules in YAML file with endpoints that says something like, if these regular expression patterns exist, suggest these other patterns or these other words. and uses natural language processing also to do that. This really helps with lowering the burden of technical writers and also the burden of engineers because they can even use those tools in the IDEs.
before the writer actually gets to edit anything. And lastly, I would say you can test co-snippets. There are several interesting solutions out there, proof of concepts for testing the co-snippets, see if they build. or against your code base, for example, or if they have an issue. Those are the three main dimensions I would say you could apply testing to docs. When you say unit testing for docs, I always think of getting somebody to just go through it.
follow every step and see if they get to the end result. But I wonder with the Gen AI, will people just feed the LLM the docs and say, go through this and build something? If you run into any problems, stop. You can do that. It's a bit expensive, I would say, because, you know, the compute power required for that. But I've been testing recently the computer use feature of Claude AI.
It's fantastic. I was giving this doc and I was telling Claude to pretend to be an engineer and see if they could find some Java documentation for something. And Cloud AI just opened up, fired up this browser, Firefox, and it was taking screenshots along the way and trying to explain each step. So there's some interesting applications.
coming from that area in terms of artificial user testing. Yeah. So you also mentioned in the piece this idea that Git is really important within software and that it allows folks to ensure... changes made in one place are reflected in another, that people come to consensus when making those changes, and that software that's updated doesn't have the tech debt that might come with versioning issues.
How do you approach that for documentation? Because I do think a lot of people, when they think of documentation and they think of wikis, the first thing they think of is stale knowledge that's out of date and therefore isn't helping them. and creating a frustration or a bad taste in their mouth when it comes to docs. Yeah. Well, there are several ways you can go about it. I think, first of all, you'll want to automate as much reference documentation as possible.
It's those pieces that are trickier to follow up and detect and see if you need to update them, like small parameters, say, in an API endpoint documentation. So if you can automate part or all of those docs, that's already a win because it's documentation that will change together with the code. There are ways of doing that, like generating OpenAPI from the documentation, for example.
Recently, I've seen some promising things like type spec, where you can define API's behavior in a simpler way rather than writing YAML files. And then I would say there's really the agreement, and it's like a...
kind of an unwritten thing, but you can reach this work agreement within the team where documentation should really be part of the definition of done. And as you create, for example, the UI of the feature, or you tend to the infrastructure side of things when you release a new feature, the documentation should be within that stream from the very beginning, not as an end product.
Yeah, I also like the idea of docs as ops you had in there. You know, as technical writers, we're sort of trying to anticipate the questions and problems that people can have, but that doesn't always happen, right? Yeah. Somebody has... an incident and there's nothing there for it and they find out the answer on the fly is there have you found there's a way to sort of capture that information for the next time it happens you mean like what sort of situation for example like
if there's you know somebody runs into a mystery 500 server error and they go in and investigate the problem and they're like oh this the setting was not set correctly or whatever um you deleted a file Have you found there's a way to make sure that information is in the next version of the docs? Well, there's a bit of art there. For example, I'm a big fan of
Slack notifications based on keywords, for example. I'm in every channel here at work and I try to capture any mention of documentation, docs, errors, whatever, anything that matters to the docs.
It's a lot of noise, of course, eventually. I've seen some interesting solutions that I would like to try out there, like one I think is called Guru. Essentially, what they do is they scoop all the information from all the potential... sources of knowledge within the company like jira a wiki slack etc and they turned out into you know nuggets of knowledge that then you can organize and you can reuse
So I think it's still a problem to be solved, that one, and not just for documentation, but more in general for sharing knowledge within a company. Yeah. I mean, what you're talking about gets to a really interesting question that... I think is top of mind for us at Stack Overflow, which is where are we headed with enterprise search and...
Are we really moving towards a solution where you can ask a system to, like you said, ingest knowledge from Jira and Confluence and GitHub and all these other places, and it's able to make sense of it to the degree that... When you then want to go ask a question, it can provide an answer. And so then it's like, well, then what is documentation then? Because all of this has been...
ingested, automated, and made into knowledge artifacts. I don't think we're at that point, obviously, but just hypothetically speaking, what is the role of the documentation engineer in a world where Gen AI is that capable? Well, for one... There is the role of quality control. The documentation engineer wants to make sure that the content that comes out of any such system is valid, that complies with whatever style guides or standards are out there.
In that sense, it's not that different from what a developer augmented through AI would do in the sense of, you know, you can ask an LLM to come up with a snippet. or cook some boilerplate code for an issue, at least at the initial stage. But then is the human, is the developer who is liable, who has to review and check that... that code is going to work and it's safe, etc. So the human is also becoming an editor. You can have LLMs doing lots of manual tasks, lots of boring work even.
but the human is like coordinating and orchestrating the pieces there yeah i mean i think that's that was an interesting thing that you said there the the documentation person becoming an editor um and you mentioned you know all your slack keyword things i remember i did a survey of the engineering org at my last job to be like where do you go for information and it was slack about 60 of the time it was slack and i have this feeling of you know we can create the best docs
How do you get them there? Have you found a way that you can lead your engineers to the documentation? You have to play with incentives a bit. For example,
One, of course, is that my existence means that I'm going to take some work off their shoulders. So for that, they are really grateful. And it's a nice experience to have as a technical writer, of course. But it's not enough. It's not enough. So you have to... see the documentation as something that you have to sell it in a way internally, as something that will remove, for one, problems after launch, for example, or reduce the support burden.
or help sell the feature so that the feature will perform better, which is usually also a metric that developers care about. So documentation might have several goals there. We really have to understand what piques the interest of the developer in each case, or at an organization or team level. Yeah, that's super interesting, the idea that you can lead a horse to water, but they may just ask in Slack.
because that's what they're used to. But if you tie it back to actual metrics and KPIs that maybe are associated with the work that they're doing, then they have this much bigger incentive to... interact with the documentation and then you get this positive feedback loop of oh that did help me or you know this answer wasn't quite right or we changed this and this needs to be updated
Yeah, I think a lot of what you get to is sort of the shift of awareness of internal documentation. For most of the developer documentation I wrote, my career was... for customers of that product, for people who bought an SDK or whatever. But now companies are thinking that, you know, our code is complex. We don't necessarily have the ability to understand everything.
Do you think there's a sort of technical shift driving that or is it just an awareness filtering down to everybody? It's interesting that you mentioned internal documentation because... So to start, I think the divide between internal and external documentation is more virtual than real. And we should treat internal documentation with the same amount of care we treat external documentation.
First, because we want the organization to be productive. Second, because internal documentation might become external documentation, and it happens. Internal API endpoints suddenly get exposed, and what you have is the internal docs. so you better you know you better take care of those but then the issue with complexity i mean that happens pretty much everywhere it happened the previous company where i wore that where after 10 years you know you amass this amount of knowledge
and it becomes increasingly unwieldy to update and maintain. There's no secret solution there, except that you cannot really reduce the amount of resources you dedicate to docs. Quite on the opposite. I've seen many companies... reducing their technical writing teams, the documentation teams, or the resources or the budget dedicated to documentation tooling. And I think that's a mistake. Of course, it's not about throwing money blindly at things.
But one of the things I argue with my blog posts, with my blog in general, I'm trying to advocate for more awareness that documentation matters and that it requires people and resources. Just out of curiosity, you mentioned before, one way to get folks thinking about this might be to...
connect it back to a product being released faster or a product being more performant? Are there ways that you try to measure that, that you try to be able to say, look, I can show you the ROI of the documentation because... When this happened, this happened. Do you try to set up a system to do that? Yeah, there's this concept I came up with a couple months ago, which is documentation observability.
I've been documenting observability solutions for IT and software development, so the concept was already familiar. And I was thinking, why don't we apply observability principles to documentation as well?
So currently, on the current job at Tinyberg, what we are setting up is product analytics throughout the product and documentation. And we are hoping, we're still at the early stages, but we are hoping to see the kind of interaction i'd like to see which is much more meaningful than web analytics for example picture a user using an api endpoint getting an error the error payload contains a url to the docs right
The user follow the link. You're tracking the session of the user. The user consumes the docs, goes back to the API, corrects the call, and he gets a 200. So there's already a journey there. And this might happen in CLI, might happen in UIs. There's already a journey there that path through the docs as if they were part of the product experience. And you can track success rates there, for example, or conversion or whatever.
is considering the docs part of the product journey and part of the product. That way you're measuring it in the same trace, so to speak. I know we touched on Gen AI a little bit. One of the things I hear about Gen AI being good for it would be documentation as sort of generating automated documentation. Do you have any worries of that? Either that. It's going to push technical writers out either rightfully or unjustly or that it's going to just create worse documentation.
technical writers are wondering about that my answer is that you know sure sure let's start using writer bot let's you know let's start creating documentation using ei because some documentation is better than none What companies relying solely on artificial intelligence will realize is that that documentation is not always scary. Actually, it's quite bad, usually. And we require editing. We require...
a human there to, again, do quality checks and improve the documentation after generation. So eventually it's going to happen that... companies would realize that just automated documentation is not the answer. What I recommend is that technical writers and documentaires in general should not feel afraid of LLMs and generative docs.
they should really think of the strategies to apply it at their jobs. So if you're talking, for example, of code snippets, that's something where Gen AI can be used with, I think, greater success than complete documentation. So it's constant strategy, right? So technical writers decide, okay, we have this pen here, and it's enclosed, and it's a code snippet. This is where I want to unleash GNI, but not the entire doc.
But that decision should come from whoever decides on documentation strategy. Great. All right, everybody, it is that time of the show. Let's find someone on Stack Overflow who came on and shared a little knowledge or a little bit of curiosity and in doing so helped the community. How to touch a file, an MK directory if needed in one line. Thanks to Srihar, you were awarded this badge because your answer was so good.
Way more upvotes than the accepted answer. And we've helped over 25,000 people. So appreciate the knowledge. As always, I am Ben Popper. I'm the director of content here at Stack Overflow. Find me on X at Ben Popper. If you want to come on the show or you want to hear us discuss something, email us, podcast at Stack Overflow. And if you like what you heard today, why don't you subscribe and get more in the future?
I'm Ryan Donovan. I edit the blog here at Stack Overflow. If you have hot takes that you want to share with us, please reach out to me on LinkedIn. I'm Fabrizio Ferri Benedetti, documentation engineer at Tiny Bird. My blog is paso.uno if you want to learn more about my documentation thoughts. And I strongly invite you to check out tinybird.co, our data platform solution. Thank you.
Awesome. All right, everybody, we'll put those links in the show notes for you to check out. Thanks for listening and we will talk to you soon.