Welcome to the Open Source Way. This is our podcast series, SAP's podcast series about the difference that open source can be. And in each episode, we'll talk with experts about open source and why they do it the open source way. Or in this case, maybe I should add why they did it the open source way. I'm still your host, Karsten Hohage, and in this episode, I'm going to talk to Christoph Roland and Harald Kuck, who ported SAP's software stack to Linux back in the second half of the 19.
Hey, Harald and Christoph, nice to have you here.
Hi, hi.
Hi, Karsten. Nice to be here.
All right. Let's look a little bit closer at who these guys are. Harald Kuck was leading SAP's ABBA platform until, I think, April this year, and now he will soon retire or is in the process of that. He initiated the move to SAP Linux and open source together with Christoph in the late 1990s. And that was definitely, as he says, his highlight or one of the highlights in his 38 years in IT.
Christoph Roland discovered Linux at university in 1993, and he was convinced that Linux had a great future immediately. He joined SAP in 1994, where he started in the support organization. Soon he joined Harald's development team, and there Christoph took care of SAP's network library. Also, he ported the SAP stack to Linux. This is what we're going to be talking about.
And he also founded the SAP Linux Lab, which later on I encountered also due to different reasons, and took on several management roles in SAP. But he left SAP in 2008 and now pursues other things. Christoph, let me start. What are other things?
Other things, yeah. I first took several management roles at Software AG and an Australian company. Later i moment for freelance consulting work how to do large-scale development work and now i'm with software ag as the technical fellow basically doing the same helping them with.
With whom Software AG we have a pretty lively both-way exchange as far as I know, right?
Yes always had that no question about that okay.
That's hopefully why we can also talk about it and mention that name, but this is not what we want to talk about for the next however many minutes. Harald, what was the situation like back in 1999?
It's been a long time since then, so I hardly remember, but SAP back then was mainly R3. This is our sort of flagship product that basically runs all over the world and all the companies. And our R3 was supporting a number of hardware operating system and database platforms. And regarding operating system, it was mainly Unix and Windows. That was just introduced a couple of years ago then, in the mid of the 90s, I think. And some more exotic platforms like Mainframes or Ace 400.
And I remember the first eight or nine years of my SAP life. I joined in 1989, we were developing on C, and the development platform was Unix. And there was a number of Unix flavors from HP and IBM and back then Digital Equipment and Sun and Siemens. And as a C developer, that meant you wanted to support all of these different flavors. There were lots of if-defs in your code, which made the code pretty nasty. And yeah, back then, Linux and open source came up.
So that was sort of once in a lifetime opportunity to move all these different Linux flavors or consolidate them to the one and only. And from Christoph's and my point of view, this should be Linux. And back then, I think Linux and open source were becoming more and more popular. But nobody thought back then that Linux would be fit already for ERP software. Really, companies depend with their life on the stability of the software. We always say it's mission-critical software.
And no one thought that SAP would be the first company to embrace Linux and open source. But then maybe I can hand over to Christoph because the world didn't plan with Christoph that.
Yeah. As I wrote in my bio, I already came to SAP with Linux as the thing which actually brought me back to software development and thought this is a great platform and met a few people at SAP with the same idea. But since I joined Harold's team and working on the low-level parts, that team is actually doing that, I was in the position to re-report it and did that as a side project. I always developed the stuff which I could do on the no-switch.
In the beginning, it was a network stack and a bit of memory management. But we're missing third-party parts, especially the database library. So I was basically preparing and having a working non-application server for two years, but was always saying, oh, I need that database layer. And that happened then in 98, when all the bigger database vendors started to bring out their software on Linux, which was actually a big thing.
I also remember people knew I was doing this, and I remember when one of my colleagues who knew that and was very favorable about what I was doing came to me and said, Oh, you need to be very disappointed because there was an interview with Hasul where he was asked about SAP on Linux, and he said, No, we will not do that, and you have to be very disappointed.
Let me jump in at this point, maybe, because I started my question there with what was the situation in 99. Correct me if I'm wrong, right now we're talking about everything that happened before 99, right?
That's true.
And what actually led to the whole thing. And also, correct me if I'm wrong, everything you're talking about up to here is basically, that was an underground project, right? A submarine, basically.
Yeah, absolutely. It was an absolute grassroots project. Harold, as my boss knew about it, when I joined the team, I told him I need a second hard disk to run Linux on that. And he said, yeah, yeah, no problem, man. But that was basically all the support. I had the SiteTarout support saying, yeah, it's interesting what you're doing. Very fascinating and all that.
Yeah, here's to all the managers out there. Support your employees if they have great ideas like that.
Oh, yeah, definitely.
Oh, yes.
So I was really jealous back then because, as I've said, I was working as a developer still. But that was exactly the time when I became a development manager. And I always had to use one of the mentioned Unix flavors and not Linux because I was working on components that needed the entire full Monty executable and, of course, database client. So I couldn't link my executable without the database client.
And that's why I had to use HP or AIX back then. I think it was even Altrix on digital equipment. And Christoph, I was a bit jealous because he was doing the network library and we had some smaller executables like subrouter and whatever. And he could link his executable on Linux. And I was just using Linux at home and I was a big fan, but it sort of came too late for me. So you were lucky.
Yeah, for me also, it was the last step to go to HPUX and try it there. And it was so much slower and the tooling was so much less modern than on the network. So that was it. But you were right, Karsten, and we were talking right now, I was talking about 98, not 99. So basically how it all came together.
And then 98, the deciding part was when the database client for Linux came out.
Exactly. And when all these vendors basically started to say, yes, we will support our database on Linux and some more reluctantly, some more actively. And yeah, coming back to that anecdote, I was talking about the guy thing. You have to be very down about how so denying that we would do SAP on Linux. I was answering, no, no.
You know, when he says that in six weeks, we will do it. Because that would happen to all the database windows, all these enterprise software, all the first denied, and then they did it. And it took a bit longer than six weeks for us, but it didn't take so much longer. And so as soon as we had the database library, to my surprise, it worked out of the box.
Actually, it's really funny because I just found yesterday all this, you can't see that, but back then I was using this endless printing paper from the mainframes to write a sort of diary for my notes. And I just found all the notes from that time. It's so interesting because Christoph and me, we were sitting there in our room and we were thinking it would be so great if SAP could offer R3 on Linux because we felt, yeah, we would be the first then. and this would be a really big thing.
And so I remember one night we said, yeah, but will customers really like that? And what about partners? So we decided we just scan all the customer support messages for the string Linux because there were lots of customers back then in support messages asking us, hey guys, is there any chance you will support Linux in the near future? And we were just then calling these customers and we got incredible positive feedback.
I can't remember exactly, Christoph, but I think it was the end of 98 where we both decided we should really go for it. And I mean, you ported all this stuff. The R3 was running.
Yeah, that was the end of 98. By the way, about the customer side, there was another one we had, the sub-router. I think Harold was mentioning that. And it was running fully. And we had published that to customers officially via our support website. So we knew about some customers already using Linux. But that was a typical Linux thing at that side for networking pieces and all that. It was already quite established, but not for these heavy-rate data processing.
So am I getting you right here? You talked to these customers first, even with the delivery of the, what did you just say, what was already running on Linux?
The sub-router.
Yeah, the sub-router, exactly. So you pushed that out to customers first. And who was the first one within SAP, you told, that you have a running system on Linux?
I can't really remember. I think it was a small group of people then that were really interested in all these Linux topics. And we might have talked to them about it. But that was only really a matter of a few days. And I think once Christoph and I took the decision, let's really go for it. And I remember Christoph was saying, yeah, we maybe should announce it in CeBIT in a year or something. And then I said, I know, but CeBIT will be in a few months and this is a huge opportunity.
But then Christoph said, oh, God, because, I mean, he was a techie back then and developing. And I was still in this development manager world where you always talk to developers and say, can't we do this quicker? And then Christoph said, yeah, but if we want to announce it on CeBIT, it really has to run stable and we have to do this and this and this and that.
And then we created a long list. But as I've said, first of all, we just collected feedback from the customers and just asked them how they would like it. And I think I didn't get one single negative feedback. And lots of really enthusiastic.
And now we're talking about the CeBIT in the beginning of 99, is that right?
Yeah, that was the plan.
So we're still at the end of 1998. And then I think this diary I was just mentioning said I talked to Hustle the first time on January 28. And I've noted here that... No, no, sorry. 29. 29. January 28, 1999.
That's an even shorter period than I remember. It's incredible.
I've noted here that I made a few notes, what I would like to tell him, and then afterwards, I did a sort of brain dump of the discussion, and I told him, yeah, we must support Linux because, I mean, technically, it's really great and stable and great performance, but most of all, because Linux is open source, and open source is a completely different development model, and this will be the main development model in future anyway, I told him.
And if we support it now we will have a great image thing there which we in the end really had and if we wait now we will in the end do it anyway and then we're just one of the others and then he said yeah but this won't work anyway so how do you think we will do support and everything for open source so then i had to explain open source to him but in the end i convinced him and he said yeah then please let's talk to all
our partners what they think and you can set up a mail and then I will ask my peers at the different companies and the partner companies and that's what he did that yeah just.
A second let's talk about timing here so you talked to hustle in late January if I recall correctly CeBIT was usually when mid February or early March or something.
In April or March I think.
You earlier than April I'm pretty sure it was either in February.
I think we had six weeks, something like that.
Okay. Yeah. So in mid-March, maybe. All right. By the way, CeBIT has ceased to exist for quite a few years now, right?
Yeah. They stopped it.
For those who do not remember it, CeBIT used to be, I don't know, the world's largest, possibly even computer and high-tech fair, which was held in Hanover, Germany. All the world all the big players made their big announcements there and everything just for the young ones out there who are possibly listening it was like the thing and so we're six weeks before that okay so how do these six weeks continue up to CeBIT yeah.
That was pretty crazy because as emerald mentioned there was this requirement we needed to have one of our partners at least one of our partners, who has a Linux and supports Linux, has a Unix and supports Unix, and will support Linux on his hardware. And so I think, especially Harold's boss thought we will never find one. And within a week, we had all of them on board. That was crazy. And then we started, really, we got from them the hardware, especially Compaq at that time was very pushy.
And they flooded me with hardware and manpower to get it running and all that. And we've set up the demo systems really live on these machines to bring it to CeBIT. That was the technical idea. We had to prepare all that. We had to prepare the booth. We had to learn about what CeBIT is, basically. I was never there before. How about you before?
I think as a guest, but not in the way we were there. Oh, yeah.
Definitely not in that way. It was crazy. We were overwhelmed, but it was so much fun that we crammed it into these six weeks we had.
But i i remember after we talked to the customers that's right we were talking to our different partners and to the people we knew there and we're just trying to get a sort of feeling would hp really support that would ibm support that and the good thing was we knew just like christoph said if we just get one of them to support linux on their hardware then this of course created pressure on the others and in the end it was really all of them who told us i mean some more reluctant than others i
think ibm was most enthusiastic and they all told us yeah we're in if you're really doing that then we will support you i.
Guess christoph there was a reason why you said compact flooded you with.
Hardware because.
They never had an own operating system right.
Yeah for them this was such a big chance and they had had a guy boris bialik who was so much a fan of this and he is this kind of can do what i want to do with this if you give him something you will do it and so we were working basically day and night and i was like wow we're suddenly getting these big server machines i have never seen before you know we have never touched before and so installing on them having don't know So at that time, four CPUs was very big,
and they had four CPUs, eight CPUs, memory like hell. Everything I could dream of as a developer and as a technician at that time was fascinating and so much fun.
Now, eight CPUs wouldn't sound that much today anymore, but we have to remind everyone again, we're talking about 1999 here. And here is not to managers this time, but to developers. If you want to be flooded with hardware, just do something really, really cool, right?
I remember all these computers and open cases in our room there. We were exchanging hardware drives.
We drove IT a bit crazy with that.
Okay.
Because we did all of them in some room upstairs, not in the computing center.
So what happened at CeBIT?
First of all, I think what was interesting, Then Hasso just said, yeah, you can go ahead, but he needed this letter that he planned to send to all his partners. And that was one of the happiest moments in my life because I set up a small text and I think he corrected it. But in the end, he wrote a small letter. To the CEOs of basically the entire IT world. It was back then IBM and HP and Siemens and Microsoft, Bill Gates, Intel, Sun, Oracle, Informix, and Compaq.
And he basically wrote, yeah, the feedback we have received from partners regarding R3 on Linux is extremely positive. And now we decided to make Linux our main and one and only development platform. And we assume that you, the partner, will provide the same degree of support for Linux as you already do for your respective Unix operating system. And my favorite line was, should you have any questions, please contact Harald Kuck, client server technology, email HQ at sap.com.
That was really cool. And I had to fax these documents in the middle of the night. And I remember Bill Gates was the last one.
And did Bill Gates contact you with his questions?
No, he didn't. But I heard later that he was asking people, who the heck is Harald Kuck?
Well, at least Bill Gates apparently has said your name. I don't think he's ever said mine.
If rumors are true, yeah.
Okay.
And then I remember that Christoph and me, we were thinking about how can we now spread the message because if we just show up there at CeBIT, nobody would know that there is something like this going on there. And so first of all, internally, we told everyone we did a port on Linux of the R3, but please, this is strictly confidential. Don't tell anyone. So this is the fastest method of spreading news within SAP.
And regarding CeBIT, we thought we had some friends at the EISA ticker and we called one of them. And so they said from informed sources, SAP might be announcing R3 on Linux, on CeBIT. And so people already knew when they came. That was really great.
Okay. And people did come, right?
People did come, yes.
Oh, yes. They did come. We were a bit of an outlier on the really big SAP booths.
And Christoph was really interested in showing off the uptime of his machine there because he was so proud that it was running through and you never had to boot it, other than other operating systems I don't want to mention. And then back then, the German chancellor, Gerhard Schröder, he came with a huge amount of people to our booth and asked some questions. And at one point in time, he looked like he would push some button on Christoph's machine. but Christoph was really defending the thing.
So Gerhard Schröder was not allowed to push anything because this would have spoiled the uptime.
Instead, you brought him a bottle of beer, I assume, right?
That was fantastic. And then in the mornings of SAP, people were always comparing their leads. This is basically after you talk to potential customers, they give you their contact card. And then in the morning, you can say, I have 23 contact cards, and I think Tristof and me, we had hundreds or even 1,000 one day. People just came, left their card, and said, so great, and then they left again.
But this was exhausting. And I think Henning Kattermann said this was for SAP the best announcement so far. That's what he said at an internal event.
Okay. We've talked about, wow, the SAP board's reaction. We've talked about some of the hardware partners' reaction. The customers absolutely loved it, it seems, if you got hundreds of business cards at CeBIT. How about the, also important in the game, the large Linux distributors like Red Hat and SUSE?
Yeah, that was an interesting one and a difficult one, actually, because we had the clear requirement about only one distribution. We can only support one. There was a prerequisite, and at that time, there were a lot of them, but basically the two candidates were Red Hat and Suze. Suze, as a German distribution, of course, thought they would be the prime, but since I had the background more in open source, I definitely said we should go for Red Hat. They have the broader appeal.
They are more open source minded also than Sousa. Sousa was playing around with some proprietary things in there. So we started with Red Hat, definitely. Sousa was pretty upset about it, read some rumors about how this came together, whether that was completely not true. Later on, they came on board too. What was interesting and also a big challenge, you know, we went out beginning of 99, saying, yeah, we had it running and all that.
And after that, we had to realize that certain pieces were missing to run such a big software like SAP R3 and the application servers, really stressing the operating system immensely. So there were two parts to it. One is actually that we were pivotal to create Red Hat Enterprise Linux at that time because we needed certain functionality, which wasn't there. We needed the support around that.
And I had for years, I don't know, 40 golden Red Hat Enterprise Linux CDs in my drawer, which they sent over to us to test it, if it's working and all that. So the original Red Hat Enterprise Linux was especially for SAP and driven by us in the Linux lab. They sent an employee into the lab at that time. and by the way that the guy who's leading the Linux lab since I left it, So that was the interesting piece on the distribution side.
We also had to realize that the Linux kernel had some limitations, which were pretty bad for what we needed to do. And that also led to myself contributing to the Linux kernel in a pretty significant part.
And that was mostly about what?
That was mostly about the shared memory in the Linux kernel. Well, actually, the current implementation of the shared memory in Linux is written by me. The tempFS, which is the basis for containers and also in Android, I just checked my phone, it's mounted that file system 10 times. It was written by me just to make SAP on Linux not only really be able to run smoothly and fully, but also be better than on the other Unix.
This is kind of interesting. I mean, if I get you correctly, there are things like shared memory. So some of the original qualities that are now making Linux the preferred system for the cloud actually have been implemented from SAP side, right?
Yes. Yeah.
And now we've sort of over the years have a bit forgotten that we were kind of heading the technology there as far as that's concerned, right?
Let's say we probably didn't forget. And I actually basically forgot about that. I had to remind myself. So I contributed that. I contributed that first from my SAP address, later from my private address. But I think there is still the copyright. Definitely there is copyright on that from my side and I think also from SAP because I did it from SAP. So you can probably look up the Linux kernel sources and see that and see the birth date of my son in there.
See, Christoph, only because we made such heavy use of shared memory in the SAP kernel back then, you got the opportunity to write this important piece because that was exactly what made Linux Enterprise ready then.
So I think really it was, we pushed both from a credibility point of view and from a technical point of view, Linux very much into the enterprise and there were significant pieces missing, no enterprise Linux support really from the distributors, shared memory, and also other inter-process communication pieces. It's still pretty rough, especially on shared memory. It's still that cold, which is working. And it was actually my most complex, but also most fun development project I ever did.
And as we've talked about these things in other episodes before, was that all ported back to the community Linux, basically? Or was that exclusive to Red Hat?
No, that was not exclusive. That was only in the mainstream.
Okay.
So I was working directly with Linux Torvalds and the other key developers around there. I was actually participating in the first Linux kernel summit where the kernel developers meet. That was, I think, 2000, beginning of 2000, where I joined them. So I met all these guys, which was actually pretty fun. And yes, it's part of the mainstream kernel life.
What I did was just picking still the... I downloaded at some point in time to remind myself the email from the Linux kernel mailing list archive, where I had sent that message and the response from Linux, which was twofold. He liked the general idea, but he hated how I implemented it. He writes, beautiful patch, except for this case. I'm really pleased with how well the POSIX shared memory code seems to integrate into the file system and VM layer. And that makes me happy.
But the implementation makes me cringe.
Was that a quote from Linus Torvald? Yes. All right. Right. Actually, I'd now like to put out a poll, but we don't have a polling function. The poll would run, would you rather meet Linus Torvald or have Bill Gates say your name? Which is kind of summarizing your two very different roles in this, right?
Yeah. I definitely prefer meeting Linus Torvald.
Harald, did you also go along?
Yeah, me too. But I talked to him. Yeah, back then I didn't write any code anymore, but I talked to Linux on one of these Linux days. And I had other fun as well. I remember because all this hype started then and a guy from the SAP press, he told me, yeah, we have to go to Köln to the WDR Computer Club. And they are doing live TV session there. And you're supposed to talk there about Linux and open source.
And I was sitting there next to the German minister for education, Brigitte Zyprius, and a few other folks. And then it was an open discussion and it was live on TV. So I was a bit nervous back then because I was not used to things like that.
That was about the concept of open source in general? What was the discussion about?
That was basically because Linux and open source were getting more and more hyped. and they just wanted to know, hey, we recently heard SAP is now supporting it and what are you doing there and stuff like that.
Okay. That was still in the days when the thinking was, these are some kind of rebels who want to put the communist approach into software, basically, and everything.
Yeah, it was sort of the time where this image changed, but was still there. So everybody said, open source, yeah, but this can't work, and not for our industry, and not for mission-critical software, and there won't be support. And yeah, I remember that we always said that Linux is great, but the main advantage about Linux is the development method, how it's implemented. And we both thought that this is really superior and some people were skeptical, even within SAP.
But in the end, now, if you look today, the role open source plays in Linux in the cloud, it's unbelievable. And even Sadia Nadella, other than Steve Barmer, who was after Bill Gates, he totally embraced that, and that's a smart move.
But that's what you have to see. It was still Bill Gates, and Barmer was then talking about GDL being a cancer and whatever. So the acceptance of that at that point in time, actually people just started to even think about it. For most people in corporate software, they had never thought about it. We had discussions with our legal department about these licenses, and they had to start completely fresh.
There was some really difficult discussions around that, as I said, with me contributing to the Linux kernel. I didn't even talk to anybody at that point in time. They did it. But, yeah, since then, so much has changed. And I think the knowledge around what does it mean, even open source was new before it was always free software and all that.
And I think by now people have implemented enough concepts that open source is not free giveaway anymore and also have understood that it's not free giveaway anymore and have understood that liability issues and everything can be solved by the distributors that you have in between and everything. So now we're fully here. But I mean, totally almost brave of you guys to do something like that in the time where all that was not established. Very cool.
I didn't feel that as brave. I thought it's fun. I thought it's the one thing which I believe. And I wanted to have this done. And I didn't care too much. And how else is the same, I would say. We believe in it. Let's do it.
Yeah. No, I wouldn't say brave. I was more proud to sort of, by more or less coincidence, be part of such a huge thing there. Because in the beginning, it was just R3 and Linux. but then we really felt that we are part of something bigger there. And that felt good.
Yep. And the future, today's presence, proved you more than right, I guess. All right. Then let's maybe look for some final words as we've almost headed for some of them with these last couple of lines. What would you tell today's developers about your experiences from this idea and project later on?
Yeah, I would definitely say what I said before. If you really believe in something, do that. And if you really want to make an impact on something, you need to believe into it.
Yeah, that's exactly right. And I think this is exactly how innovation happens. You can't plan it. You can't say next Tuesday we reserve three hours and then we are thinking about something cool. If something like that happens and there is an opportunity to fill, sort of, then you just have to make sure that you're in an environment where your bosses support you, let you run. And luckily, our bosses were really supportive back then. It was Karl-Heinz Hess and Hasso Plattner. That was our bosses.
And Arad Kork, in your case, yes. That was the main thing. And yeah, this is how innovation happens and still valid today, I would say.
Okay, great. Thanks for these more or less final words then. That was really cool hearing about this. So thanks again, Harald and Christoph.
Thanks, Karsten, for having us.
Thanks for having us. That's what you say on these American shows.
And also thank you all for listening to the Open Source Way. If you enjoyed this episode, please share it. And don't miss the next one. We kind of left our last Wednesday of the month principle but you're probably totally subscribed to our feed and so you will find out on your be that Apple podcast or Spotify or some open source client whatever anyway thanks again and bye bye bye thanks bye.