*Live* Tis the Season for SSIS

In this special holiday themed episode, we're diving into a topic that's as classic as Christmas Carols, but just as divisive as fruitcake. And that topic is the future of SQL Server Integration Services, SSIS. But wait, there's a twist. This episode was recorded live, so if you notice a different vibe, some festive banter, and maybe even a change in our usual musical interludes, that's why. Think of it as the holiday party version of our usual data driven discussions.

Together, we'll explore why SSIS, despite its vintage status, remains a cornerstone of data engineering and why dismissing it might just be a data driven mistake. So grab your cocoa, settle in by the fire or your nearest CPU, and let's get festive with some data talk. Well, hello, and welcome to franksworld.comstream. And, with me today is Andy, and I'm looking for the lower third that has us both. There we go. Frank and Andy Frank Lavinia and Andy Leonard, host of Data

Driven, which I might turn this into a podcast. I might take the audio and and turn it into a podcast. What do you think about that? That'd be good kind of festive stream and also kind of up to date on things. And it gives me some more time that to put together another episode that I had a really great conversation with a guy who does red teaming for LLMs. Nice. So which I think is a growth industry and certainly a wise career move. Speaking of career moves. Good thing.

Oh, we have a first comment. SQL dev d b a. Hey, SQL dev. Awesome. So this is this may be the first time we've done this. This feature's been around for a while. No. We did it once or twice before. Did we do it on recent? Like, months. Yeah. That we've done. So we're sharing our so Frank's audience, people that are connected to Frank, they're seeing this. People connected to me are seeing this. It's like it'll because it told me Frank started this, and, then he sent me

the link. And as I joined in, it it said, hey. You can share this with with your on your channels as well. So I was like, oh, yeah. Click that. Oh, you know what it is? We did it the other way. You were the main, and then I shared it on my channels. That's what happened. That's what happened. Yeah. Yeah. Well, it's cool, though. If you've never met me before hello? That's Frank. Frank digs data on the socials and, franksworld.com, datadriven.tv, which hopefully you know about that,

and impactquantum.com. So that's me. And, yeah. So back to the segue. Yeah. I was talking about how security and AI is a good career move. And we were talking about, speaking of career moves, 'tis the season for SSIS is the title of the stream. And this kind of goes, I'm sorry. Come on, man. It's fun. Right? It is. It's awesome. So so and I had kind of done, 2 livestreams on this already, but one of them for, like, 10 minutes, I didn't

catch the fact that I had no audio. And then yesterday, I did one for 2 minutes, so I didn't catch the fact that I didn't do the audio. So I figured I'd bring the troublemaker himself onto here. Although, strictly speaking, you're not the original troublemaker on this. Well, I participated in it. I'll I'll own my my part of the trouble. You'll own your part of the trouble. So so I definitely will. Yeah. What's the background here? Well and and I'll I'll do a plug for, for andylehner.blog.

And if you go there, you can sign up for my newsletter over on the right side. It's kinda hard to read because the widget is a little narrower than it needs to be. But if you if you do that or if you just look up engineer of data, I think it's engineer of data dot substack.com. But I I put a newsletter out today kinda talking about it. Yeah.

There's the site. Thanks, Frank. No problem. And, you see the subscribe to my newsletter down there on the right, and there's a box on the left where you type your email address and then on the right, you click it's free. And it'll it should take you, right over to Subsec, which by the way, I started using this year. And so far, I'm pretty impressed. It's it's been a a really interesting, experience for me. So the trouble here here's, here's where the

trouble happened. I I have been, reading. I caught a couple of articles just every now here and then, mostly on LinkedIn, where people would express an opinion about, you know, SSIS stinks. I don't like it. It's old. It's was so much trouble. And, you know, and they would just kind of kind of poo poo share their their negative thoughts about Azure sorry. About SSIS. And I've, of course, I've worked in SSIS since really before it came out, I got to work on that Rocks book project

with Brian Knight and I remember that book. Yeah. Yeah. 10 of us. Yes. Back when Rocks would put your picture on the cover of the book. And have a copy around here somewhere. Yeah. That yeah. Thank you, Frank. You know, it just but it's so I got yes. I got kind of a boost out of my career, and I did an awful lot in SSIS for a long time. And every now and then, I still do. I used to deliver training, as part of solid, solid quality learning is what it was called when I joined it. Solid

queue. After that, I worked with them for a few years and I delivered training developed by Eric Veerman and also did consulting gigs. And I learned a lot, about both data engineering and SSIS while I was doing both those things. When I left solid q, I think I put about a year or 2 between me and, you know, and the business. Actually, it was about two and a half years because I went to work for Unisys then as a ETL architect. I remember

that. You're up in Reston quite a bit because that's where it was. Oh, yeah. Yeah. Frank. Now an apartment complex now, that building. Oh, is it? Okay. I think so. Yeah. Okay. So Frank and I have been friends since the before times, even before SSIS came out. And, Well, no. I think you had just written the book at the time. I I'm trying to remember. So Just moved to Richmond just when I met. It was November of 2005. December 2005.

Yeah. Yeah. November 2005 is when we met. And, another mutual friend that I won't name, but we're all still friends now. And the book actually was published in January, I think, of 2006. Yes. That's right. So it wasn't it wasn't quite ready for for prime time. But oh, sorry. The the book wasn't out. It was going through the process, and it takes a couple of months from the from the time all of the drafts are finished until they they make a book out of it. It was my very first,

book project. And, yeah, I I'm pretty sure I was I was so excited. I was telling everybody, I worked on a book. Oh, yeah. Yeah. Because it was for the Richmond Code Camp, which was in May, April of Yep. 2006. Yeah. 2000 Yeah. It was 2006. You and I. Where I did A team. A team. Developers on a plane, and I had the guy I photoshopped the guy carrying your book. That's right. I do remember that. Yeah. I have to find that picture somewhere. I've been I've been using

SSIS for a a long time. I would say I learned more about data engineering, the field and did more projects probably in, in data warehousing where I used SSIS for for the data engineering, data integration. I think it's important to to to, 1, explain for those who may not know what SSIS is, and 2, explain that data engineering was not always seen as a discrete, profession or or Yeah. It's a data engineering's a

relatively new word to describe what we do. It was called the part of business intelligence. Back even before all that, I think the first term I heard was data acquisition, and it was in it was sometimes that was that phrase was used standalone. The most often, at the time when I and this is what got me into databases was doing system control and data acquisition or SCADA systems. These were manufacturing systems where you collected data from

instruments on the floor. You gotta remember, IoT was, you know, still somebody's dream back, you know, in the 19 nineties. IoT. It was just OT back then. It just was OT. You're right. It's funny. Yeah. But but we still did acquire, data from, plant floors and instruments that were mounted all over, but they weren't Internet enabled at that time. They were, most of them were hardwired. A few were using wireless. And so that's kinda what led me

into this whole this whole field. And the idea of the field, is of data engineering, data integration as we called it back then, is we do that data acquisition part. We go find wherever the data lives, we go find it there. And sometimes the data is a very static list. It it could be even a text document, created in notepad that is tab separated or, you know, delimited by character position or something like that. And a lot of old old lookups, lookup data was that way. And I'm not making

that up. It was maintained in a a text EDI. EDI. Yeah. So electronic data interchange. Yeah. So, yeah, EDI is I have an interesting stories about EDI, but but one of the things that really kept me away from the data space for a long time was I didn't wanna be DBA. And this work, I think, had traditionally been kind of merged with DBAs. Oh, absolutely. But at some point, I don't know exactly when it really evolved into its own discipline. And I remember.

Go ahead. Because I remember I tried to get you a job at a particular company. I remember that. And what do they do? And what was it? Why do we need a DBA? You don't need a DBA. Right. And I think that I'm not DBA. That was the funny part. Well, that was the fun. Well, we clearly did because at

the time there was a project going on, and I think the term data architect is what you just said. You were I'm not a DBM data architect. And then that fell on deaf ears. And, ironically, like, not like a couple months later, there was a project that we worked on that, so many stories, and I'm just trying to protect the innocent and the guilty, and myself, from from from libels. But, basically, there was a project going on that when it was basically kind of

behavioral analysis of social networks. Right? This is before Facebook. I think Myspace was around that sort of thing. But it was basically the idea of organizational networking as a discipline. And it turned out that the the software that we bought off the shelf would actually query the database, bring everything back in from the database, and then run through the filtering on the C Sharp components on the web server. Gotcha. So long story short, there was 0 optimization, hardly an

index. I mean, it was just a mess. A data architect will use the terms of the day, would have slot spotted this right away. We didn't. And it was just a massive disaster. And it's kind of one of those things where there were a number of projects that that company was taking on. Basically, one of their one of their core business models was was brilliant actually was software maintenance. So you have an existing application offshore or outsource it outsource it to

us, and we'll take care of it for you. And, you know, it was really like an an an education in kind of Jenga programming. Right? Where you had they wanted updates to this stuff, but they didn't wanna pay to redo it. So you kinda, like, had to replace rip and replace stuff. And there's one particular instance where there was a SQL query that took like 14 minutes to bring back an answer. And I'm like, it's only like like 30,000 records. Like, what what's the deal here?

And turns out there was no indexes, no nothing. Well, you know, those indexes take up space. Right? Exactly. Exactly. I mean, this is like why you should save space. Joke. That's a joke. For one reason or the other, like, there was there was no index. And I was like, well, let's add indexes. And like, no, no, no. We can't change the schema. Okay. So what I end up what I end up doing was creating temporary tables with indexes and then copying all the data, and I still got it down to 2 minutes.

Nice. Which 1 minute and 59 seconds was copying the data, and then one second was actually changing. Yeah. So, like, it was it was kind of like what I call Jenga programming or Jenga architecture. You had to like they wanted updates, couldn't touch too much, couldn't change anything, couldn't improve anything because it was just it was a time in my career that I think back of and I've kind of learned many lessons, both hard lessons and soft skill

lessons. But Sure. But we digress. But, I'm just gonna I'm just gonna take that answering your question in my usual long winded way. SQL Server Integration Services, came along, and it was probably the, again, it was the thing that spanned the longest part of my career. Before that, I worked with something called data mirror. That was the first, I'd I'd say the the first system like that. First bit of software that

way. Before that, I was writing my own. So I was reading from these plant networks and writing to all sorts of stuff. And I got into SQL Server because I crashed access back in the nineties. So I ran, I collected a 1,000 points of data every second for a long weekend. And I wanna say the access file grew to about 4 gigs. When I went to open it and start doing some analysis on it, it turned out it wouldn't open. So 4 gigs is nothing now. Right? You

can do that on a smartwatch. But back then, a server struggled, to open the file system. If you go back far enough, would have freaked out or anything over certain size unless it was, like, NTFS or something like that. Right? Yeah. And this this wasn't. This was, one of the other OSs. But so, you know, I went I went to, altavista.digital.com and typed in Microsoft database, and I saw this listing for something called SQL Server, and that's how it all started.

Well, then I I got got in as, working on a data warehouse, and part of my job moved into the database part of it. I actually was hired to do the reporting piece of it, and lots of cool lessons learned there as well. But on the database side, they use Data Mirror. I think that company is still around. I'm not sure. But this is like 25 years ago. And it was it was so cool, and I was fascinated that somebody had built software to orchestrate this collection of data. I was like, wow.

That is a good idea. You know, it always makes me feel better, Frank, when smart people come up with an idea that I've also come up with independently. It makes me feel like, okay. Maybe I'm onto something. Go through all of that, data transformation services or DTS, and then finally, SSIS and this big block. And what I've noticed and I kinda noticed this trend started maybe 4 or 5 years ago. I people complained about SSIS before that. Don't get me wrong. And a lot of it is

because are you sitting down? It's hard. We're not making it up. Comparatively, though, like, I I remember when I was at barnesandnoble.com and which just goes back a ways. So if you bought a magazine at Barnes and Noble between 1996 and probably about 2012, 13, you you interacted with the system I wrote, nice in the late nineties or at least part of it anyway. So, you

know, that's how I learned EDI, right? Because we get these feeds from publishers, literally a mainframe would dial up another mainframe, download the file over a modem. And, and this is how it worked. And what we did was we pulled down the raw EDI files and I parsed it and I had to do that and drop it into an informix database. So it was a cool writing for GL scripts to to to take that data in text format and then dump it into an actual. You were doing data engineering. I was

doing data engineering, which is kind of funny. But like, you know, data engineering as a discipline is not easy. Right? So SSIS being hard. I mean, you know, brain surgery brain surgery is hard too. Right? You you make a good point about it. And it, you know, it took me a while, especially teaching it. And I would do 4 or 5 day course, originally with solid q and then eventually on my own. I I wrote my own course. I found myself adding to Eric's content when I would deliver the

material here. And don't get me wrong. Eric is still a genius. He was then and he still is. I just I I had a way of approaching some, demos and examples that I felt kinda added to the clarity of the information we were sharing. I kind of expanded that out and wrote all my own material, my own I use my own data, that I collect as part of my, weather station here. And to this day, there are students that are going through, recordings of that class.

The last recordings I made were back in December 2020, and I recorded 3 courses on SSIS. The 4 day from 0 to SSIS course was, you know, will take you from if you can spell SSIS to being a functional, advanced beginner, low end intermediate developer. And it was built for that. It's got labs 13 12, 13 labs that you do in 2 days, of that course. And then it talks it kinda changes gears and goes to the care and feeding of SSIS and ancillary topics. So

I learned a ton about the concepts of data engineering on while as while doing SSIS training and consulting and development. So when I teach it, Frank, I share these concepts that I learned. Because you gotta keep in mind, this all came out around the same time as a data warehouse toolkit book, by,

Kimball and his crew. And the in fact, I don't know what the relationship was between Microsoft and Kimbell, but I do know from the horse's mouth that the, data flow task in SSIS was modeled to load, Kimball data warehouses. There's just a lot of functionality baked right in that, you know, targets those star schemas, and, you know, it's it's built to do that. There's so, you know, there

was that aspect of it. So at the same time, I'm reading and learning and, you know, and then going out and teaching and, you know, and and consulting. There's this nice amalgam going on. I'm getting information from books. I'm applying that information on consulting gigs. I'm figuring out new ways to solve, you know, problems I hadn't seen before, And then I'm training. So I'm just rolling all that together. When I do the training, I'm sharing with people, hey. Here's

some first principles, if you will Right. Of data engineering. And we call it data integration and BI back then. And star schemas and why you use them and how they work and, you know, kind of the trade offs that you get. Data explodes a little bit. Talking about concepts like staging, data, the benefits of it, why you like, how you would wanna build your staging tables. If you're reading from a flat file, everything in that file is text. Now the text may be

numbers. It may be dates, but it's really just text. So you built the stage tables with and bar charts. So at, you know, stuff like that because you wanna get in and get out just quickly. Memory than the way to do it would be in memory and then, like, do validation as you do the insert and things like that. There's a there's a 100 different ways to slice that. Yeah. There really are. But, you know, when you did, that was that was just pieces and parts of

saying, okay. You know, Tim, I'm teaching you how to use this mechanism, if you will. Right. SSIS. But I'm also sharing with you how you would use it and then why you would use it that way. And, you know, so there's more to it than just the data engineering. And the point I wanted to make thanks, Hector. Merry Christmas to you too, Hector. The data engineering all by itself, just that world, that's hard all by itself. Yeah. Absolutely. And then the tool itself was extremely

flexible. And, you know, from the years that you and I have been sharing about stuff, anytime you say it's it's flexible, you're also saying, the the it's a sonic way of saying it's complex. Right. And if it wasn't flexible, people would say that it's too simple. And, like, it's just one of those things where now that I'm in a job where I am in a on the product group, what Microsoft would call PG, a product group or or

team. Yeah. We call it a BU. I I understand. Like, there's only so many hours in a day that you have engineers and there's time to market. You have to kind of make these trade offs. And, you know. That's it. I mean, that's that that I mean, I had this real eye opening moment with with I think suspect was the guy who introduced us, who was an evangelist at Microsoft back in the day. And, you know, I wanted some new shiny feature in Visual Studio 2005. And, you know, I was complaining about it.

And he kind of pointed out like, look, even Microsoft has limited resources in terms of people, time and testing and material and things like that. And I was like, you know, I mean, my god, if Microsoft has that problem, then I guess everyone has that problem. You know? It turns out they're just a bunch of software developers just like the rest of us. Turns out they're all humans. Although maybe now it's mostly AI. Who knows? But, it's getting there. So so so, you know, I think we both kind

of set the stage for the controversy here. Right. SSI has been around for at least 20 years, maybe 25 and SQL Server itself. Let's let's remind folks the history of SQL Server. It was originally who was it a partnership with? Sybase? Yes. I believe it was a Sybase product, completely. And I don't know if it was like And it was like version 6. Got into the mix, and there was a collaboration or something, and then they ended up with it,

owning it. That's my best guess on it. I actually I I know I haven't spoken to her in a while, but I was I'm friends with and and have co worked with, with Caitlin Delaney. And she was with Sybase. Oh, okay. Yep. So, you know and did we have her as a guest on the show? I know we wanted to. We we totally need to because that would be Yeah. Interesting story because I first heard of SQL Server when I was at Barnes and Noble because at the time we were ready to launch in 19 this is why I

left Barnes and Noble. We're ready to launch by Christmas of 96 with a Yeah. Linux or Unix based based system based on Spark, Oracle, and a few other things. No. I'm sorry. 4 g l. It was Ultimate Formics. And, you know, we had the hardware. We had

everything set up. And then as the story goes, Bill Gates and, one of the Riggio brothers who was the CEOs of kind of co CEOs of Barnes and Noble at the time. Bill Gates had kind of I don't know what he'd done, Jedi Mind Trick. In August, September of, like, 96, basically said, no, we're ripping everything we've built so far and we're moving it over to Microsoft tooling, which at the time was not really mature. I mean, it was this is like inter dev. I think we had a beta version

of visual inter dev. Yeah. Yeah. Which was not the best product at the time. Right? It was, you know You know, I used it At the time. At the time. Yeah. I I used it, and if you came from, like, cold fusion or some other development platform. Yes. Was also awful. Yes. But yeah. So So I started on inter dev. In fact, that was the first tool that I I remember downloading for, Visual Studio. I don't think I downloaded it. I think I went somewhere and bought a

CD or something. Yeah. Yeah. I think I found it in her dev 97 CD, which was the the second or third version. But, I mean, I we we had everything written in per on CGI Pearl scripts. Like, we had everything, and it was just a very different era. But my take was and this was my I was at the meeting with the CEO and everyone else. Like, if we don't launch by this Christmas, people are not going to use us as a habit. Amazon will

take the mindshare and this and that. And then then the CEO said, sit down, s t f u. Basically, you don't know how to sell books. You may know technology, but you don't know how to sell books. Now we can look back at Jeff Bezos' super yacht and his, you know, moon missions and all that. These guys have super yachts and moon missions. Right? They do not, actually. Oh. And my well, I mean, I'm pretty sure they live in an oceanfront thing in Long Island. But,

he didn't know anything about selling books online either. So I can kinda I can sit back here, you know, some, you know, good God almost 30 years later and kind of be smug about it. Right? Right. But it's just it's just funny. Right? Like, so so what's interesting is and I think this really cuts to the bone of what this controversy is. And I have the thing queued up. I can kind of show the screen where you posted it, where the fundamentals haven't really changed. Not at all.

Right. Yeah. Binary is still binary. The debates about schema optimization and things like that are still very much the same today as they were 20 years now. The numbers are bigger. The stakes are arguably bigger. But for the most part, the fundamentals haven't changed. And and I would say this is really where it kind of boiled down to. And this is this is where the controversy starts. So buckle up, kids. Let's see. I will share the screen. There's actually

2. I think you talked about one of them. The choices? I'm only aware of 1. This is the this is the one post, and I'll drop you the link, to to one of the others. Right? Yeah. I'll put it in a chat. I'll I'll send that to you here. Just a second. Along can can understand. So this is what I saw. And it was basically Kendra Little, who was a I would say legendary. Scary smart. She's legendary in in in in the sequel kind of family, right? Hashtag sequel family. Is that still a thing? I

think so. She's legendary. She used to work at Redgate. I think she worked at Microsoft, too, at a time. I think so. But I'm not positive. Well, we can look at LinkedIn. If only we had that information. But anyway, so you basically so if you read this and she says so it says strong disagree. Don't run after every shiny thing. Again, that is good advice. But, Lord, I would assume that is her saying. But Lord, don't learn SQL Server and SSIS if you want to be a data engineer. That's 2 decades too

out of date. Sincerely, a SQL Server expert. I think that's

a bit harsh. She's right about this part. Don't try to change chase out there if you show anything. So apparently, I can't, and I can't select a thing. So I read that, and and I know there's more controversies that are in there as I as I look at the thing. And you said I humbly submit data engineering may be accomplished even in the year of our Lord, 2024, using T SQL, this foul year of our

Lord, 2024. To borrow a phrase from Hunter Thompson, T SQL, SIS, ADF, Fabric Data Factory and other technologies supported by Microsoft, which I thought clearly Microsoft's not going anywhere. Right? Yeah. And so I basically said fundamentals never grow out of style. Then I think I wrote again somewhere like when I looked at the context of it because that's not what you're supposed to do apparently in social media. You're supposed to react right away. I did that, by the way, Frank.

I'm guilty. I did not go look at the context. So this is the original context. I well and, you know, you pointed that out and I'll I'll be honest, I I'm still running on second hand information. I have not yet clicked it and gone back to, to our guest post. Now I can see it. Now you can see. So so this is what struck me is, well. This is what struck me as odd. And I know we had talked about it and I had talked about it. You talked about it. We talked to each other about

it. You know, we talked to our dogs about it. I don't know. Like, but like, it was kind of like so so when I read the thing, it gets even stranger. Right? So Yeah. He was talking to someone, and I guess strictly speaking, even this is secondhand knowledge. Right? But, so that's the data scientist in there. Like, well, strictly speaking, this data is also all right. So so look looking to someone to get a job as a data engineer. Okay?

Right. Unfortunately, he was learning about LLMs and other ML stuff. I'm like, that's not data engineering. That's a AI engineering or data science type work. That's more like I think he's he's trying to set him straight from that. He's like, you're learning the wrong things. That's how I read those two sentences. I mean, I would say you're learning the right things if you wanna be an AI practitioner.

Yeah. But I wouldn't call I wouldn't, you know, read up on Langchain, you know, Ollama and anything LLM and all that stuff and then call myself a data engineer. I mean, that's Yeah. That's like a cardiologist cutting up you know, doing your taxes. You know what I mean? Like Sure. Or or cutting open your brain. Like, I mean, I suppose there's some similarities, but it's not the same. Well, I I do like bullet number 1. Yeah. You know, let's see that. Yeah. This is something I think

that you point out quite a bit. So when you give your talks, either on, SSIS or ADF, you ask people, like, how many people here have workloads running in the in the cloud or right? And then only a quarter of the hands go up. Well, it's it grew to about 40% the last time I did it, but it's been over a year since I since I spoke live and asked that question, ran that little survey. There's a slide usually hidden in, all of my presentations that has survey up near the very

top. Right. You know, it just and that's that's what the survey is about. And often, especially say the last, I said it's been over a year. So let's say from a year ago and then back maybe 4 years of asking that question. Almost every time I did that and people didn't see everyone else's hand go up with theirs, the those people would come up to me at the end. And usually, their first comment was, I didn't know that it was most of the people here were not doing

production jobs in the cloud at this point point with data. I thought we were way behind and we're the only ones. And my response would be 2 fold. The first would be, that's because Microsoft marketing is doing an astounding job. That is not a swipe at Microsoft Marketing. If anything, they deserve a raise because they were so effective at communicating how cool this is Right. And how these larger

companies are doing it. You all of the big shows, keynotes, There's some list of big companies, and they're almost all of them or companies that you'd wanna work for because it's prestigious. That's so I don't know if you want it on my personal market. Seem like everybody's doing it. And I I know I know for a fact it's not always true because when I worked in the sales for Microsoft, we would encounter them and there was a pejorative term used internally called server

huggers. Okay. Right. Because like, oh, they're server huggers. They'll never go to Azure. Right. So so now, you know, I used to see that it's server hugger as a pejorative. Now in light of kind of maturity and working, with more customers and being more aligned in the open source kind of realm and dealing with international customers who have very real regulatory concerns. You're right. Call them smart. Right. It's not, you know, I didn't so much drink the Kool Aid is I became one with

the Kool Aid. You couldn't tell where I ended and where it began, where I kind of had this deep programing experience of. Yeah. That's not always the answer. Right. And I think that dealing with LLMs and AI and things like that, I think really makes that more obvious. Right? Yeah. I totally agree with that. And, you know, to be fair, and I wanna start with, you know, with being as positive as I can about this. If I was It's not a negative on any

from scratch. Wasn't. No. I I'm just saying. But if I'm starting today, day 1, and I wanna go, be a starter company and and work with data, I it would be foolish. Foolish to start today and not go to the cloud. Absolutely. So and and the reasons are numerous. Yeah. Here's the

thing. The companies there are a handful of companies, really large companies, mind you, that have started sent in the cloud age. Let's just call it that, or the Internet age. There's a small number of them that have gone on to be huge, but they are really huge. They're overpowering, oversized. They're larger than the companies that are previous to the Internet age companies that have made their way into the Internet. And that's that's

not an accident. However, those companies, the brick and mortar companies, are the companies calling consultants like me and asking me to help them either transition from a purely on premises environment, managing their data into a cloud environment or the and back before that, in 20 years ago, when I was first getting called to do this kind of work, they were just trying to figure out how to collect their data and then analyze it. And so, you know, SSIS was a great way to do that. T

SQL was everywhere. Azure Data Factory didn't exist. Yes. Much less Fabric Data Factory. And so we were just trying to solve this business problem. And I was trying to couch couch my responses, especially there was a thread that that got combative, I would say. And, you know, as we went went down through that, and I kept trying to say, and I did. I said over and over again that, you know, my job is to go help solve these business problems.

And what I meant by that opening line, that T SQL, SSIS, Azure Data Factory, Fabric Data Factory, even in 2024 of viable ways to accomplish data engineering. I I meant that, and I'm not back backing off that for one minute. I I misunderstood the context of the question, and I didn't really understand until I listened to your stream last night where you had gone back and done what I should have done and read the original post. And you said, yeah. It's kind of a mixed mesh

post. The guy's talking about data engineer, but he's also talking about LLMs and machine learning. And in the middle of that, he throws out, you know, this comment about SSIS, how 90 99.5% of the companies are still using. I think that estimate is high. I I think it was more of a, let's make this point that there's still a lot of companies out there using, T SQL and SSIS to accomplish this. And this is something that I can't find the comment that I put

in there. I'm looking for it now, but. Yeah. Some of the comments I can't get to anymore. I don't know why. Maybe they were reported or maybe they're. Who knows? Right. I mean, social media does weird things to people psychology. But the point that I think that I wanna say that Kendra overlooks. I think everyone overlooks it. Data and back end systems have a longer shelf life. And I say this as someone who was, what, 10, 15 years ago,

strongly ensconced in client development. Right? Whether it was your Windows, Windows Phone, or other types of Windows based devices. Right. Or web development. Right. Those technologies turn over pretty quickly. Right. You know, you're likely to get multiple updates per year on a device phone, like an app on a device, but you're likely to never see, a radical change or redesign. You'll you'll see a radical change or web redesign of a website or portions of a website

couple times a year maybe. Right? But you're never gonna see a radical redesign of a data back end system, but once or twice a decade. And It's true. Yeah. And mostly what drives that is scale, not features. Right. Not features. It's just date or just tend yeah. Exactly. Right? So if you a 100 x and who could who could accounted for that, you know, going to the project started. It's a problem. Still a

wonderful problem to have, but but a problem nonetheless. Well, and there's also the fact that, you know, it's 2,000 whatever now, and there's still mainframes running. Right? There are still not not not to to knock on IBM too hard because they are the company of Red Hat. But, d b 2 is still around, still getting updates. Still backbone of many Fortune 100 companies that also share the stage with Satya at these big Microsoft events too. Right? Like Which was mind blowing

for people from the old days of Microsoft. Right? Well,

that's a whole other thing. But, like, you know but, I mean, it it really boils down to, like, these technologies have a longer shelf life. So if something is 20 I think we get hung up. 1 of the threads sub threads in here gets hung up on, you

know, 30, 20 year old technology. We're thinking that, well, you know, there's a meme of the the little monkey puppet, like, you know, giving a side eye and then goes like a cringe face, like, and a side eye where it's like, oh, Windows is, you know, I don't know, 40 year old technology. And I'm thinking, like, some, you know, Unix people or Linux slash Linux people are, like, 40 years old is old. You know? I mean, this stuff goes back much further. So it's but

it's still like and that's not a knock. It's just No. It's just now that we're in this industry now for as long as we've been in it and the industry's been around longer this long, there's just stuff that is gonna just start aging out, but it doesn't age out as quickly as we think it does. It's not like it's not like the iPhone. Sure. Right? Where you the iPhone I don't know what number they up to. 16, 17. Right? Oh, well, suddenly my iPhone 15 looks bad, and that

happens every year or 2. You this you don't see that in database systems. Right? The only impetus to really move, say, from, like, SQL Server 2,005 to 2019 is updates stop going. Right? And that's a whole big project. Yeah. The maintenance cycle. So it goes out of maintenance. And then you worry that if something crazy happens, you can't get support for it. And that's kinda like, you know, it's it it's sort of it I'll say this. It's analogous to your phone starting to run slow for some unknown

reason. That's funny. Something something on SQL. Yeah. Something something. Sybase something. Well, and you think about all the I mean, I mean, and contrary to this, contrary to that statement of these things have long life shelf lives. Yeah. Is the fact that I mentioned Informix earlier. Raise your hand if you heard of Informix. Right? So I've heard of Informix. You've heard of Informix? I mean, we don't count.

But but, like no. But, like, I remember my first experience with Informix was because some alum of Fordham had because it was a big shot at Informix. And, and I think we had somebody who was also a big shot at Silicon Graphics. So we had SGI machines running at Formix. Right. So I remember my first UNIX I used was an IRIX system. Right. Which most people today wouldn't even know what what that means. Right. And, you know, but Informix is out of business. Sybase is gone.

I can't even think of other names. I know there's more. Right. But really, the only things that it those have probably been migrated to SQL Server or Oracle. Well, or some form of Postgres or something like that. And I I hear you. You know, there's there's an argument to be made for, you know, the the cost of maintaining old software. Right. There there definitely is.

I'll say this about SSIS. I if you learned SSIS in probably in 2,005 era, between 2,005 and 2,008, that engine I I don't know how many lines of code were changed before it was upgraded to 2008 or r two, but it changed. There were some performance tweaks in there. It was obviously, faster. And then again, that happened in the 2012, error when we saw I love that SQL tab. You know, it's dead. Long live Crystal. So was Crystal ever database or was it

just Crystal Reports? Reports is all I knew. I didn't know about it as a database. That's all I I I use Crystal Reports and my favorite thing was it filling up the drive because it kept caching things. But I remember the whole idea of just because you place it somewhere, it doesn't mean it's actually gonna end up there. Like, the whole thing is but sorry to cut off. That's okay. But SSIS in general, if you learned it even in, you know, 2006, came out

in November, I think, of 2,005. Even if you learned it then, it at at a fundamental level, it hasn't changed that much. And whereas you'll see other software you Visual Studio is, you know, a software development platform that allows you to do c sharp and v v and, you know, all of the stuff. And it allows it still supports for us. I know. I haven't tried v v in 9 years now. So it's been a while. But if you look at how much most software changes from a developer

perspective, and SSIS is software development. So as your data factory, and any data engineering, that software development, SSIS is probably in the 95% of what it was. If if you knew the fundamentals in 2006, you know those fundamentals in 2024. And Right. Part of the decision to go make the upgrade, we talked about, you know, maintenance wonders and stuff, and I I get it. And it's not the same as your phone slowing down. I said that, but that's a bad analogy. But Well, it is also

it's also, I think, also very relevant to Windows 10. Right? If you're on Windows 10, your updates are gonna stop in October. That's true. I don't wanna get on that soapbox and rant. Sure. No. But I I mean, there's I get reasons for that as well. I don't like that it's gonna change because I like Windows 10. But, but yeah. Well, there's there's I'm gonna join you in not going down that road. But I'll say this. Hey, Maddie. How are you? The, the

just the fundamentals of data engineering haven't changed. And the tool itself, you know, if you knew it back then.

And it's, you know, you know it now. And so if you learned it now, you could go back then and still work in the previous versions of it with, very little headache. And that speaks a lot speaks volumes to the, the team that designed and built that. And so in addition to the technical reasons for doing this, the business reasons, kind of revolve around one of my favorite phrases. I mentioned this in today's newsletters. It's a compelling reason. Do you have a compelling reason to make this change?

And business people think about this all day every day because the amount of money that they make, the profit is based directly on the amount of money that they, you know, they spend and are they getting this value for it. So if they can improve the performance of something, say 10 times, and a result of that is they get, 5 times as many customers,

then that's not a bad investment. That'll just work. But if you're coming to me and I'm a business that existed before the Internet, if you're coming to me and saying, I want you to change to this completely different model, where, you know, and and the way it's presented often is you can save money. And that's true because if I start a new business today, I'm I couldn't even compete. I'm not gonna be able to stand up the servers, you know, take that time and buy that hardware and float

that that inventory that I need to manage all that. Whereas, I can I can pay rent essentially every month on that service? Right? Right. I I always like to say I always like to say if I wanted to start a bookstore today, right, versus 1996, right, or 1995, depending on when you wanna say when they started.

I mean, Barnes and Noble spent a ton of money. I don't have the exact number, but I can there's probably tens of 1,000,000 of dollars, probably closer to a $100,000,000 to just before they had their first customer. Right? Wow. And that but that was the heyday of the dotcom. Right? Because they were you know? But then but if you wanted to start a bookstore today, whether or not it's a good idea, let's let's just suspend our disbelief for a second. You can probably do it on a on an average credit

card limit. Because because your IT is enabled. Right. And and you pay like I said, you pay fractions of what you would have to do in the brick and mortar. And most of the initial spend isn't gonna be your servers or hardware. It's gonna be in development and marketing. Right? Getting the word out because it's such a noisy market. It did the the market has radically changed. And I also think imagine go ahead. I'm sorry. What? Imagine? I was gonna say imagine that you've built

Right. This infrastructure on premises already. You've got all of this done. It's a sunk cost. We can debate about how to feel about sunk cost. Right. But it's there. You spent the money and it's there. And you're not gonna get that 5 x income boost when you move to the cloud. In fact, in some cases, not all by by any stretch, but in enough cases, you move to the cloud and it costs you money. Because when you're getting the presentation about starting using the metrics of this

new company being started today Right. You're, you know, you're told the truth. You're not being lied to at all in in any of this. But often, systems that were designed software and front end back end systems that were designed, you know, from the nineties through the mid early 2000. Those systems were architected in a whole different mindset of what's the prevalent mindset for today. And as a result of

that Yeah. Yeah. As a result of that, one of the things missing from the spreadsheet calculation that you're gonna get the ROI from moving off your on premises servers to the cloud is that couple of $1,000,000 and about 18 months, of the hit that you're gonna have to spend rearchitecting Yep. All of your systems so that they now fit today's paradigm. And frankly, if you are interested in doing that, you you could go do that at any you could have done that at any time in the last 10 years

and made that shift. But people didn't do it because the business people didn't do it because the ROI was not dead. There was not enough return on that investment. If they wanted to, they would have spent that money then, but it wasn't gonna improve the bottom line. In fact, it was gonna hurt the bottom line. And so you see companies now make this move into the cloud and then, yeah. Yes. That is a

that that's an astute question to ask. So for those who may be listening and not viewing this, it says is SQL SQL dev d b a says, I use Brent's and I'm assuming Brent Ozarks. Brent Ozars. Problem are you trying to solve by changing this for justifying upgrades? Brilliant. That's that is brilliant. And he's right. And the you know, but it's compelling to hear and read the case studies of of companies that, you know, were able to do use to to access

$10,000,000 worth of hardware, like you said, on a credit card. And think, wow, what would that do for us? And the answer is sometimes, yeah, it'll revolutionize your business. You'll 10 x coming out of this. But other times, it's like, no. You'll point 8 x. You know, this isn't as compelling. So it's interesting because, like, I think there's a number of and I found the article. I'll pull it up. But but one of the examples, it was either Dropbox or Box. I forget which company it

was. But but they had basically started off, I think, in AWS. Mhmm. And then they got to a certain size. They actually figured out it's cheaper for them to design their own servers that are optimized for mass storage Mhmm. Than doing it. So they started building their own hardware and their own stuff. But I could tell you, if they were a startup and they went to a VC saying, we want to start with this on prem, they would have been laughed out of the building. Yep. Today. Yes. Today they would

have. They mean, you know, and it's it just shows that the the shifting economics of cloud versus on prem and and other types of things that I don't think people really have figured out yet. So this was a really interesting I'm gonna share this tab if I can show it on the screen. Sources. But that that

use case is you can't, you know, having the compelling reason to migrate to the cloud, and you can do that upfront. It's harder. But exactly what you're showing there, you're sharing that that idea of leaving the cloud, that's

growing. And it it's growing across the board. And I one of the, metrics for that that's directly related to what we're talking about here today with with data engineering, is that there there's been an increase in 2024 in the number of, people that reach out to me to talk about, SSIS help with their systems. And, I mean, I do consulting in, you know, ADF and fabric, and most of my consulting has been in ADF. When SSIS was involved, it was in lifting and shifting SSIS into an Azure SSIS,

integration runtime. But all of a sudden, after 2, 3, 4 years of that, that shifted this year. And people started reaching out to me with SSIS on premises consulting things, and I kept up with it. So I was able to do it. And but there's other evidence that I will not share. I probably I may be able to, but I'm just not going to. But it's even better evidence than my

anecdotes about people more people reaching out to me. Right. That the amount of SSIS being executed in the world has increased, and it's a double digit percentage increase just in the past few months. And I I think I now this is where I start speculating, and I don't know the answer to that. But we have a our mutual friend that we, another mutual friend you and I connected with in November of 2025. Sorry. 2005. Like in the

future. Recently recently worked for a year and a half, 2 years for this large agency that's not part of the government, but does money supply stuff. Oh, okay. I know. I know. Yeah. After getting his MBA from Sloan, you know, which no. Sloan. No. Sloan is important. The, you know MIT. School with MIT. Right. He's a graduate with that. Super smart. He shares with me when I'm telling him this story, I give him that stat, and he says, here's what's going on. Economically,

money is more expensive today than it was. And so he said he said that as he's telling me this as a cautionary tale because he says it's gonna change. It's good. Money's gonna get cheap again, and people are gonna flock back to the cloud. That's his theory that it's all being driven by money, and I don't think he's wrong, especially I think that's one level. I think that's one lever. I think there's

more than one lever. That is certainly a big one. But I you know, as someone who I you know, my previous role at Red Hat and my current role at Red Hat, I have to think globally. Right? And we don't again, not a commercial for Red Hat even though the fedora is there. You know, one of the things we do is we basically provide a data platform end to end that can run-in any cloud on prem or, you know, one of the hyperscale. Or hybrid. Yeah. Yeah. Or

hybrid. Right? Where and there was one customer that I spoke with before I won opportunity to leave. They were, big government agency. And this big government agency, you know, they have their own data centers, even though there was a push to get rid of them all. But they also have because of way contract government contracts work in the US, they had, foot you know, money to spend in AWS, money to spend on Azure, and I think

even money to spend on Google Cloud. So the one advantage that we had that the other ones couldn't is that the he called them the soft costs of training people how to do he'd do the same thing to do linear regression in SageMaker and push them out of production in SageMaker and Azure and in Google. Right? Yeah. And this was one tool. You learn it once. You administer it once. The same glass.

It was the same thing. I think those environments are very real. Now those are probably limited to large customers or kind of the government agencies that have these kind of contracts and things like that. Yeah. But also Mhmm. You have a number of countries that it's just not a good look to move your data out of country. Right now, in the US and Canada, we don't have this issue because there's plenty of all the hyperscales

have footprints in Canada and the US. But if you're in Latin America, which is this where this example comes from, right, there's only at least as Red Hat defines Latin America, includes Mexico, and basically all the way down to Antarctica. Mhmm. And only just, like, 30 some odd countries. Right? Someone's gonna write me hate mail saying that this is the exact number, but let's just keep the math simple. It's 30. We love those mail. We do. We love we love the mail. We learn things every

time. Right. I talk about them personally when I get corrected at at the dinner table because I wanna share that with No. I mean, it's it's good. I'm not saying don't do it. I'm just trying to keep the math simple because it's Friday before, you know, basically, we're sure holidays. Sorry, Frank. I I derailed you. No. That's fine. Only 3 countries from Mexico down to Antarctica have hyperscaler presences. Now, the 4th one in the but out of 30.

Right. So it's actually 10% or less realistically. I think it's like 37 countries. I asked Wikipedia and stuff like that. So less than 10%. Right? Right. If you're in a country that doesn't have a footprint. If you're in that 90%. You have to ship it out as a country. You have to be okay with that or do roll your own solution on a thing. So there was a government we we won a big contract because they wanted to do advanced AI and they wanted

to keep it in country. Right? Doesn't necessarily have to be on prem. Could just be, like, you know, an Equinox data center down the street or something like that. Right. But within their thing. And it was a government agency, so it wasn't computer science or even data science. It was political science that really kinda was the

driver there. Right? Because if I'm in country x and I have to move my and I'm a government agency in country x, I have to move my data to a sovereign country y. Not a good look. Yeah. Right? And, you know, would it really matter? I don't think so. Like, in a but from a legal point of view, it kinda does. Like, where the data resides in the residency. And I think if you go to the Azure website

now, they'll actually tell you where the data resides. And they actually interestingly enough, they get down to granular, at least on the US side to the state. Right? So, like, it'll say, you know, Virginia and stuff like that. We have a comment from, or not. I'm gonna hide my screen so I can look up the Azure map and kind of demonstrate that. And then I have to figure out where the sources are. There we go. So you wanna read the comment while I do that while I'm distracted? Sure.

So so many comments, but there are clients who are considering the technology used in the software package, and they may escape when they see the old school, stuffs. I'm so maybe. And and that may be, you know, I hate to be that guy that says, oh, that use case is invalid. I don't think so. I'm not aware of it. That doesn't mean it's invalid. I'm not aware of a lot of things. But but maybe. And, you know, definitely

have mixed emotions, about that. If you're buying because the the if you're a client and you're buying from, some company and you decide to go to a different company because of the technology stack that's being used behind there, I don't know. I think that says more about you as a client than it does about the company. If they're delivering the service and it's, you know, the the the rules of data engineering are you get accurate data as fast as possible, and those priorities

are in that order. Yeah. I don't think the old school stuff second. The only risk of the old school stuff is it's still maintained or there's still security packages. That would I mean, if I were in that position, I would be like, oh, I mean I mean, if you're still using, say, Sybase 6. Right? You know, like Yeah. You know, got a little there's definitely a line there, and it's drawn

based on it's drawn different places first. And some of the reasons, that it is drawn in different places is security is huge these days. I mean, that's gotta be your number one concern. And and, you know, it it goes from there. But if you're delivering the service securely, I'll just pick that one, then I would say that, you know, that if if you lose a client because you're not using the new shiny, I don't know what you can do about that. I'm trying to think I'm not

gonna say the client's wrong for feeling that way. They probably have valid reasons for feeling that way. But if, you know, if they wanna if they wanna make that decision based on that. And I'm looking at Frank's graphic here of the is that the data centers? This is the one I was telling you about. Right? So this is this is just Azure, but I would say it's a pretty good proxy for the other hyperscalers. Right? Mhmm. I would say Azure at one point had more. I I haven't kept up.

Well, that was one of our talking points when I worked at Microsoft. Right? We have more than to be honest. Right? But I would say if it's not exactly more, it's close enough. Right? So there's Mexico. You see the United States is pretty well covered. So is Canada. Right? So if you were a Canadian company, you had to keep it in Canada. You had an option. Right? Yeah. If you're an American company, you have pretty good choices.

If you're Mexico, yep. But if you're in any of these countries in Latin America, down through South Wales. We only had okay. So now there's Chile. Okay. Gotcha. Right? So I'm sorry. Now there's 4. So my math is going to get more complicated right away. Right. So, there's Brazil. Actually, no, there's still 3. So there is no footprint for Azure in Argentina. These little blue things you see, that is Colombia. Those are networking

pops. So basically, from a networking point of view, if computer science were the only thing that would matter, then that be that would be acceptable. But data residency is the issue. So if I go here, the US East 1. Right? It'll tell you that its location is Virginia and it's stored at rest in the United States. Like like here. Is there any more details? There isn't. US. Yeah. They used to. Yeah. They don't talk much about that, about where they are. But one of the

US East Georgia. 1 of the Yep. The US East 2 is down in Danville, isn't it? They are or Mecklenburg. I forget which. Well, essentially, they chose a picture of Richmond. Right. I mean, these are, you know, with we kinda touched briefly on on politics in a geopolitical slash, sovereignty strategic way. Right. These are huge. And I I know I'm not the one, thinking about it. But Look at this. Yeah. There's one in Israel. Those satellites, Frank,

are getting it, by the way. Those those satellites on the graphic, those things are moving at way faster than normal satellites. Oh, yeah. Yeah. I mean, satellites are going to change things, but, like, in terms of where the data sits at rest is really where because ultimately, I think it really boils down to when are the local police going to barge down a door with a with a court hopefully with a court order and basically copy everything. Right. That's really what

matters. Right. That turns out that's really what ended up mattering. But if you look at the Middle East, right, like 1, 2, 3 countries have it. Right. And that entire region, you know, obviously with geopolitical tensions being what they have been for a number of years. Yeah. Moving your data center to any one of these countries may be an issue for you for your organization or your regulatory. Right? Europe is kind of the same thing, right, where, you

know, there's Switzerland, there's Italy. And I know that there's different kind of things in terms of Germany. It was actually I don't know if it still is now, but it might have been a, I used to live in Frankfurt, actually. Yeah. There was actually what they call a sovereign cloud because there was concern that if it was a US company owning a data center, that US courts would have jurisdiction there, which is a

brilliant move by a a past administration. I say brilliant sarcastically in case you're didn't get pick up on that. Where they thought that they could basically issue a a court order to demand something from here in Ireland. And Microsoft fought that because they realized, like, wait a minute. That would mess up our entire that would cause a lot of problems. Yeah. And, ultimately, they dropped the case before it was finally

decided. But in order that they could, thing, at one point, anyway, this is actually owned by a German company, managed by a German company, and it's leased to Microsoft to to to have that concern. I think China also operates the same way. Frank, I was commenting on the, satellites on the graphic there. Oh, that they were moving around. Yeah. Yeah. They are moving very, very fast. And it keeps they're moving with us. But, I mean, keep in mind, though, like, keep in mind, though, that

we're just talking about data residency. There's other things that if you're building a real solutions, other things to consider. Yeah. Right? Like And there's a whole lot to that. And Yeah. Yeah. Oh, absolutely. And, you know, part of the part of it is, part of what what happens when you start kinda going back to the data engineering, platforms and stuff that you use. There are sound business reasons for not making a change, and there are some unsound business reasons that will

confine you to not making a change. And I I think about this. I'll put it in context of, of SQL Server. Companies will come up and they this has happened, and I still have clients running applications on old servers because the company that so they they serve, their clients include enterprises that care an awful lot about checking boxes and auditability and all of that stuff. Regulatory type things, which is not bad. It's just the way that it is. That's their their business demands something

like this. These companies were formed. They were stood up, and they've got SQL Server 2,005 running or 2,008 or stuff that's been out of the maintenance cycle at Microsoft for a long, long time. And unknown it's also not a well known fact that if you don't wanna upgrade to version x or y, you can pay extra money, and Microsoft will maintain

and provide you patches. Right? There there's rumors that there's at least as of a few years ago, there were still Windows 95 systems that were, you know and that sounds absurd. Yeah. But Well, you walked down the, entry to, Delta flight, and there was one that's what was it? One is 97, I think, sitting there. 98. Yeah. 98, was it? Yeah. One is 98. Sorry. Yeah. One is 98, boxes sitting there for the longest time. They're still I believe they're 1 to 7 now. Still. I I

saw XP. XP. You're right. It is XP. Yeah. So, you know, you just I kinda noticed this, like, wow. I hadn't seen that in a while. But it's it's not about will the new technology run that SQL Server 2,005 database. The answer is clearly yes. Well, you can always virtualize something. Right? Like, that's something that, like I mean, there's that compatibility levels help. Right. There's a number of things that

do it. But here's the kicker. If the application is not certified to run on that and you're for you can change it, and you be maybe you have changed it and tested it and go, yeah. It works. We'll just move it to, you know, SQL Server 2019 or 2022. We know it works. But the people you're serving, people who care way more about checking all the boxes and the regulations being a 100% and auditable, they won't

allow you to. And it gets even more complex when that company that originally sold you that software 20 years ago is no longer in business. Right. So you have no path forward. I mean, the only Or they get bought by another company that you don't really like. Exactly. That's happened too. Exactly. Then A lot of mainframe companies were brought up by I don't wanna name names, but, like, were brought up, and they it really was, like,

ironically, because they what they do, they they knew they had them. And ironically, a lot of mainframe migrations happened because of that. Like, it was And so you've got, you know, you've got that

angle where people are sticking with older systems for whatever reason. And it's, you know, it goes like I'm saying, my point is that this goes beyond just the data engineering realm. There are there are compelling reasons to use, older software. It may not be anybody's, you know, satisfactory answer, but it is, you know, those reasons exist. And if you're the, you know, if you're a developer who likes using the new shiny and learning the new

stuff, I'm one of those. That's why I'm teaching courses on fabric data factory right now and watching as it kind of some days it works and some days it doesn't. We've had that happen on a number of deliveries this year, with that. So if you read what I wrote about this and you come away with Andy's against the new stuff, well, you're just as wrong as wrong can be. That's not the case at all.

You know, it's an interesting I mean, so back to the lecture at hand, what kind of kicked this all off and inspired the stream was this post where I think the short answer is everybody's a little right. Everybody's a little wrong. And as a consultant, you can appreciate these two words. It depends. Right? Because, like, you may want to upgrade to the new shiny. I know every developer wants to do that. And I think the comment for some reason I can't

say is like basically hiring managers will put in a job description. All there's also the other matter of job descriptions and, you know, job requirements are. They're always a 100% accurate. Disconnecting from consensus reality. Yes. I like to say.

But they may want someone with, like, say, ADF and and and this, but then actually have them working on systems and SSIS because the hiring manager knows that he or she may not have that open req for a while and has a in the back of the mind the idea of moving to that someday. Sure. But realistically, for the next 2 years, you're gonna work in this site. Yeah. I mean And there are still large consulting companies out there that develop brand new applications in SSIS, brand brand new data

engineering data warehouse. One worse than that or one better depending on your point of view. A few years ago, I think it was on dotnetrocks. They were talking about telemetry from Visual Studio. And this back when I cared about Windows client development. Basically, WPAF came out in 2006, 2007. Right? XAML. No. Not XAML. Metro or modern applications, UWP came out in 2011, 2012. Right? So there's been multiple

frameworks to write when and it's been a number since. But, again, don't really care about those client development anymore. Windows Forms is still the number one of all those, like, ways to develop Windows applications that run on Windows Mac. Windows Form is still accounts for 80% of development. Wow. Something someone's going to like, please email me in hate with hate mail, not hate mail, but like tell me the exact number. But it was still. He's off. Well, and they kept saying in Visual

Studio 2005 came out. They said, this is it. This is the end of the line for Windows Forms. We're not updating. We're not adding anything. And the future is from now on. Right? Until the future became something else. And then when I last installed, I think it was Visual Studio 2016, 2019. There was improved. They added stuff to Windows Forms, which is kind of funny because they said they never would. Yeah. But it just because they. Demand. Right? Customer demand. Ultimately, customer demand

is what pays the bills. So you've got to be very mindful of that. And, you know, if you if it's 2024 and you're writing a Windows Forms app, I have questions. You know, I'm not saying I disagree, but I have many questions. Arguably, you could say the same thing for UWP or WPF. Right? You know,

But, again, it really depends. Like, in the last time I had worked with WPF professionally was it was for when I was at a, between my stints at Microsoft, and there was a, you know, there was a customer who was a mortgage company, and they basically had their mortgage intake form written in WPF. Right? And they were having performance issues with it. And I I looked at the code, and I was like, this is a good lesson, I think, is that, you know, they loaded up, like, some 6, 700 controls

all at once. Right? Wow. Because there were a lot of fields and but they were all collapsed and things like that. And I was like, well, I'm looking at this, and I'm, like, testing it. And I'm like, oh, dear god. This is gonna be a nightmare because it's 600 controls. You could do lazy loading and things like that, but then Sure. There could be unintended

consequences there. And then then I happen to notice when I load the app, the CPU spikes, but the GPU was hardly touched, which the whole promise of WPF was that it would offload as much of the rendering Yeah. Over to the GPU as possible because it was basically built on XNA, which was a gaming framework. But that, again, different different sidetrack, and different lifetime ago. So I'm like, what the heck is going on? So then I'm like, I looked

at the some of the machines. I'm like, they had the generic GPU driver. So I'm like, just for grins, let me see if I can get the proper driver for this device. All of a sudden the 30, 40 seconds it took to load that initial screen went down to 5 seconds. Wow. And that's a big jump. That was a big jump. And that was like, and I said to the guy, it's like, look, just installing this driver, you get

a, you know, massive increase in it. Right? Do you really wanna architect it or are you trying to just you want this to be faster? Like, what's considered acceptable? And he said, well, under 10 seconds would be acceptable. And I was like, I could do you better. How about 5 and a half? Right. And I showed him and he goes, well, not everybody has a GPU in their device. And I was like, well, like, you know, this GPU costs about, I think, 189. For some reason, 189 is stuck

under $200. Yeah. And I'm like, so you'd have to install it. You have to think about the labor of installing, like, this cheap GPU. Right? And this, he goes, that's fine. He goes, look. The cost of redeveloping and rearchitecting and retesting this versus $200 per box, plus whatever it takes for somebody to go in with a screwdriver and update the drivers. Right. It was so like we ended up not having to touch the code at all. Right? Yeah. It was just

a matter of a driver update, which nice. Because it was like a fixed price kind of support contract. I was the hero because I solved the problem with about 3 out 3 to 5 hours of work. Nice. And the customer was happy because they didn't have to re architect anything. Everybody wins. Everybody wins. I love it. Happens. But but it it's just it just goes to show you, like, sometimes the most cost effective approach isn't is to not

touch anything. You know, and it's although it was a relatively small amount of money, and and, manageable amount of time for the client, sometimes in if you look at the, you know, kind of the the big, performance tuning picture. And I I ran into some of this, in the past where Well, it's not always a happy ending. As as an engineer, I want to, you know, to fix the the thing that I'm engineering. And if

it's software, then I wanna make the software perform better. And if it's, you know so I went through one of those experiences and then, number of circumstances where but the end result was we we threw money at it and bought, better disks. And I remember telling me about this. I remember you telling me this. Oh, it's a specific long story, but, yeah, it's back from about 12 years ago. Yeah. You were, like, just a long SSDs, and it got fast

enough. And so when I did the math on that was the enterprise level project, there were dozens of people tuning on a tiger team, and we did make it go faster. And as a test, what I did was I rolled the code back to where it was when we started. I will say now though, tiger the term tiger team after my Nobody knows what I mean. Sorry. I know what you mean. I know what you mean. Of individuals, dozens of individuals focused on solving a particular problem. It's it's like, Although in some cases

slurring. Focused on not solving a particular problem, but that's a story for another day. Different different story. You're right. But, yeah, we we did and and I rolled everything back and ran the old code that we had optimized. You know, it ran super fast on the SSDs, and it was running okay. It was acceptable, barely, on, the spindles. But when we rolled it back, it was, it was the same difference. We actually got a touch more

performance just off the SSDs. And when it you know, you do the math on that. At that time, SSDs were rather new, and the amount that I wanted was, not trivial. I asked for it on a lark, and I was surprised when it showed up. So she did work for Microsoft. Okay. I'm not surprised. Kendra is, scary as much. She did consulting. We Brent's name came up a a few times. She worked with Brent for a while. She worked at Redgate. Not everyone knows what Redgate is, but they're kind of a

big deal in this in this situation. Yeah. Yeah. They're a big deal. So, like, clearly, like, I I just find I'd love to get her, like, initial opinion after factoring in all kind of all of this is that, you know, ironically, Sync don't run after every shiny thing. But the the thing that that guy that was originally learning was is the new shiny thing, ironically. Like, so there's a lot of layers to this. There's even if you take this kind of at face value of not knowing the

context, there's a lot of layers. But, like, beyond that, there's even more layers. Like, it becomes this multidimensional problem. It's true. It's like an ogre. Many layers. Many I was wondering when we're gonna have a movie reference because it's been a while. There we go. Boom. Shrek's a classic. Shrek is a classic. The sequel is not so much, but that's a common case. Sure. Very, very few Matrix movies looking at here. It was you know, I I saw the, activity on this and it's best because

since I posted this, I can see how many people looked at that. The the link I sent you for the other one is kinda the one that started it. I don't know if you have that link in chat. I think If you wanted to click on that. I have That was that was a few days, maybe a week before this. That was the first one I commented. It was similar sentiment and I shared, you know, again, I joined the conversation and then, I can't believe is

SSIS dead? Yeah. That's the one. So that particular one has gotten, like, probably 12 to 15 times as many views. It's happened to You know, you mentioned And it was the first time I had piped up about anything like this. I'm I commented on the original one. LinkedIn will pop this up if you write a lengthy comment like like that one. And I said, that window exist. Looks like Windows XP era. Well, that's SSIS. That is a funny. Like, it's just funny. Yeah. But, And I

commented when this LinkedIn popped up and said, hey. Do you wanna repost this since you wrote this comment? And I said, sure. And it just pastes the comment up at the top of the repost. But I can see, like, the number of people that and that one drew a lot more, a lot more comments. And like I said, 12 to 15 times the views. And, and and I communicated with the original author just to touch about that at all. It it went nowhere near as I'm trying to think of the right word. No nowhere. It it

didn't get nearly as heated. I'll say it that way. And it may just be that, you know, that I'm saying heated, but there was there was one individual in particular who just very passionate about the tools that they that they used for. I I I wanted to ask, that individual, you know, how many SSIS packages they developed and put into production. Because I I I I would again, I think I know the answer to that. This in the this individual conversations like that.

Say that again? Which individual? Not in that one. Oh. Not it's in the other range. The one we've been looking at. This angle here, though. Right? Nothing against Kiwi ETL pools. What do you Yeah. Appreciate so this is this is an argument of GUI versus Yeah. Straight code. I think that's also an interesting concept. Low code, no code versus you know? I like them both, but, you know I do too. It's for me, it's which solution matches best and as a consultant who goes to work with other companies a

lot Right. A big factor for me is What they have. You know, what happens when I'm gone? Can you support it? What what are you most familiar with? Even if I'm working in a tech a technology technician, a technology like SSIS, there's really a couple of ways you can, float it. It. That's good, John. 2 was good. I'm referring to a 3, and I think there was a 4th one. Oh, really? I didn't even know there was a 4th one. So there you go. But in SSIS, you can take a more And

the matrix 2 was good as well. Matrix 2 was good as well. I was That's true. Yeah. You you could take, like, a more DBA approach, a more T SQL driven approach, or you can take a more dot net driven approach because you have both execute SQL tasks and SQL sources and destinations and transformations, and you have a script task and a script component. And, you know, which way do you go? Which is the right way to go?

Well, it depends. If you've got a bunch of dot net developers being tasked with maintaining the, ETL after I leave, no way. 5. Good god. Shrek 5 or matrix 5. Either one sounds like a terrible idea. You know, whoever's gonna be left behind support net, you gotta make sure that Right. You've done a a fair enough job of representing their preferences,

on that. And, you know, I I know I know consultants who come in and say, you know, we gotta do it this way and you're bad and wrong and you're, you know, you're gonna go out of business in 18 months if you don't listen to me. Right. That if I also think too, like, when I first read it I'm sorry. I don't say hire somebody else. Right. I mean, I when I first read it, I first read it as should you learn SQL? Okay. Right? And that's a big debate in the data science community is should should

data scientists learn SQL because, you know, Python can do everything. Just your no. No knock on my Python can do everything. Should you use Python for everyone everything, I think, is another question. Right? And I think the answer is no. Yeah. I think if you're a data scientist if you're a data scientist or even an AI kinda engineer, whatever the word is this week, you should still learn SQL

because this one, it's way less complicated than anything else you're doing. And 2, it's kind of the the the the lingua franca of anything data or data interaction. Right? And again, Frank, the context comes into play here. Right. There's the there's the mechanical tool. It's describing a software, you know, a language is mechanical is one one way of looking at it. But Right.

There's also the problem you're trying to solve. And if if you're gonna do data exploration or managing or whatever you wanna call it, then I don't really care which mechanism you use. But don't tell me one of these mechanisms is better than the other. It Without a qualifier. For this particular task. Yeah. So it could be that, you know, one of them is is good at this one particular thing, and I I would argue

this. I did in my newsletter. Oh, strike 5. Okay. In my newsletter, I argued, every single tool has something that they're stronger and there's there's some feature or some aspect of it that's better than all of the others. And they also have some weakness that's worse than all of the others. It's true. And and so you gotta, you know, you gotta strike that balance. It's gonna depend on your use case, the parameters, the things that are important to you. Sometimes it's cash on hand.

Sometimes it's the servers, hardware that you're forced to work with because you're owned by somebody and they they're not upgrading. They're not giving you that staffing concerns. The people, their experience, their languages, that their Well, how easy is it to find someone that necessarily knows SQL versus knows Python? Or, the bill rate for someone that knows SQL is probably gonna be different than the person who knows Pandas. I think so.

I I think there are differences there, but, again, it's gonna depend on the company. That's true. You're worth our job market. Right. You're worth more to, you know, this company as a SQL developer than, you know, maybe than a Pandas developer is to that that company. That's a possibility. Go go where you get paid the most, but realize especially if you're new, and I think that person that in Kendra's original, post that she was, she was jamming on, that

that was a different scenario. You're right. There's a number of squirrelly things about that original post, but if you're a young developer and just getting started, you're in college. I've got a daughter studying computer science in college right now. And my recommendation to her is first, learn everything that you can. Do as much as absolutely as you can. Pick up that knowledge. But, be aware that there's more to it than just the, you know, the the the brain exercise you get and the

thrill you get from seeing the code execute. Keep that thrill. Keep that spark alive. Right. Right. Right. Right. But, you know, real realize there's often more to it. And some of those factors you have no control over, and the person you're working for has no control over. And, Ed, you know, there's just a number of things totally external to the experience of writing code that often impact the experience of writing code. That's very true. That's very true. K. Actually, going on for,

like, 90 minutes. So Wow. Goodness. It doesn't seem like it. I've had this much coffee, and I still don't have to go to the bathroom. That's amazing. Christmas miracle. Of you. It's a festive miracle or a Christmas miracle. I don't know. Awesome. But, so this is this has been great. I think we kinda got to the bottom of this is that basically, it's a nuanced conversation. There's no simple answer. I have many questions though. Why if you're learning LLMs and now, why are you

calling yourself a data engineer? But Yeah. That's a different thing. Could just be semantics at that point. Could be. But thanks, John. Thanks, Merdad. Thanks, Hector and SQL Dev DBA. I'm sure that's not the name on on the driver's license, but you never know. You never know. I wanna get a license plate holder that has, like a, like, drop table. Like, so that way when they take a picture, they do the OCR. Boom. I've seen the bumper stickers with longer Right. Right. Right. Right. Right. Secret

hacks on them, you know, secret injection attacks. No. Hey, man. Thanks for tuning in. And, he legally changed his name. That's cool. Apparently, there's a number of people who have, like, a last name, and their last name is Null. Oh, wow. And I

was looking up on YouTube. Like, there's, like, a number of, like, problems that people have. And my first thought was, wait. Wouldn't that only be the case if they encased it in quotes or single quotes? Like, wouldn't it? Apparently, no. Some of these systems Depends. I mean, you wanna talk old systems. I'm sure DMVs have some pretty ancient technologies that are still running. I mean, I can only imagine. Robert Tables. Yes. Little Bobby Tables. Little Bobby Tables. That's the

one. I tried to name my kids something like that. But You got overruled if I remember correctly. I did in so many ways. I'm glad I didn't go with the, initials of, x a m l. Yeah. That that You could have. Xavier Anthony Marcus was, on the table. See and Lavinia? That would just Yep. That would just flow. That would be your role. Though, since XAML kinda died, it's probably a good thing. That's true. So XML is even not as in vogue as it once was. Well, you know our mutual friend, mister Kevin

Hazard, aka the Duke of Hazard. The Duke. He says he says JSON is just hipster XML. He's right, though. And there's another format called JSONL. What? Yeah. JSONLines. Never even heard of that. It I it's only the I didn't hear about it until the product I work on, Rel AI, actually uses it to store our data. And, basically, it's a different between JSON and JSONL? Basically, it's a long line where each line is a record of or a chunk of JSON. That's how I interpret it. I'm sure I'll get

corrected, but please correct me on that one. Interesting. Yeah. I'm like, Jason now? Like, what the heck? Jason is an acceptable one. That's true. Jason would've been a good name. But I didn't want him to get teased on Friday 13th for the rest of his life. So so, thanks everyone for tuning in, man. This was awesome. We should do it more often. Thumbs up on, or, you know, be sure to like, share, subscribe. I gotta do

this to all the things. Like, share, and subscribe. I can show off, this little graphic. You you know, Frank, we could do this, we could do this a lot more often if there anytime I stir up some trouble, we do a a live stream Oh, god. We do it every day. I Yeah. That's what I'm saying. You know? No. I think it's interesting. I think it's good because, like, you know, it the controversy I think people are starting I just look last time I looked at the thread

thoroughly, people were talking past each other. Yeah. Which I guess defines all of Reddit, mostly Internet. You know, it is what it is. Yeah. But cool, man. I gotta actually, now that I mentioned it, I do have to go to the restroom. See. You did it to yourself. I did do it to myself. And thank you, Miranda, for turning in. And I will play the outro graphic. Excellent.