AI Data Tool Suite for Annotation, Calibration & Validation w/ Mohammad Musa

Hello. We want to reimagine Mobility podcast series. I'm here with Mohammad Musa from Deepen. Thank you, Mohammad, for joining me today on on helping me and our listeners and viewers reimagine mobility as we look forward in these very exciting times. Mohammad, explain what you're doing. What have you started with Deepen? What are you guys doing?

And then let's afterwards jump right into how you with your technologies, your products, help reimagine mobility in all sorts of areas of the mobility sector? Yeah, definitely. So hello, everyone. Great to be here. Thank you. Stephan. For hosting me, I've started Deepen around six and a half years ago in 2017, early 2017, looking at autonomous technologies and specifically autonomous vehicles. And at the time we thought that autonomy will be sold by 2020.

So we were trying to be part of that re-imagination of mobility at scale and populating. Now it's 2023 and we're still five years out from from autonomous mobility, even though we have seen some good progress, is like Cruise and Waymo extending their operation in this go and so on.

But ultimately, as we saw in the way of transportation and mobility, specifically enabling a wider segment of the population to have mobility access without owning the vehicle, reducing the cost, allowing for more access and on on various sectors like, you know, for example, disabled people, that their vehicles cost a lot more. Now with Robotaxi, you can offer them more cost effective mobility, even less for that segment of the

population who cannot afford a car. Now they can get portable access to mobility and most importantly, also safety. Given that we humans tend to not pay alot attention on the road, especially with the mobile phones that our hands and so on. So the number of accidents and the safety impact of mobility is pretty crucial.

So at the mission of our company, we wanted to enable that safer transition to autonomous mobility for us to realize the cost advantages, the life savings and and the productivity that we get out of it. I think at the heart of of of mobility is that we as humans, we need to be in certain places. We don't want to do a zoom and video calls the whole time and in order to be in those places, it takes us a long drive to get there. We can be productive on the way during transportation.

That is nice for me. I can sleep, work, play on my way to wherever I'm trying to go. It's like it's like I'm an home and I'm just getting to where I need to be. So that time savings I think is really that's the goal that that mobility, autonomous mobility gives us time to do things that we wouldn't have access to otherwise.

And in that transition, being part of that enabling that wave of transition, we saw that there are bottlenecks in how companies are transforming their their vehicles and their fleets to be more automated, whether it's autonomous mobility like Robotaxi, as well as a level three or level two ADAS, which is advanced driver assistance.

And in that transformation, whether you need more sensor types like light hours and imaging radars and so on, or different camera types, different, the for example, thermal sensors are different that an app is a capabilities that are that are available. How do you interpret that data and how do you take that data and evaluate and validate that it's accurate, that it's performing the way you think it could perform?

So that was our first product offerings around data Adaptation is basically taking the data from the sensor suite and adding in meaning on top of it that that processes called ground truthing. And the way you generate ground truth data is through adaptation, or you annotate that data with humans as well as looking at even some AI in the loop to give meaning to through the light. Our data radar camera, thermal, etc. and that is required for thousands of miles.

So whenever you're trying to add these ADAS tasks or if ability is you need to evaluate that perception and if the vehicle is perceiving the world the right way, whether it's pedestrians, cyclists, motorcyclists, traffic signs, traffic lights, the animals, you know, whatever comes in that way at that notation challenge is quite involved. We need to annotate a lot of data and you need to have that. They are very highly accurate and the like.

The precision, the level of precision around where do you know that this as a person like your feet is on the ground? Where does that ground start and your feet and then then vice versa. So that is these kind of very fine grained pixel level accuracy at this point. You're talking about a human. At this point you are looking at the street or the ground and so on, and doing that in multi-modal ability in in the light our in the camera, in the radar and doing that over time.

So not just at a single frame but over many, many seconds or minutes or even hours. So that's the first segment of work. That's where where we start it and we look at it. We want to achieve safety. You need to understand what you're looking at and to understand what you're looking at. You want to annotate and label the data so we can evaluate its accuracy. That was the beginning of our work.

And then as we they build a lot of data, we discovered that it is the relationships between the sensors was affecting the quality of the perception. So if your camera had changed its direction by a small amount with respect to where the lighter was collecting the data, you would actually start to mis predict or miscalculate the location or the trajectory of objects.

So if you think about, for example, let's take a very common and existing technology today with emergency braking, the radar and the camera are working together to estimate the distance from the object ahead of where that vehicle's or other. So if the camera changed its calibration or meaning like let's say it was looking straight ahead, the camera started to look further down, just like degree down the calculation of the distance from the camera only.

What would differ significant, though, maybe at smaller distances, you're not going to see a big deviation. But if you're looking at something that 100 meters away, you might miss estimated by tens of meters because of the change in calibration. So we saw that as a safety critical component that we started investing in our calibration suite that detect that calibration deviation and correct the calibration between the difference that serves and between the sensor and the vehicle.

So think of it camera to vehicle light our two vehicle radar, the vehicle IMU to vehicle, all of these sensors that you need to know your location in the world and what you're perceiving. So that's the second category related to our adaptation business and ultimately to evaluate safety. It's not just about what you're perceiving and what your sensors are looking at. It's also about the behavior and the decision making that the vehicle is making. And we're not a simulation company.

We're not trying to be in that a full ADAS capability company, but we want to enable how enable companies to evaluate the safety of their operation. So we invested in a database called Safety Pool. That's a database of safety scenarios.

So ultimately, when you launch a new capability like, for example, Highway well, the adaptive cruise control, where it's hands off to level three, where the driver can cannot it can afford to not pay attention to the road and like read or watch a movie or something. And the car would drive itself on the highway.

So how do you know that your feature that you're launching is safe enough, the the process or the the latest and greatest in the validation verification space is you run a lot of potential scenarios against your sensors and against the feature that you’ve launched to to see if it's able to handle all of these edge cases, whether like, let's say a vehicle stopped all of a sudden in front of you or somebody cutting in front of you or someone is coming at you from behind really quickly.

And there is a billion actually almost infinite permutation of things that may happen. We will never be able to count all of them, but we want to be able to have a baseline to say, where are we from a safety perspective, from coverage perspective in evaluating the safety of our operation.

And so with safety pool, we're looking at enumerating that a basic scenarios that are absolutely your system must handle successfully in order to say like, yes, we think this system is safe enough to put on the highway or to put in an urban area. Think for level four robotaxi that that the number of scenarios is very large, but for level two, level three, there is some consensus around what's considered minimal level of safety.

We have been successfully working with University of Warwick on on that database. So just to kind of sum it up together, we try to enable our customers or their stakeholders who are working on enabling Safer ADAS and AV to get there faster. So we help them with labeling the data with calibrating their sensors and then evaluating the the safety use case or safety story using these scenarios from the safety database. Very good. Very good. Very good examples and explanation. Thank you, Mohammed.

Maybe let's go to the annotation for a second. Over the years and in my organization on a global basis, helping OEMs and tier suppliers with their with their sensors, with their compute platforms and to complete ADAS all the way up to level two and level three type systems. Annotation obviously is a big one because he got to look at data. We have some products in that field as well. I know we've worked together with you with with some of that.

I hear a lot of companies advertising that their annotation software or their annotation services are whatever, 90 plus or in some cases I think you even heard 99% automated. So no human interaction. What is it that yours is and where do you see the industry going? Is is 100% really possible?

Is there always a need for somebody to also look at it and take some of those maybe cases that are hard to detect or are as we develop A.I. and algorithms, neural networks, etc., are we get to a point very soon that know all the data. I can just fit into this thing. It automatically looks at it and outcomes what I missed, what I need to change in my software, etc.. Yeah. Yeah. We've been living and breathing this for, you know, six plus years.

Unfortunately, a lot of players set some very wrong expectations around the labeling process and how to what extent it can be automated and how it can be automated and so on. And I'm going to give you a very engineering answer, which is it depends. That's an economics answer, by the way. I used to use that in my economics class, in my MBA. It well, it depends, you know, it is very contextual. You know, if, if you're giving me, for example, day time.

Sunny driving on a on a beautiful highway when everybody's behaving properly. Sure, Yeah. We can automate all of the vehicle, annotation and detections, no problem. But now the devil is in the details. You're trying to do that same sensor suite and multiple driving scenarios, multiple weather conditions, and add to that maybe even different driving behaviors and then think of this is where the edge cases start to come in.

What if the sun is at a certain angle with respect to the camera and maybe it's foggy and your or your light? Our sensor is generating a lot of noise. You have some different materials in the vehicles around you that is causing reflections that are like we call it ghost points from from radars and so on. So that's where the devil's in the details and the level of automation.

At some point, if you want to get perfect automation, it will actually cost you more engineering time to get to perfect automation rather than to just do it manually. So so that balance between where do you invest in automating versus while just let's just get it done with human in the loop and buy in by having the right system in place to ensure that the human input is also not introducing no mistakes because I can make mistakes. Humans can also make mistakes.

So it's it's balancing kind of these two two operations and looking at the economics while you're doing that. So ultimately with our tooling is we say to the customer, our tools come with some built in AI to achieve the minimum basics of automation. We also allow the customer to use their own system to help in the automation so that because we would never collect thousands of miles like our customers, that's not our job.

We're pooling a technology company, but the customer has that data and they have the AI models that are more accurate and more capable than what we have so they can export the output of their into the enabling tool and have that humans just assist in the verification and validation of these labels. So we allow that in our tools.

And then on top of it, we've built a significant number of automated quality checks and validation that looks over what the AI is doing and what the human is doing across multiple people, across multiple runs and looks at heuristics and kind of different signals to bring all of that together and gives you at the end a workflow to allow you to get a productive labeling operation. So we don't sort of we don't force a recipe on you.

You can choose any recipe that you like, but we give you the tools, you know, the utensils and the equipment to cook it and cook it any way you want. But to get to the result that that is satisfactory to you. So that's our approach. And like you said, there are so many vendors, some of them come at it from extreme technical capability where they say, Yes, we're going to achieve extreme automation.

And on the other spectrum, you get some people who say we have thousands of people that we can give them to you at a very low cost. And there are ethical issues around that. And, you know, like how you treat people and how you compensate them, which we think is is not fair. And obviously, like we don't want to go there. So in between like low cost labor and full automation, where where do you draw the line?

And we try to be that balanced in the middle where we use we have our own team to do the service, we pay them market rates where they're happy working for us. Actually, we have sponsored many marriages enabled many marriages of our laboring employees because they are viewed to have a stable job with good income where they live, and also keeping the customers happy with the quality and the level of automation that we can provide. And I guess our biggest differentiation is we're very transparent.

We we kind of sit down with the customer and we tell them this is your accuracy target, here's what the humans can do effectively on average to get to that level of accuracy that you want. So where do you want us to invest? We can obviously put more human hours or we can put more engineering hours, but some something has to happen to get that level of accuracy at scale that the customer wants.

So if you have to a billion lines that need to be annotated, I think it's worth investing the engineering hours to automate and help bring the quality of the automation much, much higher.

But if you two kilometers that you want to annotate, ultimately, I think at this point we might as well just do words with humans and having that transparency in that conversation openly with with the AI engineer, with the perception team, with the safety team that is doing validation and enabling them to have just a very good understanding of the trade offs from a quality perspective on the engineering side and on the human labor side makes everyone just understand there's no magic in here.

It's just sites for that's where kind of we've proven ourselves and luckily we've had customers that have been working with us for over five years annotating their data, both for training and validation purposes and in Germany, in Austria with the AVL team, we've had that great relationship for four plus years now and we continue to evolve that and grow it over time. Yeah, very good. So let me ask you this. I mean, in the industry, right?

We're talking about the likes of Waymo and Tesla having tons of data, doing a great job with their let's just call it ADAS and autonomous systems. And then there's there's others in the world that maybe struggle a little more because they maybe have less real world data and and and to keep ability and knowledge and how to what to do with this data, how to annotate and how to use it, how to feed it back into their system.

But independent of what the company is, what are the ones from, from your perspective that you may have worked with, continue to work with or are looking at it from an industry perspective? The ones that do a really good job of, let's say, analyzing the data again, annotating, reuse and changing algorithms in their in their ADAS algorithms are autonomous algorithms, so feed it back into the vehicle.

What are the ones that are doing really good, better than the ones that are not doing so good without mentioning any any name, Just an idea of to our listeners and viewers, if you really want to do good from your perspective, from deep and perspective, from Mohammad's perspective, what do you see they do better or they do really, really well. Merce's Maybe what the industry in general is doing, yeah, great question.

And like you said, without mentioning any names, there are a pattern that we seen across the board working with the Japanese clients, German clients, French in the U.S., North America in general, and even South Korea and so on. The ones that have invested in the quality of the data get a lot higher. Return on investment, though. You want to avoid garbage in garbage outs scenarios.

And it's this is like the biggest differentiator is that customers who have took the time to ensure that their data collection is solid, to ensure that the calibration is lawless. So you're not collecting data that will be very different than what the your vehicle is going to operate it. So the more representative, the data that you are collecting to what the actual vehicle in production will look, will look at, then you will get the most benefit of the effort that you're doing.

So it's it's really around avoiding this garbage in, garbage out scenarios. And we sum it up in the Deepen language is how good is the calibration, how good is the localization, how good is that synchronization between the different sensors? So those three elements are really the key accuracy triggers that can cause your annotation cost to be much higher and can cause you to collect and spend a lot of money on data that you're going to end up throwing away.

So anything that enables you to reuse the data is obviously going to bring a lot more return on investment for you. So if you have annotated some data, you can reuse it if or even if you change the sensors, you can at least apply some transformations to help you make intelligent decisions about like, why is this sensor really better than this sensor if you don't have if you cannot freeze the variables in order to make apples to apples comparison, is this LiDAR really better than the LiDAR?

But if you collected the data differently between your evaluation of LiDAR one versus LiDAR two, you will make wrong conclusions because you don't have apples to apples comparison. Is it is it really the physics of the sensor or is it the range or is it the the frequency or is it the like the other characteristics, for example, FMC nebulizer versus a flash LiDAR or a rotating LiDAR and so on. There are pros and cons across the board.

There is no perfect solution in any in any of the sensors, and the tradeoffs have to be made. So if you are making the right decisions about those tradeoffs based on the data you're collecting, then you will be in. And they are inherently making better decisions along the way. And that's how I would summarize it with that one thing. Yeah, No, that's perfect. That's perfect. Two more questions for you. Want one?

I know you mentioned at the beginning, you know, in 17 when you founded a company, Right. Excited about that. In 2021, we're going to drive around in the autonomous vehicles. Here we are today and we're thinking maybe in five years again, right. With the mindset of what are you looking forward in five years in a can be that maybe in five years we are going to truly have a more large scale deployment of autonomous vehicles. But what specifically are you looking forward to in five years?

What's to happen in and even in the space you're working in, in the AI space, annotation space, different types of vehicles? Does it matter? Again, podcast is about mobility. So what is your excitement going to be in in five years? What are you looking forward to to happening in five years? Yeah, yeah. The, the top of my list would be ADAS. I think autonomy at scale economically, like what's Waymo and Cruise and Motional and Zoox are doing. I think it's lofty and we will get there at some point.

But to make that economic meaning that the cost of the vehicle, the density of the operation, the cost of map collection and so on to make that economic, I think that will take a while. So I'm a lot more excited and hopeful in the capabilities are going to come from driver assistance. In that it has immediate impact on safety. It will immediately save lives. It will immediately affect vehicles on the road today. We don't have to wait 15 years for that to happen.

It will it will bring benefits to the drivers as well as to the OEMs, because they can settle these features at a slightly higher cost, hopefully better margins and and bring the benefits kind of to society along the way. Sure. Sure. That's perfect. Perfect. And final question for you, What is the next car are you going to buy and why? So the next car would be electric. I think that we we've, we've gotten to the point where I am I am ready to own an electric vehicle. Now, which brand?

That is still to be determined. I, I have been driving a Volvo XD 90 for a long time. That is an electric version of it coming. It is also a number of exciting electric vehicles that are SUVs that are happening. I think Lucid is working on gravity. Obviously that is Tesla model X. I think Volkswagen has launched the ID six as well, so I haven't done that quality evaluation of electric SUVs just yet. But if you have any hints or suggestions, I'd love to hear them.

No, I mean, the problem is right. Every day or almost every month, there is more coming out, right? I mean, you have now Fisker selling cars. You have Vinfast and now you suddenly have some of that startups or other newer transplants coming in. Certainly. I mean, if you look at what Hyundai has done in electric vehicles over the last few years, very impressive.

And then clearly the local the U.S. OEMs stellantis more of a global or maybe more of a European OEM, but clearly also bringing products to the market. GM with a heavy onslaught of a lot of different options and for doing, I think, a great job with the Mach-E and with the Ford Lightning pickup truck, I think pretty soon you're going to and it's going to be more and more difficult to do.

So maybe, maybe deep needs to come up with some AI based data analysis program to help people buy the next electric car. I don't know. Maybe we're on on to some big idea. Hey, hey. Maybe we can work together on that. You know, Walmart, thank you so much for your time. Great explanation on what you guys are doing and how important data is and how important it is to have the right sensors.

And as you so properly said, it doesn't really focus on the data, collect the data, analyze the data, are the ones that are also more successful in implementing these systems are very accurate. Thank you so much for your time and thanks everyone for tuning in. Thanks for listening to Reimagine Mobility Podcast. If you like this episode, please subscribe and tell a friend.