Episode 80: Seeing AI

00:00:10 Matt Welcome back everyone to the Xamarin podcast, keeping you up to date with the latest and greatest in Mobile development for xamarin developers covering the world, the xamarin.net, Azure and more but not today. Today we're kicking out a special series for others Xamarin podcast where we talk of two creators of amazing apps built with Xamarin and 1st up. 00:00:30 Matt Is seeing AI seeing AI is an incredible app that uses artificial intelligence to help blind people see the world around them. I'm Matt Soucoup and today I'm joined by Saqib Shaikh and Alexi Struch. Saqib let's. Let's start with you, tell us a little bit more about yourself. Every time I go to. 00:00:51 Matt The seeing AI website on Microsoft I see a photo of you, so you're a little bit famous for me, so tell me a little bit more about yourself. A little bit of background and how you got into development. 00:01:02 Saqib Hi great yeah so I'm a software engineer here at Microsoft and I lead a team working on sort of tech for good solutions. In particular as you said, seeing AI. 00:01:16 Saqib Which is a talking camera app for people who are blind, so. 00:01:21 Saqib It's been a long journey. I can definitely tell you a bit more about how we get started and so forth, but. 00:01:27 Saqib Um, yeah, that's me. 00:01:29 Matt Wonderful and Alexei, How about you? 00:01:32 Alexey My role is I'm from Mobcat team which is mobile customer advisory team and we help him helping our customers to achieve more to help them with their issues. They facing while developing their amazing. 00:01:48 Alexey Apps, basically it's performance issues, architecture issues, some probably user interface layouts, and we're doing it all in Xamarin. 00:01:59 Matt Wonderful, so I want to hear a little bit more about the origin of this app where it came from, so it's like I could you fill me in on where this idea came from and then the initial building at the app tell me how you found out about this or where it came from the idea an how you started to initially build it out. 00:02:21 Saqib Wow, so you could say this or originates back to when I was in University many many years ago and you know you always have this futuristic vision. Studying artificial intelligence and thought you know, wouldn't it be great if I could have like a pair of glasses or something that would just tell me what's around me at any moment in time and I should say that I am blind myself. 00:02:43 Saqib And so you know, is as they say, scratching my own itch. And you know the stage. An idea because, like you know, sci-fi dreams. But then. 00:02:53 Saqib It's 2014, Microsoft had its very first hackathon and this is a company wide hackathon with thousands and thousands of people from across the company participating. 00:03:05 Saqib And it was a little bit special. We had so many hack funds at Microsoft. Or, you know, just a few participants. This just felt different scale and I wanted to do something big. So I kind of thought. OK, let's let's dream big. What would happen if and that was where V1? You know the very first hack, the first prototype came from. 00:03:26 Saqib And then just turn 14, then you go to the next year and you know for the next year I was pushing this on as a side project I was working on Bing and Cortana at the time and the sort of back end engineer and the next year. I was looking through the hackathon project and looking for other people who might. 00:03:46 Saqib Beach did the same thing and I found a bunch of people in Silicon Valley an and they wanted to do computer vision for blind people too, and we teamed up and sort of overtime. The whole project snowballed we that year we end up with like 30 participants. Then we got some participants over in Vancouver and really all around the world it became a bit of A. 00:04:07 Saqib A mission with more and more people joining and lending their expertise and that just continued and. 00:04:12 Saqib Yeah, eventually became part of my day job event. Soon after that we got on stage with Satya at their build 16 Conference. And yeah, they they came in app and it kept going. 00:04:24 Matt That's amazing, I had no idea that it started out from a hackathon project back in 2014. I remember the build. 00:04:32 Matt With when you're on stage with Satya an I'll put a link to the video that they did with it and it's just an amazing. It's a tearjerking video. It's so so good, but I had no idea started off with a hackathon project back in 2014. That's that's that's great. And now, as you mentioned it is here it is your job did. Was it always xamarin? 00:04:52 Matt That you did it with? Or was it something? Did you start doing it with just like objective C and iOS? We should mention the app seeing AI is an iOS app and am I right? 00:05:02 Saqib Yeah, that's right, exactly OK. 00:05:04 Matt OK, and so did you. Start off with objective C or did you start off with xamarin? 00:05:09 Saqib Haha neither so 2014. It was a Windows Phone app because that's what I knew how to develop for, so I I think I made or you know I tried making it as a web app. You know just hacking around but I thought I'm a server side depth. So how could I do this just in a web view? And then I thought no. So I've done some WPF winforms, so let's let's go to Windows Phone for that. 00:05:32 Saqib Very first prototype. 00:05:35 Saqib And then we could have the next year. We were all, I think, yeah, that version was objective C because we had an iPhone developer on the project and I also knew objective C and I did a bit of iOS dev by this point in time, but then eventually went down the route of what if we had smart glasses. So once again we switched all over again and it was all Linux code. 00:05:56 Saqib Running on a pair of glasses and, you know, commercializing something like that was hard. So then we came to this realization that really the way to get the benefit after as many people as possible is through a mobile app, 'cause that is the Super computer in your pocket. 00:06:12 Saqib And we we sort of had a big discussion of week ago. React native where we gonna go native iOS. We're going to Xamarin. It can talk a bit more about that, but especially at that time we we we just really felt that Xamarin was the ideal choice for us. 00:06:28 Matt Oh, that's great, and so I guess let let let's dive into that just a little bit more. Why? Why did you pick xamarin? What was the deciding factor? What made xamarin stick out at the end of the day? 00:06:41 Saqib There were several and like I said, I sort of built apps to like toy apps in many different languages and just just to play around. 00:06:49 Saqib But then I think now we're talking late 2000 and. 00:06:53 Saqib 15 now I guess that was productive. Probably you know mid to then 16 so at that moment in time. 00:07:01 Saqib I just I just really, really like the C Sharp language and this is before Swift and Kotlin and I was like, you know. 00:07:09 Saqib At Microsoft is gonna be much easy to find C sharp developers. I like the fact that you could develop on windows with a remote Mac build host. 'cause again not everyone in the team had. 00:07:21 Saqib How to access to a Mac and Mac laptop and so those are true factors. Plus you're just really liking C Sharp. 00:07:30 Matt Oh great. I mean, that's one of the claims of Fame for salmon, right? Is that you can write C Sharp and you reuse the business or the application logic everywhere? So Alexi, I want to turn to you. 00:07:43 Matt Back when you first started and Development, what made you pick xamarin over going with like objective C or any of the other? Like react native or something like that? What what drew you to Xamarin? 00:07:56 Alexey Uh, I actually started from Windows Phone development as being a.net developer. That was the most straight way to mobile development. 00:08:06 Alexey And at that time objective, she was not the best language you want to use to develop mobile application and I didn't really like that or understand that. So I decided to go with Windows Phone and develop in.net and then gradually when Xamarin arrived with all the SDK's. 00:08:26 Alexey I was able to reuse the same skills basically for iOS and Android, and I was Super Happy because at that time it was three platforms and you can develop cross platform app which will run natively on all three platforms and for me it was a straight learning curve which is. 00:08:46 Alexey It wasn't that hard to adopt, so I guess yes. 00:08:50 Matt The thing I love about salmon, it's if you know C sharp or you know that you're kind of. You're kinda almost a mobile developer there. There's always going to be the platform specifics that you're going to need to learn, but you're you're there just a little bit. 00:09:03 Alexey And to add, it's hard to learn everything at once, so learn every platform specific feature of iOS and Android. 00:09:11 Alexey And on top of it also language specific features while keeping.net in your pocket, you can learn easily all the native or platform specific stuff and then gradually when you need eventually switch to Swift Kotlin and that's what I'm doing now. I'm basically not no longer tide to csharpand.net. We actually doing lots of native stuff using swift and Kotlin. 00:09:36 Alexey And bindings for xamarin adoption. This all the Amazing HD case. So we are no longer tide to C Sharp. 00:09:44 Matt And he actually wrote a great article on the Xamarin Blog talking about how you can take swift libraries and bind them to C Sharp. And I'll upgrade. I'll put a link in the show notes so people can read it. It's a really interesting article. 00:09:57 Alexey And same article. About Coughlin. 00:09:59 Matt Oh yes, right, that's right. Can't forget about Kotlin. And that's actually a pretty neat cross platform. 00:10:05 Matt Framework by itself, so sick if I want to ask you about the application itself. Now I have it up and running on my phone and it has a couple different. I'd say several different functionalities for it. There are things called like short text, document, product person. So can you walk me through just not all of 'em but just some of 'em of what they. 00:10:27 Matt What they do and why they are, why they're so helpful. 00:10:31 Saqib Absolutely. So we think of this. I mentioned earlier talking camera app, so the main screen is your camera view, but then you have the ability to recognize different things and we have a number of different channels we call them along the bottom in the carousel and so we can recognize short text, which is like short snippets of text. 00:10:53 Saqib Which I just read out loud in real time. So we imagine that you might just want to glance around to see this envelope. Who's it for? Is it from or other products or just piece of paper just to get their glance? 00:11:08 Saqib And we also have channels for reading a document where you want to get a more accurate scan, and you're going to spend the time it's going to help you line up a picture by finding the edges of the document for you. 00:11:19 Saqib And you can do face recognition to identify people so you know who's around you and a whole bunch more we can sort of describe the scene and then also let you explore it with your finger. So as you move your finger over the screen, you're going to hear whereabouts the different objects in the image are. 00:11:38 Saqib And then, in addition to being camera, we can also browse the photos already on your phone or share photos from other apps, which is something users really love. 00:11:46 Matt The scene 1 where you can actually move your finger around and actually see things or have it describe particular things in a scene that's that's neat and I didn't know that when I was playing around with that I did have to admit I was playing around with the person want it, took a selfie and it told me I was a little bit older than what I am going to have to talk to you. After this showed I see if we can get that fixed. 00:12:07 Saqib As well as statement, yeah. 00:12:11 Matt If Matt then so what's your favorite functionality? What's what's your? The one that you're most proud of that you developed that's in here? 00:12:19 Saqib Wow, that's that's the timeline I think. 00:12:22 Saqib If we could have more than one, the short text is pretty cool. Is the one that I think you first launch into people really like it because it's just immediate. It's reading to you straight away. As you point, point things like using your camera as a pointer and it just starts reading the explore one, exploring, exploring photo where you describe. I'm quite proud of that cause. 00:12:43 Saqib It could involve using sort of setting different audio cues, and haptics and an also the AI for recognizing the object. So that's pretty cool. Bringing all that together. 00:12:56 Saqib And. 00:12:57 Saqib I get some sort of neural network, sort of based barcode detector which then goes to recognize products and stuff. That's kinda cool too, so I guess maybe we're going to see everything. But yeah, there's some of my highlights. 00:13:09 Matt Oh, that's really cool. So let's talk about how some of the AI in it works is it? Are you calling out to like any cognitive services, or is it all like core Emil baked in? 00:13:22 Matt We don't have to get too down into the weeds, but if you can kind of give our listeners just a little quick tip or some tricks on how everything is working within it. 00:13:31 Saqib Yeah, totally. So there's sort of a combination of different technologies and techniques used here. So when we need to use more heavy AI, which requires the computational power of the cloud. 00:13:44 Saqib Then absolutely, we're just calling cognitive services via the APIs that are available. 00:13:50 Saqib However, the best experiences on device B 'cause like I said with a short text, you want to point it at something and you want to get that immediate feedback, and so for that we are training models in the cloud. But then we are bringing them down onto device with caramel. 00:14:06 Saqib And then another factor in choosing xamarin is that we also use several native libraries and that was a factor that Xamarin lets you bind to those native libraries. So that was. 00:14:20 Saqib Very helpful as well. 00:14:21 Matt Like you said, the short text one is great and one of the things that we do in my in my day job, so to speak of. I do a lot of writing with documentation and one of the things that we like to have is 5 minutes to delight where somebody can sit down and read this quickstart doc and actually have an app that's up and running and so this short text. 00:14:40 Matt Is like 5 seconds to delight you. Download it from the App Store. You launch it and then all of a sudden you're reading things on the on the application. It's just it works. It works quick and it's it's amazing. This app is for everybody. It's not just for blind folk, but I mean it could be for everybody to make their day- to-day life easier I think. 00:15:01 Saqib Totally so we still hear from. I remember my favorite story like that is hearing from someone who wanted to read the serial number of a like the asset tag of a computer. And so you know he's crawling on the desk underneath the computer. It's all dark under the desk and so he was using seeing a short text to to read the. 00:15:21 Saqib Number around the back of the computer. 00:15:23 Matt I was talking a little bit before with Alexi an asking you know what he did to help this app work better and he told me that he moved in and did some refactoring to it. When I hear refactoring, I hear that we we bumped up against something so. 00:15:39 Matt If you could tell me a little bit more about when you needed to do the refactoring, what I'm this is all a long way of asking. 00:15:47 Matt What didn't you like about Xamarin? 00:15:50 Saqib I think that with seeing AI we bumped up against some of the edges. As the app grew. So when you file new project and everything is fresh and straightforward, everything works really well, but sort of over the years and we've been going for, you know, four years, maybe only guess now. 00:16:07 Saqib You just get more and more different types of challenges, so I think I sort of went to Alexian team to say, you know, our build times are really really slow. Can you help us and? 00:16:20 Saqib You know, especially when you're doing that awesome dev on windows across over to a Mac and the app size is also just growing really big. So Alexi came along and did some incredible work to increase the build times and simplifies our project structure and so forth. 00:16:39 Matt Alexa, you can guess what my next question is. 00:16:42 Matt How do you go about doing that? Making the build times faster? Decreasing the app size, and helping with the project structure? 00:16:50 Alexey Yeah, that was amazing journey. It was a pretty long journey an we it wasn't just a single task to accomplish to make it faster we had a. 00:16:59 Alexey List of activities to be done. 00:17:02 Alexey An refactoring was one of them. Also, we've changed some settings. We moved some classes around. 00:17:09 Alexey We switched from shared project structure to dot net standard to compile specific for specific platforms and its altogether work to to improve the development cycle and while working on the project I felt that pain when you. 00:17:30 Alexey Trying to do a small change and then wait for a few minutes while it's compiled, and then you execute that. So I was pretty engaged on getting that thing done and that was it. Intense refactoring. We worked together with psychic here also was changing code. 00:17:52 Alexey On top of mine and we work as a team for for quite a long time. 00:17:56 Matt What's what's the architecture patterns that are being used in the app? If I can ask? 00:18:01 Matt I'm gonna go out on a limb and savior. This is not a xamarin forms app. Am I correct in that? 00:18:06 Saqib No, yeah, it's totally xamarin native and that's what I love. I I I can't tell you how appreciative I am that Xamarin, in addition to forms, lets you do pure native development and that's in terms of the Accessibility. And we're just using so many iOS specific APIs for. 00:18:24 Saqib A smooth, uh, sort of? Not just move, but really exemplary accessible experience, but then also, Carmel and haptics, and a whole bunch of other things which you need to get left before. So yes, native app. 00:18:39 Matt Native app and are you using 'cause I know over on? It's been awhile since I've done any iOS development work with objective C, but it was always like an MVC pattern. There are you following any of those patterns. 00:18:53 Saqib Yeah, it's pretty much similar to MVC with, you know, bit of homegrown stuff thrown in following a typical iOS pattern with your view controllers and so forth. 00:19:03 Matt OK, very cool and we mentioned a little bit about Azure before so we're using cognitive services. Are you using any other Azure features at all? 00:19:14 Saqib And so we have our own back end, which sort of wraps are called calling service and so forth. And that's all in Azure App Service and we use like. You know, if you listed all the outputs of Azure, you probably. 00:19:25 Saqib Come up with 20 different little services that we use with that key vault or app insights or app center or. 00:19:35 Saqib Yeah, table storage, blob storage and Azure functions and digital like it keeps going on. 00:19:41 Matt What's your favorite Azure service? 00:19:45 Saqib I think it's app insights, so we use that for our Telemetry and I love the way that we can query Telemetry with this programming language called Kusto and. 00:19:58 Saqib Because I'm like command line inclined, personally I have my own little command line tool for querying them whenever I want to find out, you know what's the monthly active users or you know which country are people? Mostly like I can just go to command line, ever tell ripple type in a few commands and and you know put my finger on customer behavior and. 00:20:19 Saqib I love that. 00:20:20 Matt So I'm asking not only because I'm super interested, but also I'm picking up so I can have a next Azure app service of the month and maybe it's going to be application insights. Lxe, do you have a favorite Azure service? 00:20:36 Alexey Um? 00:20:37 Alexey That's an interesting question. 00:20:40 Alexey I think everything related to mobile services probably push notification services push notification hubs. 00:20:46 Saqib I'm also going to add custom vision. There's all the cognitive services are pretty impressive. How you can just build intelligence into your app, but you know for listeners some there's a service called custom vision. You can go do custom vision dot AI and let's let's you train your own machine learning models by dragging in images, and then you can train a classifier. You can export it to Carmel and. 00:21:09 Saqib We don't use that so much in seeing AI. We train our own models, but it's an incredible way to get started in this if listeners want to. 00:21:18 Matt Yeah, custom vision is an amazing service and what we touched on it. But you can train anything you want with it. Like if you want to be able to identify different Flowers with it, you can if you want to be able to identify different trees with it, you can. It lets you do whatever you want and you can download it a core Emily. 00:21:38 Matt Model for it and an installed on your phone or a tensorflow model as well so it's totally expandable. It's totally customizable and you train it an you give it a. After it's trained, you can upload a sample image to it and you can like say, essentially you're testing it at that point, and whether it's good or not and it's. 00:21:59 Matt It's a stupendous service an I believe there is a free trial level for it to seem to get started, and it has a wonderful user interface as well, and I wish more people knew about this thing because it's it's great and it's not like a cognitive service where you have to call out all the time where you're making web calls all the time. 00:22:19 Matt It's loaded on your phone, so you do get a little bit of a faster response time to it. Yeah, thank you for bringing that up. That's an amazing amazing service. 00:22:28 Alexey I can end it. I also like that service because we tried it for one hackathon project an it was pretty amazing. We did it it on device so it also provides you API's to. 00:22:39 Alexey Access every feature you can only not only upload images, but also trained model and downloaded everything using just API's. 00:22:48 Matt It's an amazing service and I'll put in the show notes how to documentation behind it and how to get to it and everything else. So I would highly encourage everybody to go out and try to get at it. Skipper Alexi. Is there anything else that you would want to bring up about seeing AI? 00:23:05 Saqib Well, first of all, everyone should just go out and. 00:23:08 Saqib Download from the App Store and try it of course. 00:23:13 Saqib But yeah, there's just so much more we can do. I think in our team we're really just thinking about what are the emerging technologies coming down the Pike? What is the incredible work being done across Microsoft? And then how do we use that to help more people? And you know, primarily, right now we're seeing AI. That's people who are blind. But we also hear from people. 00:23:33 Saqib With learning differences or English as a second language that the some of these technologies are also useful so. 00:23:39 Saqib Um yeah, give it a go and there's just always more that our team is looking today. 00:23:45 Matt Oh, that's you know, I'm gonna. I was wrapping up the podcast, but I'm not going to dig back in. So can I ask you about some of the future road map that you have because the ESL Alexi and I were talking about English as a second language? ESL stuff. Before we kick this off and that's that's really cool that you could work expanding, saying AI. 00:24:07 Matt To allow that or save enable that. 00:24:10 Saqib So it's not so much. This is not specifically so. Senior is remaining primarily an app for people who are blind. That's not a scenario that we are enabling per say, but it's just something that we hear from customers who have found it useful for that purpose. So I wanna be, yeah, like. 00:24:28 Saqib You know the fact that you can take picture and hear a description or you can point to an object and hear what it is in English. So I think it's one of these things. As engineers we put things out there that we built for one purpose and that might be the 99 use case. But then you hear from so many other people with so many different needs that an there. 00:24:48 Saqib Finding for your app. 00:24:50 Matt Sure, end users are ingenuous, they'll find they'll make your app work in a way that you never intended to. That's all we have to test. 00:24:58 Saqib I gotta say, wait, yes testing. Absolutely by far my favorite story about users using the app in ways you didn't imagine is in the beginning. We did not have a special channel for recognizing currency like we do now. 00:25:11 Saqib But we just have face recognition and so we heard from a group of users who had trained the face recognition to recognize the presidents on and on the dollar notes, and so they had made their own currency recognition through face recognition. I thought that was that was incredible. 00:25:29 Matt That really is something else. I mean, I guess I would never thought of doing that. 00:25:33 Matt But it makes total sense, right? 00:25:36 Saqib Exactly. Never thought of that. 00:25:39 Matt Well, very neat. 00:25:40 Matt So I guess I guess with that said. 00:25:45 Matt We should have like a pic of the pod here. This is our first one doing the what I'm gonna call the customer success stories or the customer stories podcast. So I'm going to say a pic of the pod, an Alexi. I'm going to start with you. Are there any applications out there? Any apps that you're using right now that you think are super cool that you just want to talk about? 00:26:05 Matt Give our listeners a glimpse into. 00:26:08 Alexey Just recently started to use an app with selfie a day. I don't know you doing a selfie and I do it time lapse for you. It's a pretty well developed application and I like that. So an I like the use case when you doing something. 00:26:26 Alexey Every day like it could. 00:26:27 Alexey Could be not only selfish. 00:26:29 Alexey He trying to track the progress. 00:26:31 Alexey And that's that's a very interesting concept. 00:26:36 Matt I also did an app like that or use an app like that where I took a selfie everyday. I did it for like 3 years straight, which is amazing and then it. 00:26:45 Matt It ended when as soon as we had like the, The developers stopped updating it and then it just iOS updated in the app. Anyway, when it run anymore. So that was an end of that app alright, so give anything anything you're using out there that you think is super cool right now. 00:27:02 Saqib I've been there so many, though, you know, when you ask just on the spot I'm like. Actually I can't think of a single one thing that you know I I'm sort of scrolling through my phone and thinking, you know what are the absolute recommend an I think for developers that I would definitely just recommend. Take a look at Azure Cognitive services if you haven't already. 00:27:25 Saqib There's just so much cool possibilities of things you can make out there. 00:27:29 Matt Yeah, cognitive services. It's actually expanding every time I go in there. There's like this new little not going to say little, but new functionality out there that I didn't know about before. Like form recognizer. That's been out for a little bit, but it mean. 00:27:43 Matt It's it's really neat on how all these. 00:27:47 Matt I'm going to say intelligence in the cloud is out there and is at our fingertips, says at developers that we can just go ahead an integrate and it really is. It would amounts to is just making an A rest call to these services and you can really add a lot of brains to your applications. 00:28:05 Saqib Totally my favorite maybe is only for the most recent, let's say. 00:28:09 Saqib Is the neural TTS. This is text to speech where you can give it some text and it gives you audio back and my goodness since the last time I checked the quality and the human Ness of the audio just improved so much. So definitely. Just go and explore and take a look. 00:28:25 Matt Wonderful, Alright Alexia Saqib. Thank you so much for joining us in the very first zamber podcast. Customer success stories. I appreciate it. I know our listeners will so thank you. Bye bye.