People want additional information in addition to something that specifically answers or satisfies their intent. Hi, everyone. Welcome to a new episode of Data Stories. My name is Moritz Stefaner. I'm an independent designer of data visualizations. In fact, I work as a self-employed truth and beauty operator out of my office here in the beautiful countryside in the north of Germany. And I am Enrico Bertini. I am an associate professor at Northeastern
University in Boston, where I do research and teach data visualization. Exactly. And on this podcast together, we talk about data visualization, data analysis, and generally the role data plays in our lives. Thank you. Usually we do that with a guest we invite on the show. Yes, but before we start with our guest, a quick note. Our podcast is listener supported, so there's no ads. If you enjoy the show, please consider supporting us with recurring payments on patreon.com slash datastories.
Or if you prefer, you can also send one-time donations on PayPal. by following the link on to paypal.me slash data stories. Yeah, in the last few days, we've just seen a few new donations coming in. That's always a great feeling and always wonderful, much appreciated, especially if you send us a little funny note along with it. It always makes us smile. Yes, thank you, thank you.
Yeah, and also if you can afford it and do the recurring thing on Patreon, this definitely keeps us going and it's again much appreciated. So let's get started. We have a cool topic today. We'll talk about language and data visualization and all that surrounds that. But before we dive into that, quick note on the Informationist Beautiful Awards. So some of you might remember, we had Amanda McCulloch from the Data Visualization in Society on the show a few weeks, months, can't remember, ago.
And we mentioned that the awards are being rebooted, they're a bit like the mini Oscars of data visualization, and really try to celebrate all the diversity and variety and excellence in the field. And now the winners are announced. We didn't win anything, Enrico, right? Yeah. But a lot of other great people, which is awesome, too. And yeah, on the website you can see all the winners. Shout out to our podcast colleague Ali Turvan, who won Community Leader. Much deserved. Congratulations.
Lots of other great people. Yeah, it's great to see this event is back. I'm here. Yeah. It's nice to have a celebration, you know? Yeah, but let's go to the main topic. Our guest today is Vidya Settler. Hi, Vidya. Thanks for joining us. Hi, Moritz and Enrico. Thanks for having me. Yeah, welcome on the show. And thanks so much for joining us. Can you tell us a bit about you? What's your background? What are you excited about? What are you currently working on?
Sure. So I'm Vidya Settler. I head Tableau Research. I've been with Tableau for 10 years. I got a PhD from Northwestern. My background is in natural language processing and computer graphics. So I'm generally interested in the problem space of understanding the semantics of data and how that can help inform meaningful visual depiction of information. So Tableau has has been. the perfect place for leveraging that skill set. Before I joined Tableau, I was at Nokia Research for seven years.
I manage a wonderful team of interdisciplinary research scientists who work on problems in the space of applied ML, multimodal interfaces, HCI techniques, augmented reality, and so forth. Already back in the day, augmented reality. I know, it's all coming back full circle. I know. It's all coming back full circle. Oh yeah. Oh yeah. Yeah. It was nice too today is that we have a true long distance call all around the world. Enrico is on the east coast in the US, I'm in Europe.
Where are you based right now? Right now I'm actually in Mumbai in India. I'm teaching at the GEO Institute, teaching a course on data visualization, but incorporating aspects of semantics and intent, some of the NLP with data visualization stuff to a class of 60 students. So it's been a really great experience. Nice, yeah. So that's a little break from your usual like research work at Tableau. Sort of. I mean, I'm still kind of doing both, but it's been really refreshing to interact with the
local population. And for me, it's just a way of sharing what I have learned over the years with people from my home country, India, and just helping them understand the value of data visualization and why it's important. So, I feel like India has reached that stage now where everybody understands the value of data visualization, but they don't necessarily have the training and the access to resources. Um, so Yeah, there's so much going on in India. I actually visited last summer.
And for the first time, yes, for a wedding. So for an Indian wedding. So that was really, really special. It's a good way to get started. The full package. Yes, exactly. So Vidya, I think one reason why we wanted to have you on the show is because you recently published this book together with Bridget Cogley called Functional Aesthetics. And reading these books, me and Moritz realized that there's a lot in the book that is related to semantics and language, which interestingly is not.
that much, discussed that much in the world of data visualization, right, and data analysis as well. So we thought we would focus this specific episode on the relationship between semantics and language and data visualization. So maybe we can start by exploring or understanding what are the main connections between data visualization and language and how did you get into this specific topic. Yeah, I honestly got into this topic kind of serendipitously.
In general, I think information is meaningful when you understand what it's about, the context of it, it gets enriched by additional information, and when you have a better understanding of that information, you can figure out meaningful ways of communicating what that information is it's about. And given that we are very strong visual creatures, a very logical place to start is conveying that information visually to a user.
So a lot of my previous work before I joined Tableau was exploring the space of. figuring out how to effectively communicate large pixel, megapixel imagery on small mobile screen devices. This was a time when smartphones were big and that was primarily the work I did at Nokia. And there are a lot of connections to data visualization from the graphics space.
In a lot of respects, I would say, it's a much more tractable problem because in the In the graphics world, you're trying to find semantics within pixels of information. But with data visualization, there's a lot more structure and built-in semantics already that is provided. We know the chart type, we know the mark type, we know the attributes, we might know the attribute types. So there's a lot that we can start off with.
So it really gives an extra foothold in terms of coming up with semantically meaningful ways of depicting that information. So I kind of started my journey at Tableau exploring ways in which we could provide smarter defaults to data visualization. So we noticed that on Tableau Public, users spent a lot of time going to the web, trying to find images, creating icons. and associating them with sports visualizations, with their favorite sports teams, or flags, or other types of icons.
So, this was a project that I started with Jock McKinlay when I joined Tableau is, can we actually figure out a way to use natural language techniques to effectively query image databases to get back these images and use graphics techniques to come up with a visually consistent,
yet aesthetic palette of icons and automatically suggest that to the user. So these were these little, you know, micro problems along the flow of analysis where I was exploring how can we provide reasonable defaults to reduce the friction so that people don't have to step out of their workflow. do these manually intensive tasks and come back. And so that's where I started.
And then moving on, I started exploring natural language interaction with data, which we can talk about in more detail in a bit. But there is this general structure of using language and grammar for both defining charts, you and also asking questions of charts because at the end of the day data visualizations are a manifestation of visual communication. So communication is the underlying theme and the best way to think about communication is through language.
Yeah, I think that's, yeah, there's so many cross connections there, right? It's like, Have a conversation. Even the metaphors that we use. Yeah, exactly. And that all shows us already that implicitly this is happening. And I think what's fascinating is that your work makes it explicit and really creates that tight link between language and visuals and it's very, very interesting.
And I guess EVISA is a project that was sort of seminal there, both for you personally, but then also probably for the field. Yeah. For the first time allowing, in the paper it says, it's a prototype system that enables a user to have a conversation with their data using natural language. That sounds like really exciting. Can you tell us a bit about it, how it came about and how it developed? Yeah, for sure. So the name, actually, Eviza is inspired by the MIT chatbot assistant, Eliza.
And I love the project so much that my vanity license plate is also called Eviza. So true Bay Area techie at heart. You should send us a picture. put it in the blog post. Yeah. And I, I bump into people randomly at grocery stores saying, Oh, I saw your car in the parking lot, so you must be somewhere in the store. Um, so it's great. Yeah. So, um, I, the, the underlying premise of what I think, what made, uh, advisor interesting was.
Communicating with data and asking questions of data using natural language was not a new problem. People have done it for years. I mean the database community started with it where they had these pseudo-SQL statements and they were asking questions of their data. What made EVISA unique, and I want to acknowledge that it was a collaboration with other members of the research team, was we were really looking at not just single queries to data and getting back a response.
As you indicated, Moritz, it's an actual conversation with data. So for example, if I am talking with you and I'm talking about, you know, my evening here in Mumbai, you will respond and you will respond. you will ask me a follow-up question. You might say, what is the time? And I might say, you know, it's 8.30 in the night. So there's this back and forth exchange where questions are not restated
every time. We build on the previous context and knowledge and have this sort of natural progression of dialogue between me and and yourselves. And so I was really interested in emulating that human-to-human conversation behavior with respect to a computer around data. So I looked at what we call language pragmatics, which is looking at the context of the previous query. So if I ask, show me the large earthquakes near california My follow-up question is, how about near Texas?
I'm not restating, show me large earthquakes near Texas, because that's kind of unnatural. So that was kind of the first, I would say, novel contribution of that work, is just really implementing this pragmatics model in the context of an analytical conversation. The second insight with building the system which actually got Tableau leadership excited was. There was a previous concern that these natural language systems needed to be perfect.
They needed to completely understand what the user was saying, and they needed to generate a perfect answer, otherwise, people would get annoyed. I think some of it largely came from systems like Microsoft Clippy and other places where if the system constantly made an error, then people would get annoyed. But if you actually think about the nature of human-to-human conversation, our conversation You patterns are not perfect.
I might, I mean, well, Enrico, you thought I was in Bangalore, and I said, no, I'm actually in Mumbai. And I corrected you, but I was not offensive, right? I mean, it was just a conversation. Hopefully I didn't offend you. Thank you. Maybe the system comes up with a reasonable guesstimate of what a large earthquake might mean and might set it up to a magnitude of six and greater on the Richter scale.
and surfaces that reasoning back to me and says, hey, you know, you said large, I wasn't quite sure what you meant, but I took a guess and I, you know, assumed you said six or greater. And I might, you know, come back to the system and say, no, it's actually, you know, I live in California right by the earthquake fault and I'm not joking. So, you know, large earthquakes to me is anything four or greater. And I might tweak the slider that the computer produces to me.
To me, when we actually tested it with users, A, users were actually not bothered when they had to correct the system because it was just a small tweak in a slider, and B, the system actually remembered that setting so that when a user asked you know, about large in a subsequent interaction, the system remembered that and the users were
delighted by that. It's like, oh, you know, actually, you remember the context, right? So if Enrico remembers that I'm actually in Mumbai, I'd be like, you know, that's great, Enrico, So you have a great memory, right? I mean, I'm actually going to compliment you. So... It turned out people actually love that sort of back and forth interaction.
So something that was previously viewed as a hindrance or a limitation was actually a strength in terms of modeling human to human dialogue, in terms of a human to computer dialogue with respect to data. And as I indicated, You know, data is a lot more tractable because it's not... trying to make a model of the entire universe, the universe for the system are the bounds of the data. So all of that made it a very tractable problem to move the work forward, including productization.
Yeah, that's one aspect that I really like of the type of research that you and your team have done over the years that is not, I mean, the technical contribution is remarkable, but what is really interesting is that you have done a lot of also... human factors type of research on top of it and trying to understand what happens when people
interact with this type of quote-unquote intelligent systems, right? And I think that there's so much to do in that direction and really understanding how how users' behavior also changes as they learn how to... how to work with these machines. One thing that comes to mind is, like, I guess even the way we use Google, we just adapt because we know that... We don't ask Google the questions in the same way we would ask to a person. Or to Alexa, even. Or to Alexa, exactly. That's right.
So I was wondering if you observed the same kind of behavior where after a while people just know how to ask the thing so that they will get the information. or the results that they want? That's a really excellent question and insight. So what we have found over the years, even with search systems and natural language interfaces is people don't like to be wrong. They don't like failure. So they will adapt to the computer speak of the system.
Humans really understand very quickly what a system can or cannot do. and readjust their model and their view of their interactions with the system, which is good in a way because then you don't have too many frustrated users, hopefully, and they sort of adapt to whatever the system can support. But there is a downside to it.
When you're looking at the telemetry data of how people are using systems, the window of opportunity to understand what are the types of queries the system cannot support, and what would make for useful extensions and improvements to the system is a very narrow time window.
what we have found is within a couple of days, people will change from asking questions that the system does not understand or partially understand to only the questions and the repertoire of analytical functions that the system can support. And this includes search. And so you immediately see a drop-off of failure rates for all these users.
So you... we really have to catch these new users within a one to three day window, which makes it kind of challenging when we're using customer data and telemetry data to help inform how we can improve the system. Yeah, yeah. Yeah, that's a tricky aspect, is the discoverability in terms of... What can I even ask, right? It's a bit like with gesture interfaces where you just try and figure
out, okay, maybe swiping works, maybe zooming, right? It's the same with language because first probably you ask anything and then you realize, yeah, it can't quite do everything. And then you're like, yeah, that makes sense. You know, there must be some limitation, but then you need to figure out what is the vocabulary I can use. Yeah, exactly. So one part I'm really interested in is also this transfer between research and then putting it into the product.
And yeah, EVISA in some way found its place in Tableau as Ask Data or probably some further development but I guess it all started with EVISA, right? And so yeah, I'm curious, like how does this work? How did this specific prototype and piece of research make it into being part of a product Temporal. Yeah. What's the process there and how did this work and, you know, I want to learn all about the process.
Sure. So, as mentioned, I mean, when I joined Tableau back in the day, I was fortunate enough to interact with Chris Dolty and Christian Chabot and Pat Hanrahan, you know, the co-founders of the company. And we were having these conversations when I joined Tableau, where, you know, Chris Dolty said, you know, NLP is really hard, Vidya. I don't know how we can actually crack that nut. And even if we do crack the nut in research, how are we actually going to productize it so it works
for any customer, right? Because we are not, Tableau doesn't cater to a very specific vertical. And there were, there are previous systems like IBM Watson that have catered to very verticals, including chess or healthcare, but Tableau, in a sense, is more of a generalist type of platform. So that was kind of the problem. But when I demoed Eviza, it's kind of funny because the story there was I was in California
and everybody else was in Seattle. And back then, for various reasons, I was using voice to talk with the advice. I just thought it would be so cool to talk to the data with voice. I mean... you could do text or voice. That was not kind of the main part of the story. But I was like, yeah, I'm going to demo this with voice. And I asked everybody to mute themselves. And I did my whole demo. And I said, don't interrupt me in between, because it's going to catch your voice and mess up my demo.
So I did my demo for 10 minutes. And then I had to rush off to a doctor's appointment. So I just said bye. I didn't even wait for Q&A. And I didn't make it. I didn't really think about it, right? And late in the night, I get an email from Chris Dolte. It's like, OK, can you demo this to Tom Walker, who at that time was making the call in terms of which teams are funded and stuff like that. So long story short, Tableau leadership got really excited about the project. And, um,
they decided, okay, I think it's time that we invest in NLP. And this was also a time where the tech industry was hedging its bets in this space. And this is, I think, one of the things, And I don't think it's one criteria that indicates or dictates what, you know, what's going on. you know, what makes a successful tech transfer. It's a multitude of factors. You need to have some sort of perspective that you can show that it works and the market needs to be ready for that.
The customers need to be ready for that and the company needs to be ready. So there's so much that has, I mean, all the stars need to align. And we were fortunate enough that everything did align. But the problem back then was Um, Tableau was not known to be an NLP company. I was the only NLP person in the entire company, and it was a very small startup. So how do you actually convince people who are working in companies like Google to come and join Tableau and work on, you know, NLP.
So we made the decision to look for smallish startups in the Bay Area, because I'm based in the Bay Area, to try to see if we can acquire some seed technology as foundation and then then I could help build some of the stuff that I had in Eviza on top of that platform. So after, you know, several months of vetting various startups that I was involved in, and this is the other piece, you know, when you're talking about doing tech transfer research
projects, you have to be all in. So that might involve doing technical due diligence for startups, interviewing people. You have to be all in. You have to show that you are motivated to make it work. There is kind of a larger sociological aspect to this. So, long story short, we identified this startup called ClearGraph, and they came on board and I moved from research and joined the product team, starting first as a lead engineer, working on production quality code.
to write production quality code with your peers, including junior people critiquing and reviewing your code. But it was really good for my soul. And I definitely improved my coding chops after that. And soon enough, I became the engineering manager on the team. So, I was responsible for that. you know, all the release planning, working closely with the PMs, doing code reviews, writing code myself,
sneaking in features from Eviza into the product, and writing papers. Melanie Torrey, who is currently at Northeastern, was a very close collaborator of mine. And so she and I would... you know, buddy up and in figuring out the research aspects of the project, you know, how do we actually support intent understanding and semantics so that we can build on top of this platform. And so I stayed on, stayed in the team for a few releases of AskData and then ultimately move back to research.
And like the way it's implemented now, what can you ask AskData, like let's say the earthquake data set, right? What are some of the questions you could ask and where you would get a meaningful chart back? Yeah, so AskData connects natural language utterances to Tableau's visql command stack and the feature leverages show me. to display the top recommended chart that ShowMe suggests.
So we do leverage some of the Tableau underlying architecture and combine it with the parser and the natural language input that's part of Astida. So that being said, we support five basic analytical functions that are tied to Tableau's core analytical stack. You know, you can ask questions about aggregations and groupings and drill downs, filters. and sorting and various combinations of that. We also have vague concepts, kind of like the the large and the near in the earthquake examples.
It, you know, also supports these follow-up questions. So some of the pragmatics behavior as well. Wow. Yeah. That sounds quite useful. I was, it was interesting for me to hear that initially the prototype was meant to be used through voice. Yeah. I don't know if, if it's actually possible to do that with Ask Data. Or is it only through typing? We decided to do text. You know, from a business standpoint, it makes sense.
I mean, if people are all at their desk talking to their data, You know, it may annoy, it may annoy you. No, I mean, to me, voice-to-text translation is more or less a solved problem. I was just using an off-the-shelf API to do that, so it was more of a footnote, and I just thought it would be cool to use my voice and talk to the data. as more of a cool effect.
But, you know, I am exploring, you know, voice with data stories, actually, now as part of my research portfolio, just in the space of text and charts. So it'll probably come back to me full circle. Yeah. And I was also curious to hear from you. I really like the story that you started in research and then went to engineering. And I guess now you're back full time in research. Correct. Yeah. The last mile is probably really hard. The mini-miles. The mini-miles. The mini-miles, right.
It's a big hike. How many years did it take all together, kicking it off? Um, it was about two years. Um, it was about, I mean, we shipped, we shipped our first, uh, feature on time. I would say a little over a year. And then... Mm-hmm. You have to be all in, so if you want the right type of sausage to come out, you need to be part of the messy sausage-making. It's very hard to sit in research and try to get the team to build a product.
based on what you would like to see you have to roll up your sleeves and be part of that process for a lot of reasons, because unlike research, engineering works very fast. every day there are stand-ups, every day there are decisions being made. By sitting outside the team you're not part of that conversation, you're not part of those serendipitous decisions where it's like, oh there's a bug, how do we fix it? Oh by the way, if we fix this we can add this additional
feature. That doesn't happen by setting up a formal meeting every week. So you lose out on a lot of that if you're not part of the team. And then the second thing is when you're part of the team and you're writing code and being part of the process as a team member, there is a whole notion of trust that is an credibility that is built. I will say that, you know, engineers are great people, but they are very skeptical of us researchers. They're like, yeah, you know, you guys are all smart.
You write papers, but you know, when rubber has to hit the road, I don't know that man. So, you know, you have to spend some time with the team and building that credibility in order for them to take you seriously. Get the street cred. Exactly. And I think it's a really good opportunity, especially when you are in industry research, to work on code. You know, there's nothing as empowering as being able to have an idea and being able to make it real.
It may not be the perfect prototype, but it's super power if you can actually take an idea and implement something and show it beyond a PowerPoint deck. And when you join a team like that, you understand the complexities and processes that go into shipping a feature. It's much more than what meets the eye. There's so much that goes into stake, the performance. You can't have regressions. You've got to build a very sophisticated test suite. And you have to listen to customers.
You have to be very careful of release planning. So there's so much that goes on, and you kind of develop a profound respect for the engineering profession. And it's a way of bringing that back into research. And how do you then balance not getting lost in the weeds? Like this is very much like, you know, in the trenches and like the frog's eye perspective basically.
But how do you keep that that vision then and not get lost in the, yeah, this doesn't work and that doesn't work and that's broken, you know, because engineering is fundamentally a lot about fixing something that's broken. How do you keep that big picture with you?
I think it's very important to have a good working relationship with the PM on the team because at the end of the day, the PM is the one that really hashes out the roadmap for the feature and what does it look like over the years and being involved in those conversations and going back to that document that a PM would write on the roadmap and the strategy of the feature kind of keeps you honest. And, I think it's just a calibration.
There are some days when I would be lost in the weeds because my code wouldn't work and you just can't sleep when your code doesn't work. You want to get it working. That's not good. But we made a conscious decision where, you know, I continued to publish while being on an engineering team and, in fact, actually getting some engineers to be on papers and patents with me.
you know, the research aspects of it, and you have to articulate that in prose, that really helps you look at the bigger picture holistically. And that combined with... What makes this unique, or what are the fundamental mechanisms and so on. Yes, yes. I think it's a double word score when you have a feature and it is backed and supported by well-done research, right? I think that's the best of both worlds. Yeah. Yeah. Or it invites new research, right?
Yeah. Yeah. Yeah. I think what is really interesting is that when I think about Tableau, to me, Tableau is like the thing that basically introduced a specific type of interaction for people to create visualizations, right? Before Tableau, it was basically Excel and a few other things. And Tableau introduced this idea that you basically select attributes and you drag them into a panel and and now you have some visualizations out there,
right? And then you have now with with a visa and as data, you have a new type of interaction. I was wondering, I always felt that this space we're trying to figure out how to express what we have it in our mind so that the machine can understand it and produce. data analytics and visualizations is a really interesting space.
And at the same time, it doesn't seem to be as, I would say it's not as much explored as the visual side of things, but it seems to me that is at least equally relevant, if not even more relevant. How do we express our intent? How do we tell a machine what we want, right? two really strong milestones. I would say these are two milestones in this sense because they introduce completely new paradigms on how to express what is my intent and what I would like the machine to do for me, basically.
Right, right. Yeah, and I think there is a spectrum of the type of analytical intents that people express This is the end of the video. If you liked the video, please subscribe to the channel and click on the bell icon to and what is supported. receive notifications of my latest videos. So to your point, something like a direct manipulation type interface like Tableau can be very concrete, right? I click on a bunch of attributes and I drag them to rows and shelves.
There's very little ambiguity in terms of what the user wants to see. and Now, with language, you can definitely be a bit much more abstract in terms of how you express your intent. It can be vague. It can be fuzzy. and there could also be new paradigms of analytical intent.
So, you know, one of the one of my papers that was presented at VIS with, you know, where Eamon, Gaba and Cindy and others collaborated with me was just exploring the language of comparisons and how you can actually interpret comparisons and show meaningful representations. And I would argue that through language, the whole space of comparisons can be pretty complex and nuanced when compared to direct manipulation. There's cardinality, you can compare one to one.
1 to N, N to N. You can be very specific or vague about the concepts. If I say, when is the safest time to fly? The data might indicate the morning. But there is an implicit intent there that I not only want morning as my answer, but I probably want to see how much safer is it to fly in the morning compared to other times of the day, right? I mean, if flying in the afternoon is slightly worse than the morning, I want to see that, right?
I want to see the whole distribution. So there's so many explicit and implicit ways of expressing intent that One could do more easily with language as opposed to a direct manipulation interface, which kind of opens up a range of interesting problems and opportunities.
Yeah. And I guess if I remember well, you also have kind of like a mix of, I don't remember if this is true for Revisa or us data, but I have seen systems where there's a natural language component, but then you can also interact with some elements of the sentence through direct manipulation, is that correct? That's right. So, in general, we have found that the pattern of developing mixed-initiative systems
tends to be effective, because there are certain things that I just want to tweak directly. You know, going back to the earthquake example, if the system comes back with a Richter magnitude of Sixth and Greater. I may not want to type saying, actually, I want large to be interpreted as four and greater. It's a lot easier for me to just grab the slider and tweak it down to four, right? So there are certain things that.
lends itself a lot more convenient to just click on marks. If I just want to see some data in a particular region and I see an outlier, it's just easier for me to click on the outlier rather than asking a question or I might click on it. It's called deictic referencing where you circle, maybe you lasso select some points and say, tell me more about this. And this is a very complex concept. What does this mean?
It's hard to express that in language. I mean, we do this all the time as humans, right? We point at things and say this. We don't explain and describe the object that we are pointing to. So I think there is a place for every sort of interaction and the key challenge is to figure out which flavors of interaction lend itself better for certain types of questions and intent. Yeah, that's so interesting. Yeah, I'm now curious, new ideas pop in my mind.
So going back to voice, I'm wondering if you ever explored the voice channel as an output, right? So what if I want to communicate data through say Alexa, right? Alexa is telling me, rather than visualizing something, it's telling me something about the data. How do you communicate data verbally? Yeah, so we actually had a paper at CHI last year, where we explored analytical chatbots
using this Grecian model of cooperative conversation. And we essentially studied the behavior of three flavors of chatbots using Slack. where it's text-based input and the output is a combination of text and images. We had just a text chatbot, which did not have any provision for any imagery or charts, Then we had a pure voice-based chatbot that was using the Echo device. And so we first did kind of Wizard of Oz studies to just understand the expectations
with respect to these various mediums. It's kind of the Marshall McLuhan paradigm of medium is the message, right? And it was very interesting, particularly with voice, but not surprising if you think about it. So people's working memory is very limited. Um, so if I ask what are the most expensive wines in Napa and this voice chat bot goes on and gives me a speech about all the, you know, the top five wines by the time it tells me the last one, I would have forgotten the first one.
And so what we realized is people, and this is something that has been studied in kind of general voice chat bots, where it will give you a snippet of information saying, you know, the most expensive wine is this. Do you want to hear about the others? And so it's like a follow-up question, almost like this back and forth. And you're like, sure, tell me the next one. And so it's like conversation chunking. The other thing that we noticed was the issue around trust.
When people, we were using, so we implemented these chatbots and they all shared a common parser. So while we knew that the performance of all these chatbots were comparable to one another, we found that people trusted voice chatbots less when compared to the text ones. So they wanted the voice chatbot to repeat the question. Just to make sure. Just to make sure. They got it right. They got it, exactly. So it's very interesting.
So we found a lot of differences, and we also found the types of intents that people tend to ask of text versus voice chatbots to be different. With voice, it's almost always these fact-finding questions with single responses. You know, kind of like how you ask Google, tell me what the weather is going to be, right? So it's like these single ones. You know, what are the sales looking like today for this region? But with other types of chatbots, the questions are definitely much more complex.
They are much more compound. People tend to chain multiple questions together. So it kind of goes back to that humans adapting to the computer speak of the system. But the new insight we learned was humans also adapt to their own limitations of interacting with the system, because they know that they have a limited working memory. And so they figure out, okay, to circumvent my limited working memory, let me simplify the types of questions that I will ask of these voice chat bots.
Very interesting, yeah. And do you think, like when, I think we touched a bit on that, but like some of your systems respond with a chart and then others respond with an answer or like a fact? Yeah. Like how, when, when is what being asked for or what was better when? Yeah, I mean, yeah. depend on the I think it's all of the above. When you have the luxury of a text interface with real estate, people want the single answer, But they might want additional context that supports the answer.
They might want the distribution of other regions, how well it's like, you know, especially these superlative questions, you know, the cheapest wine or the best selling product. People want to know how much better or worse is that answer when compared to the rest of the data distribution is very important to people. It's the same when you search for a product, you always get these big tables with the feature comparison and still the winner is marked, right?
But you want to see the full matrix, just know that everything's correct, even though everything is made up anyways on these sites. Yeah. So this is not a new paradigm. I mean, search systems have been and recommendation systems have been doing it for ages, right? Like if I go to Google and I type in my flight status number. it's a fact-finding question, right? I type in my flight number and it'll get me the status,
but there are other documents beneath it. The page just doesn't contain only my answer. There are articles that support, you know, my query and there's other stuff going on. The same thing with something like Amazon. I might be very specific and I want like Like a polka dot, you know, pink shoe made out of this brand, right? And maybe there's only one pink polka dot pair of shoes, but Amazon will show other related stuff.
And you could argue that's just their business model because they want to entice you to buy more stuff, but it's also just context. People want additional information in addition to something that specifically answers or satisfies their intent. Yeah. There's also some generosity in providing a bit more around it than just literally what you ask for. Right. It's like the thing you actually ask for is like the bare minimum, basically. And it's good to provide something extra with it.
Yeah. If somebody asks you, how are you doing? And you say, I'm fine. And you just stop there. It's weird. It's a German way of answering. That's a German way of answering. And then we're back to these flows and the processes and there's actually a mutual process going on. And I think that's so fascinating. You Yeah, it's called cooperative conversation and Grice came up with these, you know, principles
around cooperative conversation. There is a back and forth, there is a give and take, there is a notion of manner and relation and the quality of the content and the amount of content, and keeping it very relevant, but... making sure you're polite. So there's so much that's going on, and we have found that to a large extent, a lot of those cooperative conversation Maxines hold good even in the context of analytical chatbots. Yeah. That's interesting.
Yeah. Also in the book, you mentioned this idea of register and tone, like in natural language is not just what is said, but also how it's said and certain like, yeah, choice of formulations of words. just a certain maybe social status. social status between the participants or you know all these little things between the lines basically.
Yeah. Yeah. But I've also been very lately interested in, you know, data storytelling and the types of words and the choice of words that you use in these stories can also set the tone. And if you have any questions, please feel free to contact me. It depends on the audience.
I mean, the running joke when we wrote the book, actually, and Bridget will laugh about this, is she would always tell me, Vidya, you need to lower the register when you're writing your chapter because this is not just a PhD audience. So I appreciated her candid feedback and I use it all the time now. It's like, you have to figure out who your audience is and find the right level of information and how that is communicated. back to the user.
So it's both about content, but also again, the form in which the framing, right, the It's creamy. you know, just giving me a meta table and saying, what question would you like to ask? That's a very different sort of tone. And it really influences how people interact with these systems and the systems expectations. So Vidya, one thing I'm curious about, do you see a role of language as a way to, say, help people interpret data visualizations, maybe through annotations or some sort of guidance.
I believe there is, I can't recall if you're doing that type of research, but I've seen similar things, either from Tableau or other researchers, and, um, this idea that you can generate... Accessibility, obviously, right? Providing alternative, uh, means of accessing the same info. Yeah. Right. So, um, actually when we were writing the book, um, we had a chapter on, um, you know, in charts. And, you know, we were curious whether, because, you know, a lot of the practitioners,
and even we have been told about, you know, this whole data-ink ratio, right? I mean, thanks to Tufti and other books. It's like, oh, you know, you don't want to add too much text. The charts want to speak for themselves, just add a little bit. And then we did this, you know, project, you know, which was a vis paper that was presented by Chase Stokes from UC Berkeley. where we we really wanted to understand. Thank you. Oh yeah, that's a super interesting one. Right?
you Is there a notion of too much text is too bad? And it turns out, no, people actually love text. And quite often, you know, there are so many semantic levels of text, right? There's, you know, text that describes very low-level statistical properties, all the way to text that describes key takeaways, and, you know, there's been some work that I've done with Manish in that space as well.
But, yeah, to your point, I think there is a very exciting opportunity around figuring out how text can be used along with charts as a first-class citizen for people to understand their data. And they can be pretty effective, text annotations and titles and descriptions as scaffolds to the user.
And this kind of comes back to something I think that Moritz, you were alluding to, that at the beginning of this podcast was this cold slate problem where users struggle in terms of what they can ask of the data. And you can use text in conjunction with recommendations saying, you know what, you asked about this, This is the answer, but there are all these other things that you can ask. And so, yeah, I think it can really be an effective way of scaffolding the conversation.
It's also very helpful for answering more complex analytical intents, especially these why questions. I think why questions are very hard. that we haven't really cracked that nut. It's like, why is this phenomena happening?
If I am looking at the pound and U.S. dollar fluctuation around Brexit and I say, why is this drop happening, the data does not really have an answer and I have to go outside, you know, probably use natural language, go to access web articles and come up with a summary of my understanding of what happened and bring that back and formulate a very pithy annotation to the user. So supporting these why questions is going to be a place where text is definitely going to play a very important role.
How do you feel about the whole new generation of deep learning-based models in this context like GPT-3, the transformer model? So there's a whole – seems to be a whole paradigm shift in that space and like GPT-3 is a large language model trained on all of the internet as it seems, that seems to be able to answer like a lot of these common sense... Yeah. What's your feeling?
Language models have really come a long way over the recent years, and GPT-3 has been extremely promising for answering and learning certain types of intents and questions and being able to reason about information and even just learning stuff that goes above and beyond the initial training set.
What I have observed, though, with these large language models when it comes to data analysis and data exploration is that their capabilities around numeracy and understanding the numeracy aspects of data is still very limited. I am sure with systems like DALI, the graphicacy aspect might be teachable at some point. We might get to a point where we ask a question, and it'll learn to generate an effective chart. So DALI generates images based on prompts, actual texts.
But the numerical understanding of data phenomena is still a hard problem to learn, because it's extremely nuanced. You But I don't know, it might head to a place where it is. So that being said, being able to take these traditional grammar-based approaches that I have used as well in a lot of my research and retrofitting it with large language models.
can satisfy a good number of analytical intents. But at this point, I have found that you still have to augment these language models with additional logic that might come from these grammars or heuristics to solve numerical understanding of data that these models simply are unable to do or can do in a limited fashion. Yeah. Yeah. Yeah. But they could add that aspect of providing somebody hints with when does the gold price drop generally, right?
And might that be related to that specific piece of data I have here? That's right. You're right. Probably the synthesis needs to happen. Yeah. So I think that is where we are, but if we talk in a couple of years, it might be different. I mean, this area is pretty cutting edge and things have been evolving quite quickly, which is exciting. Yeah. Yeah, this is a space where I feel like something's going to happen, something's going to happen, right?
It's like, you can almost feel it in the air, right? Yeah. So it's, or maybe not, but I don't know. Yeah, it's fascinating. Yeah, because as you say, it feels like, oh, there must be like an amazing application for all this stuff. Yeah. know that you need, but you just get stuck at the 95, right? It's like, it's a bit of smoking mirrors, but I will say that, you know, these language models
have surpassed the initial notion of just being smoking mirrors. They actually worked pretty well for... I think the challenge and the opportunity is to get them to really understand data and just the numerical understanding of data. But as I indicated, I think it's a very tractable problem because, you know, it's already, you
figure it out to understand the knowledge of the world, right? So when you know certain properties and attributes of the data, you know, figuring out the numerical understanding of concepts It's probably teachable. Yeah. And this combination of systems is interesting, I think, to say, like, maybe there's different modules or agents that keep each other in check, or like, there's a, like a statistical like professor module that makes sure all the outputs are statistically correct. That's right.
Otherwise sends a new prompt that forces the model to update or something like that. Yeah. Fascinating. I thought maybe we could wrap it up with our last question for Vidya. Maybe you can tell our listeners if they want to learn more about how to use language in data visualization, what are the interesting directions? And I would say even practically, right? Say I'm a database designer practitioner listening to this.
How do I learn to use, to even be more aware of the role of language Edge in data visualization. Yeah. I will say your book, I guess there was an easy one. Just read the book because the book you know, various language disciplines, starting from kind of traditional NLP and information literature all the way to American Sign Language, because that's Bridget's background. That's fascinating too, by the way. Right, right.
some of the recent work that's come in the space. I mean, there's a sizable number of papers in the space of language and visual analysis that is being published at conferences like VIZ and CHI, even Euroviz, so that's another recommendation. A lot of people ask me about just kind of core NLP stuff, but there's plenty of courses on Coursera to just understand language models and intent.
So if people are very serious about implementing or trying out natural language algorithms, There are Python libraries. There's course lectures, including, you know, some really good ones by Chris Manning from Stanford. So I would start with those and then kind of work one's way down to reading the state-of-the-art literature in this space concerning visual analysis. Cool. It's a whole world out there to discover. It's one of these things once you open that box. It's a Pandora's box, right?
Fits into it. Yeah. Yeah. Yeah. But a really fascinating one. And I think one that has been underappreciated, like everybody's just focused on the visuals and the visual encoding and perception and the language is more than half of what we do, right? Like a chart with zero labels is nothing, right? So it needs that language content. Yeah, exactly. And what I tell people is when people think about NLP and Viz, they always almost think about chatbots and talking with data.
But there's so many other opportunities in the analytical workflow where natural language processing can help. It can help with intelligent data transformations under the hood. It can help with joining of tables, especially semantic joins, where the columns of values may not be identical, but they are related. Thank you. And, you know, coming up with meaningful encodings for data.
So there's so many sub-processes as part of that analytical workflow, where semantics and language understanding is very useful, in addition to kind of the more obvious applications that we talked about. So you're not going to get bored anytime soon? No, no, I'm going strong here. Awesome. Yeah. No, it's, I think that's really great advice.
And yeah, hopefully our listeners will now always look for the conversations and the dialogues and the register and the tone, because once you've seen it, you can't unsee it. Exactly. Exactly. Yeah. Wonderful. Yeah. That was really nice. for shedding light on all this fascinating stuff and especially this It was a great talking about it with you both. I enjoyed the conversation. Wonderful. And we'll check back in a few years how things have developed, right? Wonderful. That's right.
Okay. Thanks so much. See you soon. Yeah. Thank you so much for having me. And I really enjoyed it. And I look forward to listening to it and all the other podcasts that you come out with. So I'm a big fan. So thank you very much. Thank you. Thanks so much. Thank you. All right, take care. Bye. Hey, folks, thanks for listening to Data Stories again. Before you leave, a few last notes.
This show is crowdfunded, and you can support us on Patreon at patreon.com slash datastories, where we publish monthly previews of upcoming episodes for our supporters. Or you can also send us a one-time donation via paypal at paypal.me slash datastories. Or, as a free way to support the show, if you can spend a couple of minutes rating us on iTunes, that would be very helpful as well. And here's some information on the many ways you can get news directly from us.
We are on Twitter, Facebook, and Instagram, so follow us there for the latest updates. We have also a Slack channel where you can chat with us directly. And to sign up, go to our homepage at datastory.es, and there you'll find a button at the bottom of the page. And there you can also subscribe to our email newsletter if you want to get news directly into your inbox and be notified whenever we publish a new episode.
That's right. And we love to get in touch with our listeners. So let us know if you want to suggest a way to improve the show or know any amazing people you want us to invite or even have any project you want us to talk about. Yeah, absolutely. Don't hesitate to get in touch. Just send us an email at mail at datastory.es. That's all for now. See you next time, and thanks for listening to Data Stories.