Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum

James Dooley: Personalisation of large language models. How each and every search, whether it’s ChatGPT, Perplexity, Claude, Grok or Gemini, can give different answers. Today I’m joined by Benjamin Tannenbaum, who has a lot of information and evidence on this. He owns multiple AI tools, including Get AISO. So Benjamin, let’s jump straight in. When someone does a search in a large language model like ChatGPT or Gemini, why do different answers appear for different people? Benjamin Tannenbaum: That’s a very good question. We conducted research on this that we’re preparing to publish in a leading SEO publication. We approached the problem from scratch because there’s already a lot written about it. Benjamin Tannenbaum: We built a setup where we took realistic questions from real AI conversation datasets and ran them repeatedly. First, we used a clean baseline with no login and no memory. Then we gradually added more context and reran the same question many times to observe how the answers changed. Benjamin Tannenbaum: One complexity is that even without personalisation, large language models can give different answers to the same question. Unlike Google Search, where results are relatively stable, LLMs generate responses probabilistically. Benjamin Tannenbaum: If you and I ask the same question, we might receive different brand recommendations. That doesn’t automatically mean it’s personalisation. It could simply be variability in token generation, what some describe as a next token lottery. Benjamin Tannenbaum: For example, if we both ask about the best pepperoni pizza restaurant in New York, you might get a fancy Fifth Avenue restaurant and I might get a cheaper Brooklyn option. That difference might look like personalisation, but it could just be variation within equally strong candidates in the query fan out results. Benjamin Tannenbaum: So what appears to be personalisation may simply be variability in how the model selects from the top candidate sources. James Dooley: So what are the real personalisation factors? Benjamin Tannenbaum: The first straightforward one is location. Every time you send a query, your location is appended to it. This happens even if you explicitly ask the model not to consider your location. It still gets added. Benjamin Tannenbaum: That’s significant for local businesses. But even non local businesses should consider where their ideal customers are located, because localisation affects AI responses. Benjamin Tannenbaum: The second layer is memory based personalisation. If memory is enabled, which is the default in ChatGPT when logged in, the system references past conversations. If it knows you have a dietary restriction, it may automatically exclude restaurants that don’t meet that requirement. Benjamin Tannenbaum: This is a strategic focus for companies like OpenAI. More personalised answers increase satisfaction and, in the long term, improve monetisation potential. Benjamin Tannenbaum: But we should add a reality check. Around 95 percent of users are on the free tier of ChatGPT. Those models have less context window and less memory depth. So while personalisation exists, it is often limited in practice. Benjamin Tannenbaum: Gemini is beginning to integrate broader Google signals, including past search history, but this is still evolving. James Dooley: Some people in the SEO space argue there’s no point tracking AI visibility because responses change constantly. What’s your view? Benjamin Tannenbaum: It’s not random. It’s probabilistic. If you test the same query thousands of times, patterns emerge. You can measure share of voice rather than expecting deterministic rankings. Benjamin Tannenbaum: If your brand appears in 60 percent of responses, that’s a measurable advantage. Optimisation increases your probability of being selected, even if you won’t appear 100 percent of the time. Benjamin Tannenbaum: Interestingly, personalisation can actually reduce variability. When a model heavily weights certain attributes in the fan out based on user preferences, the candidate pool narrows. Benjamin Tannenbaum: For example, if your preference is highly specific, like Japanese restaurants in New York with jazz and premium whisky, the fan out becomes very narrow. Instead of 30 broadly similar candidates, you might only have two strong matches. That reduces randomness and stabilises results. James Dooley: That’s interesting. So personalisation can make responses more consistent rather than less. Benjamin Tannenbaum: Exactly. It narrows the search space. So while it introduces complexity for marketers, it also creates opportunities. If your brand strongly owns a distinctive attribute, you can dominate those personalised niches. James Dooley: I like the idea of measuring share of voice. If you appear 40 percent of the time, aim for 50 percent. It becomes a probability game rather than a fixed ranking. James Dooley: Benjamin Tannenbaum, it’s been a pleasure doing this series. We’ve covered how queries differ in AI search, query fan out, and now personalisation. If people want to follow your work, where can they find you? Benjamin Tannenbaum: The best place is LinkedIn. My name is Benjamin Tannenbaum. I post regularly about AI search, query fan out, and related research. James Dooley: Benjamin Tannenbaum, thank you again. That wraps up our discussion on AI personalisation and visibility within large language models.