#60 - How to input text into your model? Understanding tokenizers.
Dec 01, 2022•15 min
Listen to this episode in Metacast mobile app
Don't just listen to podcasts. Learn from them with transcripts, summaries, and chapters for every episode. Skim, search, and bookmark insights. Learn more
Episode description
Hello everyone, in this episode I explain how tokenizers work. They are basically what enables us to input the text into a NLP algorithm like BERT or GPT. In the episode I explain 3 types of tokenizers, word based, character based and sub-word based representation.
Instagram: https://www.instagram.com/podcast.lifewithai/
Linkedin: https://www.linkedin.com/company/life-with-ai
Huuging Face blog about tokenizers: https://huggingface.co/docs/transformers/tokenizer_summary
For the best experience, listen in Metacast app for iOS or Android