Ofir Press on AliBi and Self-Ask - Weaviate Podcast #65!

Hey everyone! Thank you so much for watching the Weaviate Podcast! I am SUPER excited to publish my conversation with Ofir Press! Ofir has done incredible work pioneering AliBi attention and Self-Ask prompting and I learned so much from speaking with him! As always we are more than happy to answer any questions or discuss any ideas you have about the content in the podcast! +Huge Congratulations on your Ph.D. Ofir! AliBi Attention: https://arxiv.org/abs/2108.12409 Self-Ask Prompting: https://arxiv.org/abs/2210.03350 Ofir Pres on YouTube: https://www.youtube.com/@ofirpress Chapters 0:00 Welcome Ofir Press 0:41 Large Context LLMs 12:38 Quadratic Complexity of Attention 19:12 AliBi Attention, Visual Demo! 24:53 Recency Bias in LLMs 28:57 RAG in Long Context LLM Training 36:27 Self-Ask Prompting 46:07 Chain-of-Thought and Self-Ask 50:47 Gorilla LLMs 58:42 New Directions for New Training Data