Dr. Klaus Schoeffmann
Institute of Information Technology (ITEC), Klagenfurt University (Austria)
Beyond Accuracy: What 15 Years of Interactive Video Search Have Taught Us About Pattern Recognition Systems
Video understanding has advanced dramatically over the past fifteen years, from hand-crafted visual features and concept detectors to deep neural representations, vision-language models, and multimodal AI. However, in real-world video search, recognition accuracy alone does not determine whether a system is successful. The way information is represented, explored, and combined with human interaction can be just as important as the underlying recognition technology. In this keynote, I will reflect on more than a decade of research on interactive video retrieval, with the Video Browser Showdown (VBS) serving as a long-running experimental platform for studying such systems. Since its introduction in 2012, VBS has enabled the comparative evaluation of numerous approaches to searching large video collections under realistic time constraints. Its evolution provides a unique perspective on how video retrieval has changed - from visual browsing and similarity search to semantic embeddings, natural-language queries, visual question answering, and increasingly autonomous retrieval agents. Beyond the technological evolution of the VBS, the talk will discuss broader lessons learned from evaluating complete pattern recognition systems: the interaction between algorithms and user interfaces, the role of expert and novice users, the design of meaningful evaluation tasks, and the sometimes surprising gap between benchmark accuracy and practical search performance. Examples from medical and surgical video analysis will further illustrate how challenges such as domain shift, limited annotations, and application-specific requirements affect the transition from recognition algorithms to usable systems. Finally, I will discuss how multimodal foundation models and AI agents may reshape video retrieval and raise new questions for the pattern recognition community: What should we measure when systems can search, reason, interact, and answer questions? And how should we evaluate the next generation of human–AI pattern recognition systems?
Dr. Klaus Schoeffmann is an Associate Professor at the Institute of Information Technology (ITEC), Klagenfurt University, Austria, where he received his habilitation (venia docendi) in Computer Science in 2015. He holds both a PhD and an MSc in Computer Science. His research focuses on computer vision and multimedia retrieval, with a particular emphasis on health AI and surgical data science. Prof. Schoeffmann has secured more than €2M in research funding from the Austrian Science Fund (FWF), the Carinthian Economic Promotion Fund (KWF), and industrial partners. He has supervised six PhD candidates and numerous Master’s theses. He is the founder and co-organizer of the annual Video Browser Showdown (VBS) and the Lifelog Search Challenge (LSC). He is a member of IEEE and ACM and serves as a regular reviewer for leading international conferences and journals in multimedia and medical imaging. He has also taken on prominent organizational roles, including Program Co-Chair of FAIEMA 2026, SISAP 2026, ACMMM Asia 2025, and ACM ICMR 2020, as well as General Co-Chair of ACM Web 2027, ACMMM 2025, and ACM ICMR 2024. At Klagenfurt University, he teaches a range of computer science courses, including computer vision with deep learning, multimedia technologies, and app development.