Target Audience: Data scientists, AI/ML engineers, and practitioners interested in applying large language models (LLMs) / generative AI to solve real-world classification problems at scale. Attendees should have a foundational understanding of machine learning concepts and familiarity with the Python data science stack. Exposure to vector embeddings or LLM APIs is helpful but not mandatory.

Takeaway: Attendees will gain practical insights and learn best practices for building, debugging, scaling, and productionizing a complex, multi-step generative AI system for large-scale product categorization. They will understand the evolution from traditional methods to LLMs, learn specific techniques for prompt engineering, batch processing, cost optimization with models like OpenAI's, and see the tangible business impact of such a system.

Detailed Outline:

This talk chronicles our journey tackling a yet challenging problem: accurately categorizing hundreds of thousands of diverse products into a fine-grained taxonomy of over 1,000 categories. We'll share our evolution from initial manual and rule-based systems to a sophisticated, production-ready Generative AI pipeline.

• Part 1: The Challenge & Initial Approaches (10 minutes)
  • Introduction to the business need for accurate product categorization at scale.
  • Overview of the limitations encountered with traditional methods:
    • Manual Curation: Slow, expensive, inconsistent, and impossible to scale.
    • Rule-Based Systems: Brittle, hard to maintain, and unable to handle nuances or new product types.
    • Fine-tuned Semantic Models: An improvement, but struggled with zero-shot generalization to new categories and required significant labeled data and retraining.
• Part 2: Entering the GenAI Era - Iterations & Lessons Learned (10 minutes)
  • Our initial exploration using LLMs for categorization, what worked, and what failed.
  • Developing the Prompt: We'll dive deep into the iterative process of prompt engineering for this complex multi-label, hierarchical classification task. We'll show examples of early prompts, their failure modes (e.g., inconsistent output format, hallucinated categories, difficulty handling multiple classification signals), and the refinements that led to more reliable results. We will discuss techniques for achieving structured output (e.g., JSON) from the LLM.
  • Early Scaling Issues: Discussing the pitfalls of naive API usage, latency problems, and prohibitive costs when dealing with large volumes.
• Part 3: Building a Robust, Scalable GenAI Pipeline (10 minutes)
  • The Hybrid Approach: Detailing our successful multi-step architecture that combines the strengths of semantic embeddings for efficient candidate retrieval/filtering and LLMs (specifically leveraging OpenAI models) for nuanced final categorization.
  • Productionization Strategies:
    • Batching: Implementing efficient batch processing using asynchronous requests and the OpenAI Batch API to drastically reduce latency and cost.
    • Cost vs. Accuracy: Strategies for selecting the right model based on complexity and cost constraints.
    • Semantic Similarity & Early Stopping: Using vector similarity to intelligently prune the search space, avoiding the need to evaluate every product against all 1,000+ categories with the LLM, thus significantly optimizing cost and throughput.
    • Automation & Monitoring: How we process updates of categories and products automatically in PySpark and monitor that the live system works as expected.
• Part 4: Measuring Impact & Looking Ahead (10 minutes)
  • Presenting the results: Showcasing the significant improvements in categorization accuracy and coverage compared to previous methodsIllustrative examples of challenging products correctly categorized by the GenAI system.
  • Discussing the tangible business value derived as measured in A-B tests
  • Briefly touching upon ongoing work and future directions.

This presentation will focus on the practical application and engineering challenges, sharing reusable techniques and hard-won lessons applicable to anyone looking to leverage the power of generative AI for large-scale, real-world problems. We aim to provide a transparent account of not just the successes, but also the crucial learnings from failures encountered along the way.