Scaling autonomous AI agents to serve over 122 million customers introduces significant challenges. This technical talk details the system design patterns and engineering principles employed at Nubank to build and operate production-grade, customer-facing AI agents. We will present our data-driven development lifecycle, which is centered on a comprehensive evaluation suite. We will cover the design of our automated evaluation pipelines, which quantitatively assess agent performance against key metrics and ensure consistent user experience at scale.
The discussion will then transition to the technical specifics of LLM prompt engineering, including strategies for structuring optimal prompts and generating precise tool schemas for complex, multi-turn actions. A primary focus will be on the implementation of feedback loops for automated system optimization. Attendees will learn how we utilize explicit and implicit feedback signals for the automated refinement of both evaluation datasets and agent system prompts. Finally, we will outline how these integrated systems create a flywheel for continuous improvement and share our online A/B testing frameworks and the key performance indicators (KPIs) used to measure in-production success and business impact.
Speaker
Aman Gupta
Principal Machine Learning Engineer AI Core team @Nubank
Aman Gupta is a Principal Machine Learning Engineer in the AI Core team at Nubank where he leads company-wide efforts for AI Agents, Foundation Models, LLMs and large-scale constrained optimization to improve the lives of Nubank’s 122M+ users. Prior to joining Nubank, Aman was Senior Staff ML Researcher at LinkedIn focused on LLMs, Deep Recommender Systems and Optimization. Aman also spent several years at Apple and Amazon as an engineer. Aman has published 25+ papers at top AI conferences like ICML, NeurIPS, ICLR, EMNLP, KDD, WWW, CIKM and filed 10+ patents. Aman obtained his MS in Computer Science from Carnegie Mellon University.
Session Sponsored By
Nubank (NYSE:NU) is one of the largest and fastest-growing digital financial services platforms in the world, serving over 122m customers.
