From Prototype to Production: Building Complex Serverless Apps with Agentic AI

Subtitle: It's not just about generating code—it's about architectural partnership, context management, and debugging the "Unknown Unknowns."

Introduction

The promise of AI in software development often focuses on speed: "Generate a react app in 5 minutes." But what happens when you need to build a complex, enterprise-grade system with legacy constraints, VPC networking, and specific performance requirements? This article explores the journey of building a production-ready Serverless API using an AI Agent as a pair programmer, highlighting the shift from simple code generation to deep architectural collaboration.

The Agent as Architect: Pivots & Constraints

In simple demos, the path is linear. In real projects, requirements change.

• The Mission: Our goal was to integrate Modern ITSM with Legacy Systems. We had a massive 9TB database dump of historical data that needed to be exposed via API for consumption.
• The Constraint: We had limited support. The legacy system was effectively end-of-life, meaning we were working with minimal external guidance.
• The Scenario: We needed to verify database mappings against this massive dataset. The initial "Plan A" was a simple local Python script.
• The Block: We hit a hard infrastructure constraint—the database was locked inside a private VPC, inaccessible to the local machine.
• The AI Pivot: Instead of just erroring out, the Agent proposed and implemented multiple architectural shifts to bypass networking restrictions and modernize the hosting strategy.
• Lesson: AI is most powerful when it understands the environment, not just the syntax. It can be a true Architect.

Context is King: The "Memory" Problem

Complex projects have specific domain rules that don't exist in public documentation.

• The Challenge: Our database used obscure table naming conventions (logical names vs. physical views) that initially confused the AI.
• The Solution: We didn't just correct the code; we corrected the Context. We established an AGENT_RULES.md file—a persistent "memory bank" for the project. By explicitly documenting these naming conventions, we ensured that every subsequent interaction (regardless of which specific AI session was running) respected these rules.
• Lesson: Treating your AI context like documentation is critical for long-running projects.

Debugging "Unknown Unknowns"

Real-world data is messy.

• The Bug: A standard query failed because the database content didn't match the documentation.
• The Complexity: The database contained over 14,000 tables and views with dynamic naming conventions. Navigating this structure without prior knowledge would have been nearly impossible for a human developer in a reasonable timeframe.
• The Investigation: The AI didn't simply give up. It instrumented the code with "Super Debug" queries, dumping schema samples and performing broad fuzzy searches to find the missing tables.
• The Discovery: We found that the data was stored using the original form names, contradicting our initial mapping assumptions.
• Lesson: An Agentic AI can be an investigative partner, writing its own tools to probe the system and uncover truth.

Optimization: Static vs. Dynamic

• The Refactor: Initially, we resolved dependencies dynamically at runtime. The AI identified this as a performance bottleneck and a reliability risk.
• The Shift: We moved to a "Static Resolution" pattern, using AWS SSM Parameter Store to cache expensive lookups. The AI updated the CDK stack, wrote the caching logic, and the discovery tools.
• Lesson: AI can help enforce production best practices (caching, separation of concerns) if you challenge it to optimize.

The Playbook: How to Lead an AI Team of One

Through this project, we crystallized a workflow that maximizes the AI's potential:

Phase 1: Strategy & Architecture

• Spec First, Code Later: Don't ask for a function; ask for a requirements document. We found that forcing the AI to write detailed specs first (and reviewing them) eliminated 80% of downstream bugs.
• Layered Architecture: Build the house from the ground up. We defined the overall architecture first, then drilled down layer by layer (Database -> API -> UI) rather than trying to generate everything at once.
• The "Meta" Plan: Before starting a complex task, we asked the AI: "Create a detailed implementation plan for this." We reviewed and molded this plan. You'll be surprised at how well the AI can anticipate dependencies (like security and deployment automation) if you give it the space to think before it acts.
• The Air-Gapped Architect: In enterprise environments, you often can't give the AI direct access.
  • The Topology Trap: The hardest challenge was configuring the Lambda-to-RDS networking blindly. The AI knew the theory (Security Groups, Subnets, NAT) but couldn't see the reality of the AWS console. We acted as the bridge, describing error messages ("Task timed out" vs "Connection refused") which allowed the AI to deduce that a Security Group rule was missing. You provide the clues; the AI provides the deduction.
• Economics & Ergonomics: Don't underestimate "lower" tier models like Gemini Pro. We found them surprisingly capable and cost-effective, often delivering the concise code we needed without the excessive verbosity or documentation-heavy output of models like Claude.

Phase 2: Workflow & Hygiene

• The Safety Net (Feature Branches): Create feature branches even for small changes. Sometimes the AI can get lost, and it becomes impossible to bring it back on track without a clean state to revert to.
• Protected Blocks: Implement specific blocks where you can write custom code without the AI overwriting it, using markers like:

  # START_USER_CODE: imports
  # You can add custom code here
  # END_USER_CODE
  

• Context Volatility: Be aware that restarting the IDE often loses the active context. You may need to "prime" the AI again after a restart.
• The AI Handover: Always create a "Handover Document". Treat your future self (or another AI session) as a new team member. A concise summary of "Current State," "Active Tasks," and "Known Issues" ensures the next session picks up exactly where the last one left off.
• Persistent Memory (Interaction Logs): Ask the AI to log important interactions or decisions into a project file. This allows you to revisit the history of why changes were made, not just what was changed.
• Capability Manifest: Explicitly document what the AI can do in your AGENT_RULES.md. If the AI knows it has access to specific tools (like run_command or a database shell), remind it in the rules. This prevents it from hallucinating constraints ("I cannot access the terminal") that don't exist.

Phase 3: Execution & Verification

• Visual Iteration: For the frontend, we didn't write React code immediately. We asked the AI to generate image mockups. We iterated on these visuals until they were perfect, then asked it to convert those approved pixels into code.
• The Coherence Check: You must remain the master of your design. In a complex stack, a change in the UI often triggers a cascade: UI -> API -> DB Schema -> Seed Data. The AI might fix the first two and forget the rest. It is your job to know the architecture well enough to say, "Did you remember to update the seed script for that new column?"
• The Test Generator: Ask the AI to create automated tests—it works like a charm. It can generate comprehensive test cases faster than you can type the boilerplate.
• Local Replication: Try to always have a local setup. In this project, the AI was quick to set up a local Docker environment with an API Gateway simulation and local DB, which drastically sped up the feedback loop compared to cloud-only debugging.
• Cognitive Audit Trails: While not mandatory, asking the AI to "think out loud" or document its reasoning is invaluable. It helps you catch logic errors early and understand why a decision was made.
• Automated Visual QA: The AI can drive the browser via Chrome extensions to auto-detect defects. It acts as a tireless QA engineer, spotting layout shifts, broken links, or console errors that human eyes might glaze over.

The Rough Edges: It's Not Magic

Working with Agentic AI is a skill that requires patience. It's not magic, and there are pitfalls:

• Model Drift: Sometimes the model crashes or loses context mid-stream, forgetting its objective. This usually manifests as confusion or hallucination. It can be handled reasonably easily by reiterating the current task boundary, but it requires vigilance.
• Forced Restarts: The AI integration is tightly coupled with the IDE. We found that some IDE updates force a restart, which kills the active AI session. You need to be prepared to pick up the pieces (hence the importance of the "Handover Document").
• The Infinite Loop: Occasionally, the AI gets stuck in a loop of "Deciding -> Acting -> Deciding" without making progress. We saw this in a recent update. It is safe to kill the process and reissue the command, but you must monitor it to ensure it doesn't burn tokens on a treadmill.

The Human Impact: A CTO's Perspective

After 20 years of software development, I can confidently say that this workflow is a paradigm shift. If I had delegated this project to a team of junior developers—my default mode of operation until recently—it would have taken at least 3x the time.

The "Force Multiplier" effect is real. The AI didn't just type faster; it navigated chaos that would have stalled a human team. The ability to instantly query and map a dynamic schema of 14,000+ tables without getting overwhelmed is a superpower that no single junior developer possesses.

Finally, a personal confession: This is the first technical article I have written in my 20-year career. I have always avoided documenting my work because the friction of writing felt too high. This article itself is a testament to the AI's capability—it didn't just build the code; it helped me articulate the journey. Without this partnership, these learnings would have remained locked in my head, undocumented.

Conclusion

Building with AI isn't about replacing the engineer. It's about elevating the engineer to a Director role. The AI handles the "How" (Python syntax, CDK constructs, IAM policies), allowing the human to focus on the "What" and the "Why." The result is a system built faster, but more importantly, built robustly.

---

> Disclaimer: This article was created by the same AI which created the application.

Tags: #Antigravity #Gemini #Claude #VibeCode #Serverless #AWS #SoftwareArchitecture #AgenticAI #LLM