How to Build a Great AI Product: Insights from Aytekin Tank (Jotform Founder)

This event holds special significance for me personally. Hosting Aytekin Tank at our AI Meetup Ankara wasn't just another meetup. It was an opportunity to learn from someone who has been my mentor for over two years, guiding us through Popupsmart's growth journey. 🙏

Aytekin Tank has been instrumental in shaping how we think about product development, user experience, and now, AI integration. His mentorship has been invaluable, and having him share his insights with our Ankara AI community was truly an honor.

In this post, I've extracted the key learnings from his presentation about building great AI products—lessons drawn from Jotform's 2-year AI journey with 30M+ users. Whether you watch the full session or read through this summary, you'll find practical, actionable insights that we're already applying to LiveChatAI and our other products.

Let's dive into what Aytekin shared about the fundamental differences between building traditional products and AI products.

• You can switch the dubbing to listen english version of the presentation.

The Fundamental Question: What Makes a Good Product?

"If we build a good product, we will succeed. It may take time... but if we build something good, it spreads."

Key Philosophy:

• Focus on product quality above all else
• Good products naturally attract users through word-of-mouth
• Long-term success requires patience and persistence

The New Challenge: Building AI Products is Completely Different

Aytekin Tank emphasized that building AI products requires an entirely different playbook:

"The answer to building a good AI product is completely different. Old playbooks, old rules—nothing works. AI has completely different rules."

Three Core Challenges of AI Products

1. AI is Non-Deterministic

• Unlike traditional software with if-then logic, AI produces unpredictable results
• Real example: Aytekin tested ChatGPT's Atlas agent to create a Jotform contact form three times
  • First attempt: Created blank form, then added questions
  • Second attempt: Found a template from template library
  • Third attempt: Used AI prompt feature to generate form
• Each approach yielded different results

The mindset shift:

"We're not creating software—we're training a person or providing a service."

2. AI Makes Many Errors

• Hallucinations are inevitable
• RAG (Retrieval Augmented Generation) doesn't eliminate all errors
• Errors increase significantly as data scales
• Reality check: Reducing errors to zero is impossible, but minimizing them is achievable

3. User Expectations Are Extremely High

• Users expect AI to do everything
• People request complete solutions: "Build my entire startup, make a mobile app with these features"
• High expectations create both challenges and opportunities
• The ChatGPT era has trained users to expect comprehensive responses instantly

Jotform's AI Journey: From 25% to 75% Resolution Rate

The Starting Point: Jotform AI Agent

Product Overview:

• 15,000+ active users
• Supports multiple channels: Chatbot, Gmail, Instagram, WhatsApp
• Trains on company documents to answer customer questions

Initial Challenge:

• Resolution rate started at only 25%
• AI couldn't answer 75% of questions correctly

The Cheat Code: Using Your Own Product

Aytekin revealed their secret weapon:

"We have a cheat code: we can develop our product by using our own product."

The Strategy:

• Deployed their AI Agent on Jotform's own support system
• Received thousands of daily customer interactions
• Created immediate feedback loop

The Systematic Improvement Process

Daily Review System

Resource Allocation:

• 20 support team members dedicated full-time to reviewing AI conversations
• Review 3,000+ daily conversations between AI and customers
• Support team's workload decreased as AI handled more queries

Review Protocol:

• Mark each conversation as "resolved" or "not resolved"
• Document specific problems encountered
• Identify biggest pain points

Clustering Problems

• Group similar issues together
• Prioritize based on frequency
• Distribute to product teams for resolution
• Initially done with machine learning, now using AI for clustering

Incremental Improvements (25% → 75%)

Breakdown of Gains:

1. RAG Improvements (~10%)
   • Enhanced retrieval methods
   • Better model selection
   • Optimized information retrieval
2. Knowledge Base Enhancement (~20%)
   • Improved documentation continuously
   • Fixed incorrect information
   • Added missing content
   • Created new guides daily based on gaps
3. Tool Calls Implementation (~20%)
   • Enabled AI to use tools via API
   • Example: Check email status, find blocked addresses
   • Example: Search deleted forms, help restore them
   • Made AI act like human support agents

The Critical Success Factor:

"We reviewed each one individually. Someone read 3,000 conversations every day. We dedicated resources for this."

Why Most AI Products Fail to Improve

Common Mistakes:

• Testing only internally
• Releasing and hoping for the best
• Only fixing issues when complaints arrive
• Not systematically reviewing user interactions

Aytekin Tank's Insight:

"AI is so fragile that solving problems randomly by finding them is impossible. You must examine all problems, cluster them, and solve them systematically."

Jotform AI: The Creation Tool

Product Features

Three Types of AI Capabilities:

1. Creators: Generate complete forms from prompts
2. Copilots: Chat-based assistants in builders
   • "Change my form's background color"
   • "Get logo from my website and add it"
3. Smart Tools: PDF import, automation improvements

The Daily Sprint Methodology

Current System:

• Think in daily sprints, not weekly
• Every morning at 9 AM, automated email arrives
• Email contains clustered problems from past 24 hours
• Product teams work on most frequent issues immediately
• Expectation: Problems found today are solved within 1-2 days

Email Content:

• Analysis of yesterday's reviews
• Clustered problem categories
• Frequency data for each issue
• Example: "1,000 forms created yesterday, 50 had issues with name field"

Continuous Learning Loop

• Teams focus on highest-priority problems first
• Fix is deployed quickly
• Monitor if fix resolves the issue
• Move to next problem cluster

Technical Insights

The 75% Ceiling

Why can't resolution rate exceed 75%?

• Users ask increasingly diverse questions
• Some problems too unique to solve cost-effectively
• Edge cases proliferate
• Important: Human support agents don't have 100% resolution rates either

Prompt Engineering Evolution

Controversial take:

"Prompt engineering as a future career? That's gone. It's no longer needed because AI now understands what you mean, even if you write nonsense."

Reality:

• Developers can learn prompting like any new language
• Models understand natural language well enough
• No need for specialized prompt engineers

Practical Advice for Startups

Getting Started Without Users

Question from audience: How do you get data without users?

Aytekin's Answer:

• First year of Jotform was completely free
• No account creation required initially
• Prioritize getting users over monetization
• 15,000 users by end of first year

Key Principle:

"If you're a startup, my recommendation is to make it free. At least let them try it easily."

Enterprise vs. Mass Market

Warning about enterprise-only approach:

• Hard to build good products with only 3 enterprise customers
• Enterprise products often aren't good because of limited feedback
• Mass market provides necessary feedback volume

The Support Founder Advantage

Aytekin's early years:

• Handled all support himself for first 4-5 years
• Benefit: Deep understanding of user needs
• Saw exactly how and why people used the product
• Learned which problems were most important

AI Product Development Mindset

Think of AI as an Employee

Critical perspective shift:

"Perhaps we should think of AI as a human employee. Review what they do. See them as an employee who knows nothing and train them constantly."

The analogy:

• You wouldn't hire someone, talk for a few hours, then leave them alone for 6 months
• Many startups do exactly this with AI products
• AI requires continuous training and oversight

Balancing AI with Traditional Code

Important distinction:

• Don't leave everything to AI
• Traditional software still drives the system
• AI makes decisions within framework
• Software handles structure and reliability

Example: Form Creation

• AI decides what questions to include
• Software handles how fields are rendered
• AI can modify existing fields
• Collaboration between AI and code creates best results

The Pragmatic Approach

On perfection:

• Users don't expect perfection from AI
• Human support isn't perfect either
• User expectations for AI are actually lower
• People understand AI limitations from experience

On fearless deployment:

"If we're not checking enough, if we're not putting the product in front of users quickly enough, we can't get that feedback."

Key Metrics and Benchmarks

Jotform AI Agent Performance

• Active users: 15,000+
• Initial resolution rate: 25%
• Current resolution rate: 75%
• Improvement timeframe: ~3 months
• Daily conversations: 3,000+
• Review team size: 20 people

Jotform AI (Creation Tool)

• Daily creations: 3,000+ forms
• Review sample: ~1,000 daily reviews
• Team size: 30 people on AI reviews
• Process: Daily email with clustered problems
• Resolution time: 1-2 days from discovery to fix

Future Outlook

Voice Integration

• Working on voice for over a year
• Now bearing fruit
• Users can create forms by speaking
• Voice works across 20+ products
• AI department handles deep tech challenges

Complete Automation Vision

Goal:

• Beyond form creation to complete workflows
• Generate follow-up documents automatically
• Create approval processes
• Add e-signatures
• Produce customized documents from form data

Philosophy:

"The purpose is automation, right? Software's purpose is automation. So people don't have to struggle."

MCP Ecosystem

• Building out MCP integrations
• Goal: Connect to thousands of services easily
• Enable ChatGPT and other AI assistants to use Jotform directly
• Reduce friction for users

The Path Forward

Aytekin's closing insights for AI product builders:

Core principles:

1. Ship fast and iterate - Don't perfect in isolation
2. Use your own product - Create immediate feedback loops
3. Review systematically - Dedicate resources to examining every interaction
4. Cluster and prioritize - Focus on highest-impact problems
5. Think daily, not weekly - AI requires constant attention
6. Balance AI with code - Use traditional software as the framework
7. Accept imperfection - Aim for continuous improvement, not perfection

Final thought:

"Building a good product is hard. But if you have users, you have data. Use whatever data you have. If you have 200-300 reviews, you can still find clusters and patterns."

About the Event

This was the 6th AI Meetup event organized in Ankara, bringing together AI developers and enthusiasts to share knowledge and build community. Special thanks to Aytekin Tank for sharing Jotform's journey and practical insights.

Here is the full recording including my presentation:

See you in next AI Meetup event!
Emre Elbeyoglu