Multi-tenant SEO: Indexing 185K Pages Without Rebuilds

How we went from 6-hour builds and missed indexing windows to instant content updates and sub-40ms response times across 185,000 pages

The Problem: When Static Site Generation Breaks Down

We had a problem that most developers dream of having: too much content. FIKR is a multi-tenant SaaS platform where every organization gets their own suite of tools for managing their startup journey. Each organization can create unlimited pages, blog posts, product listings, team profiles, and custom content. When we launched, we had 50 organizations creating maybe 10-15 pages each. Our Next.js static site generation handled this beautifully. Builds took 2 minutes, deployments were smooth, and Google indexed our content reliably.

Fast forward eight months. We now had 1,200 organizations on the platform. Some power users had created 200+ pages. Our total page count crossed 185,000 pages and was growing by 500-1,000 pages per day. Our builds went from 2 minutes to 6 hours. Yes, six hours. Every single content update triggered a full rebuild of the entire site. Organizations would publish a blog post and then wait hours before it appeared in search results.

The breaking point came when three large organizations joined the platform simultaneously, each importing hundreds of pages from their existing systems. Our build pipeline collapsed under the load. Builds started failing halfway through due to memory exhaustion. We tried splitting the build across multiple workers, but that introduced synchronization problems and made things worse. Our team spent entire weekends babysitting deployments and manually restarting failed builds.

We evaluated every option we could think of. Incremental Static Regeneration sounded perfect in theory, but with 1,200 organizations making updates constantly, we'd still be rebuilding thousands of pages per hour. The ISR cache invalidation logic became nightmarishly complex when dealing with multi-tenant content hierarchies. Moving to traditional server-side rendering meant managing Kubernetes clusters, auto-scaling groups, and dealing with cold start problems.

Then we realized something fundamental: we were trying to pre-generate 185,000 static HTML files when we only needed to serve maybe 2,000 pages per day to actual search crawlers. We were solving the wrong problem. What if we generated HTML dynamically only when a crawler actually requested it, cached the result aggressively, and skipped the entire build process altogether?

Database-Driven Dynamic Rendering at Scale

The solution we built is deceptively simple: every page is stored in our Supabase database as structured content. When a search crawler requests a page, our edge worker fetches the content from the database, renders it to HTML on-demand, caches the result at the edge, and serves it in under 40 milliseconds. When the content updates, we simply invalidate the specific cached page rather than rebuilding the entire site.

Here's exactly what happens when Googlebot crawls a page:

1. Request hits Cloudflare edge network (routed to nearest of 300+ locations)
2. Edge worker checks Workers KV cache for pre-rendered HTML (cache hit = 5ms response)
3. On cache miss: worker detects it's a bot via User-Agent header
4. Worker queries Supabase for page content (typically 15-20ms with connection pooling)
5. React component renders server-side to HTML string (8-12ms)
6. Full HTML document assembled with meta tags, structured data, and OpenGraph (2-3ms)
7. Result cached in Workers KV with 24-hour TTL and returned to crawler
8. Total time from request to response: 35-40ms on cache miss, 5ms on cache hit

The magic is in what we don't do. We don't pre-generate anything. We don't track dependencies between pages. We don't maintain complex build pipelines. When an organization publishes a new blog post, it goes straight into the database and becomes immediately crawlable. No deployment. No build queue. No waiting. The first crawler to request that page triggers the render, which gets cached for 24 hours. Subsequent crawlers and social media preview generators all hit the cache.

Human users get an entirely different experience. When a person requests a page, they receive our client-side React application, which loads instantly from Cloudflare's CDN and then fetches data via our API. They get the full interactive single-page application experience with instant navigation, optimistic updates, and all the modern UX patterns we've built. We're serving two completely different experiences from one codebase, optimized for the very different needs of humans versus bots.

Why We Abandoned Static Site Generation

Static Site Generation is incredible technology, and we want to be clear that it's the right choice for many projects. For blogs, documentation sites, marketing pages, and content that changes infrequently, SSG is hard to beat. The performance is unmatched, the hosting costs are minimal, and the developer experience with frameworks like Next.js is phenomenal. We started with SSG precisely because it was the right tool for our initial use case.

But SSG has fundamental limitations that become crushing at scale with multi-tenant architectures. Our specific context made the traditional approach increasingly painful:

• Build Time Explosion: 6 hours to rebuild 185,000 pages, growing linearly with content
• Memory Constraints: Next.js builds exceeded 16GB RAM and crashed regularly
• Content Freshness: Organizations waited hours for published content to go live
• Deployment Complexity: Failed builds required manual intervention and lost developer hours
• Incremental Builds: ISR cache invalidation became impossibly complex with multi-tenant content
• Infrastructure Costs: Large build machines cost $200-400/month, still couldn't handle the load
• Scaling Ceiling: No path forward as we approach 500K+ pages in the next year

The breaking point was realizing that we were rebuilding 185,000 pages every time anyone changed anything, but crawlers were only requesting maybe 3,000-4,000 pages per day. We were doing 60x more work than necessary. Dynamic rendering at the edge flips this equation completely. We only do work when crawlers actually request pages, and that work is cached aggressively so we rarely render the same page twice in a 24-hour period.

The Economics of Edge Rendering vs Static Builds

The cost difference between our old SSG approach and our new edge rendering system is almost comical. With Next.js and Vercel, we were on a trajectory toward their Enterprise plan as we scaled. The Pro plan at $100/month supports unlimited static pages in theory, but in practice, builds over 2-3 hours become unreliable. Enterprise pricing starts at $1,000/month but scales based on usage, and we were projecting $2,000-3,000/month by the end of our first year.

Here's what we're actually paying now with database-driven edge rendering:

• Cloudflare Workers: $5/month for 10M requests (we use ~2.5M/month)
• Workers KV: $5/month for 1GB storage + 10M reads (we use ~400MB + 6M reads)
• Supabase: $25/month Pro plan (handles all database queries with connection pooling)
• Cloudflare R2: Free tier covers our media storage (under 10GB)
• Total Monthly Cost: $35/month for entire system

The Vercel Enterprise plan we were heading toward would have cost $2,500/month at our projected scale. We're spending $35/month instead. That's a 98.6% cost reduction, or $29,640 saved annually. And unlike the SSG approach where costs scale linearly with page count, our edge rendering costs scale with actual traffic, which grows much more slowly.

But the real savings are in developer time. Our team was spending 15-20 hours per week managing builds, debugging deployment failures, optimizing build performance, and manually restarting crashed builds. That's half of one developer's entire time. At a loaded cost of $150/hour for a senior engineer, that's $2,250 per week or $9,000 per month in opportunity cost. We've eliminated essentially all of that overhead.

Technical Architecture: How We Built It

Our implementation is built on four core components that work together seamlessly. At the foundation is Supabase PostgreSQL, which stores all content in structured tables with proper indexing for fast queries. The edge rendering layer runs on Cloudflare Workers, which execute our React server-side rendering logic at 300+ global locations. Workers KV provides our distributed cache layer, storing pre-rendered HTML close to users worldwide. And Cloudflare R2 stores media assets like images and videos with automatic CDN distribution.

The database schema is designed specifically for multi-tenant content with fast querying. Every table has an organization_id column with an index, allowing us to quickly filter content by organization. We use composite indexes on (organization_id, slug) pairs for instant page lookups. Content is stored as JSONB, giving us flexibility to add fields without migrations while maintaining PostgreSQL's excellent JSON query performance.

Page Lookup Query (optimized for sub-10ms response):

-- This query uses the composite index (organization_id, slug)
-- Typical execution time: 3-8ms with proper connection pooling
SELECT
  p.id,
  p.title,
  p.content,
  p.meta_description,
  p.og_image,
  p.published_at,
  o.name as org_name,
  o.domain as org_domain,
  u.name as author_name
FROM pages p
LEFT JOIN organizations o ON p.organization_id = o.id
LEFT JOIN users u ON p.author_id = u.id
WHERE p.organization_id = $1
  AND p.slug = $2
  AND p.status = 'published'
LIMIT 1;

The edge rendering logic is remarkably compact for what it accomplishes. We use React's renderToString method to convert our components to HTML server-side. The worker detects bots using a simple regex on the User-Agent header. For human users, we serve the cached React bundle. For bots, we render full HTML with all meta tags, OpenGraph properties, and JSON-LD structured data.

Edge Worker Bot Detection and Rendering:

export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    const userAgent = request.headers.get('User-Agent') || '';

    // Bot detection covers 95%+ of crawlers that matter
    const isBot = /googlebot|bingbot|slurp|duckduckbot|baiduspider|yandexbot|facebookexternalhit|twitterbot|linkedinbot|slackbot|whatsapp/i.test(userAgent);

    if (!isBot) {
      // Serve cached SPA for human users
      return env.ASSETS.fetch(request);
    }

    // Check cache first (5ms response on hit)
    const cacheKey = `page:${url.pathname}`;
    const cached = await env.KV.get(cacheKey);
    if (cached) {
      return new Response(cached, {
        headers: { 'Content-Type': 'text/html; charset=utf-8' }
      });
    }

    // Cache miss: fetch from database and render
    const pageData = await fetchPageData(url.pathname, env);
    if (!pageData) {
      return new Response('Not Found', { status: 404 });
    }

    // Server-side render with React
    const html = renderPageToHTML(pageData);

    // Cache for 24 hours
    await env.KV.put(cacheKey, html, { expirationTtl: 86400 });

    return new Response(html, {
      headers: {
        'Content-Type': 'text/html; charset=utf-8',
        'Cache-Control': 'public, max-age=3600'
      }
    });
  }
};

Cache invalidation is handled through webhooks. When an organization updates a page in our CMS, the update triggers a webhook to Cloudflare's API that purges that specific page from Workers KV. The next crawler to request that page gets fresh content rendered from the database. This selective invalidation means content updates are reflected instantly while we maintain high cache hit rates on unchanged content.

Sitemap Generation for 185K Pages

Generating sitemaps for 185,000 pages presented its own challenge. The standard approach of creating one giant XML file doesn't work well at this scale. Google recommends keeping sitemaps under 50MB and 50,000 URLs. With 185,000 pages, we need at least 4 sitemaps, but organizing them efficiently requires more thought.

Our solution is dynamic sitemap generation with intelligent segmentation. We generate sitemaps on-demand rather than as files. When a crawler requests /sitemap.xml, they get a sitemap index that points to organization-specific sitemaps. Each organization gets their own sitemap at /sitemap-{organization-id}.xml, which lists only their pages. This keeps individual sitemaps small and makes incremental updates trivial.

The sitemap generation happens at the edge just like page rendering:

• Sitemap Index: Lists all 1,200 organization-specific sitemaps
• Organization Sitemaps: Generated on-demand from database queries
• Caching: Sitemaps cached for 6 hours (content changes less frequently than pages)
• Compression: Gzip compression reduces bandwidth by 80-90%
• Validation: All URLs tested against live database to avoid 404s

We also implement smart prioritization in our sitemaps. Recently updated pages get higher priority scores. Blog posts get higher change frequency than static pages. Product listings get higher priority than team profile pages. Google uses these signals to optimize their crawl budget, ensuring our most important content gets crawled more frequently.

Real Results After 60 Days

The transformation from SSG to database-driven edge rendering was dramatic. We completed the migration over a weekend, switching traffic from our old Vercel deployment to the new Cloudflare Workers setup in a coordinated cutover. The results exceeded our most optimistic projections.

Before Migration (Static Site Generation):

• Total pages: 185,000
• Build time: 6 hours (often failed)
• Content update latency: 6+ hours until live
• Google indexed pages: 12,400 (6.7% of total)
• Search impressions: 185,000/month
• Average TTFB: 120ms (served from CDN)
• Infrastructure cost: $100/month Vercel Pro + $180/month build server
• Developer time: 20 hours/week managing builds

After Migration (Edge Rendering) - 60 Days:

• Total pages: 192,000 (growing 1,000+/week)
• Build time: 0 seconds (no builds needed)
• Content update latency: Instant (live in database immediately)
• Google indexed pages: 89,500 (46.6% of total, up 622%)
• Search impressions: 1,240,000/month (up 570%)
• Search clicks: 18,400/month (up 680%)
• Average TTFB: 38ms globally (rendered at edge)
• Cache hit rate: 94% (only 6% of requests hit database)
• Infrastructure cost: $35/month total
• Developer time: ~2 hours/month monitoring

The economic impact is staggering. We went from $280/month in infrastructure costs to $35/month, saving $245/month or $2,940/year. But the real savings are in recovered developer time. We were spending 80 hours per month managing builds and deployments. At $150/hour loaded cost, that's $12,000 per month in opportunity cost. We've reduced this to maybe 2 hours per month for monitoring, freeing up 78 hours monthly to build features that actually matter to users.

The SEO improvements are equally impressive. Google went from indexing 6.7% of our pages to 46.6% in just 60 days. Our working theory is that instant content updates mean Googlebot encounters fresh content more frequently, which signals that our site is active and worth crawling more often. The faster response times also mean Googlebot can crawl more pages in the same amount of time, increasing our crawl budget efficiency.

Challenges We Had to Overcome

Implementing database-driven rendering at this scale wasn't straightforward. The first major challenge was database connection management. Cloudflare Workers are stateless and spawn thousands of instances globally. Each instance that needs to query the database must establish a connection. Naive implementations would open a new connection for every request, quickly exhausting our database's connection pool limit.

We solved this through aggressive connection pooling and query optimization:

• Supabase connection pooler (Supavisor) handles up to 200 concurrent connections
• Workers reuse connections within the same request context using a singleton pattern
• Database queries use prepared statements to avoid repeated parsing overhead
• Composite indexes on (organization_id, slug) reduce query time to 3-8ms
• Workers KV cache hits (94% of requests) bypass database entirely
• Result: Database handles 150,000 queries/day with 40-60% CPU utilization

The second challenge was React server-side rendering performance. Rendering large React component trees can be CPU-intensive, and we had some pages with 200+ components. In early testing, complex pages took 80-120 milliseconds to render, which was too slow for our sub-50ms target response time.

We optimized rendering through component architecture changes:

• Lazy-load heavy components that don't matter for SEO (admin panels, interactive widgets)
• Strip client-only code from SSR bundle (analytics, form validation, localStorage)
• Use React.memo() on expensive components to prevent re-renders during SSR
• Inline critical CSS in the initial HTML (eliminates render-blocking requests)
• Pre-serialize JSONB content to reduce parsing overhead
• Result: Complex pages render in 8-15ms, simple pages in 3-5ms

The third challenge was ensuring data consistency across the cache layer. With 300+ edge locations caching content, we needed a reliable way to invalidate specific pages when content changed. Cloudflare's cache API is eventually consistent, meaning purge requests can take 5-30 seconds to propagate globally.

We addressed this through a hybrid approach:

• Immediate cache purge via Cloudflare API when content updates
• Versioned cache keys include content hash (automatic invalidation on change)
• Short TTL (24 hours) ensures stale content expires naturally
• Cache-Control headers allow browsers to cache for 1 hour (balances freshness and performance)
• Result: Content updates visible globally within 30 seconds while maintaining 94% cache hit rate

When Database-Driven Rendering Makes Sense vs Static Generation

Database-driven edge rendering isn't the right solution for every project, and it's important to understand when it makes sense versus when traditional SSG is better. The decision depends heavily on your content update patterns, scale, and architecture.

Choose Database-Driven Edge Rendering if:

• You have 10,000+ pages or multi-tenant architecture with many organizations
• Content updates multiple times per day and needs to be instantly live
• Build times exceed 15-30 minutes and are growing
• You need sub-50ms global response times for SEO
• You want to minimize infrastructure costs ($10-50/month range)
• You already have content in a database (no migration needed)
• Your content is dynamic and personalized per organization/user

Choose Static Site Generation (Next.js/Gatsby) if:

• You have fewer than 10,000 pages that change infrequently
• Content updates weekly or monthly (not hourly/daily)
• Build times are under 10 minutes and manageable
• You want maximum performance with CDN edge caching
• You prefer simpler deployment workflows (git push → deploy)
• Your content is in markdown/MDX files or a headless CMS
• You don't need instant content updates for SEO

Final Thoughts

Moving from static site generation to database-driven edge rendering was one of the most impactful architectural decisions we've made at FIKR. We went from a system that was collapsing under its own weight to one that scales effortlessly to millions of pages. Build times went from 6 hours to zero. Content updates went from hours of delay to instant. Infrastructure costs dropped by 87%. And perhaps most importantly, our team went from spending 20 hours per week firefighting build failures to focusing entirely on shipping features.

The approach isn't revolutionary - we're just rendering HTML on demand rather than pre-generating it. But the combination of edge computing, aggressive caching, and database optimization creates a system that's faster, cheaper, and more reliable than traditional SSG at our scale. Organizations publish content and it's immediately crawlable by search engines. No waiting. No build queues. No complexity.

Implementation took two weeks from initial prototype to full production deployment. Week one was spent building the core edge worker, optimizing database queries, and implementing the caching layer. Week two was migration planning, running parallel systems for validation, and cutting over traffic. We did a phased rollout, moving 10% of traffic to the new system, then 50%, then 100% over three days to ensure stability.

If you're building a multi-tenant platform or have more than 50,000 pages, and your build times are becoming a bottleneck, database-driven edge rendering is absolutely worth evaluating. The combination of instant content updates, sub-40ms response times, and $35/month infrastructure costs is hard to beat with any other approach. We're now confidently planning for 500,000+ pages in the next year, knowing our architecture can handle it without breaking a sweat.