Bots and AI crawlers

Filter bots from your stats and measure AI crawlers with a small server-side snippet.

AI crawlers (ChatGPT, Claude, Perplexity, Gemini and others), search engines and link previews never run JavaScript, so the tracking script can't see them. A small server-side snippet reports them instead. Only bots are reported; regular visitors never trigger a request.

Set it up

1
Open Settings → Integrations → Bot tracking. The snippets there already contain your website id.
2
Add the snippet for your stack (below). It checks the user agent with a cheap pattern and, for likely bots only, sends one small request to https://trakily.com/api/bots after the response, so pages are never slowed down.
3
Turn on AI & crawler visits in Settings → Dashboard. The panel switches to Receiving once the first crawler visits.

Trakily classifies each request on its side and keeps only recognised crawlers. Query strings are removed from paths.

Next.js snippet

Next.js 16 calls this file proxy.ts (middleware.ts on older versions). Merge into your existing one if you have it.

proxy.ts

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const BOTS = /bot|crawl|spider|slurp|GPT|ChatGPT|Claude|anthropic|Perplexity|DuckAssist|MistralAI|meta-external|facebookexternalhit|Bytespider|CCBot|Amazonbot|Applebot|cohere|Diffbot|Google-|Slackbot|Discordbot|WhatsApp|Embedly|Timpibot|ImagesiftBot/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get("user-agent") ?? "";
  if (BOTS.test(userAgent)) {
    // Runs after the response is sent; never slows the page down.
    event.waitUntil(
      fetch("https://trakily.com/api/bots", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          websiteId: "pls_…",
          host: request.nextUrl.hostname,
          path: request.nextUrl.pathname,
          userAgent,
          country: request.headers.get("x-vercel-ip-country"),
        }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

export const config = {
  // Pages only: skip Next internals and static files.
  matcher: ["/((?!_next/|api/|favicon.ico|.*\\.(?:png|jpg|jpeg|gif|svg|webp|ico|css|js|txt|xml|woff2?)$).*)"],
};

Cloudflare Worker snippet

Add a route for your site (example.com/*) to this Worker. It works with any backend behind Cloudflare.

worker.js

const BOTS = /bot|crawl|spider|slurp|GPT|ChatGPT|Claude|anthropic|Perplexity|DuckAssist|MistralAI|meta-external|facebookexternalhit|Bytespider|CCBot|Amazonbot|Applebot|cohere|Diffbot|Google-|Slackbot|Discordbot|WhatsApp|Embedly|Timpibot|ImagesiftBot/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get("user-agent") || "";
    const url = new URL(request.url);
    if (BOTS.test(userAgent) && (response.headers.get("content-type") || "").includes("text/html")) {
      ctx.waitUntil(
        fetch("https://trakily.com/api/bots", {
          method: "POST",
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({
            websiteId: "pls_…",
            host: url.hostname,
            path: url.pathname,
            userAgent,
            country: request.cf && request.cf.country,
          }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};

Express / Node snippet

server.js

const BOTS = /bot|crawl|spider|slurp|GPT|ChatGPT|Claude|anthropic|Perplexity|DuckAssist|MistralAI|meta-external|facebookexternalhit|Bytespider|CCBot|Amazonbot|Applebot|cohere|Diffbot|Google-|Slackbot|Discordbot|WhatsApp|Embedly|Timpibot|ImagesiftBot/i;

app.use((req, res, next) => {
  const userAgent = req.get("user-agent") || "";
  if (req.method === "GET" && BOTS.test(userAgent)) {
    fetch("https://trakily.com/api/bots", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        websiteId: "pls_…",
        host: req.hostname,
        path: req.path,
        userAgent,
      }),
    }).catch(() => {});
  }
  next();
});

WordPress snippet

Paste into your theme's functions.php or a small custom plugin. The request is non-blocking.

functions.php

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (!preg_match('/bot|crawl|spider|slurp|GPT|ChatGPT|Claude|anthropic|Perplexity|DuckAssist|MistralAI|meta-external|facebookexternalhit|Bytespider|CCBot|Amazonbot|Applebot|cohere|Diffbot|Google-|Slackbot|Discordbot|WhatsApp|Embedly|Timpibot|ImagesiftBot/i', $ua)) return;
    wp_remote_post('https://trakily.com/api/bots', [
        'blocking' => false,
        'timeout'  => 1,
        'headers'  => ['Content-Type' => 'application/json'],
        'body'     => wp_json_encode([
            'websiteId' => 'pls_…',
            'host'      => parse_url(home_url(), PHP_URL_HOST),
            'path'      => strtok($_SERVER['REQUEST_URI'] ?? '/', '?'),
            'userAgent' => $ua,
        ]),
    ]);
});

Vercel log drain (no code)

On Vercel you can skip the snippet. In your Vercel project go to Settings → Drains → Add Drain → Logs, choose JSON, select the static, edge and lambda sources, and use this endpoint (your site's id is filled in on the panel):

https://trakily.com/api/bots/vercel-drain?site=pls_…

Then paste the drain's signature secret into the Vercel log drain card and click Save secret. Requests are verified with that secret and rejected until it's saved.

Recognised crawlers

Dashboard tab	Crawlers
AI answers	ChatGPT (ChatGPT-User), Claude (Claude-User), Perplexity (Perplexity-User), DuckDuckGo AI, Meta AI, Mistral, Gemini (user fetch)
Indexing	ChatGPT Search (OAI-SearchBot), Claude Search, PerplexityBot, Applebot, Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider
Training	GPTBot, ClaudeBot, Common Crawl (CCBot), Bytespider, Meta AI crawler, Google Vertex, Amazonbot, Cohere, Diffbot, Timpi, ImagesiftBot

Dashboard tab: AI answers
Crawlers: ChatGPT (ChatGPT-User), Claude (Claude-User), Perplexity (Perplexity-User), DuckDuckGo AI, Meta AI, Mistral, Gemini (user fetch)

Dashboard tab: Indexing
Crawlers: ChatGPT Search (OAI-SearchBot), Claude Search, PerplexityBot, Applebot, Googlebot, Bingbot, DuckDuckBot, YandexBot, Baiduspider

Dashboard tab: Training
Crawlers: GPTBot, ClaudeBot, Common Crawl (CCBot), Bytespider, Meta AI crawler, Google Vertex, Amazonbot, Cohere, Diffbot, Timpi, ImagesiftBot

Google-Extended and Applebot-Extended only exist in robots.txt and never appear in a request, so no tool can count them. Link-preview bots (X, Slack, Discord and others) are recorded too; X previews appear as hollow dots on the main chart.

On the dashboard

The AI traffic card has three tabs, AI answers, Indexing and Training, with hits per bot over time. Toggle Group by page to see which pages each kind of crawler reads, and click a bot to hide it from the chart. Human visits that arrive from AI assistants are a separate thing: they are regular visitors in the AI / LLM channel.