Writing

meta-externalagent and meta-webindexer: crawlers were a third of my visitors

By Mohamed Gado · · 4 min read

meta-externalagent and meta-webindexer are Meta's web crawlers, and they run JavaScript. That means your client-side analytics records them as visitors. On my site they looked like a desktop Chrome at 1920x1080, browsing from US datacenters, and they were 40 of the 43 visitors who hit an error in one week. With AhrefsBot and ShapBot, crawlers made up 31% of all "visitors". They never converted. I now drop them before any beacon is sent, and I filter older history with the same pattern.

What are meta-externalagent and meta-webindexer?#

Meta documents several crawlers. facebookexternalhit is the old one: it fetches a page when someone shares a link, to build the preview card. meta-externalagent and meta-webindexer crawl the web more broadly, for Meta's AI and search features.

facebookexternalhit usually reads only the HTML. The newer two render the page in a real browser, which runs your scripts, which runs your analytics.

How they showed up in my analytics#

PostHog showed 48 exceptions in a week, from 43 "visitors". All 48 had the same message: my support chat widget had failed to load. Forty-three people with a broken chat widget would be a real problem, so I broke the group down before fixing anything.

Grouped by browser version, screen size and city, the "people" looked like this:

  • One browser: Chrome 145, the same build every time.
  • One screen: 1920x1080.
  • Cities: Prineville, Ashburn and Boydton, which are datacenter towns, not where my readers live.
  • Zero conversions: no app installs, no contact clicks, no CV downloads.

The raw user agent settled it. PostHog keeps it as $raw_user_agent, and 40 of the 43 carried meta-webindexer/1.1 or meta-externalagent/1.1. AhrefsBot and ShapBot made up most of the rest. Together these crawlers were 128 of the week's 409 visitors.

The chat widget's errors were real but harmless: the crawler's browser didn't load the third-party script. That's an expected failure, so it's now a counted product event instead of an exception.

Why didn't the bot filter catch them?#

My client-side gate knew about lab and automation traffic: Lighthouse (Chrome-Lighthouse), headless browsers (HeadlessChrome) and anything with navigator.webdriver set. Meta's crawlers are none of those. They run a normal-looking Chrome, webdriver is false, and the user agent says Chrome/145 along with the crawler's name.

Most analytics tools' built-in bot filters match a list of known crawler names. Whether a new crawler is on that list depends on when the tool last updated it. Mine didn't have these two.

One pattern for every filter#

The fix is one regular expression in one module, read by every place that decides whether a visit is a person:

// Agents that are not people, matched case-insensitively. One copy for the
// client gate (lib/analytics.ts) and the PostHog `{humans}` macro and
// test-account filter (scripts/lib/posthog.mjs).
// No backslashes or quotes: HogQL embeds it in a string literal.
export const SYNTHETIC_UA_SOURCE =
  'Chrome-Lighthouse|HeadlessChrome|PTST|GTmetrix|bot/|crawler|spider|meta-webindexer|meta-externalagent|facebookexternalhit'

bot/ matches a product token (Googlebot/2.1, AhrefsBot/7.0), not the letters "bot" anywhere. A plain bot would also match the Cubot phone brand, and drop real people on Cubot phones.

In the browser, the gate runs before any beacon is sent:

const SYNTHETIC_UA = new RegExp(SYNTHETIC_UA_SOURCE, 'i')
 
export function isSyntheticAgent(ua: string, webdriver: boolean): boolean {
  return webdriver || SYNTHETIC_UA.test(ua)
}

In queries, the same source string becomes a HogQL condition, so history recorded before the filter existed drops out too:

export const NOT_SYNTHETIC =
  `NOT match(coalesce(properties.$raw_user_agent, ''), '(?i)${SYNTHETIC_UA_SOURCE}')`

Every dashboard tile and every ad-hoc query includes it through one {humans} macro. The PostHog project's test-account filter uses the same condition, so PostHog's own insights agree with my queries.

That's why the pattern has no backslashes or quotes: it is embedded inside a HogQL string literal, and a \. there would need different escaping than it does in JavaScript.

What changed after the filter#

Before After
Visitors in the week 409 265
Exceptions in the week 48 6

The remaining 6 exceptions came from one real visitor on Firefox whose browser blocked the chat widget. A blocked widget is now a counted event, not an exception.

Should you block Meta's crawlers instead?#

I don't. Blocking them in robots.txt or at the firewall would keep my pages out of whatever Meta builds from its crawl, and my site is a portfolio that wants to be found. The problem was never that they visit. The problem was counting them as people. Filtering them out of analytics fixes that and keeps the pages open to them.

If you do want to block them, meta-externalagent follows robots.txt. facebookexternalhit is the one that builds link previews on Facebook and WhatsApp, so blocking it breaks your share cards.

Takeaways#

  • meta-externalagent and meta-webindexer run JavaScript, so client-side analytics counts them unless you filter them.
  • Before you fix an error that "many visitors" hit, break the cohort down by browser, screen and city, then read the raw user agent. A fleet usually names itself.
  • Keep one crawler pattern and make the client gate, the query macro and the tool's test-account filter all read it.
  • Match bot/, not bot, or you drop people on Cubot phones.

Another week on the same board, the exceptions came from Firefox and Chrome for iOS injecting scripts into the page.