For site owners and admins

EarshotRecorder

EarshotRecorder is the crawler behind Earshot. It keeps a dated record of a website's public pages, so the site's owner can see what changed, when Earshot first saw it, and whether it lines up with a change in search traffic.

When it visits

Only after someone asks Earshot to record the site: the practice that owns it, or the person who builds or manages it for them. They confirm the site is theirs to record before anything runs.

A free look-back records the site once. Weekly monitoring records it about once a week, and checks once a day that the home page and booking links still open.

What it reads

  • robots.txt, always first, and the sitemaps it names.
  • Public pages from the sitemap and the home page's links, up to 150 a visit.
  • Up to 20 key pages a week opened in a browser with their images and scripts, so the record shows what visitors see.
  • Links on the site, one request each, to find the ones that no longer open.

It never fills in forms, signs in, books appointments or reads anything behind a login.

How it behaves

It names itself in every request:

Mozilla/5.0 (compatible; EarshotRecorder/1.0; +https://useearshot.com/recorder)

It waits about a second between requests, makes no more than 2 at once, and stops after 4 minutes on any one site.

It follows the robots.txt rules for EarshotRecorder, or the rules for all crawlers when none name it. When robots.txt is missing, it reads the site. When the server answers with an error or doesn't answer, it records nothing and tries again on its next visit.

How to keep it out

Add this to the site's robots.txt:

User-agent: EarshotRecorder
Disallow: /

To keep it out of one folder only:

User-agent: EarshotRecorder
Disallow: /private/

It reads robots.txt again at the start of every visit, so a change applies from the next one.

Questions

Write to [email protected] with the site's address.