robots.txt: how to check what's blocked, and when blocking backfires

1Test this page and its files 30 seconds

Open DevTools (F12) on the page, go to the Console and paste:

Copy// Tests this page, and the files it loaded from this site, against robots.txt for Googlebot
const robotsUrl = location.origin + '/robots.txt';
let res = null;
try { res = await fetch(robotsUrl, { cache: 'no-store' }); } catch (e) {
  console.log('Could not read ' + robotsUrl + ' from this page (it may redirect to another host). Open it in a new tab instead.');
}
if (res && !res.ok) {
  console.log('robots.txt returned ' + res.status + (res.status >= 500 || res.status === 429
    ? ': Google pauses crawling the site for up to 12 hours, then uses its last good copy of the file'
    : ': Google treats that as no restrictions'));
} else if (res) {
  const groups = [];
  let group = null;
  (await res.text()).split(/\r?\n|\r/).forEach((raw, i) => {
    const line = raw.replace(/#.*$/, '').trim();
    const m = line.match(/^([a-z-]+)\s*:\s*(.*)$/i);
    if (!m) return;
    const field = m[1].toLowerCase(), value = m[2].trim();
    if (field === 'user-agent') {
      if (!group || group.rules.length) groups.push(group = { agents: [], rules: [] });
      // Google reads the name up to the first character that isn't a letter, - or _ (Googlebot/2.1 = googlebot)
      group.agents.push(value === '*' ? '*' : (value.match(/^[a-z_-]+/i) || [''])[0].toLowerCase());
    } else if ((field === 'allow' || field === 'disallow') && group && value) {
      // Google compares percent-encoded forms: encode non-ASCII characters, upper-case existing escapes
      const path = value.replace(/[^\x00-\x7F]+/g, s => encodeURI(s)).replace(/%[0-9a-f]{2}/gi, s => s.toUpperCase());
      group.rules.push({ allow: field === 'allow', path, line: i + 1, text: line });
    }
  });
  // Google combines every group that names Googlebot; if there are none, it uses the * groups
  const named = groups.filter(g => g.agents.includes('googlebot'));
  const rules = (named.length ? named : groups.filter(g => g.agents.includes('*'))).flatMap(g => g.rules);
  console.log('Using ' + (named.length ? 'the User-agent: Googlebot rules' : groups.some(g => g.agents.includes('*')) ? 'the User-agent: * rules' : 'no rules (no group applies to Googlebot)'));
  const matches = (pattern, path) => {
    const end = pattern.endsWith('$');
    const body = (end ? pattern.slice(0, -1) : pattern).replace(/[.+?^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*');
    return new RegExp('^' + body + (end ? '$' : '')).test(path);
  };
  // The longest matching rule wins; Allow wins a tie
  const verdict = url => {
    const u = new URL(url);
    let best = null;
    for (const r of rules) {
      if (matches(r.path, u.pathname + u.search) && (!best || r.path.length > best.path.length || (r.path.length === best.path.length && r.allow))) best = r;
    }
    return best && !best.allow ? 'BLOCKED by line ' + best.line + ': ' + best.text : 'allowed';
  };
  const entries = performance.getEntriesByType('resource');
  const files = [...new Map(entries.map(e => [e.name.split('#')[0], e.initiatorType]))]
    .filter(([url]) => url.startsWith(location.origin + '/') && url !== robotsUrl);
  const blocked = files.map(([url, type]) => ({ url, checked: type, result: verdict(url) })).filter(r => r.result !== 'allowed');
  console.table([{ url: location.href, checked: 'this page', result: verdict(location.href) }, ...blocked]);
  console.log(files.length + ' file(s) from this site checked, ' + blocked.length + ' blocked'
    + (entries.length >= 250 ? '. The browser stops listing files after 250, so some may be missing' : ''));
}

It reads the site's robots.txt using Google's documented rules, then says whether this page is blocked and lists any blocked scripts, styles, images or other files the page loaded from the same site, with the rule that blocks each one.

It checks Googlebot's rules only, for the same host as the page. Files from other hosts (a CDN, a subdomain) are governed by their own robots.txt. It sees only files the page has loaded so far, so scroll and click first if some load late.

2Ask Google 2 minutes

In Search Console, paste the URL into URL Inspection. "Crawl allowed?" says whether robots.txt blocks it. Under Settings, the robots.txt report (for domain-level properties) shows which robots.txt files Google found, when it last fetched them and any problems reading them.

3Read the file 2 minutes

Open /robots.txt on the site and read it with the matching rules above in mind. Look especially for short Disallow paths without a trailing slash, rules for folders of scripts and styles, and a User-agent: Googlebot group that quietly replaces the general rules.