robotstxt.org favicon

robotstxt.org

https://robotstxt.org· First analyzed 20 Feb 2026· Updated 20 Feb 2026
Export
robotstxt.org
Desktop screenshot of robotstxt.org
Mobile screenshot of robotstxt.org
Mobile

Sponsored

Hosting Provider

IP Address

93.93.131.3

Server Location

🇬🇧 Cambridge, United Kingdom

ASN

AS44684

Server

Apache

Nameservers

ns1.mythic-beasts.com.ns2.mythic-beasts.com.

Performance

Page Speed

392 KB
13 requests
0–49
50–89
90–100

Architecture Stack

Monolithic
⚙️ SSR
Infrastructure
Hosting
Mythic Beasts

Technology Stack

9 Technologies Detected

CSS Frameworks

1

Icon Libraries

1

Payments

1

Web Servers

1

SEO Analysis

D41/100
5
6
2
R
robotstxt.orgrobotstxt.org

The Web Robots Pages

No description available for this page.

Title Length20 chars

Recommended: 50-60 chars

Description Length0 chars

Recommended: 120-160 chars

Issues (2)

  • Missing meta description - important for click-through rates
  • Missing viewport meta tag - page may not be mobile-friendly

Crawlability

Sitemap
robots.txt

2 rules

Allows all crawlers

Sponsored

Sponsored

F

Security Score

0%Critical
Critical / Transactional

No SSL

No WAF

nulld

Top Recommendations

Security recommendation preview

HTTP Headers

Developer Console

accept-ranges
bytes
connection
Keep-Alive
content-encoding
📦 Gzip(gzip)
content-length
1486
content-type
text/html
date
Fri, 20 Feb 2026 03:57:35 GMT
Good Warning Issue

Email Infrastructure

Custom Mail Server

Score

45

SPF
DKIM
DMARC

MX Records (2)

mx1.mythic-beasts.com
mx2.mythic-beasts.com

Domain Information

robotstxt.org

Domain Age
25y 5m

Created

04.09.2000

Expires

04.09.2026

Last Updated

31.01.2026

Registrar

Tucows Domains Inc.

DNSSEC
Not Enabled

Privacy & Compliance

Quantcast Choice

Cookie Consent Banner
Quantcast Choice
Privacy Policy
Terms of Service

GDPR Indicators

Cookie consent banner detected
CMP: Quantcast Choice
Privacy policy page linked
Legitimate interest basis mentioned

Typography & Readability

Open Sans + Verdana

51/100
Readability
/30
10/40

12px · LH --

Hierarchy
/20
15/25

1.46x ratio

WCAG
/25
15/15

Contrast: undefined

Performance
/25
13/20

2 fonts

Heading Font

Open Sans

Sans18px

Body Font

Verdana

Sans12px

Contrast Ratio

--

Reading Comfort

~39 CPL

Needs attention
18px
h1
18px
h2
12px
p
Minor Second
Scale

❌ Critical: 12px body text is too small. Increase to at least 14px.

⚠️ Weak hierarchy: Headlines should be at least 2x body size for clear visual structure.

✅ Good performance: Using 2 font families keeps load times fast.

✅ Smart font loading: Using font-display: swap prevents invisible text during load.

Business & Marketing Stack

3 Tools Detected

Enterprise

Behavior Analytics

1

User behavior and product analytics

Advertising

1

Ad platforms and retargeting pixels

Privacy & Cookies

1

Cookie consent and privacy compliance tools

3

Categories

3

Total Tools

1

Enterprise

1

Mid-Market

Assessment: Sophisticated marketing and sales technology infrastructure

Ad Networks

Google AdSenseMedia.netSovrnPubMaticOpenXIndex Exchange

Sponsored

Scripts & Styles

9 Resources

7
Scripts
2
Styles
2
Async
0
Defer

Top Script Sources

pagead2.googlesyndication.com(5)fundingchoicesmessages.google.com(2)

Sponsored

Last analyzed: 20 Feb 2026 at 03:57

Historical Analysis & Change Tracking