# RewindexBot

RewindexBot is the crawler behind Rewindex. It archives public SaaS status pages, pricing pages and trust/legal pages so that changes over time stay visible.
## How it behaves

- User-Agent: \`RewindexBot/0.1 (+https://rewindex.com/bot)\`.
- Fetches only public pages; never logs in, submits forms or executes checkout flows.
- Honors robots.txt: a \`Disallow\` for \`RewindexBot\` (or for \`\*\`) on a path stops fetching of that path.
- Status APIs: at most 2 requests per host every 10 minutes. Pricing, trust and legal pages: about one request per page per day.
- At most 8 requests in parallel across all hosts; 6 s connect timeout, 25 s max, at most 5 redirects, 8 MB body cap.
- Unchanged content is not stored again; only a new hash of the visible text creates a new version.

## Opting out

Add this to your robots.txt to stop all fetching:
```
User-agent: RewindexBot
Disallow: /
```
Or email ryan@rvbbit.ai with the domain and we will stop fetching it and say whether to keep or hide existing history.


---
Canonical: https://rewindex.com/bot · Markdown: https://rewindex.com/bot.md · Generated 2026-10-17T21:41:42.758Z