PFR’s robot: codes, new prop firms, rules and prices
PropFirmsRankingBot is PFR’s one robot. It reads the discount codes prop firms publish on their own websites, looks at newly seen websites to find new prop firms and, as PFR’s rule & price reader, re-reads the pricing, rules, FAQ, terms, payout and refund pages of the firms PFR lists or has a page about, the terms first. Here is what it reads, how often, and how to stop it.
How it identifies itself
Every request carries this user agent:
PropFirmsRankingBot/1.0 (+https://propfirmsranking.com/bot; codes@propfirmsranking.com)
What it reads for codes
For codes it reads only a prop firm’s own public pages: its homepage, promo or discount pages a PFR reviewer has approved, and a code feed the firm hosts on its own domain. It looks for codes the firm itself publishes, such as a site-wide promo code.
It never logs in, never fills in or submits a form, never types a code into a checkout and never guesses codes. It doesn’t read coupon sites, code aggregators, Reddit, X, YouTube or Discord.
How often it reads for codes
For codes, it reads a firm’s pages once a day if PFR doesn’t list the firm and every 12 hours if PFR lists it, at most five pages a firm each time. For every purpose, it sends one request at a time to a host and waits at least 10 seconds between page requests to the same host (longer if your robots.txt sets a Crawl-delay). When its copy of your robots.txt is more than a day old, it fetches robots.txt right before the page.
It asks whether a page changed since its last visit (If-None-Match and If-Modified-Since), so an unchanged page costs almost nothing.
When someone using the PFR extension presses “Look for this firm’s own codes” on a site PFR doesn’t know, the robot fetches that site’s robots.txt and homepage once. A site is read this way at most once a day.
How PFR finds new prop-firm websites
To spot new prop firms, PFR reads the public certificate-transparency logs, where every new website certificate is published, and keeps only the names that look like a prop firm’s, such as a name with “funded” and a trading word; every other name is dropped at once. Where PFR has accepted a list’s terms, it also reads lists of newly registered domain names. People can point PFR at a site too: a firm adding itself, a search on PFR for a site PFR doesn’t know, or “Suggest this firm to PFR” in the PFR extension. PFR never reads WHOIS contact details, and the server that reads the certificate logs never visits a website.
For each possible prop firm, PropFirmsRankingBot first checks that the name has an address (a name without one is checked again only in DNS, never visited). Then it reads robots.txt and the homepage, and only when the homepage doesn’t say enough, the sitemap and at most 2 more pages of the same site whose address names pricing, plans, rules, challenges or an FAQ: at most 5 requests a check, robots.txt included. It never follows a link to another site, never fills in a form and never runs the page’s scripts, and it checks at most 600 sites a day.
A site that isn’t ready yet is looked at again on a fixed schedule, then dropped: a parked domain or one for sale after 14, 30 and 60 days, a site under construction or too thin to judge after 3 days, then weekly for 8 weeks. A site that isn’t a prop firm is read again only if a new kind of source names it or it gets a new address such as app. or dashboard., and then once, at least 72 hours after the last visit. PFR keeps its name for 30 days, then only a one-way fingerprint of it for 365 days, so it isn’t read again.
A prop firm may get an Unlisted page on PFR, labelled “Unlisted · Not reviewed by PFR” and showing only what its own website says, quoted with the link and the date. The robot then looks at that firm’s homepage again to see the page still holds: weekly for 8 weeks, then every 14 days, or every 30 days once PFR staff have approved the page or the firm has claimed it — at most 3 requests each time, robots.txt included, asking whether the homepage changed since its last visit. Pages PFR staff made by hand, outside this check, aren’t looked at again yet.
Discovery follows every rule on this page: the same user agent and robots.txt, at least 10 seconds between page requests to one host, the seven-day stop at a security check, and a request to stop covers it too. When PFR learns that a site’s terms of service forbid robots, it puts the domain on a list that neither discovery nor the rule & price reader ever reads (a check of the file or tag the firm itself asks for still reads that one address).
What it reads for rules and prices
For the firms PFR lists and the firms it has an Unlisted page about, it also works as PFR’s rule & price reader. It reads their own public pages about prices and rules: the homepage, pricing pages, a checkout page as it looks before anyone fills it in, the rules, the help centre and FAQ, the terms, the payout and refund policies, the countries and trading platforms a firm offers and its announcements, plus the sitemaps, the llms.txt file and the product or help-centre lists the site publishes for every visitor (such as a WooCommerce, Shopify or Zendesk list). It reads only the firm’s own websites, and on a help centre several firms share only the firm’s own part of it. It never logs in, never fills in or submits a form and never goes through a checkout.
It reads a firm’s terms first. Until they are checked, it reads nothing on the site but robots.txt, the sitemaps, the llms.txt file, /.well-known/tdmrep.json, the homepage and the terms page. If the terms forbid robots, scrapers or other automated access, if the site reserves text and data mining (a tdm-reservation header or tag on its homepage or terms page, a tdmrep.json file, or “noai” in a robots meta tag), or if robots.txt has rules for PropFirmsRankingBot that disallow anything, neither the reader nor PFR’s search for new firms reads anything more there until PFR’s staff find that the clause doesn’t cover these pages (for example, it is about logged-in areas only) or PFR’s lawyers clear it. It looks for these signals when it first reads a site, whenever its terms page or its homepage’s links change, and at least every 30 days. If PFR can’t tell what the terms say, for example because their page can’t be read, it stops there as well and PFR’s staff decide; a site without terms is read under the rules on this page. While it reads a site, it checks the terms again whenever they change.
It makes at most 150 page requests a firm a day, every day, with no exception for a firm’s first read: a first read that needs more pages is spread over several days. Like every purpose, it sends one request at a time to a host and waits at least 10 seconds between page requests to the same host (longer if your robots.txt sets a Crawl-delay). When its copy is more than a day old, it fetches robots.txt right before the page. It follows same-host redirects and makes a rendered page’s subrequests inside that page’s slot; the gap applies between page requests. It asks whether a page changed since its last visit, so an unchanged page costs almost nothing. How often it comes back depends on the page and the firm: prices, rules and the homepage of the busiest firms about once a day, of other firms PFR lists every 3 to 7 days, of firms with an Unlisted page every 7 to 14 days, and terms and policy pages less often; a page that keeps not changing is read less often.
When a page changes, it reads it again at least 6 hours later before PFR counts the change, and may read the firm’s related pages within the hour. Extra reads, when PFR’s staff, a firm’s own edit on PFR or a report of a mistake asks for one, are limited to 3 a firm a day and count towards the same 150.
It keeps a private, dated copy of the text of each version of these pages, only to read the facts, find what changed and quote it, with e-mail addresses, phone numbers and social media handles masked; only PFR’s staff see the copies. On PFR, a fact or a change from a firm’s page shows at most 300 characters quoted from it, with the link and the date, never a whole page. PFR’s privacy policy says how long the copies are kept.
Pages that need JavaScript
Some pages show their prices or rules only once their own scripts run. When PFR switches this on, pages that need JavaScript open in a sandboxed browser that loads the page’s own scripts and data from the same site, at most 60 requests per page, inside the same one-at-a-time slot. PropFirmsRankingBot makes each of those requests itself, robots.txt first and with the same user agent, up to 8 MB in all; the browser can’t reach anything else, and images, videos and fonts aren’t loaded. The whole page counts as one page request towards the 150 a day.
In that browser it only opens tabs and “show answer” toggles. It never logs in, never fills in a form, never goes through a checkout and never gets past a security check: a page behind one is never opened in the browser.
When a firm claims its page
A firm can claim its page on PFR by proving it owns its domain. If it chooses the file or the tag, the robot reads that one address on the firm’s site, robots.txt first: when the firm asks and, while the claim waits, every 3 hours for a week, then once a day for up to 30 days, and at most once a minute and 20 times a day in all. It never follows a redirect from that address to another site. The DNS record and the e-mail code read no website.
These checks run only while PFR has claims open, and they pause for 24 hours after PFR stops all its robots, unless one of its reading switches is turned back on after that; the DNS record and the e-mail code keep working.
robots.txt
It follows robots.txt (RFC 9309): the rules for PropFirmsRankingBot first, otherwise the rules for every robot. It follows a redirect of robots.txt to another address on your own site; a redirect to another site counts as no robots.txt, and then it reads nothing. When a page redirects, it checks the robots.txt of the page it lands on, waits its turn on that host, and never follows a page’s redirect to another site. If your robots.txt can’t be reached because of a server error, it reads nothing.
To stop it, disallow PropFirmsRankingBot in robots.txt: one rule stops every purpose — codes, finding new firms, the rule & price reader and claim checks. Or write to us: a request to stop covers every host of your domain (www., promo. and the rest), for good.
Security checks
If your site shows a security check, a CAPTCHA or a bot wall, or answers 401, 403, 429 or 503, the robot stops and doesn’t come back to that host for seven days. If the answer names a Retry-After wait instead, the robot waits out that wait (at most seven days) and then looks again. It never retries with another user agent, a proxy or a browser to get past a check. For a possible new prop firm it looks again at most 3 times, each time after the block ends, and then leaves the site to PFR staff; the rule & price reader leaves a firm’s site to PFR staff once it keeps being turned away.
A PFR reviewer may then copy a code, a price or a rule by hand from your public pages, and only with a link to the page it came from.
A 401 or 403 from a list a site publishes for every visitor, such as a help-centre or shop list, only closes that list: it isn’t taken as a security check of the whole site.
What PFR does with the codes
Each code is labelled with where it came from, like “On FirmX’s site · seen 1 Oct”. A code read from the text of a page, or one that might be private, is checked by a reviewer before it appears. A code your site states in its own settings (structured data), or that you send PFR in a feed, can appear without a reviewer only once a PFR reviewer approved that page as a source and the robot saw the code on two visits at least an hour apart — and, for a firm PFR doesn’t list, after a 30-day review period. A code worth a fixed amount, or with no clear value, always needs a reviewer, and a code of 30 % or more needs two. A code is removed within an hour of its end date, and within 36 hours of disappearing from your site.
PFR earns nothing from codes on firms it doesn’t list, and doesn’t claim to know every code.
Where it comes from
PropFirmsRankingBot runs on PFR’s hosting provider, whose servers have no fixed addresses, so recognise it by its user agent above. The fixed addresses of PFR’s own servers are listed at /bot/ips.json: the list is empty today, and a server’s addresses go on it before that server starts. The server that reads the certificate logs never visits a website, and when the rule & price reader moves to a server of PFR’s own, its addresses go on the list before it reads anything.
On that server, once PFR has published its signing key, every request the reader sends is also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421), so a site or its CDN can check that it really comes from PFR. The signature only says who is asking: it never gets the robot past a security check.
This page describes crawl policy version 2026-10-22.
Stop it or correct it
Add this to your site’s robots.txt and PropFirmsRankingBot won’t read any page of it, for any purpose:
User-agent: PropFirmsRankingBot Disallow: /
Or email codes@propfirmsranking.com from your firm’s domain. A cease-and-desist stops all reading of your site until PFR’s legal team clears it.
A firm can also send a robot opt-out through PFR’s request form: confirmed from an address at the firm’s domain, it stops the robot for the whole domain, for good, and the firm’s page stays with the date PFR last checked it.
