Skip to main content
MarketCheckBook a Demo

MarketcheckBot

MarketcheckBot is Marketcheck’s web crawling bot that crawls and extracts millions of live adverts from thousands of websites for the industries listed on our website.

Version
2.0
Bot Type
Good (Identifies itself)
Category
Analytics
Obeys Robots.txt
Yes
Obeys Crawl Delay
Yes
User-Agent String
MarketcheckBot/2.0 (https://marketcheck.com/MarketcheckBot)
IP Address
35.208.195.36
Reverse DNS suffix
bot.marketcheck.com

What is MarketcheckBot?

MarketcheckBot is Marketcheck’s web crawling bot that crawls and extracts millions of live adverts from thousands of websites for the industries listed on our website. The adverts collected are distributed in the industry to various stakeholders across the value chain for use cases like search, syndication and analytics. MarketcheckBot helps the host website (the advert publisher) take the adverts deep into the industry and increase the advert footprint and publisher’s visibility without the publisher having to build the distribution tier themselves. To date, MarketcheckBot has enabled dozens of web & mobile apps and inventory syndication partners to feature these adverts in their applications or partner directories that drive traffic back to live advert pages on the publisher’s website.

How to Whitelist MarketcheckBot

You can whitelist MarketcheckBot using any one or a combination of the following methods:

  1. By User-Agent String

    Allow requests matching the following User-Agent in your WAF, CDN, or bot management platform (e.g. Cloudflare, Akamai, Imperva):

    user-agent

    MarketcheckBot/2.0 (https://marketcheck.com/MarketcheckBot)

    Note: If you had previously allowlisted the older version, also ensure the legacy User-Agent is still permitted:

    user-agent (legacy)

    Autobot (https://marketcheck.com/)

  2. By Single IP Address (Primary)

    If your firewall or platform supports allowlisting only one IP, use the primary egress IP:

    TypeIP Address
    Primary IP35.208.195.36

  3. By Multiple IP Addresses (All Egress IPs)

    For complete coverage, allowlist all of MarketcheckBot’s fixed egress IPs:

    #IP Address
    135.209.28.127
    235.208.69.97
    335.209.228.167
    435.208.126.38
    535.208.83.130
    635.208.223.13
    735.209.64.221
    835.208.201.253
    935.209.244.59
    1035.209.145.85

  4. Via robots.txt

    To explicitly permit MarketcheckBot on your site, add the following to your robots.txt:

    robots.txt

    User-agent: MarketcheckBot
Allow: /
Crawl-delay: 3

How MarketcheckBot accesses your site

Unlike other web spiders, MarketcheckBot is a focussed crawler that only crawls a certain set of web pages from the target website on a daily basis. MarketcheckBot uses an algorithmic process to fetch only the modified set of web pages from the target website to keep bandwidth usage at a minimum. For most sites, MarketcheckBot shouldn’t access your site more than once every few seconds on average, with a minimum delay of at least 3 seconds between successive web hits. MarketcheckBot also adheres to the Crawl-Delay directive in the robots.txt rules configured on the target website.

Blocking MarketcheckBot from accessing content on your website

If you wish to block MarketcheckBot from accessing all or partial content on your website, you could block it by placing Disallow directive in your robots.txt as follows – To keep it off a particular directory or path on your website –

robots.txt

User-agent: MarketcheckBot
Disallow: /directory/

To keep it off from your entire website –

robots.txt

User-agent: MarketcheckBot
Disallow: /

Reporting observations/issues with MarketcheckBot

MarketcheckBot respects the directives in robots.txt, but if you still do spot instances where it’s not playing by the rules you can always reach out to us for getting it to behave. Please report spam/observations/suspicious activity to info@marketcheck.com. We take bad behaviour by our bot very seriously and are very prompt in resolving any reported issues on MarketcheckBot behaviour.