1. Home
  2. Docs
  3. AFL UTM Tracker Documentation
  4. Developer
  5. Crawler / Bots User Agents

Crawler / Bots User Agents

AFL UTM Tracker 2.8.0 has the feature to filter out crawlers and bots from sending the admin-ajax.php request through our JavaScript frontend file based on the browser user agent.

Default List

This list is in the form of a regular expression (regex).

(googlebot|bingbot|yandexbot|slurp|spider|robot|bot.html|bot.htm|facebookbot|facebookexternalhit|twitterbot|storebot|microsoftpreview)

Add a new Crawler

Let’s say you are using the ContentKingApp.com service to monitor your website content. ContentKing’s job is to crawl the pages of your sites.

However in doing so, it may send large number of traffic to your site which also triggers our admin-ajax.php request. Because admin-ajax.php requests are not cache, it uses the PHP resources and your site may slow down.

So the idea here is to add ContentKing’s user agent to our bots list. Based on the ContentKing’s documentation, it will send the following user agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

Source: https://www.contentkingapp.com/support/user-agent-string/

Example Code

Putting the full user agent above just takes up so much space, so we grab only the term ‘contentkingapp’ and append it to our default user agent list before the closing bracket.

(...|...|microsoftpreview|contentkingapp)

Place the code below into your theme’s functions.php file.