How to Detect & Prevent Bots on Your Website

Bots are automated software applications that help humans to automate tasks that are often repetitive and time-consuming. Though not all bots are bad, and some are even helpful – think chatbots, search engine bots, web scraping bots, and so on – they can also be used for harm, and wreak havoc on a company’s analytics and security. Companies must therefore be vigilant and ready to mitigate any risks that come with bots. Here’s how.

The first step in detecting and protecting against bots is to understand how they work.

Bots are smart and can be utilized to automate tasks to improve a user’s interaction with your site. For instance, some companies use bots for automated QA or active monitoring. Unfortunately, that same technology can also be used to inflict harm. It is therefore necessary for companies to be able to differentiate between good bots and bad bots, which can be tricky.

Botnets

Botnets are used to further scale an attack as well as distribute it, which makes it harder to detect. When a cybercriminal has control over thousands of devices that they can use to run a scripted attack, it’s easy to achieve greater volumes as compared to running the attack all from a single device. This also makes attacks harder to detect, as they do not come from the same IP, user-agent, or location, which could evade detection based on simple velocity rules.

What are Good Bots?

As mentioned, not all bot traffic is harmful or illegitimate. Bots can help perform a lot of tasks that enable companies to become more efficient and productive. Some beneficial bot capabilities are as follows:

• Search engine/social media crawlers that download and index information from all over the internet

• Chatbots that are automated to provide information and assistance to users

• Automation of tasks that would otherwise take a lot of time, for example, browser extensions that automatically insert coupon codes when a user checks out at a site

• Proprietary integrations with partners, for example, aggregators and online travel agencies that can scrape prices, making for quick searches

Thus, while you want to detect and prevent bad bots, you want to make sure you are not targeting good bots at the same time.

What Are Bad Bots?

In general, the most basic bad bot attacks drive traffic to a website that is not actually coming from real users. This can take a negative toll not just on your analytics but on your overall security and the trustworthiness of your site for customers. Some examples of negative bot traffic include:

• Spamming your company by filling out contact forms with fake information

• Making it look like your website has more users than it actually has

• Tricking you into thinking you have leads when you don’t

• Engaging with competitors on social media

• Posting automated negative comments or reviews

• Ecommerce inventory hoarding

• E-gift card theft

The worst bots, though, can fully automate attacks on your network if they are not detected early enough. These attacks can include:

• New Account Fraud: Automating the creation of new accounts with stolen or synthetic identity information

• Account Takeover: Breaking into users’ accounts to make false transactions and/or steal data
  
  
  • Brute Force Attacks: A trial-and-error attack method where an attacker tries many sets of credentials until the correct one is found
    
    
  • Credential Stuffing: An attack method where credentials obtained illegally, for example from a breach of one service, are user to log into another online service

• DDos Attacks: Overloading servers to shut down a network completely, which can be detrimental to a company’s reputation and finances

• API Abuse: Automating bots to attack at the API level to extract data or perform other types of attacks, for example, inventory hoarding

• Scraping: Extracting data from websites using a crawler for nefarious purposes, such as competitive intelligence, spamming, or selling data on the dark web

• Sniping: Monitoring time-based events, such as online auctions, and entering last-second information or bids, making it impossible for other users to respond

Cost of Bot Based Fraud

As you can see, bots can be used to perform various types of fraud at scale, which makes them incredibly costly to legitimate businesses. Automated attacks can cost around 4.3% of online revenues, which equates to millions of dollars in losses for even a midsize enterprise, with the cost going up for larger companies with bigger revenues. This number is up 72% year over year and continuing to increase. As if that wasn’t bad enough, 88% of organizations say that bots have negatively impacted customer satisfaction, and decreased satisfaction leads to customer churn and decreased revenue, making bot attacks a vicious cycle that organizations must address sooner rather than later.

Bots are becoming more sophisticated as artificial intelligence develops. For example, bots have created content to push very real political agendas on social media and digital spaces, with real-world impacts.

The good news is that the intelligence to keep up with bad bots is also becoming more sophisticated.

Bots can be detected by recognizing patterns such as:

• Unusually high page views.

• Unfamiliar referral traffic.

• Traffic coming from places and/or devices that wouldn’t normally be interacting with your site. 

• Clunky punctuation and grammar.

Many companies that don’t have a good bot detection strategy in place may not know an attack occurred until they review these patterns after the fact. It takes an average of four months for organizations to detect bot attacks, and 97% of surveyed businesses say it takes over a month before they respond. This is an unsustainable approach, especially given the growth in bot-driven attacks and their mounting cost. Companies need a more effective approach to bot detection and management.

A bot detection solution should be able to detect bot activity in real time, deal with the entire spectrum of bot-related threats, and involve several detection techniques, including:

• Device and network attribute anomalies: Check for device attributes that do not match the declared user-agent, abnormal device hardware, traffic originated from data-centers, etc.

• Signature-based detection: Some bots rely on naive automation tools, leaving behind a clear footprint.

• Usage velocities: Look for abnormally high activity volume with a repeating pattern in a short period of time (e.g. same IP, user-agent, or geographic location), since bots are usually used to scale an attack by executing high volumes quickly.

• Behavioral anomalies: Look for a lack of behavioral footprint (mouse, keyboard, etc.) or an abnormal behavioral footprint (non human-like mouse trails, etc.).

So, your bot detection solution has helped you detect bots. The next step is preventing them from causing damage.

Prevention Requirements & Considerations

1. Real-time Bot Detection

To start, when moving from "detection" to "prevention", a very important requirement is added: detection in real time. Essentially, companies need to be able to detect bot traffic as early as possible and prevent bots from causing the damage they intend to cause.

2. User Experience Considerations

Before you do this, it is important to recognize that prevention also means that your company will need to consider the user experience. Detection accuracy needs to be sophisticated enough that it won’t take away from the frictionless user experience that legitimate users expect. Only bad bots need to be stopped; if legitimate users are being confused with bots, you’ll have more problems on your hands.

The good news is that there are several high-level bot prevention techniques:

Bot Prevention Method #1: Blocking Traffic

Blocking traffic that you’re sure has originated from bots is a very effective method. However, this should only be used when there is very high certainty that you’re indeed facing a bot. Remember; if you assume everyone is a bot, you might be preventing legitimate users from accessing the network.

Generally, specific solutions geared towards bot management typically feature the ability to actually block bot traffic. However, these solutions may not be great at combating other types of fraud and could put a strain on resources. Look for solutions that can be integrated to maximize your overall fraud-block potential. A powerful detection tool like PingOne Protect coupled with identity orchestration will allow you to build out automated responses to these threats, regardless of where or when in the user journey they occur.

Bot Prevention Method #2: Add a Challenge

A second method to prevent bots is to add a challenge in the form of a CAPTCHA, which is one of the most common methods. Most users have probably come across a CAPTCHA before. CAPTCHAs—if done right—are great at creating the appropriate amount of friction for users by requiring them to easily overcome a challenge that would otherwise be very difficult for a bot. Examples are a grid popping up with different images that you have to identify.

Of course, it is true that sophisticated bots can bypass a CAPTCHA relatively easily by simulating/mocking human mouse interactions. In fact, there are free code libraries out there and tutorials on how to do it.

That being said, CAPTCHAs are a very accessible solution that works to detect some bots. And, while it’s not the most beloved method amongst users, again, most of us are used to it.

Bot Prevention Method #3: Incorporate an MFA Solution

Finally, a third bot prevention method is to incorporate an MFA solution for your organization and your customers. Forcing MFA can be used for cases where you suspect a bot is trying to log in to accounts, especially bots that utilize credential stuffing to steal account information and use it to try and gain access. MFA not only helps drastically mitigate this but also won’t cause friction for legitimate users. MFA helps you to ensure your users are who they say they are while keeping bots at bay. Consider using biometric authentication for an additional level of security – many bots will not be able to mimic a unique biometric marker, and as a bonus, this is a method of MFA that legitimate customers will find convenient and easy to use.

Though the methods and technologies discussed above work well to prevent bots, there is one more thing that can help teams better face this threat: getting into the mind of the attacker.

Bots are sophisticated, but only as sophisticated as the people behind them. So, why would an attacker want to use a bot to get through your network? Begin by thinking about the attacker’s motives.

Next, think about the technologies they would use to carry out the bot attack, which often match the motive (but not always).

Generally, bot traffic may be mostly attempted account takeovers through brute force attacks to guess a password. Alternatively, bots may create lots of new accounts to commit new account fraud at scale. Generally, by looking at session characteristics, identity professionals can determine whether or not a user is human. To do this, they can look at the specific technologies attackers might use and then the implications of those technologies.

There are many technologies that can be used to implement bots, from very basic technologies that just mimic the HTTP traffic of a legitimate client (web browser/mobile application) to more sophisticated technologies that actually control a legitimate client.

A good rule of thumb is that simpler bots/technologies are easier and cheaper to scale because they demand fewer resources. This means they may be easier to detect and prevent. Likewise, the more advanced bots are, the more niche the intended attack is meant to be, and thus, the harder they may be to detect and prevent.

So, which technology would your attackers be using, and why?

Detecting and preventing bots is important to protecting your network, but when doing so, it’s necessary to keep your users satisfied with their digital experience. Keeping bots out and users in can be a difficult task to balance, but the right bot detection and prevention solution goes a long way.