Real Time Bot Detection: Bot Management Solutions
Bot detection is broken.
The biggest problem hiding in plain sight is bot attacks. It gets almost no consideration outside of the cybersecurity world, yet in a survey, nearly 50% of Web traffic was from bots. This leads to increased account takeovers (155% YoY). Malicious bots are now costing advertisers $100 billion in lost conversion.
Traditional bot management solutions using IP blocklisting, static rules engines, or the dreaded Captchas have not stemmed the rise of bots. If anything, they have served only to frustrate users and damage advertising and e-commerce conversion performance. To maintain the integrity, security, and performance of online platforms, it is crucial to identify and block malicious bots.
What is a Malicious Bot?
Malicious bots or scripts are simple software designed to attack websites, services or APIs available on the public internet. They can infect systems with vulnerabilities, steal data and identities, or commit fraud directly.
The surge of malicious bot traffic is emerging as a critical concern for businesses and individuals navigating the online landscape. These insidious bots are programmed to execute a host of detrimental tasks.
“The challenge with malicious bots is not just their sophistication but the sheer volume of automated traffic they generate. At Sardine, we focus on detecting the subtle differences between human and bot behavior, making real-time detection more effective.” — Jeffrie Joshua, Software Engineering Lead at Sardine
Common Bot Attack Terminologies
What is Credential Stuffing?
Credential stuffing is an attack where fraudsters test a list of compromised usernames and passwords through large-scale automated login requests. These credentials are usually purchased from breaches of other websites.
When an attacker uses hacked/stolen user ID/passwords to verify identity at Site A, they can eventually use this information to do an Account Takeover (ATO) at Site B. Because many people reuse the same login across multiple sites, this bot attack can often lead to ATOs at multiple sites.
What are Carding Attacks?
Carding attacks, also known as credit card stuffing, are security threats where attackers acquire stolen credit card numbers and leverage bots to verify them. These attacks often involve the use of credit card data shared on carding forums. The attackers may or may not have the CVV and may use carding to guess the CVV. They frequently seek to obtain and validate credit card details for criminal activity.
Credit card fraud is a significant concern in these scenarios, as attackers use various techniques to bypass security measures. Businesses must be vigilant to avoid chargebacks and maintain good standing with credit card processors. Attackers often use stolen credit card credentials in automated operations to authorize transactions.
Stolen credit card data is commonly traded on the dark web, where individuals share techniques for illicit use. The use of stolen credit cards for fraudulent activities can severely impact businesses, leading to financial losses and reputational damage.
What is Scalping or Ticketing Scams?
Bots purchase high-demand items like tickets or last-minute bidding at merchants like eBay. Unlike real customers, this leads to dissatisfaction among good customers, and these bots may not buy any auxiliary shopping items.
What is SMS Pumping Bot Attacks?
In this case, the attacker works with an MNO to send traffic. MNO may knowingly or unknowingly work with the fraudster. Hence, the fraudsters will create many fake accounts at merchants in the rideshare industry to generate SMS OTP codes. The best way to stop it is to blocklist by “MCC+MNC” for the phone carrier. You may use a professional bot vendor to minimize attacks from big carriers or suspected carriers. A variation of this attack is SMS toll fraud, where the fraudster sends SMS to premium numbers.
What are Content Scraping attacks?
A rival may scrape a website for competitive pricing, or they may extract data from banks or social media to build their product. Notably, the social media platform X has blocked scrapers as it began to develop its own proprietary generative AI algorithm for fear that competitors would use that data. Increasingly, news outlets are concerned about similar challenges.
What are DoS and DDoS Attacks?
Denial-of-Service (DoS) attacks flood systems, servers, or networks with traffic to exhaust resources and bandwidth, while Distributed Denial-of-Service (DDoS) attacks use multiple sources to conduct the attack. Usually, a DDOS distracts attention from actual fraud at the site.
What is Advertising fraud or Click fraud?
The fraudster may use bots to serve ads or have bots click on ads they've created to generate revenue in the “pay-per-click” model. Bots create the illusion of good ad performance on the sites displaying the ads.
What is Gaming Automation?
Many sites will automatically play games like poker for you. They often outperform humans and can extract additional winnings from services.
What are Fake profile attacks?
Many scammers may create lots of fake onboarding profiles for actual monetization at a later stage. If a bot can create 10,000 realistic-looking profiles, they can use those to Phish users, commit scams, or submit fake reviews. The fake profiles may also be created for Promotion Abuse, like getting $10 on signup. It is crucial to implement methods to distinguish between human users and bots to prevent such malicious activities.
What are Phishing Bot Attacks?
Bots send deceptive communications, such as emails or messages, to trick individuals into revealing sensitive information or performing actions that may compromise their security. This is a predicate to fraud and often targets users who use major retailers or social networks.
Bot Attack Anatomy
Bots need three core ingredients—a device, an IP address, and a script that runs the code on the device. They may optionally have a command center to control which scripts to run on the device.
Device: Bots can use almost any device. One of the most famous botnets is “Mirai”. It is believed to have taken down many websites on Black Fridays etc, since 2016. A botnet like Mirai will typically compromise routers or webcams and, hence, can get a trusted IP address to attack websites. When the malware infects an IoT device, the bot sends out signals to inform the command server that it now exists. This connection session is kept open till the command server is ready to command the bot. The botnets may use distributed command servers. They might also use dedicated devices running in a bot farm, cloud, or data center virtualized server.
IP: Bots use various tactics to conceal their true IP address or location. Many use proxies or VPNs, or when taking over residential devices, they use the IP of the device itself (since it often appears harmless). Think of this as the bot's nervous system and how it communicates without appearing malicious.
Script: Bots attack websites using “scripts” like Curl, Puppeteer, Phantom JS, and HtmlUnit. Alternatively, they emulate devices using browser tools like Selenium or in native apps they use tools like BlueStacks or GenyMotion. In order to defeat fingerprinting, they may use browsers like FraudFox, MultiLogin etc or a stealth plugin like Puppeteer stealth. This allows the bot's logic to run and is its “brain.”
Effective Bot Detection Solution Requirements
Below is an advertisement by one of the Web Scrapers. They are showing how to bypass bot protection in a popular CDN.
To detect bots, we need:
- Detect non-human behavior. Automation libraries and sophisticated bots can move faster, move instantly, and move consistently. Detecting movements and device usage outside a human profile and/or being consistent with known bot activity is a crucial technique.
- User behavior analysis. Analyzing user behavior patterns such as mouse movements, keystrokes, and page navigation can help detect anomalies that may indicate bot activity. Behavioral analysis technology is essential for distinguishing between human users and automated bots.
- Detect Residential Proxies. While legitimate users can use residential proxies for privacy reasons, bots commonly use them to conceal their location or appear as legitimate users of online services.
When dealing with advanced bot attacks, it’s essential to think beyond just surface-level defenses like CAPTCHAs or rate limiting. Implementing tools that can detect subtle anomalies in user behavior—such as inconsistent mouse movements or unusual navigation patterns—will dramatically improve your defenses against more sophisticated bots,” explains Jeffrie Joshua, Software Engineering Lead at Sardine.
CDNs vs. Bot Detection Solutions
Bot prevention can be done by your CDN/Cloud vendor and/or by Bot detection solutions like Sardine.
In one of my past jobs, bots took over a network of smart IOT devices like cameras and made a massive attack via them on Black Friday. The CDN could not stop the attack at that point, as there was always an irregular IP traffic pattern on Black Friday and the fraudsters were coming from reputable IP addresses.
In the above attack case, a security SDK that was looking for emulators was able to stop it.
CDN options have limitations that bot detection solutions can solve.
CDN/Cloud Vendors Bot Detection Vendors Detect simple bots via IP-based rate limits Detect advanced bots using residential proxies Simple JS bot detection Advanced security JS to detect abnormal behavior Very fast, can handle DDoS May rely on CDN/WAF to handle DDoS or rate limiting
CDN/Cloud vendors are paid for the traffic they must manage. There is an inherent conflict of interest where more bot traffic implies more profit for them. Their basic solutions are nonetheless worthwhile.
However, using a CDN in isolation creates a single level of defense against bots instead of multiple levels of defense.
For large merchants, marketplaces, and networks that demand best-in-class bot detection, the power of a security specialist is to step up bot detection in high-risk events and journeys.
"One of the most effective strategies to combat bots is leveraging adaptive machine learning models that can evolve with emerging threats. You can’t rely on static defenses when bots are constantly adapting their methods. Continuous monitoring and updating your detection algorithms are critical for maintaining a strong defense," says Jeffrie Joshua, Software Engineering Lead at Sardine.
Building Your Bot Detection
To build your own Bot detection, you may prioritize
- IP reputation
- IP-based rate limits
- CAPTCHA as a fallback
- Honeypot - Add hidden fields and check if they are getting populated.
Supplement it with a professional bot detection vendor who may have proxy/vpn detection, emulator detection, fingerprinting, behavior biometrics, script detection etc.
Sardine Approach to Fraud Detection and Prevention
Sardine is unique in the world of commerce in that its founders have experience across both ad tech and payments. These two worlds were historically distinct, yet both suffer from "bot" attacks and botnets.
While bots are often considered a cybersecurity issue, they're also a fraud issue. Different teams look at different problems in large organizations, and this split causes challenges
Sardine uses a Single-device intelligence combined with behavior biometrics tuned for effective bot detection, leading to better human vs bot behavior detection. Sardine has proprietary technology to detect Scripts/Emulators and detect residential Proxy/VPN in real-time. The usage of behavior biometrics gives Sardine the edge over other competitors.
Bot Prevention FAQ
How do you distinguish between good bots and bad bots?
Companies can check the UA and IP reputation to determine if it is a good vs bad bot. This requires understanding the reputation of the IP in low latency (sub 500ms).
Can I use Web Application Firewall (WF) to manage bots?
WAF may be used for IP-based rate limiting. Hence, WAF can be a good way to filter simple bots. But it should be properly tuned, or many good customers may be declined. Ideally, a waterfall approach with WF or a CDN followed by a security vendor provides the best quality traffic and the highest ROI.
How do you know if you have a bot problem?
If you see many onboarding profiles and most customers are not making any purchases, you may have a bot problem. In one of my past companies, many ads were being clicked, but there were few onboarding events, leading us to realize that they were primarily fake clicks.
What are the next steps after you detect a bot?
You may reject the customer or serve a captcha for new customer flow. For existing customer flow, you may throw a 2FA. If you are throwing a 2FA, limit it, as the bots' end goal may be to spam customers with 2FA so that they may ignore other important 2FA. In some limited cases, allow bots to continue and reject them down the line.
How do I detect that my devices like Cameras, Fire TV sticks, Android boxes etc, are impacted by botnets?
An unusual traffic spike, as seen in your bill, may help detect it. The easiest way may be to log all outbound calls by configuring your wifi network with OpenDNS. Additionally, you may add vulnerable IOT devices, like cameras, etc., to a different network/SSID. Some cable operators like Comcast provide their own routers and may have a parental control feature allowing specific device monitoring.
