Cloudflare : Why Do I Block Non-Browser Requests?

If you are managing web applications and services, you will learn that not all HTTP requests are created equal. Today, I want to share why I've implemented rules to block non-browser requests—specifically those from tools like curl, wget, and automated scripts—and the significant security benefits this approach provides.

The Problem with Command-Line HTTP Tools

While curl and wget are invaluable tools for legitimate administrative tasks, they're also the weapons of choice for attackers probing web applications. Here's what I've observed in my environment:

1. Reconnaissance Activities: Attackers use these tools to systematically map out application endpoints, looking for exposed APIs, admin panels, or forgotten development endpoints.
2. Automated Attacks: Script kiddies and automated attack tools often identify themselves with default user agents like "curl/7.x" or "wget/1.x". They're not even trying to hide.
3. Data Exfiltration: Once an attacker gains a foothold, curl and wget make it trivial to download entire databases or sensitive files without the overhead of a full browser.
4. Bypassing Client-Side Controls: Browser-based security controls like JavaScript validation, CAPTCHA challenges, or rate limiting implemented client-side become meaningless when requests come directly from command-line tools.

Why This Matters for Security

I've implemented these blocks because they provide several layers of defense:

1. Reducing Attack Surface

By blocking non-browser requests, I immediately eliminate a significant portion of automated attacks. Most vulnerability scanners, exploitation frameworks, and automated bots use these tools by default. It's low-hanging fruit that dramatically reduces noise in my security logs.

2. Forcing Human Interaction

Legitimate users access web applications through browsers. By requiring browser-like behavior, I ensure there's likely a human behind the request, not an automated script running thousands of queries per second.

3. Enabling Better Monitoring

When I force attackers to use actual browsers or sophisticated tools that mimic browser behavior, they leave more traces. Browser automation tools like Selenium or Puppeteer are heavier, slower, and create more distinctive patterns in logs.

4. Protecting Against API Abuse

Many organizations accidentally expose APIs or endpoints that should be internal-only. Blocking command-line tools adds a barrier against casual discovery and exploitation of these endpoints.

The WAF Rule in Practice

If you are using Cloudflare then you can add a WAF rule to protect against this with a rule like this:

Right, so lets create that rule with the Create Rule button:

Then we need a custom rule:

Then in that rule we need the following items, you can get the express builder code from the section below, when done change it to "Block" and "First" as the order then Deploy:

(http.user_agent contains "curl" or 
 http.user_agent contains "wget" or 
 http.user_agent contains "python-requests" or 
 http.user_agent contains "Go-http-client" or 
 http.user_agent eq "" or 
 not any(http.request.headers["accept"][*] contains "text/html"))

This rule is comprehensive because it:

• Blocks common command-line tools (curl, wget)
• Blocks popular programming language HTTP libraries (python-requests, Go's default client)
• Blocks requests with empty user agents (a common oversight in custom scripts)
• Blocks requests that don't accept HTML (indicating they're likely API calls, not browser requests)

Addressing Legitimate Use Cases

"But what about legitimate automation?" I hear you ask. Here's how I handle it:

1. API Tokens: For legitimate automated access, I require proper API authentication with tokens that can be revoked and rate-limited.
2. Whitelisting: Critical internal systems that need automated access get whitelisted by IP or through additional authentication headers.
3. Proper User Agents: I recommend internal developers to set descriptive user agents for their tools (e.g., "Bear-Monitor/1.3") which can be specifically allowed.

The Results

Since implementing these blocks, I've seen:

• 90% reduction in automated scanning attempts reaching my applications
• Cleaner logs that make it easier to spot genuine attack attempts
• Reduced server load from blocking high-volume automated tools
• Better compliance with data protection requirements by preventing easy bulk data downloads

Conclusion

Blocking non-browser requests isn't about security through obscurity—it's about raising the bar for attackers and reducing noise. Every additional hurdle we place forces attackers to reveal more about their methods and intentions.

Remember: security is about layers. This is one layer that's easy to implement and immediately effective. Combined with proper authentication, rate limiting, and monitoring, it becomes part of a robust defense strategy that's worked well in my environment.