Benchmark & Stress test: WordPress brute force attack detection plugins comparison


Two benchmark tests will be performed against 5 of the most popular security plugins for WordPress whose description indicates that they provide protection against brute force attacks, and our plugin NinjaFirewall (WP edition).

The first test will use ApacheBench, the benchmarking tool part of the Apache HTTP server package. It will be a rather quick but totally brutal attack on the WordPress login page, originating from a single IP.
During the second (and much less brutal) test, a distributed attack against the login page will be simulated, from thousands of different IPs. It will be very similar to one of those large attacks that targeted WordPress sites lately.

The 5 plugins are:

The test will also include:
  • NinjaFirewall (WP edition). If you aren't familiar with it, this is how it works:
    Attacker => HTTP server => PHP => NinjaFirewall => WordPress

    And this is how the above-mentioned plugins and, more generally, all WordPress plugins work:
    Attacker => HTTP server => PHP => WordPress => Plugins

    NinjaFirewall sits between the attacker and WordPress. It can filter requests before they reach the blog. That makes it very suitable for detecting and, most important, for blocking brute-force attacks.

  • WordPress (no plugins): this is going to be a very interesting part of this article: testing WP alone, without any security plugin. Does it really need those plugins during an attack? We'll see.

Two servers will be used: the Victim and the Attacker

  • The Victim :
  • IP / Domain: /
    CPU: Intel Core i5-3230M CPU @ 2.60GHz
    RAM: 6GB
    OS: Linux Debian 7
    Kernel: 3.2.32-1 x86_64
    HTTP server: Nginx, 1.2.1
    PHP: 5.4.4-9 (FPM/FastCGI), no opcode cache
    MySQL: 5.5.28
    WordPress: 3.6.1
    Sysstat version 10.0.5 (sar command)

  • The Attacker:
  • IP:
    CPU: Intel Core i3 M 390 @ 2.67GHz
    RAM: 4GB
    OS: Linux Debian 7
    Kernel: 3.2.32-1 x86_64
    Tools: ApacheBench v2.3, NinTechNet WP brute-force attack simulation Perl script

  • Connection:
  • Ethernet crossover cable.

  • Plugins configuration:
  • Each plugin will basically use the same configuration: try to detect and block an attack as soon as possible (>= 5 login attempts).
    • All In One WP Security (v2.5)
      Default installation and click menu "WP Security > User Login":
      • Enable Login Lockdown Feature : checked
      • Max Login Attempts: 5
      • Login Retry Time Period (min): 1
      • Time Length of Lockout (min): 2
      • Notify By Email: keep it disabled

      >> See screenshot

    • Better WP Security (v3.5.6)
      Default installation and click menu "Security > Login Limits":
      • Enable Login Limits: checked
      • Max Login Attempts Per Host: 5
      • Max Login Attempts Per User: 5
      • Login Time Period (minutes):5
      • Lockout Time Period (minutes): 2
      • Email Notifications : unchecked

      >> See screenshot

    • BulletProof Security (v.49.2)
      Default installation and click menu "BPS Security > Login Security":
      • Max Login Attempts: 5
      • Automatic Lockout Time: 2

      >> See screenshot

    • Limit Login Attempts (v1.7.1)
      Default installation and click menu "Settings > Limit Login Attempts"
      • 5 allowed retries
      • 2 minutes lockout
      • 10 lockouts increase lockout time to 24
      • Handle cookie login: No

      >> See screenshot

    • NinjaFirewall (WP edition) (v1.1.1)
      Default installation and click menu "NinjaFirewall > Firewall Policies":
      • Do not scan traffic coming from localhost ( and private IP address spaces: No

      >> See screenshot #1

      Menu "NinjaFirewall > Login Protection":
      • Enable protection: yes
      • Protect the login page against: POST request attacks
      • Password-protect the login page for "2" minutes, if attack exceeds 5 requests within 20 seconds.
      • HTTP authentication: User: whatever4134, Password: fjiEJKSKWKW

      >> See screenshot #2

    • Wordfence Security (v3.8.3)
      Default installation and click menu "Wordfence > Options":
      • Enable login security: enabled
      • Enable firewall: enabled
      • Enable Live Traffic View: disabled

      >> See screenshot #1

      Scroll down to "Login Security Options":
      • Lock out after how many login failures: 5
      • Lock out after how many forgot password attempts: 5
      • Count failures over what time period: 5
      • Amount of time a user is locked out: 5 minutes

      >> See screenshot #2

      Like NinjaFirewall, Wordfence will ignore traffic coming from private IP addresses, but it does not seem to have any option to turn it off. Its /wp-content/plugins/wordfence/lib/wfLog.php file will have to be edited, and the following 11 lines of code from the isWhitelisted() function to be either deleted, or commented out:

    • WordPress (v3.6.1)
      Default installation. Only one user was created: admin (pass: StressTest)
      The 'admin' user is important for the test: some of the plugins do not rely on the IP or an access to the login page, but rather on the user. For instance, BulletProof Security will block only if the user exists and has too many failed login attempts. If the user does not exist, it will simply do nothing at all against the attack.

    The first test will record the number of requests per second and the response time, and the second one, the number of requests per second, the duration, the CPU load, memory usage, MySQL queries as well as the number of bytes returned by the Victim.

    For each plugin, the same procedure will be followed:
    • Install and configure the plugin.
    • Delete the HTTP access log.
    • Restart Nginx, MySQL and PHP-FPM.
    • Get the current total number of queries from MySQL (SHOW STATUS WHERE variable_name='Queries';).
    • Run the sar command to collect system activity information every 5 seconds (sar 5 -rq).
    • Start the attack from the Attacker server.
    • Stop the sar command after the attack.
    • Get the new total number of queries sent to the DB using the above SQL command.
    • Collect the HTTP access log.
    • Disable and uninstall the plugin.

    To ensure accuracy, each test wiil be performed 3 times.

    First test:

    ApacheBench will run from the Attacker server with the following command:

    Each request simulates an attempt to log in to the admin console. It includes all mandatory fields: WordPress wordpress_test_cookie cookie, credentials (user: admin, pass: bruteforce) and the POST payload inside a postdata.txt file:

    It will try to make 50,000 login attempts but, if the attack takes too long, it will stop after 60 seconds. 5 concurrent requests will be used. That will be quite brutal but will match the definition of a stress test: the goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks.

    >> The ApacheBench results can be downloaded at the bottom of this article.

    The first graph shows the number of requests per second (RPS) that each plugin handled during the attack. The higher the number, the better:

    At a rate of 1,520 requests per second, NinjaFirewall is the clear winner. In order to see the other results, we need to graph them separately:

    They all processed the attack* at a rate of less than 30 RPS, BulletProof Security (whose 3 benchmarks show a very high standard deviation) and Wordfence Security being the two slowest. Note how WordPress alone outperformed 4 plugins.

    *only NinjaFirewall processed the whole attack, i.e. 50,000 POST requests in 32 seconds. All others reached the 60-second limit and processed only +/-1,600 POST requests within that timeframe.

    This second graph shows the response time. This is the time, in milliseconds (ms), it took for the server to process the request and send a reply. Obviously, small numbers are better:

    With a 3ms response time, NinjaFirewall crushed its competitors (190ms on average).

    Second test:

    This will be a distributed attack attempting to hack into the WordPress login page. It will be much less intensive than the previous one, hence very realistic.

    According to the Brute Force Attacks Build WordPress Botnet article from the Krebs on Security blog, the last big attack against WordPress sites was carried out by 90,000+ IPs (servers and/or infected home computers). The blog's article even includes a copy of the username/password list that the attackers may have been using for the attack. It contains 2,927 lines, with quite a lot of duplicates.
    It will be used for the attack.

    Simulating a large botnet is very easy, thanks to Nginx, by adding those 2 lines to its /etc/nginx/nginx.conf file:

    This tells Nginx to use the IP from the X-Forwarded-For header of each HTTP request coming from IP (the Attacker). Nginx will forward it to PHP and then, to WordPress and its plugins.

    The following simple Perl script will be used to run the attack: it will read the username/password list and will try each of the 2,927 combinations. Parameters, values and cookies are the same as those used for the first test.
    Line 29, it generates a random IP for each request and adds it to the X-Forwarded-For field.

    Below is a sample of the HTTP access log showing the distributed attack with "spoofed" IPs:

    >> All HTTP logs can be downloaded at the bottom of this article.

    The first graph shows the number of requests per second (RPS) during the attack. High numbers are better:

    At a rate of 154 RPS, NinjaFirewall is again way ahead of all other plugins. Interesting enough, WordPress (without plugins) comes second with 11 RPS, more than twice as fast as Login Limit Attempts and Wordfence Security.

    The next graph shows the total time it took, in seconds, to handle the whole attack. Small numbers are better:

    It took NinjaFirewall 19 seconds to get through the whole attack, while WordPress (without plugins) required 276 seconds. Way behind, Limit Login Attempts took around 9 minutes (529 seconds) to perform the same work, and Wordfence Security was almost pronounced dead after exactly 625.13 seconds of agony.

    This graph shows the CPU load on the Victim's server (i5, 2 cores, 4-threaded processor). Low numbers are better:

    Because NinjaFirewall processed the attack in only 19 seconds, attempting to measure its impact on the server load would be irrelevant.
    Limit Login Attempts and Wordfence Security, once again, scored poorly by increasing the load much more than any other plugin.

    The following graph shows the total amount of RAM, in megabytes, used by each plugins. The lower, the better:

    This graph is interesting because it shows that Better WP Security and Wordfence Security used less RAM than WordPress without any plugins (43Mb vs 62Mb). With respectively 82Mb and 96Mb, BulletProof Security and Limit Login Attempts are far behind.
    Here too, it would be irrelevant to attempt to graph the amount of RAM used by NinjaFirewall 19-second test.

    Next graph displays the total number of queries (read/write) sent to the MySQL database during the whole attack. No query at all, or a low number, is better:

    For better performance, NinjaFirewall does not send any query to the database during an attack. WordPress ranks 2nd with 32,234 queries sent (11 queries/POST request). Things start getting worrisome for Better WP Security with 56,461 queries, and totally critical for Wordfence Security which sent no less than 118,096 queries (40 queries/POST request). Looking at the database showed that Wordfence added 6,056 rows to the wp_options table.

    The last graph shows the total amount of kilobytes returned by the Victim during the whole attack. The lower, the better:

    NinjaFirewall returned a total of 142 Kb (44 bytes for each blocked request) while all other plugins returned around 9 MB (3 Kb/request), the same amount of data returned by WordPress without any plugin.


    NinjaFirewall (WP edition) was able to block both attacks because it sits between the attacker and the victim. A security plugin that sits behind the application that it is supposed to protect is of no use.

    On average, WordPress performs better alone than with the 5 security plugins that were tested. Most of them do not bring any improvement, the others may even increase server resources to a critical level during a large attack.

    General ranking, based on the ranking of the 8 graphs.


    • do not use 'admin'.
    • do not use a word from the dictionary (whatever, blog, computer...).
    • do not use a proper noun/name (michael, charlie, paris...).
    • do not use a password that could be easily associated with you.
    • use at least 10 alphanumeric characters, include non-alphabetic ones (dot, hash, comma etc...) and mix UPPER/lower cases.
    • change it often.

    Benchmark logs (raw):

  • ApacheBench results (Test #1): nintechnet_ab.txt (13Kb)

  • Nginx HTTP access log (Test #2): nintechnet_nginx.tar.gz (200Kb)

  • Related links:

     NinjaFirewall (WP+ edition): Brute-force attack protection in a production environment.

     New WordPress security plugins vs IP spoofing.

    Rev.: 1.00 2013-09-26 : 1st publication
    Rev.: 1.01 2013-10-12 : correction: the total number of bytes returned is 3Kb (instead of 30Kb)
    Rev.: 1.02 2014-09-12 : added related links

    The Ninja Technologies Network