|Benchmark & Stress test: WordPress brute force attack detection plugins comparison|
Two benchmark tests will be performed against 5 of the most popular security plugins for WordPress whose description indicates that they provide protection against brute force attacks, and our plugin NinjaFirewall (WP edition).
The test will also include:
Two servers will be used: the Victim and the Attacker
CPU: Intel Core i5-3230M CPU @ 2.60GHz
OS: Linux Debian 7
Kernel: 3.2.32-1 x86_64
HTTP server: Nginx, 1.2.1
PHP: 5.4.4-9 (FPM/FastCGI), no opcode cache
Sysstat version 10.0.5 (
CPU: Intel Core i3 M 390 @ 2.67GHz
OS: Linux Debian 7
Kernel: 3.2.32-1 x86_64
Tools: ApacheBench v2.3, NinTechNet WP brute-force attack simulation Perl script
The first test will record the number of requests per second and the response time, and the second one, the number of requests per second, the duration, the CPU load, memory usage, MySQL queries as well as the number of bytes returned by the Victim.
For each plugin, the same procedure will be followed:
To ensure accuracy, each test wiil be performed 3 times.
First test:ApacheBench will run from the Attacker server with the following command:
Each request simulates an attempt to log in to the admin console. It includes all mandatory fields: WordPress
It will try to make 50,000 login attempts but, if the attack takes too long, it will stop after 60 seconds. 5 concurrent requests will be used. That will be quite brutal but will match the definition of a stress test: the goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks.
>> The ApacheBench results can be downloaded at the bottom of this article.
The first graph shows the number of requests per second (RPS) that each plugin handled during the attack. The higher the number, the better:
At a rate of 1,520 requests per second, NinjaFirewall is the clear winner. In order to see the other results, we need to graph them separately:
They all processed the attack* at a rate of less than 30 RPS, BulletProof Security (whose 3 benchmarks show a very high standard deviation) and Wordfence Security being the two slowest. Note how WordPress alone outperformed 4 plugins.
*only NinjaFirewall processed the whole attack, i.e. 50,000 POST requests in 32 seconds. All others reached the 60-second limit and processed only +/-1,600 POST requests within that timeframe.
This second graph shows the response time. This is the time, in milliseconds (ms), it took for the server to process the request and send a reply. Obviously, small numbers are better:
With a 3ms response time, NinjaFirewall crushed its competitors (190ms on average).
Second test:This will be a distributed attack attempting to hack into the WordPress login page. It will be much less intensive than the previous one, hence very realistic.
According to the Brute Force Attacks Build WordPress Botnet article from the Krebs on Security blog, the last big attack against WordPress sites was carried out by 90,000+ IPs (servers and/or infected home computers). The blog's article even includes a copy of the username/password list that the attackers may have been using for the attack. It contains 2,927 lines, with quite a lot of duplicates.
It will be used for the attack.
Simulating a large botnet is very easy, thanks to Nginx, by adding those 2 lines to its
This tells Nginx to use the IP from the
The following simple Perl script will be used to run the attack: it will read the username/password list and will try each of the 2,927 combinations. Parameters, values and cookies are the same as those used for the first test.
Line 29, it generates a random IP for each request and adds it to the
Below is a sample of the HTTP access log showing the distributed attack with "spoofed" IPs:
>> All HTTP logs can be downloaded at the bottom of this article.
The first graph shows the number of requests per second (RPS) during the attack. High numbers are better:
At a rate of 154 RPS, NinjaFirewall is again way ahead of all other plugins. Interesting enough, WordPress (without plugins) comes second with 11 RPS, more than twice as fast as Login Limit Attempts and Wordfence Security.
The next graph shows the total time it took, in seconds, to handle the whole attack. Small numbers are better:
It took NinjaFirewall 19 seconds to get through the whole attack, while WordPress (without plugins) required 276 seconds. Way behind, Limit Login Attempts took around 9 minutes (529 seconds) to perform the same work, and Wordfence Security was almost pronounced dead after exactly 625.13 seconds of agony.
This graph shows the CPU load on the Victim's server (i5, 2 cores, 4-threaded processor). Low numbers are better:
Because NinjaFirewall processed the attack in only 19 seconds, attempting to measure its impact on the server load would be irrelevant.
Limit Login Attempts and Wordfence Security, once again, scored poorly by increasing the load much more than any other plugin.
The following graph shows the total amount of RAM, in megabytes, used by each plugins. The lower, the better:
This graph is interesting because it shows that Better WP Security and Wordfence Security used less RAM than WordPress without any plugins (43Mb vs 62Mb). With respectively 82Mb and 96Mb, BulletProof Security and Limit Login Attempts are far behind.
Here too, it would be irrelevant to attempt to graph the amount of RAM used by NinjaFirewall 19-second test.
Next graph displays the total number of queries (read/write) sent to the MySQL database during the whole attack. No query at all, or a low number, is better:
For better performance, NinjaFirewall does not send any query to the database during an attack. WordPress ranks 2nd with 32,234 queries sent (11 queries/POST request). Things start getting worrisome for Better WP Security with 56,461 queries, and totally critical for Wordfence Security which sent no less than 118,096 queries (40 queries/POST request). Looking at the database showed that Wordfence added 6,056 rows to the
The last graph shows the total amount of kilobytes returned by the Victim during the whole attack. The lower, the better:
NinjaFirewall returned a total of 142 Kb (44 bytes for each blocked request) while all other plugins returned around 9 MB (3 Kb/request), the same amount of data returned by WordPress without any plugin.
NinjaFirewall (WP edition) was able to block both attacks because it sits between the attacker and the victim. A security plugin that sits behind the application that it is supposed to protect is of no use.
On average, WordPress performs better alone than with the 5 security plugins that were tested. Most of them do not bring any improvement, the others may even increase server resources to a critical level during a large attack.
General ranking, based on the ranking of the 8 graphs.
Benchmark logs (raw):
Related links:NinjaFirewall (WP+ edition): Brute-force attack protection in a production environment.
Rev.: 1.00 2013-09-26 : 1st publication
Rev.: 1.01 2013-10-12 : correction: the total number of bytes returned is 3Kb (instead of 30Kb)
Rev.: 1.02 2014-09-12 : added related links