With the Firefox 4.0 release right around the corner, the big question for a lot of users is how fast is Firefox 4.0? How does the new Moz compare with Google Chrome, Opera, and the rest? If you’re curious, take some time this weekend to perform your own benchmarks and see for yourself. Consider this an audience participation article: We’re looking for your feedback as well.
Every time a new browser release comes out, you’ll see a rash of stories about browser benchmarks and how this or that browser are faster. The problem is that they usually focus on one test, and they’re usually aimed at Windows users — not Linux users.
Couple that with the fact that the benchmarks usually work with the standard builds of browsers for Windows or maybe Mac OS X, on what are largely standard systems. The hardware varies, but the system libraries and such should be the same for all users of Windows 7, or all users of Mac OS X (modulo a few variations depending on whether the benchmark systems have been updated properly).
Linux users can get Firefox from their vendor or from the official Firefox builds. They may be using 32-bit or 64-bit builds. They may have different versions of the system libraries depending on whether they’re on Fedora, Debian, Ubuntu, openSUSE, Debian, etc. In short, the mileage may vary quite a bit depending on which Firefox build you’re running and which distribution you’re running it on. Not to mention the hardware differences.
As the adage goes — if you want something done right, do it yourself. So let’s look at the tools to test browser performance in the privacy of your own home. In particular, we’re going to focus on three JavaScript benchmarks, since that’s where the real competition is these days.
One of the oldest benchmarks is the SunSpider JavaScript benchmark. This test includes a variety of categories, some math, some string processing, generating tagclouds, etc. Just head over to the SunSpider page and click “Start Sunspider 0.9.1 now” (or whatever the current version is when you check.
When finished, Sunspider will provide results something like this:
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total: 261.1ms +/- 3.3%
--------------------------------------------
Actually, it’ll give much more than this — all of the sub-tests results will be displayed as well. You can look and see not only which browser receives the best results, but which tests they do well in or poorly in. You’ll also receive a URL you can bookmark and revisit later to compare results in another browser or in the same browser. Note that lower numbers are better with SunSpider, so this result (in Chrome) is very good.
SunSpider is the oldest of the benchmarks, but there’ve been some questions about the tests that it uses and whether they’re applicable to real-world speed.
The Chrome folks use the V8 Benchmark Suite to tune V8. As soon as you load the benchmark page, it starts running. With V8, you get a set of results in a blue box on the right-hand side, like so:
Score: 3748 Richards: 6693 DeltaBlue: 4390 Crypto: 7368 RayTrace: 2886 EarleyBoyer: 3586 RegExp: 1437 Splay: 3226
The tests results are broken down, and each test has a summary on the left-hand side of the page. Save your results because V8 doesn’t provide a handy linky for returning to your results.
Finally, there’s the latest benchmark from Mozilla called Kraken. It’s based on SunSpider, with enhancements. The Kraken will take a while to download and run. As with SunSpider, lower numbers are better.
Running the Tests
For best results, bookmark each of the tests, close all other windows and apps and run each benchmark in its own window. You will likely skew the results by running the tests while doing a kernel compile while benchmarking one browser and then nothing while running the benchmark in another browser.
Run the tests several times each, as well. Don’t run them once and then assume that number is the final word.
Site Compatibility
One thing that benchmarks won’t do is tell you how well a browser matches up against the Web pages and applications that you use daily. Maybe Opera 11 blows the doors off a benchmark, but sucks tailpipe when it comes to Gmail. Maybe Chrome comes in last on a benchmark, but surprise happens to do very well at Gmail.
Nothing compares to real-world testing. If you’ve been unhappy with Firefox performance prior to 4.0, give it another shot when 4.0 comes out instead of just trusting benchmarks.
Sing Along
Remember that I said that this was going to be audience participation?
Here’s your weekend homework, if you’re willing: Close all of your other applications and run the benchmarks in the latest release of Firefox 4.0, Chrome, Opera, etc. Provide your relevant system information and the benchmark results in the comments. What system information? Glad you asked!
- Distribution and version
- Processor
- Memory
So, for example, I’d post Linux Mint 10, Intel Core i7 Q720 1.60GHz, 8GB RAM. Let us know if you’re using a 32-bit or 64-bit system. If you’re not sure what CPU you have, you can run cat /proc/cpuinfo in a terminal, or use one of the GUI system information tools to find out. If you’ve forgotten how much RAM you have, just run free -m for a hint, or refer to a GUI tool.
Sorry to give you homework on the weekend — but if even one out of twenty Linux.com readers runs the tests and provides benchmarks, we’ll have quite a lot to compare, and a much better picture of the browser landscape. Happy benchmarking!
