Author: Mayank Sharma
Despite a variety of open source testing tools, until recently there wasn’t an easy way to measure and compare the performance of two Linux-powered machines. Phoronix Test Suite (PTS), released this month, addresses this — and how! Using the suite you can gauge and compare multiple Linux-powered machines to find out if a particular setup is better than another for a particular task, such as hosting a Web server or playing games.
PTS is developed by Phoronix Media, which also owns Phoronix.com, a site popular for its Linux-based hardware reviews and analysis. The tool is available for download as a source tarball as well as a precompiled binary for Ubuntu and Debian. It has minimal dependencies and detailed installation instructions.
The tool comes with 57 tests which are logically grouped into 23 suites. So, for example, you can benchmark the performance of Ogg encoding on your system by running the encode-ogg test, or run the audio-encoding suite of tests, which benchmarks MP3, Ogg, FLAC, APE, and WavPack encoding. There are tests and test suites that can help you assess the performance of your computer for computational biology, gaming, GUI toolkits, audio and video encoding, building Apache, running PHP, compiling the Linux kernel, and more. Since the PTS is what Phoronix uses for its reviews as well, the package also contains a few Phoronix Certification and Qualification Suites (designed by Phoronix for use in their reviews) that test the graphics and motherboards on your desktops and servers.
PTS is a command-line tool, and is well documented and easy to use. Once you have it installed, you can assess your computer’s gaming performance, for instance, by running the command benchmark gaming. This will automatically fetch all the tests in the gaming suite, compile and install them, then run the test. To make sure it has the undivided attention of the processor, PTS temporarily turns off ambient services such as power management and screensaver while the tests are running.
All tests performed by PTS need a unique global identifier to individually identify each test. When a test is completed, PTS expresses the result in numbers in the command-line interface. If you prefer simple bar graphs, you can also have it display its results in a browser. In addition to displaying information about the benchmark (the type of test and what the numbers mean), PTS also records and displays details about the hardware (processor, motherboard, chipset, memory, disk space, graphics, screen resolution) and software (OS, kernel, X.Org server, OpenGL, compiler, file system) on which the test was run, along with details like the date and time the test was run, and by which user, and what additional applications were running at the time.
Perhaps the neatest thing about PTS is the online PTS Global service, which is a repository of tests performed by PTS users all over the world. You get the option to upload your test results to PTS Global after every test is completed. Test results are listed by their time of upload, and you can search the results with a basic search interface to find tests performed on a particular motherboard, processor, graphics chipset, or running a particular distro, kernel, or compiler.
But PTS Global isn’t just a collection of results. You can ask PTS to run a test on your machine and compare it with a particular benchmark on the PTS Global Web site. So, if you want to check how your desktop running Fedora compares with another similar spec (same processor, RAM, motherboard, chipset) machine running Ubuntu, while compiling the Linux kernel, you can use the search interface to find such a machine and then simply use its global identifier to run the same test on your machine and get comparative numbers. For example, if you want to compare your machine with a machine whose global identifer is root-9170-30463-10839, all you need to do is run the command phoronix-test-suite benchmark root-9170-30463-10839. That command will fetch, compile, and run the test on your computer and display the result in comparison with the one fetched from PTS Global.
PTS is a useful, easy-to-use, and well-documented application for all types of Linux users. As a reviewer it will help me back up my results with numbers. So while I find out how Fedora and Ubuntu perform in a virtualized environment, click to the next page for an interview with Michael Larabel, who owns Phoronix and is also the lead developer of PTS, in which he explains the motivation behind the project, the interest shown by hardware and software vendors, and upcoming developments.
Linux.com: Is the Phoronix Test Suite the same test tool you use internally for testing hardware on Phoronix? What was the motivation behind releasing the test suite?
Michael Larabel: With a few exceptions, yes, the public version of the Phoronix Test Suite is the same as what’s being used internally. For instance, the Phoronix Test Suite has sensor monitoring support via ACPI and LM_Sensors. This includes support for monitoring the +3.33V, +5.00V, and +12.00V rails. However, as these sensors aren’t always 100% accurate and there isn’t uniform support across all motherboards, we still prefer using digital multimeters when monitoring the voltage rails in, say, power supply reviews. There are also a few areas left that aren’t yet automated and covered by the Phoronix Test Suite, such as hooking in with Bootchart for measuring the boot performance, focusing on the qualitative side of video playback, and testing audio playback and recording. For an overwhelming majority of the articles, however, the Phoronix Test Suite is what’s being used.
One of the motivations behind the Phoronix Test Suite was to open up our testing protocols so that the community can more easily run benchmarks under Linux…. It’s disgusting to see bloggers and those on forums using glxgears and calling it a benchmark. The driving motivation, however, has been from independent hardware and software vendors. A number of them are open to having their engineering and quality assurance teams use Linux more in pre-production testing. However, because Linux isn’t the primary OS of their customers, they aren’t willing to devote many resources to it — both for the human testing and then to develop a test suite. A number of manufacturers though have agreed to use the Phoronix Test Suite if it provides them with a way for autonomous testing, easy setup and maintainability, and a collection of tests that represent common applications used by those on the Linux desktop. Phoronix Test Suite 1.0 now provides all of that plus more.
Lc: What’s involved in putting together a test profile?
ML: There is a brief tutorial included with the documentation for writing your own test profile. There is also a list of all supported XML tags. For a new test maintainer, the documentation is a good start, and then just looking at the existing test profile to see how those work.
For example, the profile from X-Plane, the multiplatform flight simulator, was the first test profile where the software vendor (Laminar Research) had worked with us to make X-Plane 9 a nice option for benchmarking. X-Plane 9.00 is a licensed pay program, though they do offer a free demo download, but even that is very large in size (somewhere around 900MB or so). What one of the developers did at Laminar Research was strip out all of the code and files (excess scenery, etc.) that aren’t needed if all you’re going to do is run benchmarks. He then created a standard timed demo file. This resulted in a version of X-Plane 9 specifically for benchmarking and is about half the demo’s size.
Not only did Laminar produce this customized version, but it went on to document the various testing options. These options were how to run an FPS test, command-line arguments for adjusting various settings, and so forth. With the customized version of X-Plane and the information the company provided us, we were able to create this profile in just a matter of minutes.
[Another profile to look at] is one we created based upon LAME for measuring MP3 encode time. All of the profiles can easily be found and examined in pts/test-profiles/* and pts/test-resources/* [from the source tarball].
Lc: How’s the feedback from hardware vendors and software developers?
ML: I’ve personally been very pleased with the feedback from hardware and software vendors surrounding the Phoronix Test Suite. Aside from Ben Supnik with Laminar Research providing X-Plane, Stefan Doesinger of CodeWeavers had provided a Wine compatibility test early on, and there are at least six other vendors we’re working with at this time to push their own tests and suites. This includes software vendors who will be bringing their Windows benchmarks to Linux via PTS and one major company that has committed to publishing its regression test harness for Linux within PTS. There are also about 10 other independent contributors that have been working on test profiles thus far. Right now in git we’re up to 58 test profiles.
Aside from vendors using the Phoronix Test Suite for hardware validation and using the software to automate their Linux tests, other companies are looking to this software for more reliable bug reports and customer testing, because it can serve as a common test environment between end users and companies. A great example of this can be found on our forums, where an end user believed he had experienced a problem with a driver and was able to clearly show a problem using PTS. The vendor could now look at this and its engineers could replicate the same software configuration and test settings. It’d be as simple as running “phoronix-test-suite benchmark ivanovic-22332-15328-27752” to see if they can reproduce the problem.
Lc: Any time frame on a searchable PTS Global interface? Apart from that, how will the suite be developed in forthcoming versions?
ML: There is a basic search interface, but a nice Web 2.0 version should arrive later this year. This includes many AJAX features for sorting the lists, paging, searching within a specific list, and more. PTS Global will also pick up some other features for further enhancing the collaboration possibilities and encouraging software projects to adopt the Phoronix Test Suite for their testing platform. I don’t have a more defined time frame than “later this year,” as the dedicated PTS server and the other Phoronix servers that PTS is piggybacking on are already being clobbered and chewing through globs of bandwidth for PTS Global, PTS downloads, and mirroring many of the tests that are downloaded using PTS. Once this infrastructure gets upgraded, I’ll be more comfortable rolling out rich Internet features in PTS Global.
The first major update to Phoronix Test Suite “Trondheim” will incorporate a module/plugin framework (currently in development) as well as support for new areas of testing. The module framework will allow the capabilities of pts-core to be expanded and opened up in new ways. Two of the planned modules include one for LCDproc for reporting the test being run and its results to an LCD panel or keyboard display. The other plugin is for setting up PTS to run in batch mode whenever there is a new commit in a git tree for the software being tested (say the kernel, Mesa, or a driver). The new tests support will focus upon areas such as qualitative video playback comparisons by capturing and saving the screen, algorithms for attempting to determine screen artifacts and other rendering oddities, and finer areas of graphics such as anti-aliasing and anisotropic filtering. Through the interaction with the community and various hardware and software vendors, we’ll also be working to better define the Phoronix Certification and Qualification Suites, as well as a set of stress tests for pushing systems to their limits.
Category:
- System Administration
