Interview with organizers of the BSD certification exam

The BSD Certification Group, (BSDCG) held its first in-person BSDA certification exam session for systems administrators during SCALE last month in Los Angeles. Subsequent tests were then held held during FOSDEM in Brussels, Belgium, and Linux-Tage Chemnitzer in Chemnitz, Germany. During the events, we were able to catch up with several people involved in the testing. Here’s what they had to say about the exam development process, the events themselves, and reasons for becoming certified.

Acting Vice President and Treasurer Jim Brown, and Executive Officer Dru Lavigne.

Linux.com: How was the first exam?

Dru Lavigne: Leading up to the exam was a very hectic time, with a million and one details to attend to. The Angoff session, a standard-setting study method [designed] to evaluate the average ratings of potential examinees, allowed technical experts a final review of the exam questions and gave the psychometrician the metrics she needed to determine which questions belonged on the exam and what the passing score would be. Then there was a registration system to set up and the processes for registration to be discussed and implemented. We had very little lead time between confirmation of space and advertising the exam launch. Many people were interested in taking the exam, but wanted to wait in order to have a chance to properly prepare for the exam.

LC: How did you choose the criteria for certification and the exam questions?

DL: This was a long process which we did our best to record as we went along. We started with a Job Task Analysis (JTA) — a survey of managers, sysadmins, and those involved in the hiring process — to find out which skills are important and to what depth for various experience levels. From there, the audience and exam objectives [were chosen]. The psychometrician assisted in ensuring the objectives were in line with the JTA, then provided instructions to the…Subject Matter Experts (SMEs) for writing good exam questions.

Then there were many sessions to improve the questions, followed by a beta period where the community had the opportunity to comment (under NDA) on the appropriateness and readability of the questions. This was followed by more sessions with the SMEs and the psychometrician to address the comments as well as anomalies that became apparent as part of the statistical analysis. Finally, there was a grueling Angoff session where the SMEs ranked every potential exam question, which gave the psychometrician further statistical data to work with.

Jim Brown: The Job Task Analysis and several other documents were created as surveys. One of the first issues we faced was how to create a survey that we could easily modify, and that would be able to work with multiple languages. We wanted to use open source tools, and finally found the PHP Easy Survey Package. This was a big help.

LC: What was the process behind the development of the psychometric assessment of the exam questions?

DL: Personally, I found the beta analysis and Angoff session the most interesting, and was impressed that both were able to pinpoint problem areas in the questions. Statistical anomalies often resulted from badly worded or confusing questions, questions with multiple correct answers depending on how you looked at the question, typos, and the occasional error where there wasn’t a correct answer. They also revealed when a question was too hard or too easy. It was also obvious which questions people were just guessing at as they hadn’t studied the exam objective associated with the question.

LC: Do you have any statistics about the people who took the test at SCALE?

DL: Due to the short lead time, there was much interest but only one person brave enough to take the exam. It will be interesting to look at the stats a year from now. During the Angoff session, the SMEs agreed that the target audience would probably be male in the 25-30 year age bracket. A year from now, we’ll know if that is the case.

JB: We’ll know more about these kinds of stats when we look at the [exams recently offered at] FOSDEM and Linux-Tage Chemnitzer. I’m particularly interested to see the international mix. BSD systems are used all around the world, and I think our exam candidates will reflect that.

Michael Dexter, open source project manager who took the exam at SCALE.

LC: Why did you choose to take the exam?

Michael Dexter: I took the BSDA exam for a combination of reasons, both personal and political. I had wanted to take a beta exam but stopped following the project. A combination of circumstances put me in the right place at the right time, and I proudly took the first official exam without any idea of what I was in for. On one level I was happy to see how my skills would “officially” rank and on another I was happy to play the role of “the first guy to take the test” in the community and provide feedback. Only by taking the exam can I have an opinion of the exam.

LC: Now that it’s over, what is your opinion of the exam?

MD: For better or for worse, the BSDA exam makes a serious effort to be a standardized test just like any other — high-school flashbacks included. I never faired well with such exams and will not speculate as to how I faired with this one. To its credit, it touches on a broad range of general system administration skills such as networking and regular expressions, but for some minutia I recommend that an administrator simply RTFM as needed. I am assured that complex internal metrics and weighting will help distinguish the key questions from the trivia. After all, the goal is to test system administration skills and experience, not memorization skills.

Among my suggestions to Dru, one was to break the test into a number of general and specific categories such as “networking” and “OpenBSD.” Having gone in without any preparation, I was concerned by questions starting with “On a Dragonly BSD system…” as I have never used one and do not anticipate a pressing need to use one in the near future. I will not speculate as to if there were “trick” questions.

I can’t recommend an employer base a hiring decision solely on the certification of an applicant, but I’m sure the BSDA exam has its place in the industry and is of value in conjunction with other evaluation techniques.

Along with Machtelt “Tille” Garrels, Axel S. Gruner managed the certification process at FOSDEM and Linux-Tage Chemnitzer.

LC: Could you give us some stats about the people who took the test you oversaw?

Axel S. Gruner: [The age range was] 18 up to 45. FOSDEM is an international event, so there were candidates from Belgium, Germany, and the Netherlands. We had 13 candidates that took the exam. The fastest did the exam in about 30 minutes [and] I think we had no one who did not finish the exam in the 90-minute time frame.

We can not say there is a typical exam candidate like, “only sysadmins or IT nerds.” One of the candidates did not have a job and… wanted to enhance his CV with the BSDA certification. Some of the candidates did the exam just “for fun.” They were interested in doing the exam, wondering what kind of exam it would be, and what was their level of knowledge about the BSDs. Most of the candidates are working as a sysadmin for UNIX-Systems, or are students who would like to have a certification.

LC: Any unexpected problems or lessons learned?

ASG: No really unexpected problems. Some of the candidates did not pay the exam fee on the registration site so we decided that these candidates could take the exam if they paid the exam fee with cash before starting the exam. One big problem is taking an exam in the morning at 10:00. That’s a bit too early, so next year we should start not before 12:00.

All candidates at FOSDEM and Chemnitzer LinuxTage, asked the same question: Where can we find some study material? I think it would be a great thing if we had an official study book for the BSDA exam.

At Chemnitzer LinuxTage, …I had the promise of five candidates which wanted to take the exam… [but] just one candidate took [it] — a student from Berlin. The good thing is, next year there is room for enhancement. So, I will be there next year and now I have one more year to push the BSDA. The organizer of Chemnitzer LinuxTage would like to see us also next year.

Visit BSCG’s Web site for additional information about BSDA certification requirements and upcoming testing events.