My Thoughts on Certification (and some related topics)

Recently, there has been increased talk in the Agile community about certifications, pro and con, though it appears mostly the latter. It has also been noted that the IEEE Computer Society will work on an exam for state licensing of software engineers. Since I have had some experience/contact with certifications (e.g., CSQE, CSM/CSP, PMP, CSDP, my wife's PA-C), I have some idea as to how they work and feel that any certification effort should be able to explain how it addresses some common certification characteristics.

What It Means to “Certify”

First, though, it seems to me to be important to define what it means to certify something/body as there are implications just in this definition. Common definitions for “certify” related to professional credentialing include:

1. a declaration by some individual, group, organization (i.e., the certifier) that
2. some other individual, group, organizations possesses/has demonstrated some
3. quality, characteristic, knowledge, ability, skill, or combination of these.

Thus, it requires a certifier to grant the certification to an applicant. Of course, the value of any certification depends on the credibility of the certifier e.g., a professional association of some sort. For this reason, there are even certifiers who certify other certifiers. For example, auditing firms (“registrars”) are themselves audited by certification bodies. These latter bodies attest to the fact that a registrar conducts audits according to some standard, usually created by a Standards Development Organization such as ISO.

So, for any certification of any kind to have an meaning/value whatsoever there must be trust and confidence in the certifier. What has to be trusted is that the certifier has actually been able to confirm that the individual, group, or organization being certified has met the criteria established for the certification. There are a few ways in which this trust can be established:

1. the certification criteria are public and easy to understand (at least by those familiar with the scope of the certification);
2. the certifier states and can demonstrate how they verify that the criteria have been met;
3. those certified are recognized as actually possessing the quality, characteristic, knowledge, ability, skill, or combination covered by the certification.

In the latter case, this means that significant question/evidence is not raised after the fact that those certified do not, in fact, meet the criteria, e.g., people certified to some skill/craft capability actually cannot apply that skill/craft as defined by the certification scope and criteria.

Scope

As mentioned above, one important matter is the scope covered by the certification.

The scope really defines what it is that is being certified. This is often defined by what is known as a “body of knowledge” which represents the domain covered by the certification. Some certifications have “guides” or outlines to perform such scope definition since actual bodies of knowledge are usually considered to be the entire corpus of published material available for the domain (e.g., the Project Management Institute’s PMBoK or the IEEE Computer Society’s SWEBoK for software engineering).

For certain very narrow certifications, a body of knowledge might be contained within some single, perhaps large, published source. But, one way or the other, a scope and body of knowledge need to be defined so they can be publically assessed for others to determine just what the scope of a certification would mean.

Another aspect of scope is whether the certification is largely based on demonstration of knowledge alone or application of that knowledge. This is where a number of people complain about some (levels of) certification that currently exists in the software field: it can be achieved by mostly passing a test (plus perhaps some minimal years of employment). Any professional certification (from medicine through hairdressing) usually involves some actual demonstration, before already certified professionals, that a person can apply the knowledge that may have been demonstrated through a test (or series of them).

The controversy will be because folks will disagree on the boundaries of that scope, especially in a skills-based certification. For example, the SWEBoK and ASQ’s Certified Software Quality Engineer BoK both note that there are important areas relative to being an effective employee and/or professional that are not covered by their BoK. They even admit there are technical and domain knowledge their exams don't cover that can and/or will matter in given work situations. So you cannot cover everything that could conceivably be important in working in a given situation.

Finally, it would also be necessary to indicate whether there are levels of certification, e.g., entry, experienced, expert or some such scale. Now each level could also be established as an independent certification on its own, of course. But it will need to be part of the definition of scope. Having different levels may help with the difference between demonstration of knowledge vs of application of that knowledge.

Nonetheless, it is important to be clear as to what scope any certification would cover.

Criteria

While not without controversy and not automatically simple to do, it can be much easier to define the scope than to actually certify one against it. Therefore, a second important matter is defining the criteria to be met to become certified. Different certifications have different criteria, but most professional certifications contain some form of the following types of criteria:

1. Evidence of some (a) training such as industry courses or formal school classes, (b) educational degree based on some curriculum of classes/ having been met, and/or (c) experience gained, often under supervision of already certified individuals.
2. Examination/testing independent of that associated with educational achievement since people in most fields can get the educational credentialing from many sources. Developing (and maintaining) these takes substantial time and effort.
3. Possible observational input from existing, certified holders of the certification based on the applicant. Not all certifications do this; however, most "professional" ones do.

The extent of any or all of these criteria may depend on any level of certification as well.

Now 2 is where must controversy exists in current software-based certifications. Most tests turn out to be multiple-choice efforts and focus on factual knowledge, not application of that knowledge. I do not see how the latter can be avoided to have any test be of real value. Of course, multiple test types could be used for multiple certification levels, e.g., a "trainee's" being largely or completely knowledge while more experienced levels having more applied expectations.

The key here is what out of all the Body of Knowledge should be covered in any test and how answers can be made "objective." Of course, even “objective” tests are “subjective” in that they represent someone’s opinion about what someone else should know. Can this even be avoided? And in judgment-based professions, can answers even be “objective” except in very narrow, technical ways?

The observational component of 3 could be like residency in medicine. Some one gets through medical school (which requires no small practical demonstration of ability) but are now to be observed by those who are already MDs. That is, for some period of time, applicants must be under the observation of or otherwise associated with someone who is already fully certified to practice the profession independently. Engineering has a somewhat different model as not everyone graduating from an engineering school ends up being licensed as a Professional Engineer. But, in one sense this is not dramatically different than their being other medical professionals certified to be other than a full MD, e.g., various nursing licenses, Physician Assistants.

Validating Scope & Criteria

Having determined what the scope and criteria will be, the next step is to validate these. This requires that there be openness in the definition of and rational for the scope and criteria and review by the professional community. The latter would be people who are currently believed to represent those who would deserve to be certified, i.e., expected to be able to meet all the criteria. (For an update to an existing certification, this would be people already certified under the prior criteria.)

The job of these people is to ensure the criteria are truly relevant to the scope the certification claims to cover such that most people who pass deserve to do so and few people who deserve to pass end up failing to do so. This can be done through peer review of the certification materials and knowledge base as well as these peer reviewers taking sample exams and assessing whether the results suggest the tests effectively cover the scope.

Verifying Criteria are Met

After this is done, the next step is to actually carry out the certification effort and verify whether the criteria are met by individual applicants or not. Concerns here are:

1. How is training, education and/or experience to be verified? Copies of certificates of class attendance? School Transcripts? Letters from institutions?
2. What pass/fail rate will exams have? Must 100% of all answers, results, etc. be "right" or is some, less than, 100% rate okay? And what is the correct <100%>
3. What kind of “objectivity” can be brought to bear on observational input? Multiple inputs rather than just one observer’s? Indeed, can this be (and do we want) purely “objective” input for this criterion? Is any input beyond the body of knowledge relevant for observational input?

For example, PMI actually audits some portion of applications on point 1.

Ongoing Certification Responsibility

Once an individual is certified, the question remains how long that certification is considered valid. For some certifications, there is yearly recertification required. Others require it every 2, 3, 5, or 6 years. This can take the form of accumulating continuing educational credits, demonstrating continuous practice in the profession, as well as, at one of the longer periods of time, taking another examination. All this needs to be defined as part of the (re)certification criteria so applicants, and others, will know what ongoing expectations exist.

------------------------------------------------------------------------------

There is a lot to consider and all of it may not be possible or make sense at this point. But I think these ideas cover most of what passes for meeting "certification" requirements in most fields. However, some other topics deserve a bit of mention as they are definitely related to certification.

Where Licensing Fits In

Licensing is a government activity. It is where some governmental authority grants some individual, group, organization a “right” of some sort. For example, in the USA, states license people to practice law, medicine, etc. This is usually a pro forma event, accompanied by licensing fees paid to the state, given that the state trusts the certifier. For example, in the USA, this would be the AMA for doctors or the ABA for lawyers. Licensing is not an issue I want to discuss here in any depth. I felt it was important, though, to make the distinction since not all certifications necessarily lead to licensing concerns. One criteria when they do is if the health, welfare or safety of the public can be affected by activities covered under the certification. The IEEE effort noted above is occurring because matters of software development in areas such as aerospace, medical devices, financial transactions, and the like can have significant impact on the public.

Training Separated from Certification

For true objectivity and ethical reasons, a certification body should not mandate/control its own training/education as the only source to meet its own criteria. In any certification program, money will always end up having to be involved, one way or the other. This is why a certification body should not have some vested interest in consulting/training related to the certification.

The same idea applies to a company that would both perform audits to some standard and supply training/consulting to that same standard. What are the odds that, if you pay for the latter, you're likely to fail the former? At least, that's the conflict of interest question. Can a group claim to effectively train/consult with a company on some standard, then turn around and fail them because they followed the training/consulting? You get the idea. Most audit certification bodies look dimly on registrars that have too close a relationship with training/consulting firms (or arms of the registrar firm).

Value of Certification: Who Cares?

Certainly, those who would make use of the products/services of organizations employing/using certified individuals could care how effective certification of those individuals is conducted or, indeed, that certification exists at all. Individuals might care about certification as a distinguishing characteristic for themselves compared to those who are not certified. Naturally, as noted above, the credibility of the certification will matter. As some note, people with too many certifications may raise questions with potential clients/employers who may wonder how valid such certifications are if a person can maintain so many of them concurrently.

If the goal of certification is to ensure (a loaded word itself) people can be (not even "are") more effective in a given domain covered by the certification scope, perhaps there are other ways to make people more effective rather than worrying about how to judge if they are? Not that the latter isn't important, just that the former might be easier to accomplish, i.e., designing/promoting excellence in education/training could be easier than excellence in certification.

[And as a final note, almost everything said above regarding certification of individuals could apply to assessment programs related to, for example, “how agile” individuals or organizations are.]