Who grades the exam a laboratory has to pass

Somebody has to be trusted to set the questions

A laboratory doing complex testing sits a graded external exam several times a year. That much is widely known.

Less known is that the organizations running those exams are themselves approved by the government, against written criteria, and can lose that approval.

This matters because the whole system rests on the grader. If the exam is easy, badly made, or graded against a wrong answer, a passing score means nothing.

The criteria are worth reading as a model for any external assessment scheme, including the ones sellers cite about their own products.

Who is allowed to run one, and when they have to apply

The first requirement is about the kind of body that may offer such a program at all.

For approval, "the program must be offered by a private nonprofit organization or a Federal or State agency, or entity acting as a designated agent for the State."

A for-profit company cannot be the approved program. A later provision extends the same restriction to contractors doing the technical and scientific work, including selecting target values, grading, reporting scores and deciding which organisms go into microbiology samples.

The calendar is fixed as well. An organization seeking approval or reapproval for the next calendar year "must submit an application providing the required information by July 1 of the current year."

And the program owes something to the laboratories using it. It must provide technical assistance to laboratories seeking to qualify under it.

Ten participants, and why the number exists

One criterion looks administrative and is actually the foundation of the grading method.

A program must "Require a minimum of 10 laboratory participants" for each specialty, subspecialty and analyte for which it seeks reapproval.

The reason appears later, in the grading rules. For many tests the correct answer is defined by consensus among participants, and a consensus among too few laboratories is not a consensus.

The program must also assure the quality of test samples, evaluate and score results appropriately, and identify performance problems in a timely manner.

The samples have to behave like real specimens

A program has to demonstrate technical ability, and the first element is sample production.

It prepares or purchases samples from manufacturers working under the applicable good manufacturing practice requirements the regulation names.

Distribution uses rigorous quality control to ensure that "samples mimic actual patient specimens when possible and that samples are homogeneous". Specific subspecialties such as cytology are excepted, and samples have to stay stable long enough to be analyzed.

The homogeneity requirement is what makes a score comparable between laboratories. If two laboratories receive materially different samples, their scores are not measuring the same thing.

Cytology is excepted because a glass slide of cells cannot be made homogeneous, which is why that subspecialty gets its own rules.

Deciding the right answer

The second element of technical ability is the one that decides whether any of this means anything.

The program must have "A scientifically defensible process for determining the correct result for each challenge offered by the program".

It must also run a program of sufficient annual challenge, at the frequency the specialty-specific sections set, to establish that a laboratory has met minimum performance requirements.

It must also have the resources to report. Statewide or nationwide reports to regulators on individual performance for gynecologic cytology, and on laboratory performance for testing events. Then cumulative reports and scores, and reports of specific failures using grading criteria the department accepts.

Those reports go to the government on a timely basis when requested.

The signature that makes cheating a documented act

One requirement sits on the report form itself, and it is the sharpest anti-gaming device in the scheme.

The requirement sits on each report form a laboratory uses to record testing event results. The program must include "an attestation statement that proficiency testing samples were tested in the same manner as patient specimens". It carries "a signature block to be completed by the individual performing the test as well as by the laboratory director".

Two signatures, one from the person who ran the test and one from the director.

The effect is that treating a proficiency sample specially is not merely a rule broken, it is a false statement signed by a named person.

Logistics, disputes and inspections

Three more criteria round out the technical demonstration.

The program must have a mechanism for telling participants the shipping schedule. A participant reports that samples did not arrive or were unusable "within three days of the expected date of receipt of the shipment". It must be able to replace samples lost or received unusable.

It must have a process to resolve technical, administrative and scientific problems about its operations.

And it must meet the specific criteria for its specialty and analyte, then give the department the information needed each year to confirm it still does.

Approval is not paper-only. "HHS may require on-site visits for all initial proficiency testing program applications" for approval, and periodically or when problems arise for already approved programs, including reviewing documents and interviewing staff.

A program hitting significant problems during reapproval can be made to reapply as if new.

What happens when a program is disapproved

The consequences of failure are written with dates, which makes them checkable.

Where a program fails any of the approval criteria, the agency notifies it of withdrawal of approval. "Approval of the PT program remains in effect for 60 days from the date of notification."

That window exists so participating laboratories are not stranded mid-cycle.

The program has to pass the news on: it "must notify all of its participating laboratories of the withdrawal of approval within 30 days from the date of notification."

And there is a specific ground beyond ordinary failure. "CMS may disapprove any proficiency testing program that provides false or misleading information" about anything needed to meet the approval criteria.

A dissatisfied program may ask for reconsideration under a separate procedural part.

What one specialty looks like in detail

The general criteria are filled in by specialty. Bacteriology is a good worked example because its grading is fully specified.

For approval, "the annual program must provide a minimum of five samples per testing event." And "There must be at least three testing events provided to the laboratory at approximately equal intervals per year."

The program tells the government what it plans to include "no later than 6 months before each calendar year". It must include commonly occurring organisms and important emerging pathogens. And it must vary its content over time across the named major groups of medically important bacteria.

One composition rule stops the exam being too clean: "At least 25 percent of the samples must be mixtures of the principal organism and appropriate normal flora."

The regulation then distinguishes a genuine mixed culture from two harmless organisms whose intended answer is that nothing significant is present. It requires both kinds of mixed sample. Ones where only a principal pathogen should be reported, and ones where every organism present should be.

How a correct answer is established, and when grading is abandoned

Bacteriology also shows how consensus grading works in practice.

Accuracy is determined by comparison. "the program must compare each response with the response which reflects agreement of either 80 percent or more of 10 or more referee laboratories or 80 percent or more of all participating laboratories."

That is the ten-participant minimum doing its work. Without enough laboratories, neither route to a consensus answer is available.

And a program cannot simply drop a hard sample. "Both methods must be attempted before the program can choose to not grade a PT sample."

The laboratory is graded against its own practice, not an abstract standard: "A laboratory must identify the organisms to highest level that the laboratory reports results on patient specimens."

Scoring is arithmetic and unforgiving of over-reporting. Naming an organism that is not there reduces the score, because the denominator includes incorrect organisms reported as well as organisms present. Susceptibility testing is graded only for the antimicrobials the laboratory routinely uses on patients.

The one exam that grades a person, not a laboratory

Gynecologic cytology is different in kind, and the differences are all consequences of the material being glass slides read by eye.

A program provides test sets of ten and twenty slides, obtained from cytology laboratories under the retention and loan rules. Each set must include at least one slide from each response category.

Testing happens where the work happens. A program "must provide announced and unannounced on-site testing for each individual at least once per year".

Failure has a fast clock. An initial retesting event is provided "within 45 days after notification of test failure", and subsequent retesting within forty-five days after remedial action is completed.

And the answer key is built to a standard nothing else in this subpart matches. For all slide preparations, "a 100% consensus agreement among a minimum of three physicians certified in anatomic pathology is required."

For premalignant and malignant slides that is not enough on its own: "confirmation by tissue biopsy is required", either by comparing reported biopsy results or by a certified pathologist reevaluating the biopsy material.

Scoring that weights the consequence of being wrong

The scoring design is the part most worth borrowing.

Slides carry point values set by how the interpretation relates to a clinical condition, and by whether the person being tested is a cytotechnologist or is functioning as a technical supervisor.

The marking is graded rather than binary. "The scoring system rewards or penalizes the participants in proportion to the distance of their answers from the correct response or target diagnosis", weighted in proportion to the severity of the lesion.

Four response categories are used, running from unsatisfactory for diagnosis, through normal or benign changes, to the abnormal categories.

One further provision matches the test to the job. A technical supervisor who only interprets slides after a cytotechnologist has screened them may be tested on a pre-screened set. One who screens as well as interprets must be tested on a set nobody has screened.

What a reader should take from this

The first point is general. An external assessment is only as good as the process that decides the right answer, and this scheme makes that process an approval criterion rather than an assumption.

The second is the attestation. Signed by both the tester and the director, it converts special handling of a test sample into a documented false statement.

The third is a useful yardstick for any claim about third-party verification. Ask who approved the verifier, against what published criteria, how the correct answer is determined, and what happens when the verifier fails.

Those questions have written answers here. In most commercial testing claims they do not.

One limit on this article. Only the general approval sections and two specialty sections were read in full. The other specialty criteria, the laboratory-side participation requirements, and the reconsideration procedure referenced were not read for this piece.

One element was read from a table that the extraction flattens into a sequence of cells. The four response categories are described from it, and no cell is quoted, because a quotation stitched across flattened cells would not be a sentence the source contains.

Sources

Each document below is named as it names itself, with the date printed on that document rather than the day it was read.

1. Title 42 Code of Federal Regulations section 493.901, Approval of proficiency testing programs, read in full for the eligible bodies, the July application date, the ten participant minimum, the technical ability criteria, the attestation requirement and the on-site visit provision — Electronic Code of Federal Regulations, Office of the Federal Register, July 2022
2. Title 42 Code of Federal Regulations section 493.905, Nonapproved proficiency testing programs, read in full for the sixty day validity, the thirty day notification duty and the false information ground — Electronic Code of Federal Regulations, Office of the Federal Register, July 2022
3. Title 42 Code of Federal Regulations section 493.911, Bacteriology, read in full for the sample count, the three testing events, the advance content disclosure, the mixed culture proportion, the consensus grading routes and the scoring arithmetic — Electronic Code of Federal Regulations, Office of the Federal Register, July 2022
4. Title 42 Code of Federal Regulations section 493.945, Cytology; gynecologic examinations, read for the slide test sets, the announced and unannounced on-site testing, the forty-five day retesting clocks, the unanimous three-pathologist answer key with biopsy confirmation, and the proportional scoring system — Electronic Code of Federal Regulations, Office of the Federal Register, December 2023