The debate over an additional exam for those applying for a state psychology license has shined the light on a nest of scientific problems originating at the Association of State and Provincial Psychology Boards (ASPPB).
The debate has unearthed new facts and a jaw dropping irony––the psychology profession, a discipline that preaches anti-discrimination to others, and that sets the bar for selection-testing, has been promoting racism at state licensing boards, and by all accounts doing it for the money.
These problems might start with the ASPPB, but legally and morally they land at the doorstep of every state psychology board. The situation hits Louisiana particularly hard. While only 4% of licensed psychologists nationwide are African-Americans, Louisiana has a 34% Black population, a group chronically underserved by mental health professionals. Louisiana is specifically in need of psychologists who understand the Black experience.
For this report we look at current facts, core problems and underlying causes of how organized psychology is failing its students, the public, and its own goal of fighting systemic racism.
“Adverse Impact” found in psychology license examination program
The psychologist license exam is called the Examination for Professional Practice in Psychology or EPPP. After finding racial differences in the New York state pass–fail rate on the EPPP scores, Dr. Brian Sharpless has now found similar problems in Connecticut.
Dr. Sharpless studied 642 applicants to the Connecticut State Board of Examiners of Psychologists. In his article, “Pass Rates on the Examination for Professional Practice in Psychology (EPPP) According to Demographic Variables: A Partial Replication,” he reported significant differences in failure rates based on ethnicity.
Whites had a 5.75% failure rate, Blacks had a 23.33% failure rate, and Hispanics had a 18.6% failure rate.
In a much larger study in New York, Dr. Sharpless discovered an even greater impact by race. He reported his findings in “Are Demographic Variables Associated with Performance on the Examination for Professional Practice in Psychology (EPPP)?”
Dr. Sharpless gathered data on 4,892 New York applicants and first-time EPPP takers. He obtained records of all doctoral-level psychology licensure applicants from the past 25 years and looked at their EPPP scores.
He found that Blacks had a failure rate of 38.50% and Hispanics had a failure rate of 35.60%. Whereas, Whites had a failure rate of 14.07%.
“Adverse impact” is the term used to describe differences in scores. An exam has adverse impact if minority candidates fail to pass at at least 80% of the majority race candidates’ rate. The results in New York classify as adverse impact and the Connecticut results clear the bar only by a hair.
Title VII of the Civil Rights Act of 1964 makes it illegal to discriminate against someone on the basis of race, color, religion, national origin, or sex. When state psychology boards deny a license based only the EPPP scores, they must prove that the test is being used in a fair and
unbiased manner.
Selection–testing and design of selection programs is most often a subspecialty in industrial– organizational and business psychology. State boards primarily deal with healthcare practitioners, and are composed of clinicians. So, expertise in selection testing is unlikely to be involved in all or most states.
“If two states have found adverse impact, it is probable that all or most states will also find adverse impact. It is typical for knowledge tests to have adverse impact anyway, and this must be handled in the overall selection program,” said one expert.
One Black candidate told the Times, “We’ve known for a long time that the test discriminates–we learned it in graduate school. But there’s nothing we, as students, can do about it.”
According to one source at the Louisiana State Board of Examiners of Psychologists (LSBEP), the board makes no effort to study their procedures for adverse impact.
Critics of the national exam have gained momentum, fueled by the ASPPB’s effort to install yet another, second examination, called the EPPP2.
Dr. Jennifer Callahan sounded the alarm as lead author in her article, “The enhanced examination for professional practice in psychology: A viable approach?” published in the flagship journal for psychologists, American Psychologist.
The EPPP2 has not been evaluated for its intended purpose, Callahan said. “For jurisdictions implementing the EPPP Part 2, failure to gather and report the evidence required for use of a test in a forensic context may also open the door for legal challenges.”
Dr. Sharpless had also been pointing to legal risks. “… given the ethnic performance discrepancies and limited validity evidence, […] it will remain open to charges of being a potentially arbitrary barrier in an already protracted path to professional independence…”.
Industrial-organizational psychologist Dr. William Costelloe, who works in the private sector, agrees. There is no other choice these days, he told the Times, “… predictive validation studies must be conducted.”
Another business psychologist said that in the private sector the ASPPB’s approach would not be accepted. “Business owners would not take the risk of having adverse impact. We would be adjusting cutoffs and adding unbiased tests to the overall selection program, so that our clients could avoid adverse impact.”
Criticisms have been mostly dismissed by officials at the ASPPB. In an answer to Callahan, also published in the American Psychologist, Drs. Matthew Turner, John Hunsley and Emil Rodolfa defended their decisions. “The standards emphasize that licensure/credentialing examinations are built from a content validation framework, and this framework is used for licensure examinations across professions,” they said.
Dr. Turner is employed by ASPPB and in charge of the exam services. He was previously employed by the Georgia school systems. Both Hunsley and Rodolfa have also worked with and provided consulting services for the ASPPB examinations.
Callahan and coauthors replied, “…Turner et al. remain narrowly focused on defense of content validity and a reliance on outdated standards that fail to meet contemporary expectations for assessment of health care professionals. […] ASPPB’s methods demonstrably foster linguistic biases and systemic racism that constricts licensure of diverse individuals as psychologists.”
Callahan urged ASPPB to take “drastic corrective action.”
Experts point to serious issues with how states use cutoff scores on the national test, especially since there is no criterion related research to help set the cutoff.
“A 50th percentile cutoff score, that automatically fails the bottom half of a sample, all who are highly qualified already, does not make sense,” said one business psychologist. “This is exactly the way you drive up adverse impact. You’re basically having a bunch of PhDs compete against each other and then flunking half of them. Is your hypothesis really that half are incompetent?”
Sharpless had earlier noted problems with the cut off scores, typically set at the 50th percentile by state boards. “Additional empirical attention should be devoted to the cut score…” he said. “…the determination of the ‘passing’ score is one of the most important, yet difficult, psychometric tasks in testing …”
ASPPB acknowledges the exam limitations. On their webpage officials state, “There is no suggestion that people who do better on the EPPP will be better practitioners.”
The connection between test score and job performance would require predictive validity research, which ASPPB does not conduct.
Dr. Costelloe, explained “… predictive validation studies must be conducted.” For instance, “… you infer that a candidate with a high Extroversion score will make more sales than a candidate who has a high Introversion score,” Costelloe said. With a predictive study, “… you realize that your inference was not only completely wrong but backwards. Why? The sales personnel are interacting with mechanical engineers who must make the decision to switch over these new valves. They don’t want to relate. They want specific engineering facts and data and they are introverts.”
Michael Cunningham, PhD, Professor of Psychology, Africana Studies and Associate Provost at Tulane, points to potential problems with item development.
“Like all standardized exams, people with the highest pass rates tend to very similar in racial and ethnic backgrounds as the test developers,” he said. “For many standardized tests, experts examine items for bias when there is an adverse impact of a question for males or females. In these cases, when bias still exists after an item analysis, the question is excluded. I don’t think similar considerations are done for racial/ethnic or SES backgrounds.”
ASPPB seems unconcerned with the scientific criticisms and standards. In April 2018, then SPPB CEO, Dr. Stephen DeMers, met with members of the Louisiana State Board of Examiners of Psychologists and representatives of Louisiana Psychological Association (LPA).
About the meeting, Dr. Kim VanGeffen, Chair of LPA Professional Affairs, said, “Dr. DeMers acknowledged that, currently, there is not really any research on the validity of the EPPP-2. There do not seem to be any plans to obtain predictive validity nor does the EPPP2 committee believe that establishing this type of validity is necessary,” she said.
Dr. Marc Zimmermann, past LSBEP board member, also attended. “He [Dr. DeMers] stated that there is no predictive validity,” said Zimmermann. “He also threw in that none of the national tests had predictive validity. He reported that content validity was the accepted standard because a test with predictive validity could not be constructed,” said Dr. Zimmermann. “… DeMers had the temerity to try to sell us something that does not meet the standard that psychological tests being published are expected to have.”
Is more regulation needed? Safety estimates for psychologists are very good
One of the arguments that critics mention is the consistently high safety ratings for psychologists, based on the low number of disciplinary actions nationwide.
“There is no evidence that the public is facing some sort of previously unheard of crisis in terms of safety from currently practicing psychologists,” said Dr. Amy Henke, who spearheaded a Resolution opposing the EPPP2 while serving as a director for the LPA.
“Trainees are already held to high standards through a variety of benchmarks,” Dr. Henke wrote in the Resolution, “… including but not limited to: APA approval of doctoral programs, multiple practicums where competency is repeatedly assessed, completion of formal internship training (also approved and regulated by APA and APPIC), and supervised post-doctoral hours obtained prior to licensure.”
However, Dr. Emil Rodolfa, then a program developer at ASPPB, questioned if these standards are enough, saying that supervisors have “… difficulty providing accurate evaluations of their supervisees to others who may have to evaluate the supervisee’s competency.”
The facts are on Dr. Henke’s side. Data from their own ASPPB Disciplinary Data System: Historical Discipline Report show rates of disciplinary actions for psychologists to be consistently low. For an estimated 125,000 psychologists in the US and Canada, the disciplinary rates remain around 1–2 per 1,000.
For 2016–2020, the total reported disciplinary actions across the U.S. and Canada ranged from 139 to 186.
Using a conservative estimate of 10 clients per psychologist per year, this translates to a safety problem of one or two per 10,000 service events.
Louisiana’s rate is similar to the national average. For the year 2019–2020 there was one disciplinary action, for 2018–2019 there were two, for 2017–2018 there were also two, for 2016– 2017 there were three, in 2015–2016 there was one, and in 2014–2015 there was also one.
ASPPB’s plans for doubling the size and cost of licensing exam
Several sources suggest that profit motives may be the main reason for the cutoff and the extra test. The current EPPP costs candidates $600 plus administration fees. The test contains 225 items, with a four-hour time limit. To compare, physicians pay $605 for an eight-hour exam and social worker candidates pay about $250 for a 170-item exam. The EPPP2 would increase cost from $600 to $1200.
Some years ago, ASPPB appears to have embraced a more aggressive corporate strategy. An insider told the Times, “In 2010 or somewhere around that time they [ASPPB] were in New Orleans and they implied that they would be making a lot of money on the new test.”
In 2012, ASPPB acquired the rights to the exam, taking over from Professional Examination Service (PES). In 2013 ASPPB wrote the boards that their contracts with PES were being “…
replaced with a contract between your jurisdiction and the Association of State and Provincial Psychology Boards.”
In that letter, ASPPB officials said, “ASPPB and PES have agreed that it would be simpler and more appropriate for ASPPB to contract directly with the 64 psychology regulatory agencies that are members of ASPPB.”
ASPPB said the change would be “…mutually beneficial because ASPPB can now provide a simplified agreement that is more specific to the needs of psychology licensure boards. In addition, the renewal of contracts is expected to be more efficient…” At the same time, ASPPB increased candidates’ exam fees from $450 to $600.
One insider thinks the corporate objective for ASPPB is to be a central source for regulation of psychologists. “They want to ultimately do all the licensing and regulating for psychology,” said the insider. “They want to regulate all the telepsychology.” And, “They want to be the Walmart.”
In 2013 ASPPB officials were instrumental in conducting and designing the 5th International Congress on Licensure, Certification, and Credentialing of Psychologists, held in Stockhom. The invitation-only conference was primarily funded by ASPPB. Dr. Emil Rodolfa, Chair of the Implementation Task Force for the EPPP 2, facilitated at the Congress.
ASPPB officials have gone through several roll-out efforts for the EPPP2, first to persuade member jurisdictions to accept the new test, and then to force the new exam on states.
In 2016 the firm announced the EPPP2 and told its members, licensing boards across the United States and Canada, that the use of the new test would be “optional.”
However, after criticisms mounted, ASPPB did an about face in late 2017 and announced in a surprise move that the new exam would be mandatory after all, and combined with the tests. And, the price would increase from $600 to $1200.
In July 2018, Dr. Amy Henke, then serving on the Louisiana State Board of Examiners of Psychologists (LSBEP), and LSBEP members sent a blistering letter to the ASPPB Board of Directors, to the ASPPB members, and to the administrators of state psychology boards across the US and Canada.
Following this, in August 2018, ASPPB President Sharon Lightfoot, PhD, announced that the ASPPB Board of Directors voted to rescind the mandate.
However, shortly after that, ASPPB decided to use a carrot and stick approach. According to an October 24, 2018 letter from Lightfoot, if Louisiana, or other jurisdictions, chose to decline the use of the new additional test, then student candidates in those jurisdictions would be prohibited from taking the test. Sources at the Louisiana state board considered this to be punitive, because many students wished to prepare for licenses in other states, which might require the second test.
Dr. Henke said that at a recent meeting of the member jurisdictions, representatives voted 100% to allow qualified candidates from any jurisdiction to take the EPPP2.
“Unfortunately,” Henke told the Times, “the ASPPB Board and staff have pushed back on both the vote and the formal request. For instance, despite this unanimous vote, ASPPB’s Board has not acted on the clear wishes of the member jurisdictions. Instead, they have decided to individually poll each jurisdiction with a survey that I felt was misleading and biased.”
ASPPB’s non-profit & financial status
The ASPPB is a private, non-profit, 501(c) 6, tax-exempt corporation located in Tyrone, Georgia.
The IRS notes that the 501(c) 6 “… may not be organized for profit to engage in an activity ordinarily carried on for profit (even if the business is operated on a cooperative basis or produces only enough income to be self-sustaining).”
The corporate mission is to “Facilitate communication among member jurisdictions about licensure, certification, and mobility of professional psychologists.” The “members” are the 64 or so regulatory boards from across the United States and Canada.
These boards pay dues to ASPPB. LSBEP records note they paid $2,660 in 2020 for annual ASPPB dues.
ASPPB’s net assets for 2018 (the most recent year available due to delays from Covid) totaled $9,137,930. GuideStar estimates their assets at $11,013,348.
Total revenue for 2018 was $6,505,651. Revenue for 2017 was $6,645,731 and $5,933,473 for 2016.
ASPPB’s main income producing product is the national exam. The exam and related services generated $6,137,348 in 2018. This accounted for 94% of the Association’s 2018 income. Exam income was $5,378,524 in 2017, and $4,916,406 in 2016.
While they paid $1,302,603 to Pierson Vue Minneapolis for exam administration in 2018, most other expenses claimed on their tax reports are for employees and employee related expenses.
They report a total of $2,278,482 for compensation of key employees, other salaries and wages, contributions to pension plans, employee benefits and payroll taxes.
In 2018, the CEO, M. Burnetti-Atwell, received pay and benefits of $255,936. In 2017, Dr. Steven DeMers, then CEO, received $270,784.
Assn Executive Officer Dr. Matthew Turner received pay and benefits of $$171,174 in 2018. He has four employees reporting to him for exam services.
Assn Executive Officer Janet Orwig received pay and benefits of $158,142 in 2018. Ms. Orwig has 12 employees reporting to her for member services.
The ASPPB website lists other staff, including a business manager, financial officer, two directors of professional affairs, and an administrative associate.
“With a lot of cash sitting on the balance sheet, the strategy is to maximize expenses,” said an MBA in reviewing the information for the Times. “The extra profits are likely to go into perks rather than price cuts,” he said.
Examples of this appear to include items like travel, which includes travel for spouses or companions. The organization spent $949,483 on travel in 2018 and $1,169,743 on travel in
2017.
Other examples are $336,175 on “technology,” $188,256 on conventions, $123,053 for “item writers and exam consultants,” $144,000 on bank fees, $60,610 on advertising, and $55,946 on dues and subscriptions.
It is not clear how oversight is established at ASPPB. The Times asked one CPA to look over the information and he said, “Of course there is influence and COI (Conflict of Interest).”
Conclusions
ASPPB appears unable to constructively answer the criticisms and mounting evidence that their exam program, marketed to the captive customers through state boards, is scientifically deficient and discriminatory.
The state boards have bought into a mess, but do not appear able to deal with the bureaucracy at ASPPB. Since state boards are typically composed of clinicians, and rarely have the hands-on experience needed for high-stakes selection testing, they may lack the knowledge to fight the problem.
Ignorance does not fly as an excuse for discriminatory practices in the private sector, so it’s ironic that it is found in the public and quasi–governmental agencies of psychology.
ASPPB appears too busy feeding off of the exam revenues, and building their bureaucracy on the backs of psychology license candidates, perhaps especially racial minorities, to wake up. State psychology boards must not ignore the problem any longer. Callahan’s call for “drastic corrective action” is on point. But it is the members of ASPPB who need to take action, with or without ASPPB bureaucrats.
In September 2020, the American Psychological Association (APA) called for “… true systematic change in US culture.” Zara Abrams reported in a Monitor article, APA “… is working to dismantle institutional racism over the long term, including within APA and psychology.”
Before preaching anti-discrimination to others, APA needs to start in their own backyard.
