Psychology’s Testing Enterprise Deserves Better: Response to Mikesell, Frederick, & Otto (2025)

In their March 2025 rebuttal of our paper which appeared in the December 2024 edition of On Board with Professional Psychology,  Mikesell et al. (2024) claim that a protective order “properly crafted” will safeguard test information from further distribution. In our paper, and in a position paper endorsed by the Interorganizational Practice Committee (IOPC; a consortium of neuropsychological organizations), we carefully laid out eight reasons why protective orders are not, and cannot be, adequately protective of psychological tests (Boone et el., 2025). Mikesell et al. completely sidestepped these issues in their rebuttal. We believe they have an obligation to all psychologists engaged in psychological and neuropsychological assessment to explain how they propose that the fatal problems with use of protective orders be addressed. If we did not make the implications of protective order limitations clear, we will further spell them out here:

1. In a digital age, in which rapid scanning, archiving, and distribution of materials can be done instantaneously, how can adherence to the terms of a protective order be monitored? Can we access attorney hard drives to see what has been stored, or their email systems to see what has been distributed? The US Supreme Court, even prior to the digital age, specifically acknowledged concerns that protective orders were not adequately protective of psychological tests (Detroit Edison v. National Labor Relations Board, 1979). In a labor case involving Coca Cola, the company opted to settle a dispute with employees rather than disclose the recipe for Coca Cola under protective order, realizing full well that the order would not protect their recipe and, by extension, the viability of the Coca Cola enterprise (Childs, 2007). So, a soda recipe is recognized as not adequately protected by a protective order, yet psychological tests, which are used to protect society and allow for fair allocation of societal resources, are to be released under protective order?
2. Release of psychological test materials to attorneys under protective order is analogous to turning over trade secrets to a competitor. Protective orders were meant to protect confidential information for those who are party to a lawsuit, with the assumption and expectation that the attorneys involved in the case have no financial investment or future interest in the materials protected. But this is far from the situation when protected test materials are released to attorneys, because such materials could be used to coach future clients to display presentations that appear credible when in fact the clients do not have the symptoms and dysfunction they are claiming. Further, a protective order does not remove acquired knowledge from the minds of attorneys, i.e., we don’t have a “Men in Black”-type “neuralyzer” to erase memories. An attorney could comply with a protective order but still use information learned by exposure to tests to manipulate the test-taking presentations of future clients. Truly, when protected test materials are released to attorneys under protective order, the “fox is guarding the henhouse.”
3. Psychological tests are more at risk to damage by breach of protective orders than are other trade secrets because the psychological tests themselves are destroyed if the public gains access to them. There is no stockpile of back up tests held “in reserve” that can be used to replace tests invalidated by test security breaches. If our tests become compromised, then how does society determine whether workers are cognitively and psychiatrically capable of carrying out job tasks involving public safety, or make fair and accurate judicial decisions and determinations regarding academic accommodations? 
4. Protective orders are not enforceable. For example, in the 2004 Zyprexa product liability litigation case, thousands of documents were released in violation of a protective order. Childs (2007) notes that subsequent judicial injunctions were ineffectual in limiting access to the leaked documents, and a year later, no contempt motions had been entered against those involved in the dissemination. 
5. The sheer number of protective orders required, given escalating attorney demands for protected test information, virtually guarantees exposure of protected test information. One-half of 1,677 neuropsychologists responding to a recent survey reported engaging in forensic practice (Sweet et al., 2021); if each of those  neuropsychologists releases test information under a protective order 10 times per year, that would result in over 8000 protective orders yearly. Moreover, the protective orders would be covering the same finite set of test materials; that is, the same test information would be released over and over, thereby allowing for repeated chances that the information will be breached. In contrast, protective orders were intended to be used to protect confidential information specific to a particular case on a one-time basis. Even inadvertent breaches of protective orders would become routine given such a volume of protective orders. 
6. Protective orders can be vacated well after the case at issue is concluded, thereby placing protected test materials in jeopardy in perpetuity. If a psychologist releases protected test information 10 times per year under protective order, that would result in that psychologist being involved in 100 protective orders over 10 years. Is the expectation that this psychologist is to track the status of all these protective orders on an ongoing basis? We are not a party to the lawsuits in which we are retained, and therefore none of the attorneys involved represent us or our tests; thus, we are literally on our own in terms of any ongoing monitoring of protective orders.
7. The number of people with access to protected test information under protective order includes not just attorneys, but all legal office staff and other experts.  In other words, upwards of 30 or more non-psychologists could view test materials produced under protective order in each litigated case. Assuming over 300,000 personal injury cases are filed per year in the US (United States Courts), if just 10% involve psychological and/or neuropsychological testing, this would result in hundreds of thousands of non-psychologists gaining access to protected test materials each year. 
8. Even the legal field is not in agreement that sanctions can be levied for protective order breaches. Some courts have found that sanctions are not appropriate. Further, the typical sanctions for protective order violations involve dismissal of a case or payment of the opposing attorneys’ fees, neither of which would remedy the damage done to the profession of psychology.

Obviously, the above-identified problems regarding using protective orders for psychological test information cannot be surmounted, and it is irresponsible for Mikesell et al. to suggest that “properly crafted” protective orders are a viable option. They assert that there is “no evidence to date” that any psychological tests have been ruined through turning over test materials to non-psychologists. This argument is flawed on its face. None would argue that a closed barndoor can be safely left open because livestock had not gotten out yet; they did not get out precisely because a barrier was in place. Is the recommendation of Mikesell et al. that we remove test protections and then wait until tests have been compromised before we take action? What would constitute evidence of test ruination? Research has in fact shown that the sensitivity of some original performance validity test cut-offs has declined over time (e.g., Rey 15-item, Poynter et al., 2019; Dot Counting Test, McCaul et al., 2018). Is that not what we would expect if exposure was impacting test effectiveness?

Mikesell et al. argue that attorneys should have an unfettered right to our test materials, even though it could result in the destruction of our tests and, ultimately, harm to society (see Boone et al. (2022) for further discussion). They fail to acknowledge that a court, when considering ordering discovery, is obliged to strike a balance, whenever possible, between the rights and interests of all parties. Courts have overestimated the importance of attorney “first hand knowledge” when critiquing our materials; attorneys have no training and knowledge regarding psychological assessment, and they introduce misinformation to the trier of fact when they attempt to craft cross-examination questions based on their own analysis of our test materials (Boone et al., 2024). As an analogy, neuropsychologists learn psychological testing methods during four years of graduate school followed by a one-year clinical internship, and two years of full-time post-doctoral training in neuropsychological assessment.  What if on day one of graduate school, a student is handed test data sheets from a neuropsychological exam and told to analyze the data, and then informed that their interpretations will be used as the basis for cross-examination questions at a trial?  Arguably, the probability that such interpretations would be accurate is close to zero.    

The Mikesell et al. recommendation to turn over protected test information to attorneys under a protective order is a direct assault on the value of psychologists’ extensive training and acquired expertise in objective assessment methods, and an attack on the importance of science-based knowledge.  Would any party in a lawsuit prefer that test data be analyzed by their attorneys or by true experts in such testing? The obvious solution, and the one strongly endorsed by test publishers and all neuropsychological organizations, is that protected test data remain solely in the possession of psychologists. Psychologists are the only ones with the skill set to accurately critique psychological test work product, which they can do while at the same time maintaining test security. A nationwide survey of 600 psychologists conducted in 2023 by the IOPC showed that >98% endorsed the withholding of protected test information from attorneys, and 94.5% of respondents agreed that a protective order issued by a judge is not sufficient to ensure test security.

Consistent with our position is the recently published APA Resolution on Protecting Psychological Test Security, Test Validity, and Public Safety (February 2025), which concludes:

 “… APA’s position is that fair and reasonable transparency and access to psychological test data and test materials in legal proceedings is best achieved and scientific validity best preserved when psychologists share test materials and test data with other psychologists and other experts properly trained in test administration and interpretation who have a legal and ethical obligation to protect test data and test materials.”