by Jennifer Kowalewski

Before she ever came to OU-COM, Joy Matthews-López, Ph.D., worked on cutting-edge research in the area of educational testing. Now, three years after leaving the Educational Testing Service (ETS), one of the works she co-authored has been published by the prestigious College Board.

“It is a nice piece of research,” says Matthews-López, Centers of Osteopathic Research and Education (CORE) research director. “The College Board, owner of the SAT, is a very competitive setting to be published in. It was exciting to be accepted.”

Matthews-López worked as a measurement statistician for ETS, a premier testing company overseeing SAT, GRE, Praxis as well as other professional and educational assessment programs. While at ETS, Matthews-López did her primary work on a Spanish adaptation of the verbal section of the Graduate Management Admissions Test (GMAT).

She studied cultural and linguistic issues and then adapted questions to make the test fairer for all test takers.

Her paper, “Using DIF Dissection Method to Assess Effects of Item Deletion,” was published in 2005 with co-authors Yanling Zhang and Neil J. Dorans. The researchers presented their findings at the National Council on Measurement in Education in 2003 in New Orleans.

And although she left ETS behind, Matthews-López will use the research she helped create to make better testing procedures for medical students at OU-COM.

“We wrote several papers looking at fairness from different perspectives,” she said. “Even with the best intentions, bias can be written into tests.”

In this setting, bias refers to questions considered unfair to a specific gender or race, based on their experiences. Matthews-López used the example of sports-related questions to assess math skills. The question may appear “fair” for males taking the test, but females that are not sports oriented may have difficulty answering the question, even though they may have the skills necessary to do so. The context of the question can bias the response. Thus, a sports-related math question may be biased against females.

Differential Item Function (DIF) is a statistical procedure used to screen standardized test items for fairness violations. Matthews-López’s article looked at DIF from a new perspective, in that both gender and “race” were considered simultaneously.

Although previous DIF research addressed bias based on race and gender separately, Matthews-López and the other researchers looked at how certain questions would affect how the respondents answered, taking into account both their race and gender. In other words, would a question be unfair to certain populations based on their gender and race. The research looked at respondents’ gender as well as their race. In all cases of review, groups were defined in terms of reference (majority) or focal (minority) status, and all items were reviewed in this light.

Also, the study had to take into account “Equally-able” test takers, which refers to test takers that have similar skills or ability. Typically, this is measured by the total raw score on a test. Unfortunately, if a test contains many biased items, then the validity of the total score is called into question.

“And that is the only way we are able to group folks into a given score group,” she said.

They initially discovered there were biases inherent in questions for certain genders and races when these two factors were considered together. For instance, African American women may be more likely to get a certain question wrong than other equally able women and African American men.

Matthews-López would like to see more research done in the field, furthering the study she did.

In addition to directing the CORE Research Office, Matthews-López is currently working on an assessment project for OU-COM. By lending psychometric expertise to the existing testing program at OU-COM, some of the ideas and screening procedures used at professional testing organizations can be applied to the college and as a result, improve the testing conditions and policies for the medical students.

“I think it’s an example of the quality people we have at OU-COM,” says Keith Watson, D.O., associate dean for postgraduate medical education. “It is a nice contribution to the literature of research out there. We can take pride in her work.”

Matthews-López has published other research papers and has presented works at national and international conferences. “Test adaptation: General guidelines and suggestions for test development” was presented at an annual meeting of the National Council on Measurement in Education in Montreal and “Using differential person functioning to detect aberrant response patterns in a standard-setting session for teachers licensure” at the Midwest Educational Research Association (MWERA) meeting in Columbus.

She also published “Navigating the Review Process for Research Involving Human Participants: An Overview and Practical Guidelines” in the Ohio Research and Clinical Review (2004) and presented “Differential Speediness; A look in to subgroup differences” at MWERA (2003).