Clariana Develops New Grading System
by Joe Savrock (June 2010)
UNIVERSITY PARK, Pa. - For instructors, interpreting exactly what students intend to convey in their written essays can be a challenge. Grammatical missteps—for example, overuse of pronouns or inconsistent subject-predicate agreement—can skew the intended meaning of a student’s essay.
Roy Clariana, professor of instructional systems, developed a text-scoring system he calls analysis of lexical aggregates (ALA) as well as a text-analysis software package, ALA-Reader. ALA-Reader provides a means for scoring essays objectively in various instructional and research-related settings.
“Today, students in our high schools write a lot of essays and teachers spend a lot of time marking these essays. I initially developed ALA-Reader to do some of the work, not to replace teacher scores but to complement their work,” said Clariana, who serves as academic division head of education at Penn State Great Valley.
ALA-Reader translates written text directly into electronic files. The program converts the text into a word map or picture of students’ essays, and each student’s map is also automatically compared to the teacher’s map for analysis that results in an objective score.
The current version of ALA-Reader is used mainly by researchers interested in automatic text analysis, especially doctoral students, but classroom teachers could use it as well. “This approach has potential not only for assessment but also for classroom instruction,” said Clariana. “For example, the software works in any language and researchers in Mexico and Finland are using it now. It can convert a Spanish essay into an English word map, and so it could be used in foreign language-learning classes. And students can run it on their own writing to see the word map of their essay and to try to improve the essay if they score low.”
Recently, Clariana and his colleagues applied ALA-Reader to the essays of students in an undergraduate business course. The investigators sought to analyze the effect of pronouns. They edited the essays, replacing commonly used pronouns with the appropriate referent, and then processed both the original essays and the edited essays using two different ALA approaches:
- the sentence aggregate approach, which analyzes text at the sentence level. ALA-Reader disregards all words in a sentence except preselected key terms. It then replaces the synonyms of key terms with the appropriate key term.
- the linear aggregate approach, in which ALA-Reader searches the text sequentially from beginning to end, looking for adjacent key terms both within and across sentences. This approach likewise disregards all words except preselected key terms and replaces synonyms of key terms with appropriate key terms.
Analysis revealed that the linear aggregate approach provided a better measure of individual essay scores.
“Besides providing more evidence that the system works, this sort of fundamental research also helps us understand the linear and nonlinear nature of text and the influence of parts of speech in texts,” stated Clariana. “It also provides a way to visually represent the structure of an individual’s knowledge that is captured in their writing.”
He added, “New tools allow us to think in new ways about things, and what we are finding now using ALA-Reader enhances the discussion about what is knowledge.”