Providing context for keywords in writing courses
A case study of UX research in curriculum design
Overview
All students, including those who use English as an additional language, can benefit from looking for patterns in language use, such as how real writers tend to use specific words or phrases in their writing. In this three-year project, I worked with a research team to design instructional materials based on real written texts. We then evaluated the effectiveness of the materials based on data collected through focus groups, surveys, and contextual inquiry.
This case study focuses on one major finding from the first year of the study.
Our goals
To develop new ways of presenting examples of real texts so students could easily discover patterns of language use
My role
UX researcher, instructional designer
Collaborators & stakeholders
project lead, undergraduate and graduate research assistants, writing instructors
Core skills
• Focus group moderation
• Contextual inquiry
• Surveys
• Ethical review
• Instructional design
• Contextual inquiry
• Surveys
• Ethical review
• Instructional design
Background on corpus-based instructional materials
Our team designed instructional materials using a corpus, which is a large collection of texts. The Corpus & Repository of Writing (CROW) contains more than 10,000 texts written by students at three US universities. Users can search for keywords in the texts using an online interface. Corpus search results are commonly shown as a list of concordance lines, which are short segments of text that include a set number of words before and after the keyword.
Concordance lines for the keyword “rather” in students’ written assignments. Each line of text shows the 10 words before and 10 words after the keyword. Users can sort concordance lines by the word before or after the keyword. Concordance lines can reveal patterns of language use. For example, these concordance lines show that the word “rather” is usually followed by “than” but can also be preceded by “but” or used to modify an adjective (e.g. “rather poor quality”).
Participants
Instructors
The instructor participants were lecturers and graduate teaching associates who teach in a university writing program. All instructors had earned or were working toward a master’s in teaching English as a second language or a related discipline.
Students
All students in the instructors’ classes were invited to participate. They were international students from around the world, most of whom spoke English as a second or additional language. Most student participants were in their first year at the university.
Methods
Focus groups
Our team conducted focus groups with groups of up to five writing instructors before and after they used each set of instructional materials. We chose to interview the instructors as a group so they could collaborate to generate topics for lesson materials and learn from each other’s experiences.
After the instructors implemented each set of materials, we held a focus group to ask them about their experiences. In these focus groups, we asked:
• How did you use the materials in your class?
• What challenges did you face as you used the materials?
• How did your students respond to the materials?
• What would you do differently in using these materials in the future?
• What challenges did you face as you used the materials?
• How did your students respond to the materials?
• What would you do differently in using these materials in the future?
Classroom-based contextual inquiry
We conducted in-person observations as the instructors were implementing the materials in their classrooms. Guiding questions for the contextual inquiry were:
• How did the instructors incorporate the materials into their classes?
• How did the students engage with the materials?
• How did the students engage with the materials?
Surveys
After the instructors had used the materials in their classes, we surveyed them and their students about the effectiveness of the materials.
Insights on concordance lines
Our observations and focus group data revealed a shortcoming of concordance lines for teaching some concepts. We found that:
• Students could observe patterns of language use immediately surrounding the keyword, but they sometimes had difficulty understanding the concordance lines because the beginning and end of the sentences were cut off.
• Instructors struggled with activities that aimed to teach transition words (like “however”) using concordance lines because the cut-off sentences made it hard to show students how some transitions were being used in the context of a sentence or paragraph.
These findings led to a key insight: Concordance lines do not provide enough context for teaching some transition words. Students understand these words better when they are presented in the context of complete sentences.
Concordance lines for the keyword “however” in multiple students' language narrative assignments. We found that the concordance lines did not provide enough context for students to understand what kind of contrast was being drawn with the word “however.”
Revising the materials
Based on this insight, I led our team in creating a new set of instructional materials for teaching the same concept based on text excerpts that were two sentences long. These short excerpts provided enough context for students to understand how transition words are used, while still being short enough to put the focus on the keyword.
An example of the revised instructional materials. Two-sentence excerpts provide enough context for students to learn how to use the transition word “however.”
Outcomes
Survey data collected after we revised the materials showed that the two-sentence excerpts were more effective and easier for both students and instructors to use and understand. Overall, effectiveness scores increased from the first semester, when the materials were introduced, to the fourth semester, two years later.
Qualitative analysis showed that student participants found it empowering to learn from model texts written by other students just like them. Many looked forward to contributing their papers to the corpus so that other students might benefit from their work.
Students' average ratings of the effectiveness of the instructional materials steadily increased from the first semester of implementation to the fourth semester. In this part of the survey, students rated the effectiveness of the materials for reaching the goals of the course, achieving the goals of the specific assignment, and improving their writing in general. (Image credit: Dr. Shelley Staples)
Next steps
As an extension of this study, our team went on to analyze students’ written texts to identify differences in language choices between students who used the materials and students in a comparison control group. Our findings will be published in an international peer-reviewed journal in 2024.
Publications related to this study
“Applying learner corpus data in second language writing courses” in Second Language Writing News
“Adapting corpus-based materials for online teaching in L2 writing courses” in Second Language Writing News
“Building genre awareness through learner corpus data in a second language writing course,” forthcoming invited article for the International Journal of Learner Corpus Research
