Previous Section | Chapter 2 Table of Contents | Next Section
from Making Meaning
Making Change, by Elsa Roberts Auerbach
One of the most important ways to develop student participation in curriculum development and evaluation is by structuring ritualized procedures, built in on a regular daily, weekly or monthly basis, for collecting and reflecting on student work. These activities give participants immediate feedback about the effectiveness of learning and teaching, allowing adjustments to be made along the way; in addition, they provide a basis for reviewing progress at the end of the cycle.
In-class goal-setting activities
Early classes can focus on tasks designed to elicit student conceptions
about language learning, student-teacher roles, and student and teacher
goals for the class. Many of the activities described in Chapter 4 can
be used in this way. In addition, "The English Class" (Unit
I, Lesson 3 of ESL for Action, Auerbach and Wallerstein, 1987)
includes activities to generate discussion about reasons for studying
English, attitudes toward language learning, and a model for ongoing evaluation
of each lesson. Andy developed a lesson (Nash, 1989) based on group responses
to "It's easy to learn when_____; It's hard to learn when_____."
From this, a set of class resolutions emerged as the basis for evaluation.
Action evaluations
Once students have addressed a problem through individual or group
action (like testifying at a funding hearing, dealing with issues of family
or classroom dynamics), they can reflect on what happened with dialogue
questions like: What happened? How did you feel about it? Why did it happen
this way? What might you do differently next time? What new issues have
arisen from this action?
Charts/checklists
Students can make charts reflecting what they can and can't do,
do and don't like, want and don't want to learn. These can be done individually
or as a group, daily, weekly or monthly. East End Literacy (1990) and
Nunan (1988) include a range of checklist formats that can be adapted
for particular groups or students. The
following questions are adapted from a format suggested by Nunan (1988,
p. 134):
This week I studied__________.
This week I learned__________.
This week I liked____________.
This week I didn't like:________.
This week I used my English in these places:
This week I spoke with these people:
This week I had difficulties with_______.
I would like to know/ work on_________.
My learning and practicing plans for next week are_________.
Individual student journals
Journals can be used for assessment in a number of ways. Balliro
(1989) suggests building in 15 minutes at the end of each class in which
the teacher and students each write journal entries. Students can reflect
on their own learning, assess their progress using English, and report
on accomplishments, writing about reactions to classroom experiences,
interactions using English outside of class, family interactions, or anything
else that is on their minds. As they develop, journals provide concrete
evidence of students' progress. Teachers can evaluate them in terms of
criteria like range of topics/ content, elaboration of ideas (including
use of details, examples, depth of analysis, emotional force, etc.), length
of entries, grammatical development (specific forms like tense markers,
fragments, etc.) and coherence as well as in terms of students' own perspective
on their learning. Students can use them for self-evaluation by reading
and responding to the finished products, noting changes and areas needing
work.
The group journal
Sauve (1987) describes a group process whereby everyone contributes
reflections at the end of each class in response to questions like, "What
happened today? What did we do today? What did we learn today?" This
provides a sense of the differing perspectives in the group, forces the
group to name what they have done, and encourages collective responsibility.
It can be done as an LEA activity; journals can be collected as a class
history.
The posted journal
Charo's class used an evaluation procedure that involved posting
a sheet of newsprint in class with the word "Accomplishments"
at the t op and two columns, one called "In Class" and the other
called "Out of Class".
Whenever anyone had something to report that they felt good about, they
wrote it on the list.
Class newsletters
Andy developed a class newsletter in which she summarized the
activities of the week as a vehicle for reflecting on learning and discussing
accomplishments. She included points covered in the lessons (grammar,
readings, etc.), reports on class discussions, attendance, and accounts
of individual students' problems or achievements. The newsletter served
an number of functions, from reviewing lesson content, to becoming a reading
text, to catalyzing action about class issues (like attendance), to documenting
the progress of the class and becoming an evaluation tool. See "Our
Class: A Weekly Literacy Ritual" in Talking Shop (Nash, et
al., in press).
Portfolios
Student writing can be collected in individual portfolios that
include everything from informal free-writing assignments to all the drafts
of each piece from the beginning to the end of the cycle. These become
records of development that both teachers and students evaluate periodically
or at the end of the cycle. Students then see concrete representations
of their own growth and they can be asked to comment on changes they note.
Teachers can look for development of spelling, grammar, coherence, organization,
elaboration of ideas, etc.
ANECDOTES
Hemmingdinger (1988) identifies anecdotes as an important tool for legitimating the many ways that literacy changes students' lives. It defines an anecdote as "an account of someone, describing what you noticed about the student in the beginning and how the student has changed since then" (p. 128). These stories describe changes that don't show up in direct paper-and-pencil-assessment procedures including affective changes in self-confidence, openness, group participation, ability to make a living, etc. In a participatory approach, they go beyond personal changes to include the ability to use literacy to address social problems: to work with others to make changes in family and community life.
Anecdotes serve two functions. First, they provide valuable feedback to the learner; because of this, they should be written with the learner in mind, using language that is accessible and content that can be shared. Second, they are a means of reporting changes to others in a systematic format. To accomplish these goals, East End Literacy suggests that anecdotes have two components, one that is descriptive and another that is analytical. The former tells the story of the incident indicating change, comparing the student before and after. The latter entails labeling or categorizing changes to provide a schema for documenting them. This is Not a Test: A Kit for New Readers (East End Literacy, 1990) describes in detail how to identify, keep track of, and summarize changes. It suggests a format in which the description is written on the right hand side of the page with corresponding categories of changes listed alongside them in the left margin. Categories of change can be summarized at the end of a collection of anecdotes with short phrases providing examples of each category.
LOOKING BACK: END OF CYCLE EVALUATIONS
The following activities for the end of the cycle invite students and teachers to reflect on what they have and haven't accomplished; again students should have the option of doing these in either English or their first language.
Peer interviews
Students can interview each other using questions they have generated
collectively. These may be framed in terms of initial goals or more general
questions like "What are the most important things you learned in
this class? What can you do now that you couldn't do before? What changes
have you made since you began this class? What did you like most about
this class? What should be changed about the class?" Students can
report each other's answers to the whole group and compare impressions.
Student-teacher conferences
Students and teachers can use the same questions they started
with at intake, comparing before and after responses.
Review and repetition of
earlier tasks
Students can review their portfolios, journals, and coursework
to see changes. They can repeat reading and writing sample tasks and compare
results; this can be done individually, with peers, or with the teacher.
Student self-evaluations
Students can use chart, checklist or narrative writing formats
to evaluate their own learning. If they used checklists to identify goals,
interests, and needs earlier, they can come back to these and determine
whether their goals have been met and what they still need to work on.
See Nunan (1988), pp. 131-134).
Class evaluations
Students can be invited to provide feedback about the class either
during or at the end of a cycle. Because students are often reluctant
to express negative feelings or criticism, questions should be impersonal;
students can be asked to write anonymous evaluations or work in groups
so that no individual's ideas are identified. In addition, it helps to
have specific questions about what participants disliked or would change.
For a beginning class, Ann used questions like "How do you usually
feel in class? Is the class too easy or difficult for you? What could
improve the class?" At a higher level, she asked, "What kind
of atmosphere did you expect to find in classes before you came? What
did you find that you didn't expect to find? What didn't you find that
you expected to find? In what situations did you use what you learned
in the class? What were the games you learned the most from?
Program evaluation
Students from various classes can come together to discuss programmatic
issues like class structure, content, use of the native language, child
care, scheduling, class size, groupings, and funding concerns. Ann Cason
details the processes and benefits of this kind of evaluation in Talking
Shop ("All-Program Evaluations").
TEACHER RESEARCH: ONGOING DOCUMENTATION PROCEDURES
Finally, teachers can document their own curriculum development processes through the following:
- Retroactive lesson plans
- Teachers' journals
- Tape recordings
- Monthly reports
- Minutes of meetings
The cornerstone of qualitative evaluation is documenting what is happening in the classroom as it happens. As Johnston (1989,p. 509) says,
"Central to this approach is the teacher's ability to know the students, and to notice and record their development in a variety of areas...the ability to set the conditions for and to notice patterns of activity and changes in those patterns is at the heart of the teacher's evaluative skill.
This kind of evaluation provides the context within which to understand students' progress, the basis for curriculum decision-making, and a record of changes that can become data for further analysis. Thus, teachers become researchers of their own classrooms.
Both the process and the product of documenting curriculum development serve important functions. Since documentation is done in an open-ended, descriptive waycollecting and recording data without predetermining what to look for (as in ethnographic research)the process itself becomes a vehicle for listening to and valuing the unexpected. It enables the teacher to stand back from the immediate moment and reflect on it, which, in turn, may lead to new insights about patterns and issues. Just by writing or talking about what is happening in their classes, teachers gain new understandings of why it is happening and what to do next.
Very often, however, insights about what is happening don't come until much later. It may not be clear how to use information as it is being gathered. Thus, the ongoing accounts can serve a retroactive function, becoming "data" for future reflection; they provide invaluable information as recorded histories of class cycles, student progress, and teacher thinking. Teachers in our project, for example, collected teaching materials, student writings, notes, and journal-like descriptions of particular teaching cycles. While they were in the middle of the cycles, they had one perspective on them but when they wrote about them later, their perspectives changed: "Putting our experiences down on paper, we have been forced to reflect on them as we may not have previously done. In this way, we've learned not only from each others' writings but from our own as well" (Nash et al., in press). Likewise, it was the cumulative, detailed documentation of day to day activities, discussions, and student work that provided the data for the analysis which has emerged in writing the curriculum guide. There are many forms of documentation ranging from very structured and schematic to open-ended and flexible. Some that we have used are described on the next pages".
Retroactive lesson plan
forms
A uniform format for keeping track of activities and student
reactions to them can be used on a daily or weekly basis. We started with
a documentation chart with columns for noting where themes came from (catalysts),
how they were developed (tools / activities), how language work was incorporated,
what new issues emerged, and teacher's reflections. Although teachers
in our project felt it was artificial and inhibited rich description,
others might find it useful, particularly in moving from a traditional
to a participatory approach.
Teachers' journals
Teachers
in our project felt that journals were a more organic way of documenting
day to day classroom interactions. They are more open-ended and personal.
Journal entries, taken together, become a kind of history of the development
of the curriculum as well as a record of particular events. While they
at first seem time-consuming, the payoff is worth while, as Lucille Fandel
(one of my graduate students at the time) writes here (personal communication,
May, 1990):
Taking time to keep a journal after every class made a qualitative difference in my teaching, helped that lessons flow better. Before, I'd do a lesson plan based on how much we'd "covered" of my previous plan. With my journaling I've found that I remember significant things that happen (often very fast, in a fleeting way) in class. This way I can pick up on them. They are the "stuff" of people's lives / of our personal interactions / of their evaluations of the class or of their own progress. Journaling has helped me see more clearly where the "flow" is going.
Family literacy teachers kept journals in which they described activities, student reactions, issues, or concerns that arose for them, their own reactions or reflections about the interactions, and ideas for future lessons.
Tape recording
Taping classroom interactions can provide raw data for future
analysis. Madeline used tape recording for the purpose of monitoring small
group discussions to get a sense of how one group was doing while she
was working with another. Listening to the tapes revealed both interesting
student issues and areas for language work. She shared the tapes with
the whole class in the form of transcriptions. Thus, students could themselves
reflect on the interactions, exploring both the content and linguistic
aspects together. In this way, the documentation fed into instruction
and students were involved in the analysis.
We also used tapes to compare reactions to materials and activities. When teachers used the same code to generate dialogue about homework, Andy taped each class on the day it was used and transcribed the tapes as a basis for teachers to analyze and compare responses. Since each group reacted differently depending on how the lesson was introduced and who the students were, the transcription enabled teachers to see concretely how the context shaped dialogue.
Monthly reports
Summaries of the month's activities provide an overview of activities,
accomplishments, and issues for groups. The process of writing these reports
can be a framework for reflection, a time for teachers to think about
what they've done and where they're going. The reports can also provide
a place to communicate problems, needs, and concerns to program administration.
Depending on the point in program development, these might address questions
about recruitment, intake, students, classroom activities, critical incidents
or anecdotes, new insights, ideas, or issues (what you've learned about
your own teaching, reflections, goals, challenges). It is also helpful
to include examples of materials, student work, journal entries, and retroactive
lesson plans.
Minutes of meetings
There are several limitations to these forms of documentation
done by individual teachers. First, they take more time than many teachers
have. Second, while each may reflect what's happening in particular classes,
separately they may be fragmented and fall
short of reflecting the broader picture for a project. Third, because
by nature they are the result of an individual process, they preclude
the kind of insight that comes from collective analysis. Thus, the dialogue
that takes place in teacher meetings and the recording of that dialogue
through minutes are central to the curriculum development and evaluation
process. The meetings provide a framework for the development of a "community
of knowledgeable peers" (Balliro, 1989) and a context for the program-based
practitioner research called for by Lytle (1988). The minutes provide
a detailed, sequential documentation of what transpires in this process;
they become the thread that ties together individual accounts.
PUTTING IT ALL TOGETHER: A CONTEXT-SPECIFIC APPROACH TO EVALUATION
Although the system of evaluation
proposed here isn't neatly packaged into a ready-made sequence we hope
it will be useful as programs construct their own evaluation schemes.
Specifically, we hope that its rationale and principles can help to challenge
demands for standardized testing and to justify qualitative approaches
to evaluation, that the "toolkit" can be a resource as programs
select and adapt tools for their own contexts, and that the documentation
processes will help teachers gather information that is useful for their
own curriculum development as well as for reporting to others (administrators
and funders). Most importantly, however, the results of this kind of evaluation
will help to develop the field of adult education. As more and more practitioners
document, analyze, and share what they're doing, our collective understanding
of what does and doesn't work will grow so that research that comes from
inside the classroom will become the basis for constructing and extending
our knowledge about adult learning.