opinions-organizations-teacherEvaluations

Some thoughts on student evaluations after reading the essay, Student Evaluations: A Critical Review. Disclaimers: I haven't studied the literature on this subject. Also, I'm a graduate student.

THE PROBLEM TO BE OVERCOME

My general feeling is that the current provides insufficient mechanisms to ensure that learning is optimized.

I can think of five ways in which classes have been suboptimal:

(1) Waste of time

Some classes that I have taken have been very suboptimal, in that me and most of the other students spent a couple of hours a week going to class and yet learned very little in that time, or, similarly, in that me and most of the other students were assigned things that seemed to not be a good use of time.

Not to say that the ENTIRE class was a waste of time, just that I could have learned everything that I actually learned in a small fraction of the time spent.

(2) Teacher is incomprehensible Some people just aren't very good at explaining thing. Also, some people have a strong accent, a very very soft voice, or a poor command of English that makes it hard for students to understand them.

(3) Too much work

Some classes were suboptimal in that everything said and assigned was very interesting, but that the total amount of work was so much that I was miserable during the time I took the class, and also I was constantly forced to choose between fulfilling the minimal requirements for that class and the minimal requirements for other classes.

(4) Good for some students, bad for others

Some classes are suboptimal in that many of the students learn a lot and are happy with the class, but a large minority of students don't learn very much and are unhappy with it, and that

(5) Bad tests

Regardless of whether the course time is good or bad, sometimes the tests are bad. By which I mean, the testing performance is relatively uncorrelated with the amount learned in the class. This means that something is wrong: if you think the purpose of testing is to encourage students to do the work, then it isn't serving as a very good motivator when students who didn't learn anything often do better than students who worked really hard; on the other hand, if you think the tests are a gold standard designed to certify who knows what, and the purpose of the class is to help learn the stuff, that the class was not taught well if it did not help the students learn what they were supposed to learn.

Some more comments/examples of some of these three failure modes:

(1) Waste of time

I've been in some classes where the professor just didn't bother to spend sufficient time preparing for classes. Often, the result is that the class is impossible to follow, and time is wasted sitting in class listening to a disorganized and incomprehensible lecture, although little is learned.

Other classes have assigned readings which were very long and not very useful.

Other classes have had 3 hour long lectures on technical topics. Generally, no students could pay attention after the first hour and a half. So, half the time spent sitting in class was wasted.

(4) Good for some students, bad for others

For example, many times there are multiple subpopulations of students taking a class, with different backgrounds. Sometimes a professor skips some prerequisites for some material, which is fine for one subpopulation of people who already know what is skipped, but disasterous for another subpopulation. (some would say, well, that subpopulation shouldn't be taking the class, but what if that subpopulation is required to take that class, at that time in their program? in this case it's pretty clear that the department expects these people to learn what they need to within that class)

Other times, the professor doesn't like some students for weird personal reasons, or other times only likes the very best students (for instance, i know of one professor who responded to very on-topic emailed questions by publically making fun of the question-er in class for being "so behind that you don't understand even this basic fact" -- and then not answering the question. this was a question that many students had been wondering about, but were afraid to ask -- with good reason, it turned out).

SUMMARY OF THE PROBLEM TO BE OVERCOME:

Sometimes classes:

• provide a small amount of learning per unit time
• don't make any sense
• take too much time
• are optimal for some students but very suboptimal for many others
• give tests that do not correlate with the amount learned

OTHER WAYS TO OVERCOME THE PROBLEM

Besides student evaluations, some other ways that faculty's teaching is assessed (that I am aware of) are:

• Faculty members asking students whom they trust what their opinion is. I think this is a good method but has problems with small sample size and bias (bias because "students who they trust" are sometimes smarter or better prepared than the median student). But it really isn't that much different that just doing the surveys, except that less people are being surveyed.
• Other faculty members sitting in on one or two lectures. I think there are many problems with this. First, the "sitting-in" faculty don't do the homework and don't take the tests, so they don't evaluate huge parts of the course (in some classes, the time spent on the homework and studying dwarfs class time). Sometimes, the sitting-in faculty may look at the homework and tests and say, "that look allright", but this will lead them to miss nuts-and-bolts problems that come up in practice, like, "gee, the weekly problem sets are interesting and they look easy but they took me 20 hours to actually do them and write them up", or "hmm, these problems looked good but when i tried to do them i realized that half of them were written ambiguously or incorrectly, in small ways that made it impossible to answer them", or, "huh, the professor didn't grade the first weekly problem set until the last week of class".

Second, since the sitting-in faculty doesn't attend the entire course, they can't tell if the professor has adequately explained the prerequisites for whatever they were talking about the day that the evaluator attended. Perhaps the evaluator hears the professor talking about X, and in order to understand X you must already understand Y, and the evaluator assumes, "oh, s/he must have talked about Y last week", when in fact Y was never mentioned. Or perhaps the professor tries extra hard to prepare well for the lectures when the evaluator attends, but in general does not adequately organize hir lectures. Or perhaps each lecture is good, but the course only covers 2/3 of what the department expects it to cover, leaving the students underprepared for future courses. Or perhaps the professor spends 2 weeks presents their own minority viewpoint of some controversy as the unquestioned consensus of the field (but not the weeks when the evaluator is there, of course). Or perhaps the professor frequently cancels class and reschedules it for inconvenient times like 6am or 8pm.

Third, it's always hard for someone who knows the subject to gauge how good a lecture is at teaching it to someone who does not know it. Often it is exactly those lectures or reading assignments which students find too fast and dense to follow that other professors will compliment as "going straight to the heart of the matter, while also generating excitement by covering current controversies and ongoing research".

• Looking at grades on the final exam. This can catch some but not all forms of suboptimality. I expect that in the worst case, if very few of the students are learning the material, this may be seen in the exam grades. However, this won't catch the following problems:
  • Wasting of time. The final exam won't show if students had to spend 3 hours reading or listening to irrelevant or incomprehensible material for every 1 hour of learning.
  • The class was too hard. Evaluators might see mediocre grades on the final and moderately hard questions, not realizing that the breadth of material covered was so great that students have been spending too much time on this class in the last few months.
  • The class was good for some students but very bad for others. Unless extensive statistical analysis is done, the cluster of grades from students for whom the class was ill-suited will probably be indistinguishable from students who performed poorly because they didn't try hard.
  • Bad test. If the test measures a significant amount of knowledge that wasn't even covered in class, but if a substantial proportion of the class already knew that material going in; or, if the test is sufficiently easy; then the test grades could conceivably be distributed in a bell-curve with a believable average.

HOW STUDENT EVALUATIONS HELP

So, basically, student evaluations provide a way to detect all of these sorts of problems that other methods do not detect.

BUT WHAT ABOUT THE ARGUMENTS AGAINST STUDENT EVALS?

Namely, that their results are contaminated by other factors unrelated to amount of learning, for instance the speaking style of the professor. Well, be that as it may, they are still the only way to get information on the problems mentioned above.

RECOMMENDATIONS FOR IMPROVING STUDENT EVALS

Some people recommend that the evals focus on essay answers rather than numbers. I agree with this to an extent, however, when there is a big problem with the way a class is taught, I expect that it is hard for faculty to believe the extent of the problem unless they are faced with very clear evidence. Having 30 essay answers (which most of the evaluator faculty will only skim) probably won't be as persuasive as having a numerical tally of answers to questions ("80% of students polled said this person was a "very bad" speaker").

Another recommendation is for the questions to be very specific, i.e., "does the professor arrive on time" rather than "is the professor good?". This is also good, but there also still needs to be some general questions to catch problems which the survey-writer didn't think to ask about.

Another recommendation I would make would be to create a process where the students in the class collectively (electronically) add two or three questions to the eval survey for their class. This would allow questions to be specific, while also providing a mechanism to catch issues that the survey-writers didn't anticipate. For instance, students might add a question, "Were questions on homework assignments frequently riddled with errors?".

ANOTHER RECOMMENDATION

I propose taking student evaluations very seriously. But I have a further recommendation; have faculty member evaluators sit in on THE ENTIRE CLASS, and be responsible for doing all of the work, and all of the tests (their homework and tests will even get graded, preferably blindly; these grades will be used to assess the EVALUATOR to make sure s/he is putting sufficient effort into hir evaluation duties). This will deal with the problems mentioned above with "tourist" evaluators. Many of the failure modes could be caught by this method:

• waste of time: other professors will be very aware when they are forced to waste time listening to or reading something they deem worthless
• don't make any sense usually anyone can tell when someone's speech or presentation is simply incomprehensible.
• take too much time this one might be hard to catch sometimes, because professors who know the material already can do many assignments much faster than students who don't. But even so, I bet that about 60% of the time, other professors will become aware of an overly large workload if they have to actually do it.

The major issues that cannot be caught by this method are only:

• Class moves too fast
• Class skips background material Because the evaluator will already know the background material, and may not be conscious of hir application of it.
• Good for some students, bad for others

A further refinement would be for the evaluator to be from a different but related department from the teacher of the class -- so that, hopefully, the specific material in the class will actually be new to the evaluator.