Shortly before the end of 2010, the USFA sent out a series of e-letters regarding SEEQ. These e-letters touched on three concerns about SEEQ repeatedly raised by USFA members – the privacy of results and how they should not be used to compare units, statistical issues related to using the questionnaire in small classes, and the use of anonymous comments in collegial processes. We received a tremendous number of responses to our e-letters from members who had both positive and negative things to say about SEEQ and who raised a number of additional issues for discussion. This Collectively Speaking is written in response to the questions and comments received from members and attempts to clarify a number of issues regarding SEEQ.
What is SEEQ? The SEEQ (Student Evaluation of Educational Quality) is a standardized evaluation instrument developed by Marsh to assess student course satisfaction. It is one of the more widely used and validated instruments available to assess student satisfaction in post-secondary institutions (Gibbs & Coffey, 2001).
How is SEEQ structured? Using a five-point scale (from strongly agree to strongly disagree) and 32 standardized questions SEEQ assesses student satisfaction across nine dimensions of teaching. These dimensions and the number of questions associated with each dimension are: Learning (4), Enthusiasm (4), Organization (4), Group Interaction (4), Individual Rapport (4), Breadth (4), Examinations (3), Assignments (2), and Overall Satisfaction (3). A 33rd question has been added to SEEQ asking if the instructor should be nominated for a teaching award. These 33 questions comprise SEEQ.
Are there other questions that can be added to SEEQ? Yes, there are a number of additional questions that can be added to SEEQ at the discretion of the instructor or department. There is also an option for an open-ended question for students to provide comments and feedback. Note, however, that only the answers to the 33 questions mentioned above can be used anonymously for assessment purposes.
Is SEEQ a required course evaluation instrument? No, SEEQ is not a required course evaluation instrument. In June 2007 University Council approved SEEQ as a valid instrument that can be used to assess student satisfaction in courses. This does not mean that it is the only valid instrument or that it must be used. Council simply indicated that SEEQ is one of many possible validated instruments that can be used to assess student's satisfaction. If departments or colleges wanted to adopt a different validated instrument to assess student satisfaction they are free to do so. A number of departments (e.g., Psychology) use other instruments that are much more specific to their needs and equally well validated. Other departments (e.g., English) are in the process of developing and validating their own assessment instrument.
What is the difference between confidential and anonymous? Confidential refers to the fact that individual responses and written comments can be directly connected to the person who provided them, but the ability to make this connection is restricted. Restrictions usually include who can make the connections and under what circumstances. Thus when something is confidential there is a way of tracing the information back to the person who provided it. Anonymous means that individual responses and written comments can not be traced back to the person who provided them.
Are student responses on SEEQ anonymous? Student responses on SEEQ are not anonymous although they should be for the 33 questions comprising the formal SEEQ instrument. All answers can be traced back to students using their identification number. We have been assured, however, that this will not be done.
How can SEEQ results be used? The results from SEEQ can be used in one of two ways. The results can be used in a formative way to help instructors better understand student satisfaction in the courses they are teaching. If open-ended questions are included then the comments can also be used formatively to gain feedback on specific aspects of the course. When used formatively the results do not form part of the collegial processes for members.
The results can also be used for summative purposes. In this case the results from the standard 33 questions can be presented in a summative manner and used anonymously as part of collegial processes. If results from open-ended questions are to be used then they may not be anonymous. All written comments must be signed or in some way linked back to the person who made them. The comments remain confidential, but it is necessary to ensure that comments can be sourced so that the principles of natural justice can be applied.
How to interpret SEEQ results? In order to fully understand individual results they must be compared to an appropriate normative sample. For example, receiving a score of 2.1 out of 5 does not tell you anything about how well you performed on the item without an appropriate comparison group where the mean and standard deviation are known. There will be different comparison groups for different departments and for different levels of courses. For example, it is likely that there should be different comparison groups for 100, 200, 300, and 400 level courses. An instructor teaching a 300 level course should have his individual results compared to others at the same level and not all courses in the unit. The mean and standard deviation of the comparison group are needed in order to meaningfully compare individual results to the comparison group. In fact, given applied test theory it only really makes sense to present individual results in the context of confidential intervals along with a qualifier to indicate where the instructor's results fall (e.g., average, above average). Scores that are "˜floating' without this can not be reliably interpreted.
How should SEEQ results be used in collegial processes? SEEQ results should form one part of the information put forward regarding teaching. They are not a substitute for peer-evaluations. The results provide information on student satisfaction in one course at one point in time – typically at the end of a course. When SEEQ results are presented they should be presented in the form of confidence intervals and in relation to an appropriate comparison group. Descriptors should also accompany the numerical data (e.g., average, above average).
Open-ended comments and any other feedback can only be included if it is signed (i.e., can be attributed to the student who wrote them). It is important to remember that open ended questions are not part of SEEQ, they are extra, and can never be validated. These comments should also only form one part of an evaluation of teaching and should be provided in summary form. SEEQ results and responses made to open-ended questions should form a part of an overall comprehensive evaluation of teaching. For information regarding the use of teaching evaluation information in collegial processes please see articles 12.1.1, 14.5.7(ii), 15.12.8(ii) and 16.5.6 (ii) in the Collective Agreement.
Are there misuses of SEEQ results? SEEQ results should not be interpreted without an appropriate comparison group. Results should not be understood as single numbers that define teaching but as estimates of student satisfaction within a range of confidence limits relative to an appropriate comparator group. SEEQ results cannot be compared between departments or colleges until empirical research is done to assess variability between departments, establish norms, etc.
Are SEEQ results public or private? SEEQ results that can be traced back to an instructor are not public information. The information should only be accessible to course instructors and Department Heads (or Deans).
We hope that this Collectively Speaking answers some of your questions about SEEQ and provides you with a sense of the concerns raised by other USFA members in response to our e-letters. The USFA is currently in discussion with the Employer about ways to effectively address concerns about SEEQ and its use at the University. We will keep you posted.
The USFA continues to be interested in hearing from you regarding SEEQ. We invite you to share your thoughts with us. Send us an email (usfa@usaskfaculty.ca), call the USFA office (966-5609) to arrange to meet with a member of the USFA Executive, or speak to your Faculty Association Representative.