Category Archives: CoursEval

At the request of the SNHU Faculty Senate I attended the February 17th meeting to discuss concerns with the online delivery of course evaluations using CoursEval. This invitation gave me the push (deadline) I needed to complete the comparative analysis of paper and electronic delivery of evaluations which I had begun and address some of the concerns raised by faculty.

For a brief history of CoursEval at SNHU, please see my December 17th blog post.

Some of the concerns that have been raised by faculty since the implementation of CoursEval include:

Comments taken from December 16, 2009 Faculty Senate Minutes:

• • The response rate is low – does not provide enough information regarding the course
  • Paper evals were anonymous but this [online] is so anonymous that students will say more and maybe be extra critical
  • Respondents may say things online that they would not say face to face, or in a classroom environment
  • There are too many highs and lows, not enough middle comments
  • Faculty does not mind being evaluated, but these are not valid
  • Rate of return is a real issue
  • Do we need a “carrot and stick” approach to increase participation? That might really help. Ways of encouraging students to respond to the surveys: do not display student’s grades unless the survey has been completed by the student.

While these are legitimate concerns, I wanted to see if there was anything to them by conducting a comparison between paper evaluation data from Fall 2008 and electronic evaluation data from Fall 2009. Other CoursEval universities including SUNY Buffalo, Purdue, U. of Miami, U. of Texas – Austin, and BYU have conducted similar analyses. Click here to see a summary report. How does SNHU compare?

To conduct an analysis I asked each school at SNHU to provide me with a sample of evaluation data from Fall 2008 to use as my “paper” sample. The only criteria is that the evaluations had to be from faculty who taught the same courses in Fall 2009 which was my “electronic” sample. I received evaluations from the School of Liberal Arts and the School of Education for a total of 14 course sections. “Paper” evaluation data was not received from the School of Business or School of Community Economic Development. The College of Online and Continuing Education (COCE) was excluded from analysis as they recently changed their evaluation instrument so a legitimate comparison was not possible. Of the course evals received, only 12 sections were usable as 2 were for instructors or classes not also taught in Fall 2009. Data from Fall 2008 was matched to data from Fall 2009 by instructor and course and independent sample T-tests were completed on each pairing. The entire 2008 sample was also compared and graphed against the 2009 sample. Comments were also analyzed for quantity and type (positive, negative, neutral).

Findings:

• Response rates were lower for electronic evaluations using CoursEval than for paper evaluations. For the Fall 2008 paper sample the average response rate was 89% with a high of 96% and a low of 71%. For fall 2009 the average response rate was 64% with a high of 94% and a low of 18%.
• Answers to the the Likert Scale questions (questions 1 – 19) showed no statistically significant difference between answers given on paper and those given using CoursEval. Responses tended to be more positive on the electronic evaluations than on the paper evaluations.
• Comments (question 20) were slightly lower in quantity on the electronic evaluations but showed no difference in the proportion of positive, negative, and neutral responses.

Response to concerns (see above):

Even though the sample size was small (12 Fall 2008 courses and 12 Fall 2009 courses), the findings are consistent with those found at the other universities mentioned above.

• Response rates are lower but this does not seem to have a negative impact on the quality of evaluation data
• The transition from paper to electronic evaluations has not resulted in lower or more critical ratings of faculty and their courses. If anything the ratings may have gone up. The comments are not more critical or negative
• The distribution of “highs”, “lows”, and “middles” remained the same
• Not really sure what the “these are not valid” concern means and why the electronic evaluations would not be valid
• Rate of return for electronic evaluations is lower than paper. While this doesn’t have a negative impact, it would be good to increase response rates. Higher response rates will likely come with familiarity as has been seen with COCE evaluations whose response rates have increased.
  • We can increase response rates by encouraging student participation. If faculty explain to their students the importance of the evaluations and the value they place in the feedback then response rates will likely rise.
  • We cannot tie access to grades to evaluation completion.

A copy of the slides presented to the SNHU Faculty Senate can be accessed by clicking here.

Overall, the transition from paper to CoursEval appears to be a positive one. There has been no negative impact on results as feared, faculty have quicker access to evaluation results, staff efficiency has increased, and the ability to secure and access evaluation data is better.

You can see more information about CoursEval at SNHU by clicking this menu above:

If you have any questions or comments, please post them by clicking the comments link at the top of this post.

 

As the fall terms draw to a close, I want to take this opportunity to provide an update on our new course evaluation process using CoursEval.

First a brief recap… in September 2009 the university transitioned from paper based course evaluations to an online system called CoursEval, a product of Academic Management Systems. This product was chosen after extensive research and a pilot in the spring of 2009 for its scalability, flexibility, and reporting. Instructional Support spent time over the summer getting CoursEval configured and began using it to deploy all course evaluations starting in September with COCE Term 09EW1.

Where are we now… since September we have deployed 21 separate course evaluations to a little over 29,000 students. Note that “students” doesn’t mean 29,000 distinct students. This is the cumulative number of registrants across all classes.

The overall response rate for all of these evaluations is around 55% with the response rates varying by term, location, and class. A more detailed breakdown can be found on our blog’s CoursEval page. More detailed stats will be posted in the coming weeks. Making the transition from paper to online, we expected the response rate to be a bit low and this is something that we will work on and anticipate will increase as the new process becomes fully integrated into the university culture.

From a staffing/budgetary/administrative standpoint CoursEval appears to be a good investment. Paper evaluations required the time and effort of approximately 12 employees and 4 work study students across campus to print, distribute, collect, scan, and transcribe all submissions. Add to this the cost of paper and ink. The same evaluations using CoursEval requires less than one full-time equivalent employee and no paper. The Instructional Support team (3 people) is now managing CoursEval as part of its normal workload. While it has increased our workload somewhat, it has not created an unreasonable burden. Also, having one centralized system means that university-wide data is available quickly. It no longer has to be collected from various locations across the university. The staff that did manage evaluations have not been negatively impacted by this change. They have been able to devote their time and energy to other projects important to the university and from what we hear they don’t miss the evaluations.

Course Evaluation Results… Once the deadline date for final grades is reached, course evaluation results will be released to faculty. Each faculty member will receive an email with instructions for viewing their course evaluation results online in CoursEval. The email subject line will be “Your course evaluation results are available”. Term 09TW1 faculty were sent an email on Thursday, December 17th inviting them to review their results. Faculty who taught in Terms 09FADAY, 09EW2, 09EW1-16wk, and 09SW1 will receive an email on the afternoon of Tuesday, December 22nd inviting them to review their results.

Where are we going… CoursEval will continue to be used for all course evaluations. Some tweaking of system settings and reports will be made on an on-going basis. For spring terms we hope to implement some additional features such as evaluations for student teaching/internships, incorporation of custom questions from faculty, report triggers and more.

Any comments or suggestions on the course evaluation process and CoursEval are welcome. If you have used CoursEval as student or faculty we would like to hear your thoughts. Please post a comment on this post using the “Leave a comment” link above. If you would prefer your comments to be private, please email them to me at a.flint@snhu.edu

Thank you to everyone who assisted with the process of implementing CoursEval.