Its not all about the data: teachers’ experience of the CES

Post by Dallas Wingrove

Image by pgcap

The previous post, Do students read your feedback? looked at the issue of feedback from the student perspective.  This post considers the view of the teacher, and particularly in relation to the Course Experience Survey (CES).

Conversations with teaching staff have driven home that the focus on teacher scores and rankings (GTS & OSI)  and the design and administration of the CES can lead to significant degrees of personal and professional angst for some staff. What is heartening though is that teaching staff have begun to share their experiences and concerns, moving beyond self-deprecation and feelings of deflation that can occur when the CES report is received following the semester’s teaching.

The purpose of the CES is to systematically capture feedback from students about their course experience to in turn provide feedback to assist teachers to continue to improve the quality of learning and teaching in their course. Yet whilst the CES, and the data it produces, carries so much weight, particularly in relation to university systems of academic promotion, teaching awards and wider career development there is some work to be done if the survey is to foster improvement in teaching practice and for CES data to provide useful feedback to teachers.

Following discussions with staff at my School’s Learning and Teaching Committee it was decided that a project team be formed to investigate how to do things better. As discussions about the CES unfolded in the school, a colleague and I starting exploring whether there was room for changes to improve how the survey was administered and ultimately experienced by both staff and students. We discussed the issues with teaching staff and had a series of follow-up one on one discussions which resulted in the development of a CES Project brief. Our project brief includes a summary of the views staff expressed regarding how they experience the CES.

Fundamentally, teachers expressed:

  • Feelings of disillusionment, and concerns in relation to the rigour of the survey, its design and implementation
  • Feelings of concern about the rigor of a survey where the very staff administering it are not briefed about, nor supported to engage with its purpose and its role in improving learning and teaching
  • Concern as to the use of the Likehart scale, with not all points on the scale marked on the CES
  • A strong sense of disenfranchisement with the survey questions having been delivered top down, with no input from teaching staff on the ground
  • Questions as to the appropriateness of the survey questions for the first year cohort
  • The experience of receiving the CES report demoralising and disillusioning, with no guidelines for Heads of School/Senior Academic Management regarding how they debrief with teachers about their scores and the CES data the survey generates.

I suspect that what staff in this school articulated both in relation to design and administration for the CES is common, including how student feedback is communicated to them. As this university moves to a full on-line administration of the CES in 2013, there are some burning unanswered questions. And I hope some unrealised opportunities for lasting positive change.

To begin with we decided to explore further the issue of survey administration in the school. This we believed had relevance not just to the current roll out of hard copy surveying, but perhaps also to the next phase of the CES as this university moves to a near 100% on-line surveying of its students by 2013. The administration of the CES in the school in which I work is delegated to administration staff. I understand from colleagues in some other schools across this university that this is common practice. In reality, this means that the administration of the survey really is ad hoc since without briefing, induction, many variables can come into play.  Commonly, the survey is devolved to admin staff who may not be appropriately briefed as to the design purpose and significance of the survey, and who may or may not introduce the survey in accordance with the guidelines.

From the student perspective, it must be at the very least perplexing to be faced with surveys for which there is minimal context, nor rationale, and yet which also carry such weight which they may or may not be aware of.

We met with our Head of school and the College Associate PVC Learning and Teaching; both were supportive, with the latter referring us to practice within the Australian sector in which undergraduates are engaged in the course administration process. Our next step was to meet with the Director of the university’s Student Survey Centre. This discussion highlighted the siloed ways we practice in higher education as the centre has, through no fault of its own, little interface with teaching staff.  We learnt much, including that the university is reviewing its administration of the survey, and already changes have been made to the Graduate Destination Survey Likehart scale with the middle point now clearly marked as neutral. Given the close alignment between the GDS and the CEQ, and in turn the CES, it seems that some positive change may be around the corner.

The three of us agreed to develop a project involving the administration of the hard copy CES which would in turn inform the process for on-line delivery. This would involve volunteer undergraduate students inducted as mentors, which we intend to enhance rigor and consistency in the administration of the CES and we hope an enhanced understanding from our students as to the importance and purpose of their feedback. We have committed to embark on a pilot in which we will compare uptake, response rates etc from the hard copy and on line sample within a course, with findings to inform the roll out of the administration of the CES on-line. It may be that our students who completed the survey on-line talk to one another via video prior to commencing the survey for example.

What can be left as an unspoken in tertiary education is how teachers experience the CES, both in terms of how the data is collected and received. Our project is only emerging, but it highlights that teaching staff need to feel supported and listened to, to put on the table their experience of the CES. As we move into a world of increasing quality assurance, it’s a win for all if the instruments we use to measure the quality of teaching and learning replicate rigor in their design and administration.

Do students read your feedback?

Post by Kylie Budge

Image via UBC Library

When asked this question, many teachers would probably be tempted to respond “Not likely!” or “I can’t see any evidence of it”. Even though it may feel like students don’t read our feedback on their work, take on board our comments, or value it in any way it’s useful to look at what the research in this area tells us.

Most of us are probably aware that students report a great deal of dissatisfaction with the feedback they currently receive on their work. This is a sector-wide phenomenon, not one just linked to your university. A colleague and I were involved in some local research on this topic recently and discovered some interesting information (see references below). In doing this research we found that student feedback surveys in Australia and the UK report student dissatisfaction with the quantity, quality, and timing of feedback. While there has been quite a bit of research into feedback generally, until recently little was known about how students feel about the issue.

What we’ve been learning is this: students value feedback on their work when the timing and frequency, quantity and quality, and the form that feedback takes is considered.

Timing is critical in terms of students being able to apply the feedback in their work. Feedback early on in the semester is very important to first year students, but all students can benefit from this too.

Students are saying they want constructive, quality feedback that tells them what they need to improve on rather than just an indication of what they did right and/or wrong.

Feedback can of course be provided to students in number of forms including verbal face-to-face (teacher to individual student/teacher to group/peer); hand written (teacher to individual/teacher to group/peer); and electronic feedback (teacher to individual/teacher to group/peer). A good feedback strategy will use a combination of different methods, including peer feedback, to encourage students to seek and use feedback from a variety of different people (ie. not just the teacher). Teachers are busy people with lots of competing demands on our time. A feedback strategy with multiple components can help us provide the feedback students need for learning in a manageable way.

Interestingly, the discipline context is also important in terms of how students value and use feedback on their work. The little research that has been done in this area from the student perspective tells us that students from creative disciplines (such as art and design) value feedback highly. Students in creative disciplines are engaged in an active feedback culture (where work critiques with their peers and lecturers is common) and often producing a product (of some description) where feedback on work-in-progress is critical. They are often eager to get feedback and value it because they are also immersed in a discipline culture where it is seen as everyday practice.

This may or may not be the case with the way students see feedback in other disciplines. Research in this area is limited so time will tell us more.

What do you think? Do you have the sense that students read and apply your feedback? And what feedback strategies work for you?

Here are some useful references if you want to learn more from recent research on feedback:

Boud, D., Cohen, R., & Sampson, J. (1999). Peer Learning and Assessment. Assessment and Evaluation in Higher Education. 24 (4), December, 1999.

Budge , K. and Gopal, S. (2009). Feedback: working from the student perspective, refereed conference paper presented at Assessment in Different Dimensions, 2009 ATN Assessment Conference, RMIT University, Melbourne, Australia, 19-20 November.

Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research. 77 (1), 81-112.

Nicol, D.J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31 (2), 199-218.

Rowe, A.D. & Wood, L.N. (2008). Student perceptions and preferences for feedback. Asian Social Science, 4, 3, 78-88.

Rowe, A.D., Wood, L. N. & Petocz, P. (2008). Engaging students: Student preferences for feedback. 2008 HERDSA Conference Proceedings, 1-4 July 2008, Rotorua, New Zealand.

What Changes our Teaching?

This is our second guest post by Felicity Prentice.  Felicity has morphed from being a Lecturer in Health Sciences, to Primary and Secondary teacher, to Marketing in the Commercial World, to Curriculum Designer, to currently being an Educational Developer in DSC College RMIT University. Apart from her career indecisiveness, she is passionate about learning in any space.

Image by: Memotions from Flikr

There have been a number of drivers and critical incidents that have changed my approach to, and practice of, teaching and learning. Each powerful, and equally valid.

The Expert Outsider
To begin with, I taught the way I was taught. It was BC (before computers), and I would spend hours carefully writing text on sheets of acetate. These sentences of all-knowing were then revealed to the students line by line (courtesy of a piece of paper progressively pulled away). In sepulchral tones I recited the text, while students with idling brains diligently transcribed the words. These were called “Lecture Notes”.

What made me change? It took an almighty shove, courtesy of a Diploma of Education (Tertiary) conducted at the, then, Hawthorn Institute of Education. Bunch of hippies, with their new fangled ideas about small group work and Butcher’s paper. Trying to peddle the notion that students can construct their own learning, collaborating with others to share and develop the growing body of knowledge. Rather grudgingly, I wandered into the large tiered Lecture room, with index cards in hand, broke the students into professional role groups, and we staged a “Grand Rounds” case study meeting. Resistance was futile, we were all assimilated into the collective, we all wanted more. I discovered that the sound of humming chatter in the lecture theatre was as warm and welcome and productive as a hive of bees. The students had time to recover from their note taking inducted Repetitive Strain Injury. From then on every lecture would be a gymnasium of pedagogy.

What changed me? Being introduced to new ideas, by someone (my lecturer) experienced and bold. I discovered that I wasn’t the font of knowledge, the restricted source that dealt out facts bit by bit. I gave up the delusion of power and importance, and joined in the journey.

The Fellow Travellers
And from there, the students had tasted the blood of being the centre of the learning experience. They were given a voice, and they used it. They wanted organisation and structure so they could freely and confidently navigate their learning. They wanted the notes available so they could stop scribbling and start interacting in the large lecture. They wanted to know what was coming, assessment had to be clear from the outset, and had to make sense of what was stated in the learning outcomes (damn, there goes my famously cryptic essay questions on exams).

There was, of course, the day I arrived with yet another roll of Butcher’s paper and they collectively revolted. Apparently there is a caveat on educational aerobics, occasionally students appreciate the quiet and reflective opportunity to sit still and listen. I heard that too.

The Harsh Reality
Although it would be ideal if all pedagogical change arose from the actual experience of teaching and learning, often the pressure comes from the outside. The Large Class had been the zone of amazing change, now it was time to tackle a new site of learning. Diminishing resources and increasing student numbers meant that the traditional approach to clinical learning, through direct and supervised patient contact, could not be financially sustained. The annoyance and frustration led to an opportunity for change, and once the resentment that change was imposed rather than inspired, the opportunity was seized upon. Enter a new approach – creative situated learning outside of the clinic. We trained the students to be SimPats (that’s “Simulated patients”). The students were given the resources and guidance to learn how to act as patients (with a wonderful myriad of conditions), and in hastily reconstructed learning spaces, they played out the scenario in pairs. Each taking the role of the patient and the practitioner, learning how to take histories and conduct examinations, with peer support and feedback featuring large. Good Lord, I had nearly written myself out of the picture!

Peer learning expanded from there, with small group problem solving happening in so many ways. Now it was PC (post computers), and multidisciplinary groups of students were undertaking entire units through collaborative case studies presented online. They still wanted to regroup in the occasional lecture, to become a cohort and experience the face to face interaction that augmented their online identities. But it was all about them, and I could not have been happier.

So, it is nearly thirty years since I abandoned my overhead projector. I can’t say that there wasn’t a bit of kicking and screaming as I experienced the revolution and evolution of teaching and learning (and sometimes it was the students doing the kicking), but I have changed, and I think I would like to keep changing.

What has stimulated you to change your approach to teaching and learning?

Resources to inspire you:

7 things I learned from being a student or Statistics is really Mean

Post by: Ruth Moeller

Image by: s.schmitz @

As I was teasing my students about their anxiety over the assessment they had to complete, I urged them to remember this when their own students ask obvious questions that they should know the answer to “if they had read the assessment sheet properly, or at all!”

Reflecting on this exchange, I was reminded of the learning experiences I have had and how humbling it can be to be put in to the role of student.

There was the kayaking course I took, where every time I even looked at the kayak I ended up in the water, but I succeeded, even getting accreditation to teaching kayaking up to grade 3 rapids. This was due to the kindness, patience and persistence of the instructor, who never once suggested that perhaps white water wasn’t for me and would always make sure that the group waited for me and fetched my paddle when, once again, I ended up separated from it and the kayak. Note to self: patience, support and a sense of humor go a long way to helping students.

Or the practical mosaics course I did, where the teacher, passionate about her subject talked about mosaics, their history, design, options for use, while all the time a pile of plates and tiles sat in front of me begging to be broken. My learning styles are that of kinesthetic activist, I love to touch and do, so sitting and listening for two hours takes me to the end of my patience and I would not have been the only one. Note to self: some want to listen, some want to break, others watch or even read about it; remember there are a range of learning styles, try to cater for them all.

In terms of formal education, my most profound learning experience came in the field of statistics. Having studied on the edge of psychology, a couple of years ago I decided to do an undergraduate course in psychology. As part of this, I had to complete three core statistics units and this has been some of the most powerful learning for me as an educator. I came to the field as a true novice and mathsphobe and left battered, bruised and with three passes.

What did I learn?

• You can go to the lecture and tutes, but if you don’t know how to use the scientific calculator the stats worksheets are still a mystery. Learning should be seen as a process of building,  you need a stable foundation before you start to add – ensure students have the underpinning knowledge and skills required before you add to them. If you don’t teach it, direct them to support services and resources.

• If they aren’t interested in me, why should I be interested in them and their subject? In tutes, at least, find out the names of your students, their experience, interest in your subject, humanize them and yourself: teaching is much about the relationship as it is about the content, especially for those who are struggling.

• You can learn and copy the formula for Anovas, and other stats tools but if you don’t know what they actually mean, when and why and how they would be used in real life what have you really learned? We learn better with a context, knowing the formula and being able to do get the right answer is one thing but being able to apply and work with it is a deeper and ultimately more useful form of learning.

• If you don’t understand it all, being asked if you have any questions doesn’t help. Find ways to check understanding; activities such as the muddiest point can help, or try questions such as “People often struggle with this, what would like me to go over again?” This gives me permission not to know, and lets me know that you will help me to understand.

• When the tutors offer extra time and help, it’s probably a good idea to take them up on it rather than try to work it out yourself. This is the ongoing frustration for teaching staff – you offer help and many students don’t take you up on it. Perhaps it’s a pride or embarrassment thing or perhaps they are unsure of how you can help. This could be why peer support and online forums can be so successful.

• Being told you are “wrong and does someone know the right answer?” is humiliating. All students deserve to be treated with dignity, and the way you treat those who struggle will be noted by all and can have a profound effect on class engagement.

• Even the best of us turn up with our calculator, bottle of water and 3 HB pencil to take the Stats exam that was on yesterday. Students, like the rest of us are human, they make mistakes, usually not deliberately. Take a breath and solve the problem, don’t create a punishment.

In my opinion, being a life long learner make you a better teacher.

What has been your learning experience? How has is impacted on your teaching? Does anyone want to go kayaking?

A writing challenge – the first AcBoWriMo has been announced

The following post is by guest contributor Dr Narelle Lemon. Narelle works in Arts Education in Teacher Education programs within the School of Education, RMIT University in Melbourne. Narelle is 17 months post doctoral thesis and just beginning to write again and put into practice what she has learnt from this experience. She tweets @rellypops

Image via Ted_Major.

Life at university is a constant juggle – it is dealing with students, writing curriculum, teaching, researching, administration and pastoral care. Now, in Australia at least, the teaching is done, the marking is almost complete and an academic’s mind can turn to thoughts of research and writing and in the distance, publishing.

Yesterday was the 1st November and with that comes a month’s supply of tea and chocolate, washed trackies, slippers, favorite music blaring from the stereo, and an assortment of cafes selected that are conducive for writing. I have been preparing for November, like Triple J and their Australian Music Month – planning, preparing, inviting and motivating others, inquiring, and most of all thinking.  A writing challenge has been set and I, along with several friends and colleagues have signed-up for the first AcBoWriMo. As academics busy most of the year with teaching, many of us are now winding down that part of our work and dedicating an entire month to writing.

November is hereby declared the first Academic Book Writing Month or AcBoWriMo. Created by @PhD2Published , and the organizer and co-participant @charlottefrost, AcBoWriMo takes its inspiration from  National Novel Writing Month or  NaNoWriMo the spin is that instead of writing a novel, we are writing for an academic audience and can be flexible in terms of where our total word count goes –  journals, book chapters, books, doctoral thesis, or academic reviews that we just haven’t got around to. The challenge also establishes itself for one to learn how to be a writer, to learn what it means to be a writer, and to set this as a focus in order to succeed in producing writing that shares your ideas, research and message.

In taking up this challenge, AcBoWriMo is igniting explicit thinking and action about a writing task. So if you are taking up the challenge, or thinking about joining in, here are some tips for approaching this new approach to learning how to write and share your research.

1. Write about something you are passionate about. This excitement will support those moments during November when resistance to this project come into play.

2. @charlottefrost recommends a decision needs to be made ‘upon a target word count and to try and make this something that would really push you beyond anything you ever thought possible’. The challenge has been set for 50,000 words in a month – ‘for an academic this could be a challenge but one that allows for a stepping up that could result in some amazing output’. But in setting a target you also need to be honest about what could be and can be achievable for you, so go for a lesser target if that is more achievable for you.

3. Look at your diary and block out time each day so you can meet this word total. If you want to work 5 days a week, with a total of 22 days dedicated to writing, then you are aiming for approximately 2, 273 words a day. But if you are like many, and want to write on the weekend as well then you have the focus of 1,666 words a day. @charlottefrost recommends for those who want to focus on thisbit of a nutty goal for academic writing in one month’ that it ‘works out at something like 2,500 words a day’. I have gone through the entire month and dedicated full days (hopefully for extra words) and blocks of time so I can set myself up to succeed in this project. I’ve learnt from my doctoral writing times that I can get easily distracted

4. Plan what you are going to do with your words. For me I’m going to focus on a book proposal I have been thinking about but never find the time to prioritize it amongst my teaching, research and administration tasks. I have set this as my goal plan and have also shared it with a few colleagues who are participating so we have a reference point for checking in on progress.

5. Once decided what your AcBoWriMo project is, create a writing framework and allocate words to sections, and even apply these to subheadings. These guides will assist greatly for the word target and also your own project management for success. As a successful NaNoWriMo author has reported, planning is essential.

6. Invite others to undertake the challenge. Celebrating and learning from each other allows for seeing what others are doing in how they approach writing. This support is a great way to share perspectives on the learning and teaching associated with participating in AcBoWriMo. As has discussed before on this blog, opportunities to learn and to see things from a different perspective leads to discovering some really interesting perspectives.

7. Share your writing successes each day. The Twitter hashtag for AcBoWriMo is already beginning to generate some supportive advice, tips and hints for like minded people. @PhD2Published and @Teachingtomtom are linking fellow writers and are not short of encouragement for challenging your month of learning to be a productive writer.

8. Set yourself a personal learning goal for AcBoWriMo participation. Open yourself up to alternative ways of writing, perhaps even implement some techniques that you haven’t had the chance to yet but would like to try. See how you go for progressing your writing.

9. Spice up your daily writing time. Shutupandwrite sessions often utilize the Pomodoro technique for 25 minutes write, 5 minutes break, followed by 25 minutes writing. If you want some incentives for this, and a tracker who will make your pauses transparent, think about downloading the Pomodoro App for your idevice.

10.  Add some exercise into your plans for writing – chocolate, coffee, and sitting at a computer will add more than words to your achievements in the month of November if some walking, running, cycling or swimming aren’t a part of your regime.

11. Find a place or places where you know you can write, and think about what time of the day works for you. I write well in the morning, and I know which cafés work for me for dedicated productive writing while I enjoy a coffee when I need a change of scenery from my office.

12.  It’s all about words, not about tools. There have has been much debate about tools that can assist in framing your writing (Scrivener has been mentioned a few times via @ThesisWhisperer and @ResearchWhisperer). Repeat NaNoWriMo writers have reported that for the month of November, if you haven’t used these tools before then now is not the time to begin as you end up spending more time trying to work out which one is right for you. I would have to agree with this. I spent several hours looking at the apps available for my iPad and before I knew it I really didn’t have one tool that I could understand its full capacity to support me for my writing. I’m going to suggest noting these possibilities and set December as a month to play and discover for writing.

13.  Start fast and target those word limits each day, and if the words are a flowing let them flow. If daily targets are busted then this allows for some release of pressure on other days.

14.  Work through writer’s block and set yourself the challenge to learn how to ‘blah write’ – just get something down on paper, don’t worry about spelling and grammar at this stage, just write when the ideas flow.

15.  Back up those precious words you have produced, as there is nothing worse (and we all have our horror stories) in losing your writing. This is a great learning curve in approaching all your learning and teaching, and research endeavors.  I have been utalising Dropbox for backing up my writing and for allowing access across multiple computers.

16.  Set a time in December to share your writing with a fellow AcBoWriMo colleague. Co-read and give each other feedback. The month of November has been dedicated to producing an amazing amount of text, so some feedback will be part of the celebration and academic integrity to gain perspective. Hopefully you haven’t lived on coffees to reach your target words, but just in case those fingers have typed overtime, this sharing will allow for some proof reading as well.

As the month progresses it will be fascinating to see where our writing takes us. Who else is up for the AcBoWriMo challenge? Follow and be inspired through Twitter using the #AcBoWriMo hashtag or blog.