Sara M. Wills

Lately there has been a lot of chatter about wanting to incorporate more pre- and post-surveys into our work as educators.  Perhaps this is fresh in my mind after attending an assessment conference, but I found myself asking a lot of questions to presenters who argued to use pre- and post-surveys were the best way to measure student learning. The 'best' way? I'm not so sure.

As I have engaged in dialogue with colleagues about why they want to use pre- and post-surveys (because I don't!), it seems that one of the most common answers is because they are easy.  You build a survey, administer it before the learning experience and then administer it again, and poof ... you can prove that your students learned more after than they knew before.  But that's actually not accurate.  You see, a pre- and post-survey is an indirect measure of student learning because it is completely based on student opinion, attitude and perception.  Learners are not being asked to demonstrate their knowledge, we are simply asking them what they THINK they now know.  It would be like if you asked a student if they could identify 3 campus resources that could help them be more academically successful in the future.  They (hopefully) would note a higher level of perceived learning after they attended a training session on campus resources; however, if you asked them to actually identify which ones, they may only be able to recall one or two.  Clearly, there is a disconnect between what they think they know and what they can actually demonstrate.

Indirect measures seem to be used a lot more than direct measures of assessment, likely for that same reason ... they are easy.  They often involve surveys that ask students what they think they know or what they are satisfied with because it is far easier to administer quantitative assessments than analyze and code qualitative.  Especially when you are working with hundreds or thousands of students, these indirect methods probably seem better than nothing at all.  While of course it is better to do something than nothing, asking students to self-reflect and self-assess should not be reported on without direct assessment to validate the results.  Even worse is when these indirect methods have low response rates, because not only does it mean that the results do not account for the majority of the population, it doesn't account at all for what students are directly learning (UNC, 2012).  These types of surveys also provide little depth or understanding of why a learner indicated what they did.  For example, if they are still testing low on their perception of leadership theories after the workshop, is it because they actually didn't learn leadership theory or because they don't realize that they did? Indirect assessment can't tell us that.

When I first started in the field of assessment, I thought everyone and every program should have a pre-and post-survey.  It was easy, I could use language from the workshops and goals for what I wanted students to know and do as a result, it would be quick and easy to evaluate, and it would help me understand student learning.  As I continued in my assessment journey, I started to realize how little we actually knew about student learning as a result, and how misinformed people probably were.  Often times, I was using it as a sole assessment method (Santa Rosa Junior College, 2006), which I now know was quite irresponsible and did not truly demonstrate what our students knew or the effectiveness of our learning experiences.  Then I used it in combination with a program evaluation, which still only measured what students thought of a session, including common questions like: Was this engaging? Do you feel like you learned something? How can this be improved in the future? ... still all about feelings, perceptions and satisfaction.  So again, no direct evidence to support any of these perceptions among learners.

The one circumstance that I think is different, and is where I would suggest to a program that really wants to use a pre- and post-survey format can look is to self-confidence surveys.  You can find an overview of this assessment technique in Angelo & Cross' (1993) book, Classroom Assessment Techniques.  What makes this different, in my opinion, is that it does not ask students to consider what they think they learned; rather, it asks them to highlight their confidence in a specific area. Academic self-confidence - that is, confidence in one's own ability to understand a specific topic - has been proven to be a significant predictor of learning performance (Briggs, 2014). In fact, in a study conducted at the National Institute of Education, it was proven that, "confidence is a much better predictor of students' achievement than any other non-cognitive measure ... it acts in a way that overcomes everything else; so confidence is very important" (Stankov). Although the test was conducted specifically for English and Math, the strong relationship suggests that it is applicable to learning of a variety of topics and disciplines. Not only does this help educators gain semi-direct methods of assessment from an indirect tool, it also helps students engage in self-reflection by recognizing the areas that they need to gain more confidence in, and therefore, learn more. 

I think this distinction between perception and opinion vs. confidence is important in the work that we do as educators.  I acknowledge that assessment can be difficult, especially when we consider administrative load and resources.  However, I do think it is important that we try to think about our efforts in a way that supports the 3Ms of assessment - is it measurable? is it meaningful? and is it manageable? A pre- and post-survey may be measurable and manageable but it may not be very meaningful if we can't get at the real learning or understand why students have the opinions they do.  Thinking about adding in a piece around confidence means that we can maintain a similar setup of the form, continue to look at data quantitatively, but know that it is backed by research which draws direct relationships between a student's confidence and their ability to learn.  That is far more correlation that we can make between a student's perceived learning and their actual learning using a single technique.

For institutions that have the resources to make a pre- and post-survey one of many techniques, with at least one being direct, then it may be a good choice.  However, I would caution that there is a lot of correlating that needs to happen to be able to demonstrate that student learning is positively related to the pre- and post-tests, which means that the direct assessment needs to be directly aligned.  That is quite a bit of work, and I would argue should be done by someone who has graduate experience in statistics and research methodology, and understands how to effectively code qualitative data to make it quantitative.  Otherwise, it may be easier and more effective to consider using a confidence-related approach.

Finally, I think that this comes down to language.  The way that we talk about assessment, and just education in general, includes a lot of terminology.  My last post was related to referring to our work as pedagogy when what most of us probably mean is andragogy.  For institutions that are truly doing a curricular approach, it can be very frustrating when an institution who is actually doing programming calls themselves curricular.  When we are sending out satisfaction surveys and calling it assessment, what we really mean is evaluation... you get the point.  But I think the same should be said about pre- and post-surveys.  They seem to be a buzzword right now for folks who are integrating assessment into their work without actually understanding much about them.  I think we need to ask ourselves: is pre- and post-survey really what we mean? Because if we want to show valid learning over time, then that is not the "best" way to do that, especially if it is the sole method of collecting data.  Furthermore, if that is truly the most measurable, meaningful and manageable way to do it, then I think we should be talking about self-confidence surveys, and not pre- and post surveys.  Although it may seem silly, language is important, and as assessment continues to become an integral part of sharing our stories, validating our work, and creating a culture of accountability to our students, then we need to start getting it right.

For quite some time I have questioned if the term 'pedagogy' was appropriate for the work that we are doing as educators at the post-secondary level.  For me, it has felt like there is something juvenile about saying that our work with young adults should be considered pedagogical.  While I believe in the importance of having a teaching philosophy and framework, I keep asking myself if that is the correct terminology when we shouldn't actually be dictating their learning; rather, our students should be self-directing it.  As I have been progressing through my Teaching and Training Certificate, I have welcomed and embraced the emphasis on adult learning as something other than pedagogy; it is called andragogy.  This concept is directly related to Knowles (1968) argument that "the biggest obstacle to the achievement of the full potential of adult education has been that it has been tied to and it has been hamstrung by the concepts and methods of the traditional education of children" (p. 350-351).   Instead, Knowles argued that adults and children learn very differently, introducing the term andragogy, which is considered the science of teaching adult learners with a focus on helping them learn appropriately.  In his mind, the distinction is clear: pedagog means the learner is dependent on the teacher, whereas, andragog encourages the learner to become autonomous in their aspirations for continued learning (Levitt, 1979, p. 53).  After all, isn't that a key outcome we want our students to be able to know and do as a result of attending our institutions?

According to Knowles (1976), one of the most significant challenges in moving towards a more andragogical approach in higher education was the transformation in the student-teacher relationship.  Although this sounds fairly easy, why is it so difficult? Brookfield (2006) and Knowles (1976) have both argued that it can be a struggle to have teachers see themselves as facilitators of learning, instead of instructors, because very few have been exposed to that style of educating.  In my mind, this makes perfect sense because at this level of education, particularly thinking of learning outside of the classroom, our role is to facilitate learning experiences and environments that are self-directed by students, not instruct their learning.  Based on my experience working with faculty, I would hazard to say that some would disagree and emphasize the importance of instruction on ensuring students are learning appropriate techniques and knowledge within the field.  While it is hard to disagree with this stance, I think the important point is that young adults are still provided with opportunities to have control over their own learning.  This nicely ties into the provincial mandate to focus on more opportunities for students to have experiential based learning in their degree programs, and I think we will start to see that andragogy organically becomes more relevant in our academic teaching because of this.  As was noted within the Interdisciplinary Studies Program at the University of Southern Mississippi, students expressed that an andragogical approach to teaching actually spurred a renewal of their "love of learning" (Harper & Ross, 2011, p. 166) and that educators saw learners becoming more autonomous as both beneficial and rewarding. 

If you still aren't convinced that we should be using the language of andragogy, instead of pedagogy, in our work in higher education, I would encourage you to review this chart and ask yourself which you think is more aligned with your educational values and philosophy. This chart is sparking important conversations about our roles and is starting to change the way educators are thinking of themselves as instructors vs. facilitators, especially within the arena of online learning. I hope the same will soon be said about co-curricular learning as well.

Furthermore, there is becoming an increased amount of literature that is pointing to the relationship between coaching and andragogy.  Rachal (2002) further defined this relationship, stating that, "The learner is perceived to be a mature, motivated, voluntary and equal participant in a learning relationship with a facilitator whose role is to aid the learner in the achievement of his or her primarily self-determined learning objectives" (p.219).  This definition supports the notion that a learner is self-directed in the process, and the educator supports the student in this journey.  Cox (2015) takes it a step further, noting that coaching also includes more than learning, such as "unlocking potential, a collaborative solution, a powerful alliance, a collaborative and egalitarian relationship, or a life-transforming experience". This really got me thinking about our educators outside of the classroom, especially someone like a conduct officer because of the work they do to help a student see them as a support, ally and collaborator on a student's journey to repairing the harm and becoming a more positive contributor to their community.  However, those same connections couldn't be made if we thought about coaching from the lens of pedagogy, which would suggest the student is only motivated because of pressure and they are being told what to learn, rather than reflecting on the experience and being motivated to be better through self-actualization.  

To validate the importance of thinking about andragogy and coaching, Cox (2015) drew comparisons between the two.  Considering Knowles' (1984) definition, "Adults have a self-concept of being responsible for their own decisions, for their own lives.  Once they have arrived at that self-concept, they develop a deep psychological need to be seen by others and treated by others as being capable of self-direction" (p.56) she applied it to her work (2015) in coaching:

1. The need to appeal to an adult's self-concept as an independent learner is "at the heart of coaching" (Cox, Bachkirova and Clutterbuck, 2014, p. 149).  The coach is there to support and guide, but the coachee should direct the learning and relate it to real world problems. 

This can be seen in almost every form of co-curricular learning, whether it be in a student conduct meeting, career advising session, leadership development workshop, or with a conversation in a residence hall. If educators are encouraging learners to direct the conversation, apply their learning, and reflect on their experiences, students can feel supported and empowered.

2. Knowles emphasizes that self-direction implies that the learner should be taking responsibility of their own decisions and is not influenced by others.  However, sometimes this needs to be massaged and nurtured in a learner, which a coach can help foster and encourage.

There are a lot of student conduct bells ringing for me right now.  These educators invest so much time in developing and facilitating a conduct program that is grounded in maintaining the rights and responsibilities of a community, and emphasizing the importance of a student taking ownership for that.  If a conduct educator sees themselves through this coaching lens, they can better understand how they can help foster that conversation and perspective with the learner.

3. Adult learners should consider their prior life and experiences, which is directly relevant to Kolb's (1984) experiential learning model. If that model is applied to coaching, Cox (2013) argues that it expands this learning cycle, as the coach helps a learner "integrate those experiences and concepts to facilitate understanding, provide direction, and support action" (p.30).  Furthermore, it helps a student be more open to learning experiences in the future.

As educators, it is not our responsibility to reflect for the student; rather, I believe it is our job to create environments and opportunities for students to move through the reflective cycle.  Whether that is with student staff in training, in meetings with students about mental health, conduct, academic or career support, in an impromptu one-on-one conversation, or through an electronic portfolio, we need to coach from a lens of reflective practice and transformational learning.

4. The andragogical idea that adults learn when they are ready and need to strikes another commonality with coaching.  Typically, a learner seeks out an opportunity for new understanding when life suggests they need to or when they face uncertainty.  Coaching allows learners to address a goal or problem, look at it through a realistic lens, explore a variety of options, and decide what will be done as a result.  Through this support, learners can make informed decisions that are driven by their own motivation and uses their previous experience to help make important decisions.

I think we see this regularly from students who reach out for support, whether academically, personally, socially, or professionally.  Once those learners do seek a coach, it is our job to help them identify concerns, consider options and come to a decision on their own.  Educators are not there to give answers, as pedagogy may suggest; rather, we are there to help learners use their experience and knowledge to find a self-directed solution.

5. Considering learning through this lens also emphasizes the importance of a life-centred approach to learning.  For coaches, this suggests that learners are interested in working through problems or learning about ways that new ideas, knowledge or skill can help enhance their lives.

As educators, it is important for us to recognize this and take advantage of it.  Again, student conduct bells are ringing because of the emphasis many educators place on having learners think about how their experience may impact their future.  The same can be said for career coaches, leadership coordinators, or trainers.  We are far more effective when we approach learning from a andragogical lens, clearly linking how learning moments can be directly applied to life (in the present or the future).

6. Unlike in pedagogy where the motivation to learn is most often external, learners from an andraogical lens are internally motivated.  Although some find empowerment through the thought of a job promotion or increase in pay, many learners are looking for some sort of "internal payoff" (Knowles et al., 2011) which tends to be a more dominant factor.  Thinking about educating from the lens of a coach helps a learner understand the connection between their values and the outcomes of the learning experience.  In other words, coaching can empower the coachee to develop relationships between their own philosophies and learning, which will hopefully continue to be strengthened in the future.

As an educator, trying to instill a notion of lifelong learning in students is important but also can be difficult.  Thinking about educating from this lens really places the emphasis on getting to know the values and philosophies of the learner we are working with, and then helping them understand how it relates to this particular learning experience.  Not surprisingly, I see a lot of connections to student conduct programs again, which often seek to help students identify friction between their behaviour and their values, resulting in a learning experience that helps them explore how to better align these in the future.

I could talk about this forever and continue to reference literature in the field that argues we should be considering andragogy in our work with students, instead of pedagogy.  While perhaps some folks would make the argument that a student coming into college or university should not yet be considered an adult learner, I would challenge them to consider how we treat our students.  They have signed our institutional contracts that essentially consider them as an adult, requiring third party consent to speak with parents or guardians, and requiring them to take responsibility for their own academic success and behaviour.  It does not mean we don't help or support them when they need it, as we all know we do, but if we are asking students to behave as an adult, then perhaps our learning experiences should facilitate and encourage that. And most of all, if our goal is to develop students who are good citizens and neighbours, have gained important life skills, and believe in lifelong learning, then shouldn't we be talking about our work as andragogy? 

​Food for thought.

If you follow along with my blog, you likely know that last February I decided to step outside of my comfort zone and take on a position on the academic side of campus.  I was offered the opportunity to cover a parental leave and focus on program-level assessment, similar to what I had been doing in student affairs, but within the quality assurance office.  There were a lot of mixed feelings at first ... a lot.  I enjoyed the opportunity to talk to faculty, ask them questions about what they think students should be learning in higher education, and try to understand why some had such negative views of student life and housing.  It allowed me to advocate for the great work educators do outside of the classroom, and the leadership skills, life lessons, and community engagement that students are afforded to compliment their academic knowledge.  However, it also made me yearn for the purposeful work and intentional projects I had in student affairs, and I missed that culture of excitement. Talking to student affairs professionals about curriculum development, program assessment, and teaching techniques was fun, engaging, and often they shared their enthusiasm to learn.  Early on in my acting position, I learned that generally speaking, faculty tend not to share that same enthusiasm and often see the culture of assessment as more work than beneficial.  Some challenge the difference between objectives and outcomes, question why each course needs to map back to an outcome, and ask who is responsible for collecting and analyzing the data when they are already on administrative overload.  While I was frustrated at first that these were issues, as I have progressed in this role and taken time to learn, I now can appreciate their concerns, perspectives and struggles. I am beginning to understand why some faculty may be apprehensive and feel like they don't always have the support and resources to implement these practices effectively.  At times, I even appreciate why some feel as though curriculum should only be discussed and used within an academic setting, and how student life should offer learners real-world, unstructured learning experiences.  I now welcome the opportunity to talk with them about their concerns, help them better understand the value of outcomes-based assessment, and enjoy working towards changing the culture of assessment on campus.  

The first few months of my role, I couldn't wait to get out of the academic realm.  It wasn't me; it wasn't true to my purpose and values; and it didn't align with my career goals.  There were times when I came home and cried, wishing that I could just return back to my job in housing ... but I knew that I had made a commitment and I needed to follow through with it.  And I am so glad that I did.  Now, 8 months later, that discomfort I faced at the beginning when I felt inexperienced, incompetent, frustrated, bored, and insecure has dissipated.  I am learning to lean into that uncomfortable feeling, and to use it as a learning experience to challenge myself to continue learning more.  Learning how to communicate more effectively with faculty; learning how to find meaning in work that I wouldn't have before; building networks and relationships with faculty and staff who I would have never crossed paths with otherwise; and contributing to some of my institution's highest level committees and councils.  Perhaps most importantly, I am learning to love my colleagues and have never felt such a sense of success, camaraderie and friendship with all the people I work with.  The uncomfortable is starting to become very comfortable ... and it is nice.

Particularly over the past month or so, I have really leaned into that feeling and used it to challenge myself to take on new projects, share my struggles with my supervisors, and found opportunities to bring my background in student affairs to the forefront to be successful.  As you may know, the Ministry of Advanced Education and Skills Development (MAESD) just published a document that outlines what types of practices are considered to be "experiential learning", which includes work-integrated learning.  We have known for quite some time that the province was going to require institutions to have experiential learning opportunities for students, but that definition has previously been quite vague and defined by each campus.  Now with these guidelines, we have a better idea of what would qualify as an experiential opportunity in our academic programs.  The key to this is to have students reflect on these various experiences and demonstrate how their learning through these practices is directly connected to what they are learning in their programs: How are they applying these theories and skills?  How do we know they are doing that effectively? How are they demonstrating this application? How are we capturing these reflections? How can the reflections be used for career development and readiness? How do we ensure that all experiential opportunities, regardless of type, provide students with the same reflection, skills and career preparation?  So. Many. Questions. Thankfully, I have been assigned some significant project work relating to these questions and have been able to wear both hats - considering how this connects to our academic programs, and how theories like Kolb (which we tend to use more in the co-curricular realm) can come together.  Best of both worlds.

Taking on this type of project has been so rewarding because it has allowed me to flex some different muscles, and bring together my experience from the co-curricular and the incredible academic work we are doing on our campus.  As institutions move towards integrating these principles and concepts into our academic experiences, I also envision these becoming key components of our co-curricular and student life experiences as well.  I already have a brainstorm of ideas for how we can do a better job at integrating simulations, capstone projects, and performance based learning into our staff training and student development experiences.  Being able to see how these two worlds - the academic and the co-curricular- can come together to be more complimentary in the future makes me excited.  It brings promise to me that we can continue to work together, instead of siloed and apart, and that everyone can see how valuable it could be.  I am hopeful that just as assessment has moved from the confines of the classroom to become an integral part of the out-of-class learning experience, perhaps experiential learning will be the next  catalyst to bring faculty, staff, students and administration together to realize that we shouldn't be talking about graduation, grades, and programs separately from life skills, personal development and student life; rather, we could be and should be talking about all together and focusing on the holistic student.

Are these ideas novel? No.  Would I have thought of them even if I was still in my housing position? Probably.  Do I feel like my perspective and lens has changed through this often uncomfortable experience? Absolutely.  

So, if nothing else, ask yourself what makes you uncomfortable. What makes you feel a little outside of your comfort zone, insecure, or unknowledgeable? What challenges you to think differently, network with people outside of your scope, and flex different muscles?  When you find it ... embrace it. Lean into it. Learn from it. Embrace it.

If there is anything that I can share from my own experience leaning into the discomfort, it is that I am so much better for it.  I am a better professional with a much broader breadth of understanding for how my institution works and the barriers that keep our student life and academic life separate at times.  I am happier because this has forced me to speak up, share my knowledge, and find ways to fuse together my past and present experience.  And, speaking of experiential learning, I see firsthand the value of applying knowledge in work settings and different environments every day, as I reflect regularly on how so much of my graduate work and previous experience helps me be a better educator in my current role.  I try to remind myself often that you have to practice what you preach, and what better way to role model to our students the value of a holistic student experience and experiential opportunities than to jump in ourselves to become more holistic professionals?