Self-Assessment in the Health Professions: A Reformulation and Research Agenda

Abstract

Many researchers and educators have identified self-assessment as a vital aspect of professional self-regulation.1,2,3 This rationale has been the expressed motivation for a large number of studies of self-assessment ability in medical education, health professional education, and professions education generally. Unfortunately, the outcome of most studies would seem to cast doubt on the capacity for self-assessment, with the majority of authors concluding that self-assessment is, in fact, quite poor.4 In a recent article, Ward and colleagues suggested that this conclusion must be questioned because the methodologies used to evaluate self-assessment are fraught with methodological weaknesses.4 However, even studies that have attempted to address the weaknesses within the methodological paradigm have produced little evidence for effective self-assessment.5 Thus, the health professional education community is left with a conundrum that can only be resolved by deciding either that the conclusions of the studies are wrong, or that a critical premise underlying the concept of "self-regulation" in the professions is unsupportable. The current paper addresses this conundrum by arguing that there is a problem with the literature on self-assessment, and that this problem is more fundamental than a list of easily correctable methodological flaws. Rather, the roots of the problem in the self-assessment literature involve a failure to effectively conceptualize the nature of self-assessment in the daily practice of health care professionals, and a failure to properly explicate the role of self-assessment in a self-regulating profession. Until such an articulation of self-assessment is elaborated, it is difficult to know even which literatures might be informative in addressing this issue, and impossible to develop programs of research that operationalize the concept of self-assessment ability in a form that can be effectively studied. Thus, we will begin with a brief reflection on the various functions of self-assessment for a practicing health care professional and the manner in which these functions operate. The Purposes of Self-Assessment in Practice Self-assessment has been defined broadly as the involvement of learners in judging whether or not learner-identified standards have been met.6 While attractive due to their concise and encompassing nature, we fear that such simple definitions risk being misleading as they can cause underappreciation of the complexities of the construct. Self-assessment functions both as a mechanism for identifying one's weaknesses and as a mechanism for identifying one's strengths. Each of these mechanisms can be considered to have distinct, albeit complementary, functions. As a mechanism for identifying weaknesses or gaps in one's skills and abilities, self-assessment serves several potential functions. First, in daily practice, the identification of one's weaknesses allows the professional to self-limit in areas of limited competence. For example, in many circumstances the professional can quickly reject certain plans of action because she recognizes that she is unlikely to be able to complete the component tasks necessary to enact the plan. In other circumstances, a professional might recognize that he is "over his head" in a particular case and decide that it is time to recruit additional resources: to "look this up," to obtain a consultation, to recruit additional support, or to refer the problem to another individual who is more competent in this domain. Second, in reflecting on one's practice in general, the ability to identify weaknesses can serve the function of helping the professional set appropriate learning goals. That is, the traditional model of self-regulated continuing professional development presumes that an individual will select ongoing learning activities that fill professional gaps, but this presumes that the professional can effectively self-assess. Thus, in this role, the identification of weakness can help a professional to decide what must be learned. As a corollary to this, effective self-assessment is vital for setting realistic expectations of oneself, to avoid setting oneself up for failure. Thus, the identification of weakness also helps the self-regulating professional to decide what not to try learning, what should be accepted as forever outside one's scope of competent practice. There is a complementary set of functions served by the ability to accurately self-assess one's strengths. First, in daily practice, having a clear and accurate sense of one's strengths allows the professional to act with appropriate confidence. For example, knowing one's strengths provides the professional with the confidence to move forward on a fitting plan of action without inappropriate hesitation or trepidation. Similarly, it ensures that the individual will choose to persist on an appropriate plan of action in the face of initially negative feedback. The right path is not always smooth even if it is right, and early abandonment of an appropriate plan of action is as costly as selecting an inappropriate plan in the first place. Second, when reflecting on one's practice in general, an appropriate assessment of one's strengths ensures that one can set appropriately challenging learning goals, pushing the edges of one's knowledge rather than choosing professional development courses that merely reiterate what one already knows. At the same time, by knowing one's strengths, a professional can select learning objectives that are within her grasp, and therefore will be able to enjoy the motivational influence of attaining her goals and experience the satisfaction of a job well done. Together, then, the ability to accurately assess one's weaknesses and one's strengths generates a capacity for finding an effective balance both in daily practice and in setting personal learning goals. In daily practice, it generates a balance of confidence and caution, of persistence and flexibility, of experimentation and safety, and of independence and collaboration. In establishing learning goals, it generates a balance of learning enough but not too much, of starting neither too high nor too low, of knowing what to tackle and what to abandon. And in reflecting on accomplishments, it generates a balance of satisfaction and incentive, of self reward without self delusion. In order to fulfill these various functions, it seems that self-assessment must be effectively enacted in three forms: summatively, predictively, and concurrently. Enacting self-assessment summatively, a professional must reflect on completed performances both for the purposes of assessing the specific performance and for the purposes of assessing his abilities generally. When evaluating performance on a particular task, the professional can often assess the overall quality of the completed job as a question that may come in various forms. That is, the individual might ask how good this performance was relative to what she could have done; relative to what her peers might typically do; relative to the best that could have been done (a gold standard); or relative to some minimally acceptable standard. Alternatively, there are some situations where the mechanisms for objectively assessing the outcome are not immediately available, in which case the professional might ask herself how confident she is in the conclusion or outcome generated (is it right? will it stand up? could there have been a better solution given the situation?). The professional might then use her assessment of the specific task to draw summative conclusions about herself or her abilities in this domain generally. Again, such conclusions may be in absolute terms (am I good enough in this domain? am I minimally competent?) or in relative terms (am I average, above average, or below average, and against whom should I be comparing myself?). In drawing general conclusions about her abilities from a particular performance, the professional must also make determinations about whether this particular episode should be taken as an appropriate reflection of her general skills: were there extenuating circumstances that led to a particularly poor (or good) performance that might lead one to discount this outcome as reflective of overall ability? In addition to these summative functions, self-assessment must be used predictively. Professionals are constantly required to assess their likely ability to manage newly arising situations and challenges. In this predictive role, self-assessment leads to questions such as: Am I up to this challenge? Should I be starting this task (now, alone, in this way)? What are realistic goals for accomplishment in this context (what would I consider to be a good or acceptable outcome for me)? How much better might I imagine performing with some additional preparation and is the increased preparation worth the anticipated increase in performance? What additional resources should I recruit (either internally or from the outside) to complement my strengths and shore up my weaknesses? Finally, self-assessment plays a vital role in its concurrent mode of functioning. In this concurrent mode, self-assessment acts as an ongoing monitoring process during the performance of a task. It is self-assessment in its concurrent mode that leads to questions such as: Is this coming out the way I expected? Am I still on the right track? Am I in trouble? Should I be doing anything differently? Should I persist in the face of negative feedback from the situation (that things are not going the way I thought they would or as easily as I thought they would)? Do I need to recruit additional resources (internal resources such as attention or external resources such as advice/assistance)? Do I need to reassess my original goal or my original plan? Thus, self-assessment is a complicated, multifaceted, multipurpose phenomenon that involves a number of interacting cognitive processes. It functions as a monitor, a mentor, and a motivator through processes such as evaluation, inference, and prediction. Given this elaborated description of self-assessment, it is unlikely that simplistic questions such as "are health professional trainees effective self-assessors?" will lead to insightful discoveries about the nature and value of self-assessment. Rather, researchers must ask questions such as: On what basis do individuals make these decisions? What factors affect their reasoning? How fine tuned does the assessment need to be in order to be useful? A first step toward addressing these questions must be to determine who is already asking them and what insights we may borrow from their discoveries and reflections. Our search has led us to several literatures that seem particularly relevant: self-efficacy and self-concept; cognitive and metacognitive theory; social cognition; models of expert performance and the development of expertise; and the concept of reflective practice. In the following sections we will briefly touch on each of these literatures and suggest how they might inform our understanding of self-assessment. Our intent here is not to provide a systematic review of each literature, but to provide an overview of questions being addressed by researchers outside medical education that should inform our conception of self-assessment as a regulatory strategy. For each new literature we will define the area, provide examples of the issues under consideration, and then summarize the implications for self-assessment in the professions. We will end with a proposal for a program of research that has the potential to move the field beyond our current paradigm of repeatedly concluding that self-assessment is generically poor. Self-Efficacy and Self-Concept In studying the accuracy of self-assessments, education researchers in the health professions have tended to focus conceptually on what we have labeled the summative function – the ability to draw general conclusions about one's skills or knowledge in specific domains: How well do I understand endometriosis? Am I able to communicate effectively with other members of the health care team? Practically, this has usually been operationalized in research studies as a request that students try to estimate how well they will/did perform on an immediately following/preceding task. Yet, there is an important distinction between general assessments of one's ability in an area and the more specific question of how one did on a particular task. Researchers in the field of personality theory, for example, usefully distinguish between judgments of self-efficacy and the development of self-concept. Self-efficacy is the belief in one's capabilities to recruit the resources and execute the actions required to manage prospective situations. Self-concept is the relatively sweeping cognitive appraisal of oneself that is integrated across various dimensions.7 Thus, self-concept beliefs are context free, generalized judgments of self-worth that involve cognitive self-appraisals independent of a specific task or goal (but not necessarily independent of domain). By contrast, self-efficacy is a context specific assessment of competence to perform a specific task or range of tasks in a given domain (i.e., an individual's judgment of her capabilities to complete a given goal). Self-efficacy is, by its very nature, driven by an interaction between self-concept beliefs about one's skills or abilities and the specific context in which those skills or abilities will be applied for the attainment of the particular goal. It is concerned with the contextually embedded orchestration of skills that lead to performance. Self-efficacy differs importantly from the concept of self-assessment as currently envisioned in the health professions education literature in that self-efficacy is not only influenced by direct and indirect feedback, but also influences the future performance of tasks (the choices we make, the effort we put forth, how long we persist when confronted with obstacles or in the face of failure). Thus, there is an important reciprocity between self-efficacy and success. Not only will success lead to a strong sense of self-efficacy, but self-efficacy will also lead to an increased likelihood of success. Self-efficacy beliefs are not merely passive reflections of performance, but part of a self-fulfilling prophecy that affects performance. As a result, there is an advantage to high self-efficacy beliefs even in circumstances where such beliefs may not be warranted by past performance. Clearly there is a logical disadvantage to continually overestimating one's abilities, but this obvious disadvantage must be balanced with the value of believing that one can achieve more than one has in the past and that one can manage the challenges that one will face.8 As a result, researchers in the field of self-efficacy appear to be less worried about the "accuracy of self-assessment" and more worried about its impact on impending problem solving situations. They unconcernedly alter the situational self-efficacy of study participants through manipulations such as: varying the order in which people consider hypothetical levels of future performance,9 having subjects contemplate various positive or negative performance-related factors,10 altering the "anchor" values representing high or low levels of performance,11 or providing false performance feedback.12 Such manipulations regularly alter subjects' expectations of success on future events within the context of the study, suggesting that subjects will take contextual information into account when judging (either explicitly or implicitly) the likelihood of future success on tasks within that context. Again, for researchers engaged in the study of self-efficacy, the important point to be taken from these studies is that "trivial" factors alter self-efficacy and can affect future performance.13 For them, the fact that one can radically alter an individual's self-assessment of future performance appears to be simply taken for granted, rendering the question of "accuracy" somewhat nonsensical. Early on, Bandura provided a taxonomy of origins from whence information that would influence self-efficacy could be received.14 It included personal experience, vicarious experience, verbal persuasion, and physiological state. In addition, Cervone has argued that fundamental cognitive mechanisms (including common heuristics, as will be discussed in the next section) will influence the extent to which information from any given source will be weighed.13 In general, Cervone argues that self-efficacy judgments are not simply driven by an active, motivated distortion of facts in the service of ego protection ("hot cognition"), but rather that fundamental cognitive processes (i.e., those regularly used for a wide variety of judgment tasks – "cold cognition") influence self-efficacy beliefs quite independently. Overall then, it appears that researchers in the self-efficacy literature offer several theoretical and methodological approaches that can inform research in self-assessment. They acknowledge, in fact presume, the instability and situational specificity of self-reflective judgments, they examine and explicitly manipulate the factors that affect these judgments, and they concern themselves with the consequences of these judgments for future behavior. Cognitive and Metacognitive Theory In contrast to the focus on "accuracy" in the self-assessment literature and the focus on "consequences" in the self-efficacy literature, cognitive psychologists interested in metacognition (knowledge of one's own knowledge) tend to focus on delineating the mechanisms that allow us to mentally supervise and control the way in which we process information. Of particular interest for our purposes are questions of how people form metacognitive judgments, and what cues influence people's judgments of how well they have learned something. It is a fundamental assumption of this work that we do not have direct introspective access to our own memories or knowledge base. Rather, just as we must infer others' level of knowledge and motivations from their behaviors and other cues, so too we must use peripheral cues to make inferences about our own level of knowledge and learning. In fact, it is argued that our judgments of our own abilities are often based on the same inferential cognitive strategies, or heuristics, that we use to judge others. For example, the easier it is to process a piece of information, the more likely we are to judge that we will remember that information later (a fluency heuristic).15 Such heuristics are cognitive short-cuts that make us extremely effective and efficient at operating within a complex world despite our limited mental resources. However, they can also bias us in a way that leaves us susceptible to errors in decision making and, when applied to ourselves, errors in trying to identify our own strengths and weaknesses. Studies from this field suggest that, when trying to judge one's ability in a domain or when trying to judge the likelihood of success on a task, the accuracy of these metacognitive judgments is dependent on the extent to which the apparent difficulty of learning mimics the actual difficulty of eventually retrieving the learned material from memory. For example, research demonstrates that, when people are trying to learn a piece of information (such as a list of words) for later recall, several factors affect their judgments of having succeeded in their learning efforts. Metacognitive judgments are more accurate if the repetitions of each word are spaced apart and interspersed with other words than if repetitions of each word are blocked together.16 People appear to use the cue of fluency (i.e., ease of understanding) in judging the extent to which they have learned material and, as such, overestimate the amount they have learned when fluency is increased by blocking repetitions together. Similarly, metacognitive judgments are more accurate when there is a delay between study of the words and efforts to during practice. In general, people overestimate their learning if the words are blocked or if too on study of the because these of the task are easier than the actual task they will eventually be to perform a long The the task during the learning the better the of the amount of learning that merely the list and during practice are often to metacognition if people are left to their own during learning. That is, in order to recognize one's to the words it is necessary to try to them and make in these errors as feedback, people to overestimate their ability to the participants are unlikely to in themselves the that better judgments of learning. For example, judgments of learning tend to be more accurate participants are to provide a and the word than if they are to This that, without external to do the participants did not try and but rather simply did not and in doing an important cue that they might have used to their This finding is with the between performance and self-assessment in the health professions literature when judgments are relative to as a the from this literature the of beyond questions of people self-assess to that explicitly focus on the various factors that affect judgments of learning or knowledge or In the of direct access to our mental we are to make metacognitive judgments based on a variety of and external Metacognitive judgments tend to be more accurate when these cues accurately reflect the factors that affect but there are many in which the cues used for judgments of learning predictive or systematic between performance and actual A better understanding of which cues are used and which should be used in health professional education well as the impact these cues have on study might better and our understanding of the concept of into the of cues that are often misleading in world situations for the of into the inappropriate use of specific has been from researchers in the field of social the focus of the next in social has led many to that much of what we to know about outside of Each of us an that much of our and This part of the is labeled because we have access to the and of we do not enjoy such access to the mental processes that are We have a to for our but these are often and more than are about the of our This is because there are to people themselves to be more a better than the While it is impossible that we are above average, at the individual level such positive can be in individuals who such are less likely to be and more likely to persist at difficult and of the the we will to a sense of well and How we is somewhat but as one example, a number of mechanisms some in fundamental cognitive by which people can develop and many of which are to an of what is required to accurately self-assess one's own strengths and As one example, and evidence that the common belief that they can the because they or their On the focus more attention on their and remember them better than They the belief that they are by the on the they should have if not for some the being As a result, come to of as and the belief that they can the likely themselves in a It is very to an of one's own ability by making the but the question or I a in to that This to discount information, with the of feedback the likelihood that in will be Given that the goal of self-assessment is to avoid such of oneself, social psychologists suggest that it is necessary to at one's own and how to it rather than simply reflecting When reflecting on our knowledge and abilities we have a of information to us that is not to but our capacity for that do not with our of can the of that additional information in a misleading of confidence in the accuracy of our of such a we have to more than on and being more likely than external to situational influences on our the to do so being broadly as the fundamental This a for self-regulating in that it one must and the of and in order to develop an accurate of question the of are also to but the more the of information, the less susceptible our self-concept might be to search for This that self-assessment is for the of accurate self-concept is with the finding that peers tend to be better of performance than do individuals both in health and social This questions about self-assessment quite from the simple question of accuracy that has self-assessment researchers in professional what extent do health care out assessments from What them to do How can we with the of to a and appropriate sense of what extent is for such to be This question has been a focus in the of expert performance. of and the of social psychologists have argued that the is a the serves more as a fact this is also with current models of a being as the of approaches to both a of new and a more (i.e., of specific In the current the question of interest is what role, if does this or process of in the development and of In the study of on performance, and colleagues have been able to on in the of practice – on specific tasks by practice is from the of as up reflective and work to social to its is the of an who can students beyond their current ability by to or approaches that are likely to if one on In fact, by its in of the health professions is an on learning. to the role of early and

Ähnliche Arbeiten