PROGRAM EVALUATION:
The evidence-based approach of our model requires a stringent program evaluation for the purposes of quality management, research for future program development, and accountability towards our donors. With program evaluation, we plan to research the short and long term influences of our program on the participants' professional development, as well as on their experience of empowerment and organizational/community involvement. At the same time, our research will track changes in the client base and general situations of the regions in which we work, in order to adapt our programs accordingly. The leadership program should be relevant for short and long term goals of organizations, communities and individuals, and specifically applicable in work AND everyday life situations. For this purpose, we define core competencies that we have determined to be essential for applying leadership, facilitation and coaching skills. These competencies are to be used as criteria by which to measure the pedagogic effectiveness of our programs. We have chosen the most accepted leadership development The-New-Road-to-the-Top evaluation methods, but expanded upon them with our own research approach. All components, including defined competencies and evaluative methods, are under our ongoing scrutiny and in the process of continuous, updated improvement based upon feedback from the process of research, evaluation and training, as the programs unfold.
Methodological background
The evaluation of the Process Oriented Leadership Development Program will include what we consider to be the two most established and respected training evaluation models, Kirkpatrick’s (Kirkpatrick 1994) Four Level Model and Phillips' (Phillips & Phillips 2005) ROI Model, in connection with Davis's (1996) Interpersonal Reactivity Index for assessing empathy.
Data that is generated from Kirkpatrick’s model will provide general, essential information about the quality of learning in our program, based on direct participant feedback. Questions on each of the four levels address essential aspects of any training program, and will help us to gauge our progress based on our own standards as well as on those of other programs. General questions enable participants to provide personal feedback data. Data will be fed back into the system to provide categories and directions for adjusting training strategies.
The ROI model will help us to determine categories for evaluation other than financial gain, or directly measurable outcomes. This is essential for our program, as many of the benefits of training manifest in changed attitudes and perceptions that lead to changed actions, which finally bring greater results. However, the most direct measurement of these benefits is through the evaluation of these initial subjective qualities. This model considers alternate methods for measuring such essential qualities.
Interpersonal Reactivity Index
The Interpersonal Reactivity Index (IRI) (Davis, 1996) is the most widely used of the many methods for researching empathy. It assesses the complex and multi-faceted concept of empathy through measuring the following subscales: Perspective taking, Empathic concern, Fantasy, Personal distress.
We feel that IRI is well suited to evaluate our programs for a number of reasons. First, the close connection between the four distinct components of empathy with the defined core competencies of our program provides an obvious fit. The Index has been validated within a number of cultural contexts, suggesting a potential applicability for other non-Western cultures where DDI is active.
We have custom tailored this index according to the core competencies of our program. Although we plan to use this evaluation to gage the effectiveness of our courses and the development of the students in the specified areas, we will make an additional analysis and use of the results within the framework of our own training system.
The core of Process Oriented Leadership is the idea that disturbances are meaningful. Applied to this evaluation process, this means that we need to unfold the meaning of a low score and find out what is good or right about an apparent un-empathetic aspect so that it can be used consciously and effectively. We do not think of a low score as bad. A high or low score simply provides information that we can use to enhance the system and the individual learning process.
An example is a participant that scores low on the fantasy scale: the person does not have a propensity for fantasy but tends to stay within the practical, tangible world. We would try to discover the wisdom in this process for this individual. Maybe she or he lives in a high-pressure conflict zone in which you cannot afford to day dream, you must stay alert for the sake of survival. This quality is clearly valuable and needed. We do not want to get rid of this attribute. We want to help the person become conscious of it and use it more fully in her or his life. This might also allow the space for the opposite attribute, the ability to fantasize, to be developed and used as well.
For our purposes, however, the significance of this scale is to create awareness around these defined processes of empathy. Our goal is not to create more empathetic leaders but to encourage self awareness about the component experiences of empathy and reactivity.
For additional information on DDI's use of the IRI please click here.
Kirkpatrick’s Four Level Model
This model proposes evaluation of four levels. Our evaluations will address questions that correspond to each of these levels:
- Participant reaction: How did program participants feel about the training content and setting?
- Participant learning: To what degree were the participants able to achieve the skill, knowledge and attitude objectives of the program?
- Participant behaviour: Transfer of learning: To what extent were participants able to transfer the learned leadership skills, attitudes and mind-sets into professional and private life settings during and at the completion of the leadership program?
- Results: Did the skills and mindsets taught in the program result in an increase in participants' personal access to socio-economic resources? Have UN millennium development goals been impacted?
Phillips ROI Model
The process of identifying and defining training benefits is complex, and does not always relate directly to identifying monetary benefits. Although financial benefits may be a secondary result of identified intangibles, this causal relationship is difficult to prove and relatively insignificant. Phillips (2006) recommends a financial evaluation step for only 5 to 10 percent of all programs. However, intangibles, those aspects that are difficult to measure monetarily but significant components of a well functioning workplace, are one of six types of data named in the ROI methodology, described as follows: “Intangibles are those measures that are not converted to monetary values, and usually include other hard-to-value measures. If these measures cannot be converted to money credibly and with minimum resources, they are identified as intangibles. Intangibles are very important because they represent the human dynamics elements in the work environment, such as commitment to organizational goals, teamwork, and communication. That they cannot be feasibly or credibly converted to monetary values does not undermine their significance in the workplace.” (Phillips & Phillips 2006, p. 160).
We have defined the following intangibles as significant standards of measurement for the Process Oriented Leadership Development Program: centeredness, coherence, ability to support various belief systems, ability to manifest creativity in the world, and conflict awareness.
Our Own Research on Evaluation Methods
We are also working on our own evaluation methods, in conjunction with European and U.S. colleagues, measuring parameters such as organizational field awareness, presence of meta-communicator, ability to maintain an overview, and eldership. We will continue to bring into operation other aspects of the multi-dimensional leadership core competencies based on results and onsite research. Please visit us again at this website for current updates on this front!
References
Behrens, T., & Benham, M. (2006). Evaluating Community Leadership Programs. In Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.
Davis, M. H. (1996). Empathy: A Social Psychological Approach. Boulder: Westview Press.
Kirkpatrick, D.L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.
Phillips J. (2003). Return on Investment in Training and Performance Improvement Programs (2nd ed.). Woburn, Mass.: Butterworth-Heinemann.
Phillips J. J. & Phillips P. (2006). Measuring Return on Investment in Leadership Development. In: Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.
