The evidence-based approach of our model requires a stringent program evaluation for the purposes of quality management, research for future program development, and accountability towards our donors. With program evaluation, we plan to research the short and long term influences of our program on the participants' professional development, as well as on their experience of empowerment and organizational/community involvement. At the same time, our research will track changes in the client base and general situations of the regions in which we work, in order to adapt our programs accordingly. The leadership program should be relevant for short and long term goals of organizations, communities and individuals, and specifically applicable in work AND everyday life situations. For this purpose, we define core competencies that we have determined to be essential for applying leadership, facilitation and coaching skills. These competencies are to be used as criteria by which to measure the pedagogic effectiveness of our programs. We have chosen the most accepted leadership development
evaluation methods, but expanded upon them with our own research approach. All components, including defined competencies and evaluative methods, are under our ongoing scrutiny and in the process of continuous, updated improvement based upon feedback from the process of research, evaluation and training, as the programs unfold.
Methodological background
The evaluation of the Process Oriented Leadership Development Program will include what we consider the two most established and respected training evaluation models, Kirkpatrick’s (Kirkpatrick 1994) Four Level Model and Phillips (Phillips & Phillips 2005) ROI Model. Data that is generated from Kirkpatrick’s model will provide general, essential information about the quality of learning in our program, based on direct participant feedback. Questions on each of the four levels address essential aspects of any training program, and will help us to gauge our progress based on our own standards as well as on those of other programs. General questions enable participants to provide personal feedback data. Data will be fed back into the system to provide categories and directions for adjusting training strategies.
The ROI model will help us to determine categories for evaluation other than financial gain, or directly measurable outcomes. This is essential for our program, as many of the benefits of training manifest in changed attitudes and perceptions that lead to changed actions, which finally bring greater results. However, the most direct measurement of these benefits is through the evaluation of these initial subjective qualities. This model considers alternate methods for measuring such essential qualities.
Kirkpatrick’s Four Level Model
This model proposes evaluation of four levels. Our evaluations will address questions that correspond to each of these levels:
- Participant reaction: How did program participants feel about the training content and setting?
- Participant learning: To what degree were the participants able to achieve the skill, knowledge and attitude objectives of the program?
- Participant behaviour: Transfer of learning: To what extent were participants able to transfer the learned leadership skills, attitudes and mind-sets into professional and private life settings during and at the completion of the leadership program?
- Results: Did the skills and mindsets taught in the program result in an increase in participants' personal access to socio-economic resources? Have UN millennium development goals been impacted?
Phillips ROI Model
The process of identifying and defining training benefits is complex, and does not always relate directly to identifying monetary benefits. Although financial benefits may be a secondary result of identified intangibles, this causal relationship is difficult to prove and relatively insignificant. Phillips (2006) recommends a financial evaluation step for only 5 to 10 percent of all programs. However, intangibles, those aspects that are difficult to measure monetarily but significant components of a well functioning workplace, are one of six types of data named in the ROI methodology, described as follows: “Intangibles are those measures that are not converted to monetary values, and usually include other hard-to-value measures. If these measures cannot be converted to money credibly and with minimum resources, they are identified as intangibles. Intangibles are very important because they represent the human dynamics elements in the work environment, such as commitment to organizational goals, teamwork, and communication. That they cannot be feasibly or credibly converted to monetary values does not undermine their significance in the workplace.” (Phillips & Phillips 2006, p. 160).
We have defined the following intangibles as significant standards of measurement for the Process Oriented Leadership Development Program: centeredness, coherence, ability to support various belief systems, ability to manifest creativity in the world, and conflict awareness.
Our Own Research on Evaluation Methods
We are also working on our own evaluation methods, in conjunction with European and U.S. colleagues, measuring parameters such as organizational field awareness, presence of meta-communicator, ability to maintain an overview, and eldership. We will continue to bring into operation other aspects of the multi-dimensional leadership core competencies based on results and onsite research. Please visit us again at this website for current updates on this front!
References
Behrens, T., & Benham, M. (2006). Evaluating Community Leadership Programs. In Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.
Kirkpatrick, D.L. (1994). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler.
Phillips J. (2003). Return on Investment in Training and Performance Improvement Programs (2nd ed.). Woburn, Mass.: Butterworth-Heinemann.
Phillips J. J. & Phillips P. (2006). Measuring Return on Investment in Leadership Development. In: Kelly Hannum, Jennifer Martineau, & Claire Reinelt (Eds.), Handbook of Leadership Development Evaluation. S.F., CA: Jossey-Bass.
