(Validating the Return on Expectations Evaluating Model (ROE
Subject Areas : New theories and models of human resource training and developmentAlireza Dehghani 1 , Bijan Abdollahi 2 , hossein Abbasian 3 , Mohammad Reza Behrangi 4
1 -
2 -
3 -
4 -
Abstract :
The purpose of this paper is to validate the Return on Expectations Evaluation Model (ROE). The study population was the stakeholders of training program at IRIB which 465 of them were selected by proportional stratified random sampling. The content validity of the measurement tool was based on the expert's view, and CVI and CVR forms, and its reliability was based on the Cronbach's alpha coefficient of 0.975. Collected data were analyzed using the structural equation modeling and LIZREL software. The results of the first-order confirmatory factor analysis indicated that path coefficients between all variables were significant at P<0.01 level. The results of the second-order factor analysis and path analysis, while fitting the model, showed that 17 path coefficients of the hypothesized between the different variables of the model were significant at p<0.05 level. The ROE model adapt to the latest developments in the field of training evaluation. It is result and process-oriented model, provides a new definition of the training effectiveness concept based on stakeholder expectations for training programs, and can evaluate the effectiveness of IRIB training programs based on that organization stakeholders' expectations.
1.McCoy M, & Hargie ODW. Evaluating Evaluation: Implications for Assessing Quality. International Journal of Healthcare Quality Assessment. 2001; 14(7): 317-327.
2. Verschuren, PJM & Zsolnai L. Norms, Goals, and Stakeholders in Program Evaluation, Human Systems Management. 1998; 2: 155-160.
3. Werner JM & DeSimone RL. Human Resource Development. 5th Ed. 2009. USA: South-Western.
4 .Khorasani A & Doosti H. Measuring Training Effectiveness in Organization. 2010. Tehran: Industrial Research and Training Center of Iran. [in Persian]
5. Bazargan A. Evolutional Evaluation. Concepts, patterns and operational process. 2010. Tehran: SAMT. [in Persian]
6. Kirkpatrick DL & Kirkpatrick JD. Evaluating Training Programs, the Four Levels 3th ed. 2006. San Francisco: Berrett Koehler.
7. Abd Rahman A, Imm NS, Sambasivan M & Wong F. Training and Organizational Effectiveness: Moderating Role of Knowledge Management Process. European Journal of Training and Development. 2013; 37(5):472 – 488.
8. Nickols F. Why a stakeholder approach to evaluating training. Advances in Developing Human Resources. 2005; 7(1): 121 – 134.
9. Kucherov D & Manokhina D. Evaluation of training programs in Russian manufacturing companies. European Journal of Training and Development. 2017; 41(2): 119 – 143
10. Windfred AJ, Bennett WJ, Edens PS & Bell ST. Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology. 2003; 88(2): 234-245.
11. Wang GG & Wilcox D. Training evaluation: Knowing more than is practiced. Advances in Developing Human Resources. 2006; 8(4): 528-539.
12. Rice AA. Evaluation of the Impact of a Large Corporate Leadership Development Course. 2011. Michigan: Western Michigan University.
13. Twitchell S, Holton EF & Trott JWJ. Technical training evaluation practices in the United States. Performance Improvement Quarterly. 2000; 13(3): 84-109.
14. Giangreco A, Carugati A & Sebastiano A. Are We Doing the Right Thing? Food for Thought on Training Evaluation and Its Context. Personnel Review. 2010; 39(2): 162-177.
15. Bates R. A critical analysis of evaluation practice: the kirkpatrick model and the principle of beneficence. Evaluation and Program Planning. 2004; 27(3): 341-348.
16. Stuffelbeam DL & Zhang G. The CIPP Evaluation Model, How to Evaluate for Improvement and Accountability. 2017. New York: The Guilford press.
17. Ford, D. J. Bottom Line Training: Performance Based Result. 2005. USA: Training Education Management LLC.
18. Murray LW & Efendioglu AM. Valuing the investment in organizational training. Industrial and Commercial Training. 2007; 39(7): 372 – 379.
19. Phillips JJ & Phillips PP. Handbook of training evaluation and measurement methods. 4th Ed. 2016. New York: Routledge.
20. Nickols F. A Stakeholder Approach to Evaluating Training. 2004. https://www.nickols.us/stakeholder.pdf
21. ATD. 2018 State of the Industry Report. https://www.td.org/research-reports/2018-state-of-the-industry (Thursday, December 08, 2018)
22. Kirkpatrick JD & Kirkpatrick WK. ROE’s Rising Star Why Return on Expectations is getting so Much Attention. 2010; T+D, 64(8): 34-38.
23. Chalofsky NE, Rocco TS & Lane Morris M. Handbook of Human Resource Development. 2014. USA. New Jersey John Wiley & Sons.
24. McLinden D & Trochim MK. Getting to Parallel Assessing the Return on Expectations of Training. Performance Improvement.1998; 37: 21-26.
25. Kirkpatrick DL & Kirkpatrick JD. Implementing the Four Levels. 2007. San Francisco, California. Berrett-Koehler Publishers.
26. Kirkpatrick JD. Return on Expectations: The ultimate demonstration of training value. 2009. https://www.trainingzone.co.uk/develop
27. Kirkpatrick JD & Kirkpatrick WK. Training and Trial How workplace learning must reinvent itself to remain relevant. 2010. USA: American Management Association.
28. Throgmorton C, Mitchell T, Morley T & Snyder M. Evaluating a Physician Leadership Development Program – a Mixed Methods Approach. Journal of Health Organization and Management. 2016; 30(3):1-31.
29. Fierceinc. The Big Picture of ROE a Supplementary White Paper to the ROI of Skillful Conversation. 2015. https://www.fierceinc.com
30. United State Office of Personnel Management. Field Guide to Training Evaluation. 2011.https://www.opm.gov/policy-data-oversight/training-and-development/reference-materials/training_evaluation.pdf
31. Husaini Usman S & Jaedun A. Impact-Based Training Evaluation Model (IBTEM) For School Supervisors in Indonesia. International Education Studies. 2016; 9(2): 149-160.
32. Moorhous C. A Strategic Conversation Model to Optimize Return on Occupational Training Expectations. (Doctoral dissertation, University of South Africa). 2013.
33. Larsen TA & Schultz MA. Transforming Simulation Practices A Quest for Return on Expectations. Clinical Simulation in Nursing. 2014; 10: 626-629.
34. Padilla Rodriguez B. C. Instructional interactions and online course effectiveness at a large Mexican organization. (Doctoral dissertation, School of Education: University of Leicester). 2014.
35. Shams Mourkani, G., Mirahmadi, K., Abbasi Kasani, H. Dashti, A. & Mohseni, P. Designing and Implementing Training Outcome Evaluation Program Using ROE Logic. Managing Education in Organizations. 2019. 7 (2): 89-111.
36. Dehghani, A., Abdollahi. B., Abbasian, H. & Behrangi, M. Forthcoming. Designing the Return on Expectations Training Evaluating Model (ROE): A Qualitative Research. Quarterly Journal of Educational Measurement and educational Evaluation. 2020; 10(29).
37. Guerci M. Percpective on Practice Training Evaluation: an Analysis of the Stakeholders’ Evaluation Needs. Journal of European Industrial Training. 2010; 35(4): 385-410.
38. Erina I, Ozola IO & Sarkane AG. The Importance of Stakeholders in human Resource Training Projects. Procedia - Social and Behavioral Sciences. 2015; 213: 794-800.
39. Fenwick J & McMillan J. Management Development and Co-Production: Myths and Realities. Journal of Management Development. 2013; 32(9): 971-983.
40. Anderson V. The value of learning: a new model of value and evaluation. 2007. London: Chartered Institute of Personnel and Development (CIPD).
41. Teddlie Ch & Tashakori A. Foundations of mixed methods research: integrating quantitative. 2009. London: SAGE.
42. Seif A. Measurement, measure and evaluate education. 2016. Tehran. Doran. [in Persian]
43. Sarmad Z, Bazargan & Hejazi E. research methods in the behavioral sciences. 2018. Tehran. SAMT. [in Persian]
44. Hooman H. Structural equation modeling with LIZREL Aplication. 2008. Tehran. SAMT. [in Persian]
45. Sobhani Y. Foundations and Applications Factor Analis and Structural Equation modeling. 2017. Tehran. Emam Sadegh U. [in Persian]
46. Kirkpatrick JD & Kirkpatrick WK. Creating ROE: The End Is the Beginning. ATD. 2011. December 01. https://www.td.org/Publications/Magazines/TD/TD-Archive/2011/11/Creating-ROE-the-End-Is-the-Beginning.
47. Chen, T. Training Mode of Innovative Talents in Higher Engineering Education in China. Educational Sciences: Theory & Practice.2018. 18(6): 3254-3265.
48. Laing N, Roberts P, Kestle L, Puolitaival T, Brenton Rule T & Bryan A. Development of a Collaborative Training Partnership in the New Zealand Construction Industry. EPiC Series in Education Science. 2017; 1:156–164.
49. Deo Kodwani A. Decoding training effectiveness: the role of organizational factors. Journal of Workplace Learning. 2017; 29(3): 200-216.
50. Chukwu GM. Trainer attributes as drivers of training effectiveness. Industrial and Commercial Training. 2016; 48)7(:367-373.
51. Lindsey A, King E, Hebl M & Levine N. The impact of method, motivation, and empathy on diversity training effectiveness. Journal of Business and Psychology. 2015; 30(3) 605–617.
52. Kirkpatrick JD & Kirkpatrick WK. The Four Levels of Evaluation – An Update. ATD Learning and Development. 2015; 15(1502): 1-19.
53. Powell KS & Yalcin S. Managerial training effectiveness: A meta‐analysis 1952‐2002. Personnel Review. 2010; 39 (2):227-241.
