An ICT Performance Evaluation Model based on Meta-Synthesis Approach

Bamary, Khatrehe; Behboudi, Mohammad Reza; Abbasnjad, Tayebeh

doi:10.52547/jist.16445.10.39.229

کد مقاله : 2021070316445 بازدید : 5030 صفحه: 229 - 240

10.52547/jist.16445.10.39.229

20.1001.1.23221437.2022.10.39.8.2

نوع مقاله: مروری

An ICT Performance Evaluation Model based on Meta-Synthesis Approach

محورهای موضوعی : IT Strategy

Khatrehe Bamary ¹ , Mohammad Reza Behboudi ² , Tayebeh Abbasnjad ³

1 - Department of Industrial and Public Management, Faculty of Management, Accounting and Economics, University of Hormozgan
2 - Department of Industrial and Public Management, Faculty of Management, Accounting and Economics, University of Hormozgan
3 - Department of Industrial and Public Management, Faculty of Management, Accounting and Economics, University of Hormozgan

تاریخ دریافت : 1400/04/12 تاریخ پذیرش : 1400/08/09 تاریخ انتشار : 1401/05/12

کلید واژه: Information and Communication Technology, Meta-synthesis, ICT Performance evaluation, Strategic, Quality, Sustainability,

چکیده مقاله :

Information and Communication Technology (ICT) is one of the key determinants for today’s organizational success. Therefore, companies spend a significant amount of money each year on ICT, while not being sure that they will get a good result. The purpose of this study is to identify the dimensions and indicators of ICT performance evaluation and suggesting a model for assessing it in organizations. This research is mainly a qualitative study with a meta-synthesis approach which uses the seven-stage qualitative method of Sandelowski and Barroso to systematically review the literature to find sub-indices (codes), indices (themes) and dimensions (categories) of ICT performance evaluation. The search of scientific databases with appropriate keywords found 516 articles, among them, 89 articles were chosen finally and used for analysis. Moreover, a questionnaire has been designed and answered by ICT experts and managers to determine the importance of each of the indicators of the model. Based on data analysis, the proposed ICT performance evaluation model has three dimensions: strategic, quality, and sustainability. The strategic dimension includes indicators of organization strategy, IT strategy, and alignment. The quality dimension includes maturity, and performance indicators; and finally, the sustainability dimension includes environmental, economic, and social indicators. For each of these indicators detailed list of sub-indices (104), which are substantial for evaluation of ICT performance in organizations, were identified and explained.

چکیده انگلیسی:

منابع و مأخذ:

[1] J. L. García-Alcaraz, V. Martínez-Loya, J. R. Díaz-Reza, J. Blanco-Fernández, E. Jiménez-Macías, and A. J. G. Lopez. “Effect of ICT integration on SC flexibility, agility and company’ performance: the Mexican maquiladora experience” Wireless Networks, Vol. 26, No.7, 2020, pp. 4805-4818.
[2] M. Arslandere, İ. Tuncer, and A. D. A. Serkan. “The Impact of ICT Use in Promotional Activities on Export Performance: An Empirical Investigation”, Business & Management Studies: An International Journal, Vol. 8, No.2, 2020, pp. 2384 – 2413.
[3] A. Rimmimgton, G. Dickens, and C. Pasqire. “Impact of Information and Communication Technology (ICT) on construction projects”, Organization, technology & management in construction: an international journal, Vol. 7, No.3, 2015, pp. 1367-1382.
[4] X. Chaojun, Y. Wei, H. Chenlin, and S. Congying. “Design and Validation of School ICT Monitoring & Evaluation Framework: A Case Study”, American Journal of Education and Information Technology. Vol. 4, No.2, 2020, pp. 78.
[5] T. B. Gyamerah. “Using e-learning to improve the performance of first year nursing students in ICT”. GSJ, Vol. 8, No.6, 2020, pp.-250-257.
[6] I. Apulu, and A. Latham."An Evaluation of the Impact of Information and Communication Technologies: Two Case Study Examples". International Business Research, Vol. 3, No. 4, 2011, pp. 3-9.
[7] M. Okumu Ujunju, G. Wanyembi, and F. Wabwoba." Evaluating the Role of Information and Communication Technology (ICT) Support towards Processes of Management in Institutions of Higher Learning", (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No.7, 2012, pp. 55-58.
[8] A. M. Sayaf, M. M. Alamri, M. A. Alqahtani and W. M. Al-Rahmi. "Information and Communications Technology Used in Higher Education: An Empirical Study on Digital Learning as Sustainability". Sustainability, Vol. 13, No. 13, 2021.
[9] H. Toya, and M. Skidmore. "Information/communication technology and natural disaster vulnerability", Economics Letters, 2015, pp. 143-145.
[10] A. André." The Information Technology Revolution in Health Care", In: André A. (eds) Digital Medicine. Health Informatics. Springer, Cham, 2019.
[11] S. Dutta, and B. Bilbao-Osorio. “The Global Information Technology Report 2015”, World Economic Forum, Vol. 1, No.1, 2015, pp. 200-42.
[12] K. Milis, and R. Mercken. “The use of the balanced scorecard for the evaluation of information and communication technology projects”, International Journal of Project Management, Vol. 22, No.2, 2004, pp. 87-97.
[13] S. Haanappel, R. Drost, F. Harmsen,, S. Brinkkemper, and J. M. Versendaal. “A framework for IT performance management”, Available from www: http://www. Cs. Uu. Nl/research/ techreps /repo/CS, 2011(006).
[14] J. Becker, R. Knackstedt, and J. Pöppelbuß, "Developing maturity models for IT management", Business & Information Systems Engineering, Vol.1, No.5, 2009, pp. 213-222.
[15] J. McKay, P. Marshall, and L. Smith. “Steps towards effective IT governance: strategic IS planning, evaluation and benefits management”, PACIS 2003 Proceedings, 65.
[16] E. H. Arribas, and P. S. Inchusta. “Evaluation models of information technology in Spanish companies: a cluster analysis”, Information & Management, Vol. 36, No.3, 1999, pp. 151-164.
[17] A. Albadvi, A. Keramati, and J. Razmi. “Assessing the impact of information technology on firm performance considering the role of intervening variables: organizational infrastructures and business processes reengineering”. International Journal of Production Research, Vol. 45, No.5, 2007, pp. 2697-2734.
[18] B. C. Stahl. “Information technology, responsibility, and anthropology”, In Proceedings of the 35th Annual Hawaii International Conference on System Sciences. IEEE, 2002, pp.1-10.
[19] K. L. Kraemer & J. N. Danziger, "The impacts of computer technology on the worklife of information workers". Social Science Computer Review, Vol. 8, No. 4, 1990, pp. 592-613.
[20] C. A. Gorse, & S. Emmitt. Investigating interpersonal communication during construction progress meetings: challenges and opportunities. Engineering, Construction and Architectural Management., Vol .10, No. 2, 2003, pp 233-245.
[21] O. Zizlavsky. “The balanced scorecard: Innovative performance measurement and management control system”, Journal of technology management & innovation, Vol. 9, No.3, 2014, pp. 210-222.
[22] N. Y. Wijayanti, W. Setiawan, and R. A. Sukamto. “Performance Assessment of IT Governance with Balanced Score Card and COBIT 4.1 of Universitas Pendidikan Indonesia”, Journal of Physics: Conference Series, Vol. 812, No. 1, 2017, p. 012072. IOP Publishing.
[23] M. Uii, H. Gusaptono, M. Effendi and N. Charibaldi. “The Information Technology (IT) Adoption Process and E- Readiness to Use within Yogyakarta Indonesian Small Medium Enterprises (SME)”, International Journal of Information and Communication Technology Research, Vol. 2, No. 1, 2012, pp. 1-12.
[24] F. Salehi1, B. Abdollahbeigi1 and S, Sajjady. “Impact of Effective IT Governance on Organizational Performance and Economic Growth in Canada”, Asian Journal of Economics, Finance and Management, Vol. 3, No. 2, 2021, pp, 14-19.
[25] M. Simonsson, and P. Johnson. “Defining IT governance-a consolidation of literature”, The 18th conference on advanced information systems engineering, Vol. 6, 2006, pp, 1-19.
[26] P. Weill, and J. W. Ross. “IT governance on one page”. No 349, 2004, pp, 1-18. Available at SSRN: https://ssrn.com/abstract=664612.
[27] B. Tuttle, and S. D. Vandervelde. “An empirical examination of CobiT as an internal control framework for information technology”, International Journal of Accounting information systems, Vol. 8, No.4, 2007, pp. 240-236.
[28] I. Arfyanti1, N. Nursobah, R. Rajiansyah.“IT Governance Of Senayan Library Management System (Slims) Library and Archives Department Of East Kalimantan Province Using COBIT 5.0”, Jurnal Ilmiah MATRIK, Vol. 23, No.2, 2021, pp. 159-166.
[29] G. Ridley, J.Young, and P. Carroll. “COBIT and its Utilization: A framework from the literature”, 37th Annual Hawaii International Conference on System Sciences, Proceedings of the IEEE, 2004, pp, 1-8.
[30] R. P. Dhaniawaty, and E. Susilawati. “The Evaluation of Information Technology Investment Management using the Domain of Portfolio Management (PM) Val IT Framework 2.0 in PT. XYZ. IOP Conference Series”, Materials Science and Engineering Vol. 407, No. 1, 2018, p. 012129. IOP Publishing.
[31] M. Kozina, and D. Popovic. “VAL IT framework and ICT benefits”, Central European Conference on Information and Intelligent Systems. Faculty of Organization and Informatics Varazdin. 2010, pp. 221.
[32] S. Sahibudin, M. Sharifi, and M. Ayat. “Combining ITIL, COBIT and ISO/IEC 27002 in order to design a comprehensive IT framework in organizations”, Second Asia International Conference on Modelling & Simulation (AMS). 2008, pp. 749-753.
[33] M. B. Al Mourad and M. Hussain, "The Impact of Cloud Computing on ITIL Service Strategy Processes," International Journal of Computer and Communication Engineering vol. 3, no. 5, 2014, pp. 367-371.
[34] M. Marrone, F. Gacenga, A. Cater-Steel, and L. Kolbe. “IT service management: A cross-national study of ITIL adoption”, CAIS, 34, 49.
[35] M. H. Koniyo "Identification of components for evaluation e-government governance framework according to regional government characteristics" The 5th Annual Applied Science and Engineering Conference. vol. 1098, No. 5, 2021, pp. 1-8.

متن کامل:

Title

An ICT Performance Evaluation Model based on Meta-Synthesis Approach

Khatrehe Bamary1, Mohammad Reza Behboudi1*, Tayebeh Abbasnejad1

1. Department of Industrial and Public Management, Faculty of Management, Accounting and Economics, University of Hormozgan

Received: 03 Jul 2021/ Revised: 25 Oct 2021/ Accepted: 31 Oct 2021

Abstract

Keywords: Information and Communication Technology; Meta-synthesis; ICT Performance Evaluation; Strategic; Quality; Sustainability

1- Introduction

ICT offers a great opportunity to acquire flexibility in globalized markets, ICT helps companies to recover resilience in uncertainty demands because sharing information is associated with a fast-decision-making process [1]. The fact that the developments in the internet and information and communication technologies (ICT) have made a great contribution to the internationalization of enterprises is a generally accepted approach. For instance, in addition to accelerating the internationalization of corporate ICT, it has also created tools for companies with sophisticated operating structures to communicate more effectively with their customers.

Moreover, it provides new ways for enterprises to conduct their business, has an exchange of ideas and information and transfers them [2]. With the increasing need to have immediate information and flexible working practices in a global market, information transfer tends to be electronic [3]. Management Information System in the education system, using, blog surfing and publishing, social media interaction, which are natural record of student and staff ICT activities. With the help of big data (BD) technology and internet of things (IoT) technology, people’s daily lives are logged and kept in binary data [4]. Information technology in a way has influenced changes to the method, purpose and perceived ability of education [5]

ICTs are being integrated into structures, procedures, and products throughout companies, governments, and communities. The use of ICT increases the supply of information as ICT plays a key role in information sharing and dissemination [6]. Most organizations endeavor to employ Information Communication Technology as a tool for competitive advantage for the accomplishment of the objective of organization as well as enhance the alignment between Information Communication Technology and management strategy. To achieve the former, ICT has been leveraged to improved service and lower the cost of conducting strategic management functions [7]. Generally, technology has made it much easier for people all around the world to gain access to resources, for example, these days technology-assisted learning has been rapidly expanding due to the Coronavirus pandemic [8].

The ICT revolution that has happened over the last several decades has transformed societies across the world. ICT has improved communications, expanded social networking, and has made information more accessible than in any period in history [9]. Information theory and its different applications have profoundly changed our industry, services, and daily life [10]. Indeed, over the past decade, the world has become gradually “hyperconnected.” We live in an environment where the Internet and related services are readily available and accessible, where people and businesses can communicate instantly and machines communicate equally. The exponential development and growth of mobile devices, social media, and big data, are all drivers of this process of hyperconnectivity. Consequently, we are beginning to see fundamental transformations in society [11].

There is an increasing requirement for stronger cost control and a demand for higher returns while diminishing risk in investments, in today’s increasingly competitive business environment. Recognition of the potential impact of ICT on the strategic power of businesses and increasing levels of ICT-expenditure have made the evaluation, justification and control of ICT investments a critically important issue [12].

IT performance management is defined as the areas of goal setting, accountability and monitoring, analyzing, governing and improving ICT performance. IT performance management to achieve business goals requires effective IT management. So, an IT performance measurement process must be selected. Performance measurement frameworks should be balanced, multidimensional, comprehensive and integrated into the framework of organizational performance measurement [13].

For each aspect of the company’s IT under investigation, the questions arise what needs to be measured and how, and what to compare it with, to assess the as-is situation of a company and to assign it a specific quality or degree [14]. Currently, management faces some real dilemmas with respect to IT. Because of competition, organisations invest substantially in IT, even when economically is not justified. Moreover, as IT infrastructure becomes an integral part of an organization's processes and structures, it becomes very difficult to distinguish the impact of IT (positive and negative) from other assets and activities.

Thirdly, it would appear that comparatively few senior executives feel that they understand IT adequately, despite high levels of expenditure [15].

The evaluation of Information Technology (IT) has been seen as a complex task owing both to the difficulty in determining the value of information as well as to the special economic characteristics of the technology which produces it. One proof of this complexity is the dichotomy that exists between the practice and the theory [16]. There have now been many studies on the relevance between the application of information technology (IT) and organizational efficiency or firm performance. It has been shown a significant and positive correlation between IT and firm performance [17.

ICT is the main factor of communication, cooperation, development, knowledge and information management, project management, etc. Therefore, a deep understanding of the impact of ICT on the performance of the organization and its management is very important and very useful for organizations. Large companies need ICT to manage, coordinate, communicate and reduce administrative costs, also they need it to develop skills for social interaction, civic participation, information retrieval and processing, and professional success.

Alignment with the goals of the organization is possible when the application of new technology begins with a study of the needs and strategy of the organization. Therefore, evaluating the performance of organizations in terms of organizational technologies is very important and increases the ability of the organization to progress.

Reviewing the literature regarding the evaluation of ICT performance in organizations indicated there is no consensus on how to evaluate the performance of ICT. Moreover, studies in this respect evaluate ICT performance from different perspectives based on the field of each organization which may confuse organizations to apply them. Therefore, there is a need for a comprehensive model which include all important component and indexes for evaluation of ICT performance in organizations. A model that can play a guiding role for organizations in evaluating their ICT performance and preventing deviation from the right path.

Also, due to the wide range of ICT areas that encompass the areas of strategy, efficiency, maturity, security, and so on, the need for a wide-ranging model that encompasses the key indicators of ICT is necessary. This model can guide an organization to use ICT to achieve organizational goals. This study uses the Meta-synthesis method to provide a model for evaluating ICT performance. Therefore, the purpose of this study is to identify the indicators and dimensions of ICT through the Meta-synthesis method. This research seeks to find answers to these questions:

1. What are the dimensions of evaluating ICT performance in organizations?

2. What are the main and secondary indicators of evaluating the performance of ICT in organizations?

2- Literature Review

Finding a precise definition of what information technology is would probably be a difficult endeavor. If one includes communication technologies then the scope of possible theories and artefacts contained in information technology becomes hard to delimit. One possible definition can be found in Mason et al. stating that IT is "the tangible means by which information is manipulated and carried to its ultimate users". They continue by pointing out some further ingredients modern information systems contain; those are "hardware, software, people, data, and procedures - designed to deliver services intended to improve a social system"[18, p80].

ICT is defined and discussed differently in construction writings but Kraemer and Daniziger, [19, p.593] define the communication element of technology as ‘the actual hardware employed to perform a basic information-processing task’. Gorse and Emmitt [20] also take the view that communication within organizations and between them is concerned with the exchange of information and the management of it. There is a need to distinguish between ‘information technology’ and ‘communication technology’ since information technology is essential in construction for the storage of information but its use does not necessarily mean communication has to take place [3]. Figure (1) shows the typical flow of information through a medium of communication:

$C:\Users\hp\Desktop\Capture.JPG$

Fig. 1. Typical flow of information through a medium of communication [3]

The old adage says: “You cannot manage what you don’t measure.” [21]. By the importance of using information technology for almost every scope, measurement is needed to identify how optimal the IT governance is [18]. By knowing the IT’s importance, it’s necessary to have a measurement or assessment of IT [22].

2-1- Models Presented in the Field of ICT Performance Evaluation

Balanced Scorecard: The balanced scorecard is a method to determine IT performance management that contains both financial and operational measures as solely using financial measures is not enough anymore. The scorecard can be described as an effective tool for performance measurement, organizational assessment and operational alignment. The balanced scorecard can provide organizations with a measurement and management system that supports the IT governance process through a combination of the business balanced scorecard and the IT balanced scorecard. The balanced scorecard contains financial views and is complemented by operational measures of customer satisfaction, internal processes and the organization's innovation. These operational measures are the drivers of future financial performance [13].

IT Governance: During the last decade, IT has been playing a significant role for organizations in achieving their objectives. Recently, IT governance (ITG) has become a critical issue for many organizations in different industries. ITG is a component of the decision-making structure, including processes, as well as relational tools to manage and control [23]. Many organizations suffer from expending too much on IT and other processes due to inefficient alignment between IT and business strategies which leads to the weak performance of the organizations [24].

IT governance is the selection and use of relationships such as strategic alliances or joint ventures to obtain key IT competencies. This is analogous to business governance, which involves make- vs.-buy choices in business strategy. Such choices cover a complex array of interfirm relationships, such as strategic alliances, joint ventures, marketing exchange, and technology licensing [25]. Weill and Ross [26] mentioned that IT governance encompasses five major decisions related to the management and use of IT in a firm:

1. IT principles: high-level decisions about the strategic role of IT in the business.

2. IT architecture: an integrated set of technical choices to guide the organization in satisfying business needs.

3. IT infrastructure: centrally coordinated, shared IT services providing the foundation for the enterprise’s IT capability and typically created before precise usage needs are known.

4. Business application needs: business requirements for purchased or internally developed IT applications.

5. Prioritization and investment: decisions about how much and where to invest in IT, including project approval and justification techniques [26].

Control Objectives for Information and Related Technology (CobiT): One commonly used framework for developing and evaluating technology-intensive information systems is CobiT. This framework was originally a benchmark of best control practices developed and maintained by the Information Technology Governance Institute, the umbrella organization to the Information Systems Audit and Control Association [27]. It covers operational, financial, implementation, planning and monitoring issues for each IT process [28]. COBIT is a comprehensive framework with 34 control objectives that has been developed from 41 international sources. IT processes are clustered into four domains: planning and organization, acquisition and implementation, delivery and support and monitoring. As the framework considers all aspects of information and its supporting IT, management can use COBIT to help provide an appropriate control system for IT [29].

VAL IT Framework: This Framework is a comprehensive and pragmatic organizing framework that enables the creation of business value from IT-enabled investments. In optimizing business strategy and Information Technology (IT) to generate value for the company, the implementation of Val IT Framework is required. Val IT framework helps increase the probability of optional investment and create value by the highest potential [30]. There are three domains in VAL IT framework: Value governance, Portfolio Management and Investments Management. Each domain has a specific number of processes, goals, inputs, outputs and process metrics [31].

ITIL (Information Technology Infrastructure Library): ITIL is a de-facto standard which introduced and distributed by the Office of Government Commerce (OGC) in the UK and includes all IT parts of organizations [6]. Presently, ITIL is the most widely accepted approach to IT Service Management in the world. It has an iterative, multidimensional and lifecycle form structure [32]. ITIL enables organizations to deliver appropriate services and continually ensure they are meeting business goals and delivering benefits. The ITIL best practices are currently detailed within five core publications that introduce five Service Lifecycle stages: Service Strategy, Service Design, Service Transition, Service Operation, and Continual Service Improvement [33]. Identified ITIL adoption benefits include: improved focus on ITSM, more rigorous control of testing and system changes, more predictable infrastructure, improved consultation with IT groups within the organization, smoother negotiation of service level agreements, reduced server faults, seamless end-to-end service, documented and consistent IT processes across the organization, an effective change advisory board, and consistent logging of incidents [34].

Models and frameworks like: Committee of Sponsoring Organizations (COSO), Project Management Body of Knowledge (PMBOK), Project Resource Organization Management and Planning Techniques (Prince 2), Capability Maturity Model Integration (CMMI), Technology acceptance model (TMA), The Open Group Architecture Framework (TOGAF), Zachman Framework, Next Generation Operational Support Systems (NGOSS) and ICT standards have also been used in the proposed Meta-synthesis model.

By reviewing and studying articles in the field of ICT, we came to the conclusion that studies and models of evaluating and managing ICT fall into three dimensions: strategic, quality and sustainability. In table (1) each of the models that have been examined in the field of ICT performance evaluation and the degree of emphasis of each model on the dimension is shown.

Table 1: Models relevant to the field of ICT performance evaluation

Reviewed models	strategic	quality	sustainability
Balanced scorecard
IT Governance
Control Objectives for Information and Related Technology (CobiT)
VAL IT framework
ITIL (Information Technology Infrastructure Library)
Committee Of Sponsoring Organizations (COSO )
Project Management Body of Knowledge (PMBOK)
Project Resource Organization Management and Planning Techniques (Prince 2)
Capability Maturity Model Integration (CMMI)
Technology acceptance model (TMA)
The Open Group Architecture Framework (TOGAF)
Zachman Framework
Next Generation Operational Support Systems (NGOSS)
External quality standard of ISO 9126 software (ISO / IEC 9126)
IT Corporate Governance Standard-2008 ISO / IEC 38500
Information Technology Software Development Standard ISO / IEC 12207 - 2008
Information Security Management System Standard (ISO / IEC 27001)

3- Research Method

This research is fundamental in terms of purpose. It is qualitative in terms of how data is collected. In this research, the meta-synthesis method is used to identify the indicators of ICT performance evaluation. The study of indicators in this study started in April 2019 and lasted until January 2020.

Considering that the meta-synthesis method has been used to identify the indicators for evaluating the performance of ICT, qualitative research method was considered for its implementation.

In order to identify the criteria for evaluating the performance of ICT through the meta-synthesis method, the seven-step method of Sandelowski and Barroso (2007) has been used. In the first step purpose and research question has been set. In the second step, relevant researches were extracted over a specified period of time by systematic searching of the databases. So, the appropriate keywords were used in combination and individually, then the literature was reviewed systematically. In the third step, in searching and selecting appropriate texts, the researchers eliminated a number of articles in each review, based on ten CASP criteria for each of the rated articles, and finally, 89 final papers were selected. In the fourth stage, the extracted codes are presented in a table and submitted to the experts to reach the final conclusion. In the fifth stage, after reviewing the codes and classifying them, 108 codes were selected and categorized into 9 themes and 3 categories; actually, in this stage, the qualitative findings were analyzed and composed. In the sixth stage, quality control, the kappa coefficient of the SPSS software was used to evaluate the reliability of the findings. The kappa coefficient of 0.733 showed the reliability of indexes and dimensions classification. The last stage was the presentation of findings.

In this research, various databases, journals, conferences and search engines have been examined. Table (2) shows the research database

Table 2: Models studied in the field of ICT performance evaluation

Scientific and public resources	Informational Base	Documents and journals reviewed
International scientific databases	www.scholar.google.com	Scientific-research extension and conference articles
	www.sciencedirect.com
	www.proquest.com
	www.springer.com
	www.emeraldinsight.com
	www.isi.edu
	www.elsevier.com
Persian scientific databases	www.sid.ir	Scientific-research, extension and conference articles
	www.noormags.ir
	www.magiran.com
	www.civilica.com
	www.ensani.ir
	www.elmnet.ir
	www.ganj.irandoc.ac.ir
Public Database	www.google.com	Valid scientific-research articles
Public Database	www.yahoo.com	Valid scientific-research articles

In this research, 32 different keywords related to the evaluation of ICT performance have been used for search. These are preliminary main search terms: Performance Evaluation of Information and Communication Technology, IT Evaluation, IT Governance, ICT ability, IT Information, IT Assessment, Configure ICT systems, IT Change, Electronic government performance, IT Measurement, IT Alignment in organization, Efficiency and effectiveness in IT, Evaluation of Technology, IT Control, IT Strategy, Coordination in IT, Virtual system evaluation, IT Monitoring, IT Management in IT, Reliability, Maturity, IT Risk, IT Engineering, Evaluation of hardware in organization, Evaluation of software in organization IT. Both “information and communication technology” and its abbreviation namely ICT and IT were searched. Table (3) indicates the criteria for accepting articles.

Table 3: Acceptance criteria for articles

Acceptance criteria	This study
Scope of studies	field of IS, ICT & management
Research language	English and Persian
Period of Studies	1997 to 2018
Study method	Qualitative, quantitative, review
Subject of studies	Evaluating ICT performance
Study conditions	Evaluating ICT performance
Type of studies	published in journals & conferences

A search of databases using the keywords mentioned found 516 articles. Among them, 89 articles were used for analysis. 40 articles were related to domestic researches (Persian) and 49 articles were related to international researches. Appendix 1 shows the list of selected articles. Figure (2) shows a summary of the article selection process

Fig. 2: Article search and selection results

The final articles were evaluated in terms of research objectives, methodology, research design, sampling, data collection, perfection, ethical considerations, the accuracy of analysis, clear expression of findings, research value and categorized from excellent to poor. The chart (1) shows the ranking of articles based on the ten criteria mentioned

Chart.1. Ranking of reviewed articles based on the ten criteria

In the analysis process, the researchers identified 920 sub-indices (codes). Based on the frequency of 104 sub-indices (code) in the form of an online questionnaire (based on the Likert scale) which was distributed among 22 ICT experts. The detailed information about these experts who were ICT managers and specialists are presented in Table (4).

Table 4: Detailed information about the research participants

Row	Age	Sex	Experience	Field Of Study	Education	Position
1	31-35	M	6-10	Software Engineering (S.E.)	M.S.	Mid-level Manager
2	25-30	F	1-5	Hardware Engineering (H.E.)	M.S.	Expert
3	36-40	M	6-10	IT Engineering	Ph.D	Mid-level Manager
4	50-60	M	16-20	Math	Ph.D	Top Manager
5	31-35	M	6-10	S.E.	M.S.	Expert
6	36-40	M	16-20	S.E.	M.S.	Operational Manager
7	41-45	F	6-10	Computer Science (C.S.)	M.S.	Top Manager
8	31-35	F	16-20	IT Engineering	M.S.	Mid-level Manager
9	46-50	M	11-15	IT Engineering	M.S.	Mid-level Manager
10	31-35	M	11-15	C.S.	M.S.	Operational Manager
11	31-35	M	11-15	Industrial Engineering	B.S.	Mid-level Manager
12	25-30	M	1-5	C.S.	B.S.	Mid-level Manager
13	25-30	F	11-15	Business Management	Ph.D	Mid-level Manager
14	40-45	M	6-10	S.E.	M.S.	Expert
15	31-35	M	1-5	H.E.	M.S.	Expert
16	31-35	M	11-15	Executive Management	M.S.	Expert
17	31-35	M	6-10	S.E.	M.S.	Mid-level Manager
18	36-40	M	6-10	C.S.	Ph.D	Expert
19	31-35	F	6-10	Business Management	B.S.	Expert
20	41-45	F	11-15	C.S.	B.S.	Expert
21	41-45	F	6-10	H.E.	B.S.	Mid-level Manager
22	31-35	F	11-15	H.E.	B.S.	Expert

Finally, according to experts, 104 sub-indicators (code) were classified into 9 indicators (themes) and 3 dimensions (categories). In this research, the main author used his comparison of opinions with another expert to control the classification of the extracted indicators and for this purpose, It was used the method of agreement between two coders. In this research, in addition to the researcher, another expert of the same text encodes separately without knowing the classification of the researcher codes. In the present study, this evaluation was performed on 116 codes extracted with the help of the expert in which the Kappa index value in SPSS software was equal to 0.733. According to the significance coefficient of 0.000 of this coding and category, classification is accepted. The results of SPSS software calculations are shown in Table (5).

Table 5: Test of agreement between the researcher and one of the experts in coding indicators

	Value	Asymptotic Standard Errora	Approximate Tb	Approximate Significance
Measure of Agreement Kappa	0.733	0.45	19.251	0.000
N of Valid Cases			104	Number of samples acceptable

4- Research Findings

By examining the research background related to ICT performance evaluation and also reviewing the models that have been presented in this field, it is realized that each of the models and researches relatively evaluates ICT in a one-dimensional and somehow different way. For example, models such as Balanced Scorecard, Qubit, Val IT, IT governance, Zakman framework, etc. examine IT from an organizational and strategic perspective. Models and standards related to the system and software such as ITIL, further evaluate the quality or operational dimension of ICT. However, the issue of sustainability (environmental, economic and social) is unnoticed or a little considered in most ICT evaluation models and studies. In this study, bearing in mind the many advantages of using a hybrid approach to identify the most important indicators of ICT, we provide an evaluation model for organizations to be aware of the state of their ICT and manage it well.

Using the meta-synthesis approach 104 sub-indices (code) were classified into 9 indices (theme) and 3 dimensions (category). Indicators related to ICT performance evaluation are presented in Table (6). Its main dimensions include strategic, quality and sustainability. The strategic dimension is related to long-term goals and proper planning for the implementation and use of technologies and, if necessary, replacement of that technology and considering whether this technology meets long-term and short-term organizational goals, its alignment. It also shows the level of organizational participation in the proper use of technology. The strategic dimension includes the strategy of the organization that shows the general orientation of the organization regarding ICT. And ICT strategy, which is used to synchronize technology with the goals of the organization, and finally, alignment and participation are a subset of the strategic dimension which aligns the correct orientation of the ICT strategy and the organization's strategy against each other and cooperation: It is a two-sided relationship. Participation is the amount of effort of personnel in order to use the right technology and in the direction of the right organizational goals for maximum efficiency.

The quality dimension includes maturity and Performance. Maturity of IT indicates the degree of progress and potential and actual ability of technology in the organization to advance organizational goals. In general, the maturity of ICT means the extent to which it can adapt to change in order to achieve its goals. The “performance” part is related to the quaintly or operational and engineering dimension related to ICT.

Sustainability is one of the new areas in the field of ICT and includes three environmental, social and economic dimensions that go beyond organizational goals and show how much information technology supports the environment and, how much economically and socially contribute to the society. Indicators in this article extracted from 89 studies; The subject of these articles is related to the most repetitive articles in the field of evaluation of ICT performance. Most articles are about the strategic, quality or operational, and sustainability dimensions. By studying articles and extracting direct and indirect indicators (derived from the general concept of articles), the indicators have been extracted; and then reviewed by experts in this field and selected based on the importance of the indicators in Table (6), which is categorized based on strategic, quality or the concept of sustainability.

Table 6: Strategic Dimensions, indicators and sub-indices of ICT performance evaluation

Indices

Sub-indices

Organizational strategy

1. The extent of ICT support for the organization's strategy

2. ICT support for organizational structure (business organization)

3. ICT support for the organization (coordination of activities within the organization)

4. Alignment of ICT with the priorities of the organization

5. Clarity of ICT goals and strategies for the organization and vice versa

6. ICT compliance with key policies and regulations of the organization

7. The level of organizational readiness in improving the field of ICT and vice versa

8. The impact of ICT on the strategic development of the organization

9. ICT attention to the long-term and short-term vision of the organization

10. The impact of ICT on the development of organizational relationships (internal and external relations)

11. Identify and meet the needs of stakeholders through ICT

12. ICT attention to organizational values and norms and organizational culture

IT strategy

1. Support IT strategy

2. The extent of ICT governance in the organization

3. Clarity of criteria and standards related to the ICT sector

4. Paying attention to the architecture of ICT-related departments (coordination of ICT activities inside and outside the organization)

5. The degree of attention to ICT goals

6. Compliance with ICT requirements (laws, policies, etc.)

Alignment

1. Coordinate and support ICT of the organization's results and goals

2. Coordinating ICT with the needs of the organization

3. Coordinating ICT activities with the organization's vision

Participation

1. Involvement of senior executives in the field of ICT

2. Participation of all managers in the field of ICT

3. Employee participation in the field of ICT

4. Participation of foreign stakeholders in the field of ICT

Table 6: Quality Dimensions, indicators and sub-indices of ICT performance evaluation

indices

sub-indices

Maturity

1. Human Resource Management ICT Project (Organizational Planning, Employee Recruitment)

2. Time management of ICT projects (definition and estimation, sequencing, creation and development, time control)

3. The impact of ICT on reducing organizational risk

4. Risk level in the field of ICT

5. Scope management of ICT projects (defining the steps from start to finish)

6. Integration management of ICT projects (program creation and integrated change control)

7. Procurement management of ICT projects (selection of supply source and contracts for the project)

8. Communication management (information distribution and reporting) ICT projects

9. ICT project environment (creating a suitable environment for implementation)

10. Prioritization of ICT (implementation of the most important projects as needed).

11. Existence of innovation vision for ICT (considering the technological needs of the organization in the future)

12. Innovation in ICT system (software and hardware)

13. Innovation in organizational processes through ICT

14. Innovation in individual and team ICT performance

15. Data and application architecture (coordination between data and applications)

16. Management of technical and physical infrastructure (hardware)

17. ICT installation management (creating infrastructure and platform for installing alternative technologies)

18. Ability to maintain equipment in the field of ICT

19. Contract management of new technology systems (contracts with individuals, organizations and institutions to implement and create technology)

20. Performance of software computing and software networks

21. Interchangeability (Ability to replace and replace with new technology)

22. Testability (existence of test courses before the implementation of new technology)

23. Improving the software environment

24. Consistency and continuous improvement of ICT processes

25. Process maturity management

26. The degree of intelligence of ICT processes

27. ICT Process Reengineering

28. Technology change management (competitive, cultural and organizational change)

29. Life cycle management of ICT products

30. Portability or (Transition scheme: the degree of ability to create a new ICT system)

31. Interoperability of different areas of ICT with each other

32. Ability to analyze information

33. Flexibility in the field of ICT

Performance

1. Ability to understand information and software trends

2. Ease of use of ICT systems

3. Accuracy of information in the field of ICT

4. Management of accidents and problems in the field of ICT

5. Security and use of security equipment in the field of ICT

6. Privacy in the field of ICT

7. The attractiveness of the software environment for users and employees

8. Avoid mistakes and rework

9. Timely information

10. User and staff accessibility

11. Reliability (validity) of the ICT field (fault tolerance and repair)

12. Usefulness (usefulness) for the organization

13. Up-to-date information

14. Satisfaction of users and employees with ICT

Table 6: Sustainability Dimensions, indicators and sub-indices of ICT performance evaluation

indices

sub-indices

Environmental

1. Use of technologies in line with reducing environmental pollution

2. Green IT training

3. Green supply chain management in information technology

4. Use technology for energy efficiency

5. Equipment aligned with the green landscape

Economic

1. The amount of investment in the field of ICT

2. The amount of profit through ICT

3. Investment effectiveness in ICT

4. Budgeting according to the cost of ICT

5. Return on investment through ICT

6. Increase the market share of the organization using ICT

7. The amount of budget allocated to ICT

8. Cost reduction in ICT

9. Effectiveness through ICT

10. ICT audit

11. ICT asset management

12. The impact of ICT on creating a competitive advantage

Social

1. Specialized training courses in the field of ICT

2. Creating knowledge through research and development and learning in the field of ICT

3. Motivate the acquisition of knowledge in the field of ICT

4. Attract and retain specialized and capable personnel in the ICT sector

5. Update ICT staff skills

6. Educate users on the use of ICT technologies

7. Staff training in the field of ICT

8. Observance of ethical principles in the field of ICT

9. The impact of ICT on the social environment and vice versa

10. The extent to which ICT is affected by the legal environment and vice versa

11. Social support for ICT

12. The impact of ICT on strengthening social relationships

13. Transparency and accountability in the field of ICT

14. Acceptance of new technologies and technologies used

15. Establishment of reward and incentive system in the field of ICT

The final model for evaluating ICT performance is shown in Figure (3).

Fig. 2: ICT Performance Evaluation Model

At this stage, the selected indicators to evaluate the performance of ICT in the form of a questionnaire were offered to ICT experts and managers in this field. The questionnaire consisted of 113 questions based on the number of indicators extracted from the meta-synthesis stage. In this questionnaire, the importance of indicators was asked in order to be rated from insignificant to very significant. The participants also were asked to mention their suggestions regarding the indicators and the model detentions. Chart (2) shows the importance of each of the ICT performance evaluation indicators based on the opinion of study participants. This data was obtained using SPSS software.

Chart 2: The importance of ICT performance evaluation indicators

5- Conclusion, Discussion, and Suggestions

Today, ICT is one of the main enablers of any organizational success, as well as one of the most important features of an advanced and modern society. Organizations need to have sound performance in the field of ICT. Hence, due to some unfavorable experiences of organizations in the field of ICT, a comprehensive evaluation of ICT performance is an undeniable principle.

With the hybrid approach, which is one of the most widely used methods for reviewing and summarizing the literature, a relatively comprehensive model to evaluate the performance of ICT is presented. This qualitative meta-synthesis research, which is conducted in seven steps, identified the detentions, indicators and sub-indicators for evaluating the performance of ICT in organizations. In the meta-synthesis method, 516 articles were reviewed, of which 85 articles were used for the final analysis. Out of 85 final articles, about 920 sub-indexes (codes) were extracted, then 104 sub-indexes (codes) were selected from them according to the opinions of ICT experts and managers. These codes were classified into three dimensions namely strategic, quality and sustainability and 9 indicators. The extracted indicators are: organizational strategy, IT strategy, alignment, participation, maturity, performance, environmental, social and economic. Among the extracted indicators, performance and maturity, according to experts in the field of ICT, were more important.

In general, the contribution of this study is offering a model that includes the most important and practical indicators for evaluating the performance of ICT in all organizations. Every organization regardless of the type of operation, whether in the field of manufacturing or service, can evaluate its ICT performance based on the model presented in this study. Paying attention to strategic, quality and sustainability dimensions can help the organization to gain the advantage of ICT and consequently has better performance in all aspects including social responsibility.

To examine the three dimensions of the model in an organization, the most important goal of the strategic dimension of ICT is the strategic alignment of IT and business, which means the development of plans and activities in a way to enable the realization of the goals and business strategies of the organization. The operational dimension of ICT refers to storage, processing speed, IT updates, maintainability, memory and more on the technical and physical aspects of information technology. This dimension is related to ICT infrastructure.

ICT not only forms an industry; rather, it affects all sectors of the economy and works to integrate and activate technologies. ICT has a serious impact on society and uses the important consequences of development in the economic, social and environmental fields. In terms of sustainability, attention to the environment, economy and social aspects are very important. Moreover, ICT effects on almost all aspects of a country, it should be considered at the micro and macro levels of the country. ICT, as an enabler of reforming the public sector, has been implemented to reinvent governments to improve performance and create public value.[35]

In the social and cultural field, technologies including ICT should be evaluated because they have a great impact on the culture of a society. Many technologies, if not used properly, can lead to cultural poverty. The environmental aspect of technology is very important for the new generation and the next generation. Environmentally friendly technologies can save society from destruction. Due to this consideration, the presented model in this study covered this aspect as well.

Organizations should use ICT that align with their strategic goals. The application of ICT tools and technologies must help organizations to achieve better its goals which can be gaining competitive advantages or financial performance or improvement in internal processes which all of these criteria are included in the presented model.

Evaluating the performance of ICT shows how much the technology that is used has taken into account the strategic, quality and sustainability dimensions. The evaluation of the ICT performance can indicate the need for ICT to be changed or stabilized or improved. In all organizations even societies, ICT evaluation and updating play a very important and key role; it can guarantee the success of the organization in today's competitive society.

This research has made every effort to suggest a relatively comprehensive model for evaluating the performance of ICT in organizations; and the indicators provided can, to a large extent, inform the organizations of its situation and progress in using the right ICT services, technologies and tools. As an implication for practice, managers can apply ICT performance evaluation indicators which found in this study in their organizations.

Due to the fact that the performance index of the model has the highest percentage of importance among other indicators, it is necessary for the organization to work on the attractiveness of the software environment for users, up-to-date information and its availability to users. Furthermore, the maturity index of the model found as the second most important index in evaluating ICT, so to improve the maturity index, organizations need to improve the architecture of data and applications, the management of the configuration of ICT components, the interoperability of different areas of ICT in the organization.

The distinguishing feature and contribution of this meta-synthesis research is the presentation of relatively a comprehensive model of ICT evaluation from a managerial and professional point of view. Because the developed model out of the literature review, were evaluated and confirmed by ICT specialists and managers in the field. Although, as a study limitation, it must be mentioned that we did not have access to all of the scientific and specialized resources.

Future researches can examine this model in evaluating some organizations in different fields to further refining the model if necessary. Moreover, this study applied a static approach in evaluating ICT performance, future researches can use a dynamic approach in this respect. Carrying out more detailed research in the field of ICT maturity can be a useful study.

References

[1] J. L. García-Alcaraz, V. Martínez-Loya, J. R. Díaz-Reza, J. Blanco-Fernández, E. Jiménez-Macías, and A. J. G. Lopez. “Effect of ICT integration on SC flexibility, agility and company’ performance: the Mexican maquiladora experience” Wireless Networks, Vol. 26, No.7, 2020, pp. 4805-4818.

[2] M. Arslandere, İ. Tuncer, and A. D. A. Serkan. “The Impact of ICT Use in Promotional Activities on Export Performance: An Empirical Investigation”, Business & Management Studies: An International Journal, Vol. 8, No.2, 2020, pp. 2384 – 2413.

[3] A. Rimmimgton, G. Dickens, and C. Pasqire. “Impact of Information and Communication Technology (ICT) on construction projects”, Organization, technology & management in construction: an international journal, Vol. 7, No.3, 2015, pp. 1367-1382.

[4] X. Chaojun, Y. Wei, H. Chenlin, and S. Congying. “Design and Validation of School ICT Monitoring & Evaluation Framework: A Case Study”, American Journal of Education and Information Technology. Vol. 4, No.2, 2020, pp. 78

[5] T. B. Gyamerah. “Using e-learning to improve the performance of first year nursing students in ICT”. GSJ, Vol. 8, No.6, 2020, pp.-250-257.

[6] I. Apulu, and A. Latham."An Evaluation of the Impact of Information and Communication Technologies: Two Case Study Examples". International Business Research, Vol. 3, No. 4, 2011, pp. 3-9.

[7] M. Okumu Ujunju, G. Wanyembi, and F. Wabwoba." Evaluating the Role of Information and Communication Technology (ICT) Support towards Processes of Management in Institutions of Higher Learning", (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No.7, 2012, pp. 55-58.

[8] A. M. Sayaf, M. M. Alamri, M. A. Alqahtani and W. M. Al-Rahmi. "Information and Communications Technology Used in Higher Education: An Empirical Study on Digital Learning as Sustainability". Sustainability, Vol. 13, No. 13, 2021.

[9] H. Toya, and M. Skidmore. "Information/communication technology and natural disaster vulnerability", Economics Letters, 2015, pp. 143-145.

[10] A. André." The Information Technology Revolution in Health Care", In: André A. (eds) Digital Medicine. Health Informatics. Springer, Cham, 2019,

[11] S. Dutta, and B. Bilbao-Osorio. “The Global Information Technology Report 2015”, World Economic Forum, Vol. 1, No.1, 2015, pp. 200-42.

[12] K. Milis, and R. Mercken. “The use of the balanced scorecard for the evaluation of information and communication technology projects”, International Journal of Project Management, Vol. 22, No.2, 2004, pp. 87-97.

[13] S. Haanappel, R. Drost, F. Harmsen,, S. Brinkkemper, and J. M. Versendaal. “A framework for IT performance management”, Available from www: http://www. Cs. Uu. Nl/research/ techreps /repo/CS, 2011(006).

[14] J. Becker, R. Knackstedt, and J. Pöppelbuß, "Developing maturity models for IT management", Business & Information Systems Engineering, Vol.1, No.5, 2009, pp. 213-222.

[15] J. McKay, P. Marshall, and L. Smith. “Steps towards effective IT governance: strategic IS planning, evaluation and benefits management”, PACIS 2003 Proceedings, 65.

[16] E. H. Arribas, and P. S. Inchusta. “Evaluation models of information technology in Spanish companies: a cluster analysis”, Information & Management, Vol. 36, No.3, 1999, pp. 151-164.

[17] A. Albadvi, A. Keramati, and J. Razmi. “Assessing the impact of information technology on firm performance considering the role of intervening variables: organizational infrastructures and business processes reengineering”. International Journal of Production Research, Vol. 45, No.5, 2007, pp. 2697-2734.

[18] B. C. Stahl. “Information technology, responsibility, and anthropology”, In Proceedings of the 35th Annual Hawaii International Conference on System Sciences. IEEE, 2002, pp.1-10.

[19] K. L. Kraemer & J. N. Danziger, "The impacts of computer technology on the worklife of information workers". Social Science Computer Review, Vol. 8, No. 4, 1990, pp. 592-613.

[20] C. A. Gorse, & S. Emmitt. Investigating interpersonal communication during construction progress meetings: challenges and opportunities. Engineering, Construction and Architectural Management., Vol .10, No. 2, 2003, pp 233-245.

[21] O. Zizlavsky. “The balanced scorecard: Innovative performance measurement and management control system”, Journal of technology management & innovation, Vol. 9, No.3, 2014, pp. 210-222.

[22] N. Y. Wijayanti, W. Setiawan, and R. A. Sukamto. “Performance Assessment of IT Governance with Balanced Score Card and COBIT 4.1 of Universitas Pendidikan Indonesia”, Journal of Physics: Conference Series, Vol. 812, No. 1, 2017, p. 012072. IOP Publishing.

[23] M. Uii, H. Gusaptono, M. Effendi and N. Charibaldi. “The Information Technology (IT) Adoption Process and E- Readiness to Use within Yogyakarta Indonesian Small Medium Enterprises (SME)”, International Journal of Information and Communication Technology Research, Vol. 2, No. 1, 2012, pp. 1-12.

[24] F. Salehi1, B. Abdollahbeigi1 and S, Sajjady. “Impact of Effective IT Governance on Organizational Performance and Economic Growth in Canada”, Asian Journal of Economics, Finance and Management, Vol. 3, No. 2, 2021, pp, 14-19.

[25] M. Simonsson, and P. Johnson. “Defining IT governance-a consolidation of literature”, The 18th conference on advanced information systems engineering, Vol. 6, 2006, pp, 1-19

[26] P. Weill, and J. W. Ross. “IT governance on one page”. No 349, 2004, pp, 1-18. Available at SSRN: https://ssrn.com/abstract=664612.

[27] B. Tuttle, and S. D. Vandervelde. “An empirical examination of CobiT as an internal control framework for information technology”, International Journal of Accounting information systems, Vol. 8, No.4, 2007, pp. 240-236.

[28] I. Arfyanti1, N. Nursobah, R. Rajiansyah.“IT Governance Of Senayan Library Management System (Slims) Library and Archives Department Of East Kalimantan Province Using COBIT 5.0”, Jurnal Ilmiah MATRIK, Vol. 23, No.2, 2021, pp. 159-166.

[29] G. Ridley, J.Young, and P. Carroll. “COBIT and its Utilization: A framework from the literature”, 37th Annual Hawaii International Conference on System Sciences, Proceedings of the IEEE, 2004, pp, 1-8.

[30] R. P. Dhaniawaty, and E. Susilawati. “The Evaluation of Information Technology Investment Management using the Domain of Portfolio Management (PM) Val IT Framework 2.0 in PT. XYZ. IOP Conference Series”, Materials Science and Engineering Vol. 407, No. 1, 2018, p. 012129. IOP Publishing.

[31] M. Kozina, and D. Popovic. “VAL IT framework and ICT benefits”, Central European Conference on Information and Intelligent Systems. Faculty of Organization and Informatics Varazdin. 2010, pp. 221.

[32] S. Sahibudin, M. Sharifi, and M. Ayat. “Combining ITIL, COBIT and ISO/IEC 27002 in order to design a comprehensive IT framework in organizations”, Second Asia International Conference on Modelling & Simulation (AMS). 2008, pp. 749-753.

[33] M. B. Al Mourad and M. Hussain, "The Impact of Cloud Computing on ITIL Service Strategy Processes," International Journal of Computer and Communication Engineering vol. 3, no. 5, 2014, pp. 367-371.

[34] M. Marrone, F. Gacenga, A. Cater-Steel, and L. Kolbe. “IT service management: A cross-national study of ITIL adoption”, CAIS, 34, 49.

[35] M. H. Koniyo "Identification of components for evaluation e-government governance framework according to regional government characteristics" The 5th Annual Applied Science and Engineering Conference. vol. 1098, No. 5, 2021, pp. 1-8.

* Mohammad-Reza Behboudi

behboudi@hormozgan.ac.ir

اشتراک گذاری

آدرس مقاله

An ICT Performance Evaluation Model based on Meta-Synthesis Approach

رایمگ

پیوندهای سایت

مراکز مرتبط

پشتیبانی

صفحات رسمی