Development and Psychometric Evaluation of Nursing Audit Tool

Copyright © 2020 The Author(s). This is an open-access article distributed under the terms of the Creative Commons Attribution License (http:// creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 1. Background Health services are provided to patients with various diseases and needs; patient care services that are provided in a health institution are both large-scale and complicated. Patients, however, do not get the opportunity to control the treatment and healthcare services supplied to them, measure the quality of that service, or decide what is necessary for them. For this reason, the responsibility of determining what is good for patients and monitoring the quality of treatment and healthcare services belongs to the managers who work at all stages of healthcare.1 Many studies concerning the measurement, evaluation, and improvement of the quality of health services have been conducted over many years. As far back as 1863, Florence Nightingale talked about gathering and evaluating the outcomes of patient care and developing the quality of this care.2 Today, most hospitals have an internal control system, and they use various methods to monitor the quality of the patient care service they provide. Although some studies have evaluated monitoring indicators related to the outcomes of health services, the number of studies concerning the monitoring of the healthcare process is quite limited in the literature.3-5

predetermined standards, rules, and purposes. 6,7 The control stage has three steps: preventive control, concurrent control, and detective control. Preventive control means prevention of an error before its occurrence. Definitions and performance standards of nursing jobs are often designated by hospitals and include processes, standards of nursing activities, and distributional standards for the available resources. To provide preventive control, recruitment, orientation, training, and development standards of the nursing staff are also stated by each hospital itself. Concurrent control enables periodical monitoring of ongoing nursing activities and their compliance with the quality standards. The aim of concurrent control is to secure the desired standards in nursing service outcomes. It also has a corrective role as a precaution against possible mistakes or errors. Detective control is the evaluation of the outcomes of nursing services and whether they met the established standards. Such controls rely on patient care outcomes such as patient satisfaction, falls and injuries, medication errors, decubitus ulcers, extravasations and urinary catheter infections, etc. and nursing staff outcomes such as job satisfaction, number of staff conflicts and bullying complaints, intention to leave, turnover, etc. 6,8 There are many studies in the literature on preventive and detective control, but there are a limited number of studies on concurrent control, which is observing ongoing nursing services at clinics in order to compare practices with nursing care standards, identify the conditional risks of nursing service, prevent medical errors and problems, and extend good practices in the workplace. [3][4][5] Data on healthcare outcomes in health services is obtainable from frequently observing ongoing healthcare activities, gathering data, and assessing outcomes. [5][6][7][8][9] Accordingly, clinical nurse supervisors have an important role in observing clinical nurses while they deliver nursing care, guiding them if needed, giving feedback about their performance, identifying the risks of healthcare, and preventing nursing care errors and problems.
Many upper level nurse managers in our country have not observed, audited, or controlled an ongoing nursing service in order to evaluate nursing outcomes. Moreover, the data on healthcare outcomes that was gathered by the Ministry of Health in order to assess the quality of patient care in hospitals was not based on periodic measurements, but on annual predictions from the hospitals' healthcare quality department. 10 Consequently, many hospitals in Turkey have not conducted studies to analyze healthcare data, have not identified and rectified adverse events or workflow failures related to patient care outcomes, and have not improved the quality of healthcare services. Due to insufficient control of nursing care quality, a nursing services audit is needed to guide the nurse managers in hospitals. This study aimed to develop a nursing service audit to observe and assess ongoing nursing services.
Checklists have been shown to improve care and reduce morbidity and mortality in the healthcare setting. Control of the internal audit is made possible for hospital and nursing managers with an evaluation and benchmarking of the ongoing nursing services and their outcomes according to the predetermined goals, intentions, regulations and standards of services. 3,8 The relevant literature shows that regular monitoring of the health services process contributes significantly to improving health service outcomes. [3][4][5]9

Objectives
This study is methodologically planned to establish the basic principles of nursing care standards and practices in wards, develop a nursing audit tool for monitoring and controlling ongoing nursing activities periodically, and evaluating nursing care quality. Upper and middle level nurse managers can monitor and control nursing services with the help of the developed nursing audit tool hierarchically, evaluate the basic outcomes of the nursing service, benchmark the differences between wards in the hospital, and improve the quality of nursing services.

Sample of the Study
The current research was carried out in all wards (medical, surgical, maternity, cardiovascular surgery, cardiology, intensive care) of a 93-bed private hospital in Ankara, Turkey, over an eighteen-month period between November 2017 and April 2019. The nursing service audit, used as a data collection tool, was used according to in-patient clinics. Seventy-seven nurses worked at the hospital on a three-shift schedule: 07.00-16.00 (day shift), 15.00-24.00 (evening shift) and 23.00-08.00 (night shift). Supervisor nurses worked evening and night shifts on weekdays and day, evening, and night shifts on weekends. The hospital nursing administration was run by a director of nursing, a deputy director of nursing, and four supervisor nurses at the time of the research. A nurse manager was responsible for administering each ward in the hospital.
When supervisor nurses completed the nursing audit tool they used observation, patient file review, patient room visit, interview with the patient, examination of records kept on the work related to the service (for example; crash cart checklist, narcotic records, etc).
Nurses were assigned patients by the nurse manager and provided nursing care to five or seven patients during their hours at the clinic. Therefore, every nurse had 5 to 7 patients and every patient had a nurse in the ward. To-date, the hospital has been accredited three times by the Joint Commission International (JCI) accreditation program, and some healthcare studies have been conducted in order to monitor and enhance patient care processes and outcomes ( Figure 1).

Nursing Service Audit Tool
A nursing service audit tool was used to collect data. Based on the literature and expert consideration, a conceptual structure of the nursing audit criteria was previously developed and published. 11 After examining the literature on the establishment of the indicators to be monitored, assessed, and compared for the measurement of the performance of nursing services offered at hospital departments, an item poll including 120 questions was created. [12][13][14][15][16][17] During development of the tool, discussions regarding all the criteria for quality of care were held with nurses and nurse managers. Then, the Delphi method was used to gather expert considerations. The research findings were published. 11 The criteria are related to nursing care, keeping regular and full records of nursing, the layout of the ward, checking and remedying narcotic medicines, crash cart and supplies, and control by head nurses. Although it is accepted that some criteria can change between departments in a hospital, the aim was to identify the basic criteria for any ward. The nursing audit criteria were also employed to create a form to guide upper and middle level nursing managers in their service department visits. The nursing service audit has a total of 63 items, including 28 items related to patient care, eight items related to indirect patient care, four items related to wards, and 23 items related to head nurse performance. The nursing service audit summarizes parameters (described in Table 2) into key categories, and used as a data collection tool, it is employed according to in-patient clinics.
Each item of the audit is evaluated by interviewing patients, relatives, and nurses, and by determining whether a particular duty was discharged. The items are then assigned points as performed (10 points), partially performed (5 points), or not performed (0 points), or as yes (10 points) or no (0 points). The overall score is then divided by the number of items. The final ward score is, therefore, out of ten points.
The hospital supervisors visited each ward with the nursing audit tool and evaluated each item for a possible ten points. Two supervisor nurses collected the data using the nursing audit tool during their shifts. Supervisors were trained on the use of the nursing audit tool to avoid any difference between evaluators. The data was gathered between December 2017 and April 2019. Moreover, two supervisors evaluated and scored the wards' performance between January and July 2018 with the nursing audit tool on the same day but at different times in order to analyze the inter-rater consistency of the tool.

Nursing Outcomes of the Units
The literature suggests that monitoring health service processes has a positive effect on patient care outcomes. 3,4,18 Nursing outcomes of the units were used to evaluate the predictive validity of the nursing audit tool. Patient identification errors, near-miss patient identification errors, medication errors, near-miss medication errors, patient falls, decubitus ulcers, extravasations, transferring inappropriate patient for surgery, sending inappropriate sample to laboratory, and sending inappropriate supplies for sterilization were identified as nursing service outcomes of the units for this study. These nursing service outcomes from January to December, 2017 and 2018, were analyzed and compared as before and after the study.

Statistical Measurements
Before and after measurements of nursing outcomes of units were used to assess the predictive validity of the nursing service audit tool, the conceptual structure of which was previously proven. 11 A Mann-Whitney U test was used to determine the differences between before and after measurements in order to assess the predictive validity of the tool. The nursing service audit tool's total and the sub-dimensions' internal consistency reliability were measured using the Cronbach's α coefficient. Relationships between the sub-dimensions of the tool were measured using Pearson's r. Kappa and Spearman correlations were used for inter-rater reliability. Mean, standard deviation, and independent t-tests were used to determine differences in the nursing service audit scores given by nurses.
The data which was collected from each items of the nursing audit tool, were analyzed through means and standard deviations (SD), total item correlation and Alpha if item deleted values, inter-rater reliability outcomes, difference between supervisors' scores in items and preand post-study.

Unit Level Performance Assessment (Nursing Service Audit) Tool's Validity
A measurement tool's validity is regarded as the degree to which the tool measures what it claims to measure or the degree to which evidence and theory support the interpretations of the test scores. 19 There is more than one method for assessing a measurement tool's validity.

Content Validity
Content validity is an assessment to measure the concept of the measurement tool based on expert rates according to their judgements. For this purpose, thirteen experts gave their assessment of the tool's items using the Delphi technique, designating them according to the content validity index. 11

Criterion Or Concrete Validity
Criterion validity is the extent to which a measure is related to an outcome. Criterion validity is divided into concurrent and predictive validity. Predictive validity is a comparison of the measure in question with an outcome assessed at a later time; however, simultaneous assessment is referred to as concurrent validity. 20 Nursing care outcomes of the wards before and after the nursing audit practice were used to compare the nursing performance on the hospital wards in order to assess the tool's predictive validity (Table 1). Based on a comparison of before and after the audit (Table 2), there was a statistically significant decrease in medication errors, bed sores, extravasation, transferring inappropriate patient for surgery, sending inappropriate samples to laboratory, sending inappropriate supplies to sterilization (P < 0.01), and identification errors (P < 0.05); there was also a decrease in slips and falling accidents (P < 0.010) at the hospital. Moreover, Table 2 shows that there was a statistical increase in near-miss identification errors (P < 0.05) and near-miss medication errors (P < 0.010) between before and after use of the nursing audit tool. These results demonstrate that regular control of the nursing service could improve nursing outcomes.

Unit Level Performance: Assessment Tool's Reliability
The reliability of a tool indicates how accurately it measures the characteristics it aims to measure. Therefore, it is important that when the tool is used constantly to measure the same characteristics, stable and consistent results are achieved. In order to assess the reliability of the tool, interrater reliability and internal consistency were used. 21,22

Internal Consistency
The Cronbach's alpha (α) coefficient was used to assess the internal consistency of the developed nursing audit tool and sub-factors. Table 2 shows the Cronbach's alpha (α) of the tool and sub-factors, the total item correlation, and the alpha if the item was deleted. The table resulting from the Cronbach's alpha test shows how the factor will affect the reliability of each question if the question is deleted. There is no harm in erasing the problem; that will significantly increase the reliability of the factor if removed from the questionnaire. A Cronbach's alpha value greater than 0.7 is a sign of the reliability of the factor. Sometimes, however, the value of 0.5 is accepted as sufficient in social sciences. The "alpha if item deleted" and "total item correlation" values in the column are compared with each other in the table. The total Cronbach's alpha (α) coefficient was 0.90 and sub-factors were 0.75, 0.71, 0.73, and 0.94 respectively. All total item correlations were 0.34-0.78, and all items correlated statistically significantly with each other, except item 17. In this research, the total score of the tool was an average 6.89 (min 5.57-max 8.33) out of 10 with a standard deviation of ±0.71 (Table 2.) Sub-factor correlations within the tool. The correlations of the sub-factors with one another were between 0.78 and 0.34 (P < 0.01), which indicates that the measurement was done at the same level.

Inter-rater Reliability
Inter-rater reliability is connected with the content of the tool and the ability to reach the same results when it is used repeatedly. 23 Inter-rater equivalence was used to assess the inter-rater reliability of the tool. The nursing audit tool was used for concurrent control by the supervisor nurses on the same day but at different times for each ward. Over six months, 309 items of data were gathered. The results of each of the wards obtained by both observers were analyzed using the Kappa and Spearman coefficients ( Table 3). The Spearman coefficient was between 0.75 and 0.85 and the Kappa coefficient ranged between 0.63 and 0.73. These results indicate that the nursing audit tool was reliable within the limits. 21,23 Furthermore, an independent sample t-test was carried out to assess the differences between observers (Table 4). It was determined that the two supervisors gave similar scores to the wards, and there was no statistically significant difference between the observers' scores of the wards (P > 0.05).    Table 2 shows that near-miss identification and medication errors had statistically increased in this research. This situation indicates that nurses pay high attention during preparation and giving medication, with improper nursing processes reported so as not to make any mistakes and to take necessary precautions. It is concluded that the tool is related to nursing outcomes and is sufficient to measure the subject which it is intended to measure. In other words, it indicates criterion or concrete validity.

Nursing Audit Tool's Reliability
It is expected that the item correlation values will total more than 0.30 for assessing an instrument's reliability. 25 Item total correlation values were discovered to be between 0.35 and 0.78 and within the expected limits, except for item 17 (r: 0.19). If item 17 is discarded form the tool, Cronbach's alpha (α) increases from 0.89 to 0.90. Cronbach's alpha (α) is a measure of the internal consistency and homogeneity of the tool. A higher Cronbach's alpha value indicates that the tool made the same feature measurement, and consistent measurement was made with each of the items. 26 The Cronbach's alpha of the tool was 0.90 (when item 17 was discarded), and the sub-factors of the tool were 0.75, 0.71, 0.73, and 0.94, respectively; this was sufficient to meet the required criteria. The high correlation and the total score of the tool show that the tool was measuring the same size. 26 Correlations with each of the sub-factors ranged between 0.78 and 0.34 (P < 0.01), indicating that the tool measures were of the same size. This result shows that the tool can measure conceptualized items of the same size in terms of the nursing services on the wards.
The Kappa analysis method was used to assess how to obtain consistent results when the same situation is evaluated with the same criteria by different people. 23 Therefore the nursing audit tool was used for six months by the supervisor nurses, and inter-rater agreement was analysed with Kappa's and Spearman's r (Table 3). If the result shows more than 0.60, it is accepted that the tool has good inter-rater agreement, according to Kappa. 21,23 In this study, it was determined that the tool had good interrater agreement because of its distribution between 0.63 with 0.73. It also had high correlation. Moreover, all scores given by the supervisor nurses were close to each other (Table 4), and there were no statistical differences between the scores (P > 0.05). According to these results, the nurses achieved consistent results, and this is compatible with the literature. 21,23 5.4. Implications for Nursing It is important for hospitals to establish quality standards and deliver standard quality. Health service outcomes are monitored through indicators, yet research on monitoring whether health workers adhere to the standards is limited. The developed tool will help nurse managers monitor the nursing care process in accordance with the quality standards. Nonetheless, monitoring of the process alone is not enough. Improving a defective nursing service process is another important aspect of better nursing care outcomes. Nursing care processes and outcomes should be monitored regularly in tandem.

Limitations of this Study
This study was only conducted in one hospital, which is considered to be a significant limitation. During the development of the nursing audit tool's criteria, evidencebased and detailed structure of the healthcare services, such as available resources, staff, equipment, tools, materials, quality standards, and processes, was not assessed. Nevertheless, the extent of the research process and the results were satisfactory. This tool is for monitoring nursing services in general wards, such as medical and surgical clinics. The nursing audit criteria will be different for specialized units, such as intensive care, newborn intensive care, obstetrics, emergency department, etc, and this tool is not suitable for such use. Having been conducted in only one hospital, it is suggested that the nursing audit tool can be used in other different hospital units following assessment of the validity and reliability of the tool.

Conclusion
The obtained findings show that the level of validity and reliability of the nursing audit tool was within acceptable limits, and the nursing audit tool can be regarded as a valid and reliable tool for monitoring nursing care processes in general wards of a hospital.
Although it is accepted that some criteria could change between departments in a hospital, the aim was to identify basic criteria for a ward. The nursing audit criteria established were also employed to create a form to guide upper and middle level nursing managers in their service department visits. Therefore, because there is no other tool for auditing nursing care in Turkey, it is estimated that the tool will encourage more studies on this subject. Also, designing and issuing a nursing audit checklist is unlikely to have an impact on its own, as to be successful it requires a program of communication, education, and ultimately culture change.

Conflict of Interest Disclosures
The authors declare that they have no conflicts of interest.

Ethical Approval
An application indicating the objective and scope of the study was submitted to the Ethical Committee of Gazi University, and the Consent of the Ethical Committee was awarded on October 10, 2017 under reference number

What Is Already Known?
• The basic tasks of nursing managers include directing and controlling the nursing services of the hospital. • Control of internal audits provides nursing managers with an understanding of the possible mistakes, identifies risky conditions, generalizes good practices, and enhances the management system and processes.

What This Study Adds?
Nursing managers should develop effective internal control systems, including preventive, concurrent, and detective controls, in order to observe ongoing nursing activities and rectify healthcare errors or problems in the workflow.

Research Highlights
77082166-302.08.01. The hospital in which the study was to be conducted was sent a written request explaining the purpose and the method of the study. After obtaining written permission from the hospital, data collection began.