When Is Info 188 Going to Be Offered Again
- Research article
- Open Access
- Published:
Evaluation of the Performance of Routine Information System Direction (PRISM) framework: show from Uganda
BMC Health Services Inquiry volume ten, Article number:188 (2010) Cite this article
Abstract
Groundwork
Sound policy, resources allocation and day-to-day management decisions in the health sector require timely information from routine health information systems (RHIS). In most low- and heart-income countries, the RHIS is viewed as being inadequate in providing quality data and continuous data that can be used to help improve wellness system functioning. In add-on, there is limited evidence on the effectiveness of RHIS strengthening interventions in improving data quality and utilise. The purpose of this report is to evaluate the usefulness of the newly developed Performance of Routine Information System Management (PRISM) framework, which consists of a conceptual framework and associated data drove and analysis tools to appraise, blueprint, strengthen and evaluate RHIS. The specific objectives of the study are: a) to assess the reliability and validity of the PRISM instruments and b) to assess the validity of the PRISM conceptual framework.
Methods
Facility- and worker-level data were collected from 110 health care facilities in twelve districts in Uganda in 2004 and 2007 using records reviews, structured interviews and self-administered questionnaires. The assay procedures include Cronbach's alpha to appraise internal consistency of selected instruments, examination-retest analysis to assess the reliability and sensitivity of the instruments, and bivariate and multivariate statistical techniques to assess validity of the PRISM instruments and conceptual framework.
Results
Cronbach'southward blastoff analysis suggests loftier reliability (0.7 or greater) for the indices measuring a promotion of a culture of information, RHIS tasks self-efficacy and motivation. The study results also advise that a promotion of a culture of data influences RHIS tasks self-efficacy, RHIS tasks competence and motivation, and that self-efficacy and the presence of RHIS staff take a directly influence on the use of RHIS information, a key aspect of RHIS performance.
Conclusions
The study results provide some empirical support for the reliability and validity of the PRISM instruments and the validity of the PRISM conceptual framework, suggesting that the PRISM approach can be effectively used past RHIS policy makers and practitioners to appraise the RHIS and evaluate RHIS strengthening interventions. However, additional studies with larger sample sizes are needed to further investigate the value of the PRISM instruments in exploring the linkages betwixt RHIS data quality and utilise, and health systems performance.
Peer Review reports
Background
Sound policy, resource allocation and day-to-solar day management decisions in the health sector require timely information from routine health data systems (RHIS) in society to rail the delivery of quality health care services and related support systems, including equipment and supplies, finance, infrastructure and homo resource [1–5]. However, previous assessments in developing countries signal that the RHIS is oft in disarray [6]. Problems constraining RHIS operation at the state-level include: poor information quality [7, eight]; limited use of available data [9, ten]; weaknesses in how data are analyzed [8, 11]; and poor RHIS management practices [12, 13].
In improver, health organisation managers in developing countries tend to miss the very purpose of the RHIS - to provide data that can aid runway the operation of both programs and the overall health system, as the information are not typically used every bit part of the performance appraisement of facility staff or for the achievement of district and facility targets.
Despite the keen need to improve the availability, quality and apply of RHIS data at the local-level, at that place is a paucity of studies investigating the determinants of RHIS performance and the effectiveness of RHIS strengthening interventions. Previous studies shed calorie-free on various aspects of the RHIS but neglect to provide a comprehensive picture of the RHIS, how it is organized and how diverse RHIS components interact with each other to influence RHIS performance. The dearth of studies in this area is probable due to a number of factors, including limited attention to RHIS as a research topic by wellness services researchers, the unavailability of evaluation frameworks to assess RHIS operation, inadequate research designs, and inadequate funding. Therefore, there is a clear need for placing higher priority on RHIS research and developing methodological approaches for assessing RHIS functioning [6, 14].
To help better the evidence-base on RHIS performance and its determinants, Aqil, Lippeveld and Hozumi (2009) recently developed the Performance of Routine Information System Management (PRISM) framework. PRISM consists of a conceptual framework and associated information collection and analysis tools to assess, design, strengthen and evaluate RHIS [6]. Equally depicted in Figure one, the conceptual framework hypothesizes that technical, behavioral and organizational determinants (inputs) influence information collection, transmission, processing, and presentation (processes), which in plow influence data quality and use (outputs), health system performance (outcomes), and ultimately, wellness outcomes (impact). Based on the framework, four survey instruments and associated sampling procedures and assay guidelines were developed to appraise RHIS performance, processes and technical, behavioral and organizational determinants at the facility-, commune-, and country-levels.
PRISM (Performance of Routine Data System Management) conceptual framework.
The purpose of this paper is to assess the overall usefulness of the PRISM framework. The specific objectives are ii-fold. First, nosotros investigate the reliability and validity of the PRISM instruments, which provide measures of the determinants of RHIS functioning. 2d, we investigate the validity of the PRISM conceptual framework by assessing whether organizational, technical and behavioral factors are meaning determinants of RHIS performance, as hypothesized.
The information for the study come from Uganda, a sub-Saharan African land that has introduced extensive wellness sector reforms over the past twenty years. These reforms include: further decentralization of public health care services from the cardinal government to districts and sub-districts; increased investments in the availability and quality of main wellness through a Sector-Wide Arroyo (Bandy); and further integration of support systems, including RHIS, as described below [xv]. The process of decentralization is particularly noteworthy in Republic of uganda, as information technology has meant that districts and authorities-run health facilities have more authority and decision-space to carryout planning and managerial roles and responsibilities, which presumably tin exist conducted more effectively through the use of routine health data at the local level [fifteen, sixteen].
In 1997, Republic of uganda introduced the Health Management Information Arrangement (HMIS). The purpose of the HMIS is to amend on the pre-existing Health Information System (HIS), introduced in 1985, by incorporating vital direction information, such as staffing levels, infrastructure, health facility direction, medical equipment availability, fiscal information, and drug information. By integrating this data with surveillance and service delivery data already routinely collected through the HIS, the aim for the HMIS is to be "a comprehensive source of wellness and management information for planning, monitoring and evaluation of the health sector strategic plan. It focuses on strengthening: a) data drove and compilation of health events; b) timeliness, abyss, and accurateness of reported data: c) analysis, interpretation, and utilization for show-based decision-making and activeness; d) regular broadcasting of feedback to all stakeholders; and eastward) enhancement of feedback to all health workers in all aspects of information management, analysis, and utilization at all levels of service delivery" [17].
The effectiveness of the HMIS in office depends on data reporting and feedback relationships too as on trained and motivated staff at each level that properly conduct out their data drove, reporting and utilise responsibilities. In Republic of uganda, the authorities-run referral system is comprised of the following levels. The Health Heart 2 represents the first level of interface between the formal health sector and communities and typically provides just convalescent services at the parish level (with a standard population of 5,000 individuals). The Health Centre III, which provides first referral cover for the sub-county (standard population of 20,000 individuals), offers bones preventive, maternal and curative care and is responsible for the support and supervision of the community and Wellness Heart II facilities. The Health Centre IV is a referral hospital at the canton or district level (standard populations of 100,000 individuals and 500,000 individuals, respectively) which also includes on its premises the direction team of the Health Sub-District Health Office. In add-on to second-level referral services, the Health Heart IV provides the same types of bones services equally Health Centre II and Health Center 3 facilities. For tertiary services, referrals are fabricated to regional and national hospitals. Routine data are collected in each of the types of facilities to a higher place using standardized forms issued by the Ministry building of Health. This information in turn is supposed to be reported to District Wellness Offices and so to the Central Level Data Depository financial institution, which is operated by the Ministry building of Health'south Resource Middle. HMIS guidelines stipulate that feedback is to and then be provided from the central level to Commune Health Offices, from District Health Offices to Sub-District Wellness Offices, and from Sub-District Health Offices to health intendance facilities. Uganda'southward HMIS collects data from both public and private sector health facilities and is probably the just instance of an integrated RHIS in Africa.
According to the Ministry of Health'due south most recent health sector strategic plan, a number of problems limit the effectiveness of the HMIS (Democracy of Uganda 2005). Data collection and reporting forms are viewed as not fairly distributed to heath care facilities and district health offices. Moreover, at that place is recognition that reporting forms are non properly filled and submitted, nor are data properly analyzed, fed dorsum and utilized by the Commune Health Offices and health facilities for planning and managerial controlling. The Ministry of Health also has experienced shortages of health information personnel, and the Resources Centre in Kampala has suffered from shortages of basic computers and software to facilitate the assay of routine health data [15, 17]. In the subsequent sections of this paper, we use the acronym RHIS to refer to Uganda's HMIS.
The integrated nature of Republic of uganda'due south RHIS too as the increased corporeality of conclusion-space at the commune- and health facility-level make Uganda an splendid context to appraise the reliability and validity of the PRISM tools as well equally the validity of the PRISM framework. It is hoped that the results of the study will contribute to future RHIS assessment efforts as well every bit to assist Uganda's Ministry of Health to strengthen its RHIS.
Methods
Data
Information for the report come from health facility and staff surveys administered in Uganda in 2004 and 2007. The survey instruments used were adapted from those in the PRISM tool packet [xviii]. The following is a summary of the instruments.
➢ Diagnostic Tool: This tool collects information from health intendance facilities and district health offices on RHIS data quality and use, RHIS procedures, supervision, data engineering science and user friendliness of data collection registers and reporting forms. The tool consists of a review of documents and observations of resources and displays of RHIS data.
➢ Facility Checklist: The facility checklist collects data from facilities and district health offices on the availability of staff, RHIS-related supplies, equipment and infrastructure.
➢ The Management Assessment Tool: The tool collects information through a review of documents from district wellness offices and health care facilities on a range of management back up services, including governance, planning, preparation, supervision, employ of performance tools, and financial resources.
➢ The Organizational and Behavioral Assessment Tool (OBAT): This is a self-administered tool completed by health workers at different levels on their perceptions of behavioral and organizational factors thought to influence RHIS functioning. The behavioral factors include: RHIS knowledge, RHIS tasks competence, trouble solving skills, confidence in carrying out RHIS tasks (cocky-efficacy) and motivation. The organizational factors include various questions used to assess the promotion of a culture of information within the health section.
The theoretical basis for each of the instruments is described elsewhere [6].
For the 2004 facility and district survey, which was conducted as function of a RHIS situational analysis, all vi regions of the country were identified and two districts from each region (n = 12) were selected (Arua, Bugiri, Bundibugyo, Gulu, Keyniojo, Kamuli, Kumi, Luwero, Masindi, Mbarara, Mubende, and Rukungiri). The conclusion of which districts to be included in the study was made by officials at the Ministry of Health. Therefore, the pick of districts is purposive and not random. Nevertheless, facilities in each selected district were randomly selected using Lot Quality Assurance Sampling (LQAS) methods. The sampling plan called for ten wellness facilities in each district to be visited: two health centre II, four health center III and four wellness heart Four facilities. Despite transportation bug and security concerns, 110 facilities were successfully interviewed (27 wellness center II, 48 health middle III and 26 health center 4 facilities), yielding a response rate of 92 percentage. In 2007, the sampling plan called for revisiting all health intendance facilities surveyed in 2004. Of the 110 facilities surveyed in 2004,100 facilities were successfully re-interviewed, and ten facilities in which staff were unavailable were replaced by near-by facilities. Power calculations suggest that the sample size is adequate to conduct multiple regressions analyses with 25 or less independent variables [xix, twenty].
For the cocky-administered organizational and behavioral assessment tool, the questionnaire was administered to ane health worker (the facility in-accuse) per facility in 2004 (n = 110). In 2007, the questionnaire was administered to as many as two health workers per facility in an endeavor to collect data from a wider diverseness of staff (northward = 197).
It should be noted that the PRISM tool bundle evolved substantially between the 2004 and 2007 surveys. New questions were added to improve the measurement of several components of the PRISM framework, including RHIS information quality and use, RHIS processes, and their technical, behavioral and organizational determinants, equally described beneath. However, all questions asked in 2004 were repeated in 2007 in order to assess changes betwixt the two surveys, to study the utility of new questions, and to ensure that the survey instruments fit the Uganda context.
Measures
A key measurement effect of the study concerns the multidimensional nature of near of the RHIS determinants depicted in the conceptual framework. As nosotros describe beneath, virtually inputs of RHIS performance (technical, organizational, and behavioral factors) are measured through a series of continuous or Likert scale indicators, which are then used to generate indices post-obit the PRISM assay guidelines [eighteen].
The cocky-efficacy scale (behavioral) incorporates 4 dimensions: collection, analysis, estimation and use of information. Each dimension is based on two to four indicators, as specified in the results department. The respondents were asked to rate their cocky-efficacy for various RHIS tasks on a scale of zero to one hundred. For each dimension, all indicators and their ratings were added together and so divided by the total number of indicators and multiplied by one hundred to obtain a percentile score.
The scale of the index of motivation (behavioral) is based on viii items and a percentile score was calculated using the same procedure described above for the culture of information score. The scale incorporates indicators on a variety of dimensions, including perceptions of whether RHIS data are: satisfying; needed to monitor facility performance; and appreciated by fellow workers and superiors.
RHIS task competence (behavioral) was measured by a pencil and paper test that measures the ability of respondents to perform calculations, and to interpret and utilise RHIS results.
The promotion of a culture of data (organizational) is operationally defined as an system having the capacity and control to promote values and beliefs among its members to promote collection, analysis and use of data to accomplish its goals and mission. For assessing whether health facilities promote a civilisation of information, the construct is operationalized every bit having five dimensions - the promotion of: 1) data quality; ii) testify based determination making and accountability; three) reward mechanisms for good work; 4) the use of data; and v) efforts and activities to change things for the better. Each dimension is measured by ii to viii items describing behaviors that are thought to directly or indirectly promote a culture of information. Each action argument or detail related to these dimensions is assessed using a Likert scale of agreement, ranging from one (very weak) to seven (very strong). All items belonging to a specific dimension and their ratings are added together and divided by the total number items and multiplied past i hundred to create an overall percentile score.
To measure the two components of RHIS performance - data quality and the use of information - indices were constructed based on indicators common to the 2004 and 2007 surveys, and on an expanded listing of indicators bachelor in the 2007 survey only. Observation of records for checking data quality is considered to be the gold standard for measuring RHIS performance and their validity is well established [3]. To measure the availability and accuracy of RHIS information in our study, nosotros compare the information contained in monthly RHIS reports with those of facility registrars for three types of services: the handling of pneumonia, antenatal care, and HIV/AIDS services. For each service, percentile scores are generated to mensurate data availability and accuracy.
Similarly, the utilise of information is observed through a review of documents that verifies whether and how RHIS data were used in controlling processes. The use of RHIS information is operationalized by a series of dichotomous indicators, including: whether RHIS information was discussed in staff meetings; whether RHIS information was used to aid brand decisions; whether RHIS information was used to help take follow-up actions or to refer issues for action; and whether updated data on various topics was displayed. Following the PRISM analysis guidelines, these indicators were aggregated to generate a composite continuous alphabetize of the use of RHIS information [21]. This approach gives equal weight to each of the indicators used in the alphabetize. We tested whether this assumption makes a difference in the analysis by applying principal components assay (PCA) to generate the index. PCA is a well-established method to create summary indices using weighted sums [22]. PCA generates the weights that maximize the variance of the resulting composite alphabetize. In generating an index of RHIS information use, the advantage of the PCA approach over the simple improver approach is that it imposes fewer restrictions - the PCA approach generates weights while the simple aggregation arroyo is merely a weighted sum where all weights are restricted to have the value of i.
For 2004, this alphabetize could not be generated because the facility diagnostic tool independent much more restricted information. Specifically, data were collected merely on whether RHIS information was displayed through maps, charts and tables, and non on whether RHIS information was used in determination-making processes. To create an index of the use of RHIS data for our pooled data analysis (described below), we created a dichotomous indicator of whether a facility had on display a map, chart or tabular array based on RHIS data at the time of the survey.
Assay
The internal consistency of the self-efficacy scale and the seven dimensions of the culture of information scale were estimated using Cronbach'south alpha. Separate sets of Cronbach'southward alpha coefficients were calculated for the 2004 and 2007 samples. The test-retest reliability and sensitivity of the scale scores on cocky-efficacy, motivation and civilization of information was assessed by conducting t-tests on the equality of the ways from the 2004 and 2007 surveys. Typically, exam-retest reliability is conducted by comparing the scores of each scale among a matched sample of individuals over a curt time interval. However, our data were gathered three years apart and consist of individuals who may or may non be the aforementioned, but could not exist matched. This prevents us from generating correlation coefficients of reliability using matched respondents. Equally a result, we accept an culling arroyo by conducting examination-retest assay based on group means, along the lines suggested past Cooke and Szumal (1993) [23]. One potential threat to the internal validity of these examination-retest results is that at that place may have been RHIS interventions introduced during the period between the surveys that contributed to real changes in the levels of the scales investigated. Nosotros explore this upshot in the discussion section.
Criterion-related validity is examined by assessing bivariate correlations among the behavioral instruments, organizational instruments, and the RHIS performance instruments described in a higher place. Correlation analyses were conducted at the private- and facility-levels. For the facility-level analyses, the calibration scores of the sample wellness workers were averaged for each facility to obtain facility-level scores.
In addition to bivariate analysis, multivariate analysis techniques were used to appraise construct validity. Two types of models are estimated: Ordinary Least Squares (OLS) and probit models. The OLS models were estimated based on 2007 cantankerous-sectional data, with the dependent variable consisting of the continuous index of the use of RHIS data, as described above, and the independent variables consisting of indicators of the technical, organizational, and behavioral factors described above. The probit models were based on pooled 2004-2007 data, using as the dependent variable a dichotomous variable that measures whether a tabular array, chart or map based on RHIS data was displayed in the facility at the time of the survey. Model results were evaluated at the ane percent, 5 per centum and 10 percent levels of statistical significance. The analysis was carried out using Stata Statistical Software: Release 10 [24].
Results
Sample characteristics of respondents
We begin by briefly describing descriptive characteristics of sample respondents selected in both 2004 and 2007 for the self-administered organizational and behavioral questionnaire. In 2004, men were a greater percent of the sample than women (57 percentage vs. 41 per centum), while the opposite was true in 2007 (48 percent vs. 52 percent), although difference in the sexual activity composition of the two samples was non found to be statistically significant. We are unable to compare the educational level of staff beyond years due to differences in the response codes between the two surveys.
The mean age of the respondents was 33.8 years in 2004 and 34.0 years in 2007, indicating no meaningful divergence in the age distribution between the two samples. The range of ages reported was similar in the 2 surveys (varying from 20 to 85 years in 2004 and from 21 to 59 years in 2007). The lack of significant differences in socio-demographic characteristics indicates that both groups were similar and that no specific feature need to be controlled when investigating the hypothesized relationships.
Internal consistency
In developing the PRISM data drove instruments, face and content validity were assessed through a review and consultation with technical experts. The diagnostic tool that checks data quality and information use through record review and ascertainment is considered to be a gold standard for assessing validity, equally is the facility checklist which is used to mensurate the availability of infrastructure and equipment through ascertainment. Thus, the validity of these tools is well-established. On the other paw, the reliability and validity of the organizational and behavioral assessment tool, which is comprised of scales of the promotion of a culture of data, motivation, and cocky-efficacy, was assessed through an analysis of internal consistency and past testing the hypothesized relationships depicted in the PRISM conceptual framework. Cronbach's alpha was used to measure the internal consistency of these scales, all hypothesized to exist determinants of RHIS performance (Table 1). In exploratory research, alpha scores of 0.6 or higher are typically accepted as showing adequate reliability and alpha scores 0.7 or college as showing high reliability [25, 26].
To assess the questions on cocky-efficacy, the confidence level of respondents in conveying out RHIS tasks was categorized with multiple indicators nether the dimensions of data assay, data interpretation and data use. For both the 2004 and 2007 samples, the indicators for each dimension had blastoff scores to a higher place 0.viii, indicating a loftier level of reliability. Since reported self-efficacy for the tasks "data collection" and "checking data quality" are each based on a unmarried question, blastoff levels were not computed. For the overall self-efficacy scale for RHIS tasks, the blastoff levels in both years are 0.95, indicating a high level of reliability.
The promotion of a culture of information is measured with a scale that includes self-reported perceptions on iv dimensions: the promotion of data quality, the use of RHIS information, evidence-based conclusion-making and accountability, and the presence of rewards for better performance. The 2d block of information in Table i presents the results. Since the promotion of data quality was assessed with a unmarried question, its alpha could not be calculated. With one exception, blastoff scores for the remaining dimensions emerged every bit 0.half-dozen or higher, indicating high reliability for both the 2004 and 2007 samples. The 1 exception was the blastoff score for the "prove-based decision-making" dimension based on the 2007 sample, which is 0.53. For the overall civilization of information scale, the alpha levels are 0.87 in 2004 and 0.85 in 2007, indicating high reliability.
Based on the applied experience of applying the PRISM framework in Uganda [27] and Pakistan [28], additional questions for assessing the promotion of a culture of information were included in the 2007 questionnaire, assuasive us to create revised indices. Changes included: omitting the dimension "rewarding amend performance" due to its relative lack of specificity; and adding new dimensions on "a sense of responsibility", "accountability/empowerment", "feedback" and "problem solving". As shown in the third block of information in Table ane, the alpha levels for the scales of the overall culture of information, use of information, trouble-solving and sense of responsibility dimensions are 0.8 or higher, indicating high reliability. Falling under the 0.6 threshold for adequate reliability are the alphas for the dimensions evidence-based controlling, feedback and accountability/empowerment.
A scale was also constructed for "motivation for performing RHIS tasks". Equally indicated by the 4th cake of information in Table 1, the alpha level for this scale is 0.68 in 2004, indicating adequate reliability. Notwithstanding, the comparable blastoff level for the 2007 sample is 0.55, falling merely below the 0.6 threshold for acceptable reliability.
Test-retest reliability and sensitivity
Tabular array 2 presents the test-retest analysis findings for the scales of the use of information, a promotion of a culture of information, self-efficacy, motivation, and RHIS job competence. The analysis is based on indicators common to the 2004 and 2007 datasets. The employ of RHIS information is measured by a dichotomous indicator of whether RHIS information was displayed in the facility at the time of the survey. The results suggest that the use of RHIS data did not change significantly from 2004 to 2007 (0.61 in 2004 and 0.51 in 2007).
Turning to the potential determinants of RHIS performance, the results suggest that the mean levels of the indices measuring a promotion of a culture of information, motivation to perform RHIS tasks and RHIS chore competence were significantly higher in 2007 than in 2004. However, the index of perceived self-efficacy was significantly lower in 2007 than in 2004. These results show changes over time, which were picked up by the measurement tools, indicating either the measurement scales are not reliable or stable or the measurement scales are not only reliable but sensitive enough to option up the alter. We further discuss this event in the discussion section.
Data quality and the apply of information were measured through a review of existing records and reports. Was there a change in data quality? Figure 2 presents the findings of record availability, every bit measured past the facility having records available at the time of the survey, and data accuracy for pneumonia and antenatal intendance services for both 2004 and 2007. The results show that record keeping for pneumonia cases (47 vs. 74 percentage) and ante-natal intendance cases (48 vs. 69 per centum) improved substantially over fourth dimension. Of those facilities where records were bachelor, the accuracy of data reported for these selected health problems was above 75 percent in 2004. However, in 2007, when tape keeping improved, accuracy was institute to be substantially lower than in 2004. Before concluding that the data accuracy of the bachelor records declined over the interval, we re-examined the data based on the assumption that all facilities with unavailable records had inaccurate data, and classified them accordingly. Based on this re-analysis, we found no statistically significant difference in information accurateness for pneumonia (χii = 0.004741, df = 1, p = 0.95) and antenatal care records (χ2 = 0.000, df = 1, p = 0.999) between 2004 and 2007.
Comparison of record availability and record accuracy by selected services, 2004 and 2007.
The diagnostic tool measures the completeness of data bachelor at the facility-level by reviewing how many data elements were filled in the monthly written report of the selected calendar month. To reduce the time needed to deport the survey, it was decided that survey enumerators would count ten percent of the unfilled number of information elements and if the unfilled number exceeded ten per centum, they would note the facility has having incomplete data. Withal, the instructions were not followed properly by the surveyors. Thus, it was not possible to investigate the completeness of RHIS monthly reports at the facility-level. Nonetheless, commune-level information on timeliness and completeness were collected in both years. A district was classified as having timely data if at least 75 percentage of facilities under their authorisation submitted the last monthly report on time, and every bit having complete data if at least 80 percent of facilities nether their dominance submitted the monthly report for a pre-specified month (on time or not on fourth dimension). The results indicate that the percent of sample districts classified as having timely data dropped (from 63 percent in 2004 to xl percent in 2007), while the percent of districts classified equally having complete information increased (from 22 per centum in 2004 to 55 pct in 2007).
Using test-retest analysis, we also looked more than closely at the changes in organizational determinants, behavioral determinants and RHIS performance. On average, 74 pct and 78 percent of respondents perceived that their department promotes data quality in 2004 and 2007, respectively, while on average the data accuracy levels were around 35 percent for both 2004 and 2007 (with missing records classified equally inaccurate), indicating that the gap between what respondents perceived and the actual situation of data accurateness remained abiding over the time interval. Similarly, the comparisons between the indicators of a promotion of a civilization of information with indicators of RHIS tasks competence and observed use of information showed wide gaps in both 2004 and 2007, indicating that perceptions among the respondents that their section promotes the utilise of information was not aligned with actual competence to use information or observed employ.
Construct validity
To assess construct validity of the PRISM framework, we conducted bivariate analysis to investigate the hypothesized associations. Three enquiry questions were investigated. First, do RHIS organizational factors, especially the promotion of a culture of information, affect RHIS behavioral factors? Second, does the level of confidence in performing RHIS tasks (cocky-efficacy) affect RHIS tasks competence? And tertiary, are RHIS organizational and behavioral factors associated with RHIS performance, equally measured past indicators of information accuracy and the use of RHIS information.
Tabular array 3 presents Pearson correlation coefficients of the associations between indices identified through Cronbach's alpha analysis for 2007. The unit of assay is the wellness worker. The results suggest that the two alternative indices of an overall culture of information are significantly associated with the RHIS tasks conviction level (cocky-efficacy), merely not with respondents' RHIS tasks competence. Both "culture of data" indices are also found to be significantly associated with the index measuring motivation to perform RHIS tasks. In addition, at that place is a statistically pregnant association between RHIS confidence level and RHIS competence indices. These relationships are all positive, as hypothesized by the PRISM framework, and are found to be significant in both 2004 and 2007, every bit hypothesized in the conceptual framework, indicating construct validity. To save infinite, results are presented for 2007 only.
To appraise the bivariate associations between RHIS functioning and its organizational and behavioral determinants, nosotros computed facility-level averages of the wellness worker-level indices institute to be reliable using Cronbach's blastoff analysis. Nosotros then merged these information with facility-level data on utilize of RHIS information and its potential determinants. The results of the analysis are presented in Tabular array 4. Of the potential determinants of RHIS performance included in the assay, only the self-efficacy index was establish to exist significantly associated with the use of RHIS information, as measured by the blended index calculated through PCA (Appendix i). Models of the determinants of data accuracy, an indicator of data quality, were not estimated due to the very limited variation in our sample. Nosotros talk over this upshot later in the newspaper.
We conducted multivariate assay to investigate the relative roles of organizational and behavioral factors on RHIS performance after controlling for other structural factors. Models were estimated using 2007 cross-sectional data and 2004-2007 pooled data. Table five presents the results of the OLS models of the determinants of the use of RHIS data every bit measured by the composite index generated through PCA, as described in Appendix i. Models one, 2 and 3 include equally independent variables the mean cocky-efficacy, motivation, and culture of information indices, respectively, besides every bit a common set of independent variables, including: the type of wellness care facility; the availability of electricity; whether a RHIS assistant is on staff; the availability of a figurer; and whether a district supervisor was reported to have visited the facility in connectedness with RHIS activities in the quarter prior to the survey. Descriptive statistics for the variables included in the models are presented in Table S2 (Additional file 1: Tabular array S2).
Equally indicated in the table, the mean self-efficacy index was found to exist positive and significantly associated with the apply of RHIS information at the 0.10 level of significance. The hateful motivation index and the mean civilisation of index were also found to be positive, after controlling for other variables, but neither emerged as statistically significant. Of the other independent variables, simply the presence of a RHIS assistant on the staff was found to be statistically significant.
Models of the determinants of RHIS data use were also estimated using 2004-2007 pooled information. Because the 2004 survey had fewer questions on RHIS information apply compared to the 2007 survey, the dependent variable is a dichotomous indicator of whether a table, map or chart based on RHIS information was displayed in the facility at the time of the survey. The probit model results are presented in Table 6.
While the index of motivation was found to exist statistically insignificant at the 0.10 level, the indices of self-efficacy and a civilisation of data did emerge as positively associated with the utilize of RHIS data, as hypothesized, and are both statistically significant at the 0.11 level, simply outside the 0.10 threshold level. With respect to the other independent variables, the presence of a RHIS assistant on the staff is again found to be statistically meaning in each of the models. This finding is non surprising, as displaying information is presumably part of the chore responsibilities of RHIS assistants.
Discussion
The objective of this article is to investigate the reliability and validity of the PRISM framework based on sample data from health care facilities and wellness workers in Uganda. The framework is innovative in that it ane) stresses RHIS operation equally well every bit organizational and behavioral determinants that typically receive inadequate treatment in the RHIS and wellness policy literature, and two) includes data collection and assay tools for empirical testing. Because previous information arrangement frameworks do not provide tools for empirical testing [29–31], the written report is the start of its kind.
Overall, the results of the study propose that the internal consistency of the scales of the constructs for organizational and behavioral components are high, indicating that the tools are reliable for assessing RHIS tasks self-efficacy, motivation and the promotion of a culture of information. These results likewise suggest that these tools are sensitive and suitable for assessing changes over fourth dimension, indicating that the changes betwixt 2004 and 2007 identified through test-retest analysis are real.
In addition, the changes are internally consistent, as hypothesized by the framework. The gaps between loftier-perceived self-efficacy for RHIS tasks and lower observed RHIS tasks competence were expected to exist filled over time, as health workers become aware of them. Therefore, it would be reasonable to look that, over time, respondents would be become more objective in assessing their perceived cocky-efficacy and objective RHIS tasks competence. The results showed that was the example, as the gaps between perceived self-efficacy for RHIS tasks and objective RHIS tasks competence narrowed from 2004 to 2007. Similarly, respondents might accept improved their perceptions of a promotion of a civilisation of information by observing that senior direction had revised data collection forms and reports by including data on HIV/AIDS services and by including data collection and reporting forms that disaggregate data past age and gender to address emerging information needs of the wellness department. This perception along with their better RHIS tasks competence levels might accept strengthened the motivation levels of respondents as well. We cannot dominion out alternative explanations for these differences, such as biases that event from: survey respondents, despite having like demographic characteristics, non existence the aforementioned beyond the two surveys; the replacement of some facilities included in the 2004 sample with neighboring facilities in the 2007 sample; and the issue of instrumentation (getting used to tools).
Construct validity of the PRISM framework is supported by the results of the association of organizational, technical and behavioral factors with the use of RHIS information, an important dimension of RHIS functioning. The promotion of a civilisation of information was associated with motivation, RHIS tasks self-efficacy, RHIS tasks competence, job satisfaction and use of information. Some other organizational factor, the presence of dedicated RHIS staff at the facility, was found to be significantly associated with the apply of data. In addition, the reliability and validity of the tools are further substantiated past the finding that data accurateness and the utilize of information did not change much from 2004 to 2007, which is consistent with our understanding that no major interventions were conducted in the period of time betwixt the two surveys to meliorate the situation.
Notwithstanding, the mean scores of the scales of a promotion of a culture of information, perceived cocky-efficacy for RHIS tasks, observed RHIS competence and perceived motivation showed significant comeback over time, indicating that the tools are sensitive to selection up changes in these factors. The training to familiarize staff with the revised forms, the addition of HIV/AIDS data to the RHIS and new provisions to disaggregate data by gender and age might have contributed to these changes. However, the size of the improvements was not large plenty to affect overall RHIS performance, as measured past levels of information accuracy and the use of data. These results, of low RHIS performance and depression RHIS tasks competence combined with high perceptions of promotion of a culture of information and self-efficacy for RHIS tasks, are consistent with those reported in previous assessments based on the PRISM framework in Islamic republic of pakistan [28, 32], Mexico [33], Haiti [34], Cote d' Ivore [35] and Prc [36].
The study results regarding the hypothesized relationships not only support the validation of the PRISM framework just also provide insights for possible intervention strategies, equally described in the conclusions section. However, that the magnitude of the size of many of the relationships investigated is small raises questions on the force of the relationships and the potential effectiveness of interventions that operate through these mediating factors, thus potentially diluting their direct and indirect impacts.
It is to exist noted that skewed responses for some of the scales of RHIS inputs and the limited variance in the indicators of RHIS functioning may assistance explain the limited number of indicators found to exist statistically pregnant in the analysis [37]. For case, despite finding some statistically significant associations between the employ of RHIS information and selected determinants, the variation in the use of information was limited, while almost of the respondents' ratings on the dimensions of a promotion of a culture of data and RHIS tasks self-efficacy were skewed positively with express variance, which might explain why these factors were not found to be significantly meaning. Moreover, the instruments that mensurate the promotion of a civilisation of information, including those on testify-based decision-making, feedback and accountability/empowerment, need farther refinement due to their low internal consistency. This might besides be a possible reason for these factors not emerging as significantly significant in the models of the determinants of the utilise of RHIS information. In addition, nosotros did not estimate models of the determinants of information accuracy due to the limited variation in the sample.
Based on our review of the RHIS literature, at that place are no RHIS studies that tin can exist used for comparison purposes. The most relevant comparing of our results on the promotion of a culture of information, which relates to communicating beliefs and values, could exist fabricated with studies of organizational civilisation and communication. Clampitt and Downs (1993) showed that subordinate communication and supervisor communication has correlation coefficients of 0.17 and 0.15, respectively, with self-reported productivity [25]. Hellweg and Philips (1980) in their literature review found correlations ranging from 0.2 to 0.five between organizational communication and productivity in organizations [38] and Pincus (1986) found similar results [39]. Thus, the study results of the promotion of a civilisation of information and RHIS performance are substantiated by the existing management literature. Ane reason for the modest bear on of organizational factors on performance is that these factors also act through mediating variables, and thus both direct and indirect effects are diluted. Our report results suggest that organizational factors accept stronger relationships with behavioral factors such tasks competence and motivation than with overall RHIS performance, which is in line with other studies.
At that place are a few important limitations of the study. Commencement, although the sample size of facilities included in the written report was large plenty to address the inquiry questions (due north = 110 for both the 2004 and 2007 surveys), the unavailability of RHIS records and missing information further reduced the sample size which prevented usa from using more sophisticated techniques to appraise the validity of the conceptual framework (i.due east. fixed effects models, random effects models, factor analysis) or to comport discriminant and convergent analyses of the subscales of a culture of information construct. Second, the 2004 dataset had only very limited information on the use of RHIS information. Only very general indicators of the display of RHIS information were bachelor, and no indicators of the use of RHIS data in routine meetings and conclusion-making processes were bachelor, which prevented us from assessing changes on these dimensions from 2004 to 2007. Because of this limitation, we believe the results of the probit model estimation, which is based on the more than express indicator of the employ of RHIS information available in both 2004 and 2007, should be interpreted with caution. 3rd, due to problems in the administration of the survey, we were not able to assess the abyss of monthly study data at the facility-level, which along with timeliness and accuracy, is a primal aspect of data quality. Every bit a event, the relationships betwixt data abyss and its potential determinants could non exist investigated.
Conclusions
Despite the above-mentioned limitations, the study results back up the reliability and validity of the PRISM framework and its tools, indicating its utility for the policy makers, RHIS managers, professionals and RHIS designers for creating a comprehensive motion picture of the RHIS and identifying its strengths and weaknesses. The PRISM framework can be used for assessing RHIS performance, processes and its major organizational, technical and behavioral determinants. These tools could be practical for monitoring changes in: RHIS data quality and use of information (operation); RHIS processes and task competences; and the promotion of a culture of information. In improver, the PRISM tools could be used in research designed to evaluate the effectiveness of RHIS strengthening interventions on RHIS performance. The major interventions resulting from previous assessments based on the PRISM approach in various parts of the world include: preparation to improve data interpretation and use skills forth with problem solving skills, which entails the apply of performance comeback tools; interventions to rationalize and sometimes reduce the corporeality of RHIS data collected; interventions to amend the use of it and information warehouses; and organizational interventions aimed at establishing processes to promote the use of RHIS information through meliorate communication of success stories and role modeling past senior management; and interventions to strengthen governance and financial resource in gild to sustain RHIS activities.
Boosted studies with large sample sizes are needed to investigate discriminant-convergent validity of scales measuring the promotion of a culture of information construct likewise as the 'utilise of information' constructs. In addition, the predictive value of the PRISM framework needs to exist demonstrated with farther practical research in various settings. Finally, given the potentially important role that RHIS data can play in improving health systems functioning, more enquiry is needed on further improving the PRISM instruments as well as exploring the linkages between RHIS determinants, RHIS operation and health systems performance at the country- and local-levels.
Appendix i - Index of Apply of RHIS Data
The 2007 survey included a number of questions on the employ of RHIS data, including whether RHIS issues and findings were discussed in staff meetings, whether facility decisions were based on RHIS data, whether there has been follow-up on these decisions, and whether diverse types of RHIS information were displayed in the facility through tables, charts and maps. Given that any one of these dichotomous indicators may not be sufficient to distinguish between facilities with relatively high vs. low levels of information use, summary indices were created by aggregating the indicators and through Master Components Analysis (PCA). Because the 2004 survey include very limited data on RHIS information use, PCA analysis could non be practical to that sample. Table S1 (Additional file 1: Tables S1, S2) presents the means and standard deviations of the variables used to create the index as well as the PCA results.
The eigenvalue for the commencement principal component indicates the percentage of variation explained. Equally indicated in Table S1 (Additional file 1: Tables S1, S2), the percentage of variation explained is 45 percent for the index. The factor scores in the last column of the table, which can be interpreted as weights, indicate that each of the variables entered into the PCA is positively associated with the utilize of RHIS data, suggesting the variables are valid indicators of the latent variable, use of RHIS information. The PCA results were used to construct the index of the apply RHIS information for the bivariate and multivariate analyses, presented in the results department.
References
-
AbouZahr C, Boerma T: Health information systems: the foundations of public wellness. Bull Globe Health Organ. 2005, 83 (eight): 578-583.
-
Evans T, Stansfield S: Wellness data in the new millennium: a gathering storm?. Bull World Health Organ. 2003, 81 (12): 856.
-
Lippeveld T, Sauerborn R, Bodart C: Design and Implementation of Wellness Information Systems. 2000, Geneva: Globe Wellness Organization
-
Stansfield SK: Structuring information and incentives to ameliorate health. Bull World Health Organ. 2005, 83 (8): 562.
-
World Health Organization: Anybody'south business: Strengthening health systems to ameliorate health outcomes: WHO'southward Framework for Action. 2007, Geneva: Globe Health Organization
-
Aqil A, Lippeveld T, Hozumi D: PRISM framework: a prototype shift for designing, strengthening and evaluating routine wellness information systems. Health Policy and Planning. 2009, 24: 217-228. 10.1093/heapol/czp010.
-
Mavimbe JC, Braa J, Bjune Thousand: Assessing immunization information quality from routine reports in Mozambique. BMC Public Health. 2005, v: 108-x.1186/1471-2458-five-108.
-
Odhiambo-Otieno GW: Evaluation of existing commune health direction information systems a case study of the district wellness systems in Kenya. International Journal Medical Informatics. 2005, 74 (nine): 733-44. 10.1016/j.ijmedinf.2005.05.007.
-
da Silva AS, Laprega MR: Critical evaluation of the primary care data system (SIAB) and its implementation in Ribeiero Preto, Sau Paulo, Brazil. Cadernos de Saude Publica. 2005, 21: 1821-8.
-
Chae YM, Kim SI, Lee BH, Choi SH, Kim IS: Implementing health management information systems: measuring success in Korea's health centers. International Periodical of Health Planning and Management. 1994, 341-8. ten.1002/hpm.4740090406.
-
Nsubuga P, White ME, Thacker SB, Anderson MA, Blount SB, Broom CV, Chiller TM, Espitia V, Imtiaz R, Sosin D, Stroup DF, Tauxe RV, Vijayaraghavan M, Trostle G: Public health surveillance: a tool for targeting and monitoring interventions. Disease Control Priorities in Developing Countries. Edited past: Jamison DT, Breman JG, Measham AR, Alleyne Grand, Claeson Yard, Evans DB, Jha P, Mills A, Musgrove P. 2006, Washington DC: The World Bank
-
Rotich JK, Hannan TJ, Smith FE, Bii J, Odero WW, Vu N, Mamlin BW, Mamlin JJ, Einterz RM, Tierney WM: Installing and implementing a computer-based patient tape system in sub-Saharan Africa: the Mosoriot Medical Tape Arrangement. J Am Med Inform Assoc. 2003, 10 (four): 295-303. 10.1197/jamia.M1301.
-
Kamadjeu RM, Tapang EM, Moluh RN: Designing and implementing an electronic health record system in primary care practice in sub-Saharan Africa: a case report from Cameroon. Inform Prim Care. 2005, 13 (3): 179-86.
-
Stansfield SK, Walsh W, Prata N, Evans T: Data to amend decision making for health. Illness Control Priorities in Developing Countries. Edited by: Jamison DT, Breman JG, Measham AR, Alleyne Thou, Claeson M, Evans DB, Jha P, Mills A, Musgrove P. 2006, Washington D.C., The World Banking company
-
Commonwealth of Uganda, Ministry of Wellness: Health Sector Strategic Program 2005/06 - 2009/x. 2005, Kampala: Ministry building of Health
-
Hutchinson P, Akin J, Ssengooba F: The impacts of decentralization on health care seeking behaviors in Uganda. International Journal of Health Planning and Direction. 2006, 239-270. 10.1002/hpm.848. 21
-
Kintu P, Nanjynja North, Nzabanita A, Magoola R: Development of HMIS in poor countries: Uganda as a case study. Wellness Policy and Evolution. 2005, iii (ane): 46-53.
-
Aqil A, Lippeveld T: PRISM Tools for Assessing, Monitoring, and Evaluating RHIS Performance. Mensurate Evaluation Project, [http://world wide web.cpc.unc.edu/measure/tools/monitoring-evaluation-systems/prism]
-
Cohen J: Statistical Ability Assay for the Behavioral Sciences. 1988, Hillsdale, New Jersey: Lawrence Erlbaum Associates
-
Cohen J, Cohen P: Practical Multiple Regression/Correlation Analysis For Behavioral Sciences. 1983, Hillsdale, New Jersey: Lawrence Erlbaum Assembly
-
Aqil A, Lippeveld T: Improving RHIS Operation For Better Health Arrangement Management, Trainer's guide, MEASURE Evaluation, USAID, 2008.
-
Jolliffe It: Chief Component Analysis. 2002, Springer: New York Urban center
-
Cooke RA, Szmul JL: Measuring normative beliefs and shared behavioral expectations in organizations: the reliability and validity of the organizational culture. Psychological Reports. 1993, 72: 1299-1330.
-
StataCorp: Stata Statistical Software: Release 10. 2007, StataCorp LP: Higher Station, TX
-
Clampitt PG, Downs CW: Employee Perceptions of the Relationship Between Advice and Productivity: A Field Study. Journal of Business Advice. 1993, 30 (1): x.1177/002194369303000101.
-
Nunnally J: Psychometric Theory. 1967, New York: McGraw-Hill
-
Aqil A, Lubaale Y, Orobaton N: Uganda HMIS/EMIS Situation Analysis. UPHOLD Project. 2004
-
Hozumi D, Aqil A, Lippeveld T: Pakistan State of affairs Analysis, MEASURE Evaluation Project, USAID. 2002
-
Braa J, Hanseth O, Heywood A, Mohammad W, Shaw V: Developing health information systems in developing countries: The flexible standards strategy. MIS Quarterly. 2007, 1-22. 31 Special
-
Hirschheim R, Klein HK: Four paradigms of information systems evolution. Communication of the ACM. 1989, 32 (10): 1199-1216. 10.1145/67933.67937.
-
Lind A, Lind B: Exercise of information system development and use: a dialectical approach. Arrangement Research & Behavioral Science. 2005, 22 (v): 453-10.1002/sres.718.
-
JICA HMIS Report Squad: National Action Plan: The Study of Improvement of Management Information Systems in Health Sector in Islamic republic of pakistan. SSC/MOH/JICA. 2006
-
Ascencio RA, Block AM: Assessment of the National Wellness Information Organization of United mexican states: Health Organization Information Center, National Establish of Public Wellness; General Bureau of Health Data Secretariat of Wellness. Cooperative Agreement GPO-A-00-03-00003-00; U.S. Agency for International Development (USAID); MEASURE Evaluation Projection; Pan American Health Organisation (PAHO). 2006
-
Boone D, Aqil A: Haiti HSIS Evaluation Report, MEASURE Evaluation, Ministry of Health, Haiti, USAID. 2008
-
Gnassou L, Aqil A, Moussa T, Kofi D, Paul JKD: HMIS Evaluation Written report, HIS Department, Ministry of Health, Republic of cote d'ivoire, MEASURE Evaluation, USAID. 2008
-
Aqil A, Lippeveld T, Yokoyama R: Yunnan Baseline HMIS Report, Mensurate Evaluation, CDC Yunnan, USAID; Aqil A, Lippeveld T, Yokoyama R. 2007 Guangxi Baseline HMIS Study, Mensurate Evaluation, CDC Guangxi, USAID. 2007
-
Huseman RC, Hatfield JD, Boulton WR, Gatewood RD: Development of a conceptual framework for analyzing the communication performance relationship. University of Management Proceeding. 1980: 178-182.
-
Hellweg SA, Philips SL: Communication and productivity in organizations: A state of the art review. University of Management 1981 Proceedings. Edited by: Chung KH. 1981, Wichita, Kansas: Academy of Management
-
Pincus JD: Advice satisfaction, job satisfaction, and job performance. Human Communication Enquiry. 1986, 12 (3): 395-419. x.1111/j.1468-2958.1986.tb00084.x.
Pre-publication history
-
The pre-publication history for this paper can exist accessed hither:http://www.biomedcentral.com/1472-6963/ten/188/prepub
Acknowledgements
The authors thank Sian Curtis for providing comments on the research design and methodology, Sarah Asiimwe, a individual consultant, for her aid in tailoring the survey instruments to the Ugandan context and helping to oversee the data collection procedure, and Monika Sawhney and Eva Silvestre for their excellent research assistance. Special thanks go to Samson Kironde of the UPHOLD Project for supporting the study.
This newspaper was made possible with financial support from the U.s.a. Government'due south Bureau for International Development (USAID) under the terms of Cooperative Agreement GPO-A-00-03-00003-00. The views expressed in this publication are those of the authors but and do not necessarily reflect the views of the United states Government.
Author information
Affiliations
Corresponding writer
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
The study was conceived by DRH, AA and TL, designed and undertaken by DRH, AA and EM, and written by DRH and AA.
All the authors take read and approved the last manuscript.
Electronic supplementary material
12913_2009_1325_MOESM1_ESM.DOCX
Boosted file 1: Tables S1 and S2. Table S1 provides the results of the Principal Components Analysis used to create an alphabetize of the utilise of RHIS information. Table S2 presents descriptive statistics for the variables entered in the cross-sectional model of the determinants of the use of RHIS information. (DOCX xvi KB)
Authors' original submitted files for images
Rights and permissions
This commodity is published under license to BioMed Cardinal Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/ii.0), which permits unrestricted apply, distribution, and reproduction in any medium, provided the original piece of work is properly cited.
Reprints and Permissions
Most this article
Cite this commodity
Hotchkiss, D.R., Aqil, A., Lippeveld, T. et al. Evaluation of the Functioning of Routine Data System Management (PRISM) framework: evidence from Uganda. BMC Health Serv Res 10, 188 (2010). https://doi.org/x.1186/1472-6963-10-188
-
Received:
-
Accepted:
-
Published:
-
DOI : https://doi.org/10.1186/1472-6963-ten-188
Keywords
- Ordinary Little Square Model
- Health Management Information System
- Health System Performance
- Dichotomous Indicator
- District Wellness Part
Source: https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-10-188
Post a Comment for "When Is Info 188 Going to Be Offered Again"