Pereira et al. BMC Health Services Research 2011, 0RESEARCH ARTICLEOpen AccessA cost comparison of electronic and hybrid datacollection systems in Ontario during pandemicand seasonal influenza vaccination campaignsJennifer A Pereira1, Julie Foisy1, Jeffrey C Kwong2*, Christine L Heidebrecht1, Susan Quach1, Sherman D Quan3,Maryse Guay4 and Beate Sander1, for the Public Health Agency of Canada/Canadian Institutes of Health ResearchInfluenza Research Network (PCIRN) Vaccine Coverage Theme GroupAbstractBackground: During the pandemic (H1N1) 2009 influenza vaccination campaign, health regions in Canadacollected client-level immunization data using fully electronic or hybrid systems, with the latter comprising bothelectronic and paper-based elements. The objective of our evaluation was to compare projected five-year costsassociated with implementing these systems in Ontario public health units (PHUs) during pandemic and seasonalinfluenza vaccination campaigns.Methods: Six PHUs provided equipment and staffing costs during the pandemic (H1N1) 2009 influenza vaccinationcampaign and staffing algorithms for seasonal campaigns. We standardized resources to population sizes 100,000,500,000 and 1,000,000, assuming equipment lifetime of five years and public health vaccine administration rates of18% and 2.5% for H1N1 and seasonal campaigns, respectively. Two scenarios were considered: Year 1 pandemicand Year 1 seasonal campaigns, each followed by four regular influenza seasons. Costs were discounted at 5%.Results: Assuming a Year 1 pandemic, the five-year costs per capita for the electronic system decrease as PHUpopulation size increases, becoming increasingly less costly than hybrid systems ( 4.33 vs. 4.34 [100,000], 4.17 vs. 4.34 [500,000], 4.12 vs. 4.34 [1,000, 000]). The same trend is observed for the scenario reflecting five seasonalcampaigns, with the electronic system being less expensive per capita than the hybrid system for all populationsizes ( 1.93 vs. 1.95 [100,000], 1.91 vs. 1.94 [500,000], 1.87 vs. 1.94 [1,000, 000]). Sensitivity analyses identifiedfactors related to nurse hours as affecting the direction and magnitude of the results.Conclusions: Five-year cost projections for electronic systems were comparable or less expensive than for hybridsystems, at all PHU population sizes. An intangible benefit of the electronic system is having data rapidly availablefor reporting.BackgroundMass immunization clinics are often employed by publichealth organizations when a substantial proportion of thepopulation needs to be vaccinated quickly, such as forseasonal influenza vaccination campaigns or pandemicemergencies. Planning for these high-volume clinicsincludes consideration of the approach that will be usedto collect client-level vaccination data to help ensure* Correspondence: [email protected] for Clinical Evaluative Sciences, Toronto, CanadaFull list of author information is available at the end of the articleoptimal vaccine delivery processes, and timely assessmentof vaccine coverage and surveillance statistics [1].In preparation for the pandemic (H1N1) 2009 influenza vaccination campaign, every Canadian province/territory mandated data reporting requirements fortheir jurisdictions. While many health regions collectedthese immunization data using pre-existing approaches,other jurisdictions adopted new methods. Ontario wasthe only province in which fully electronic influenzaimmunization systems were used. Thirty of Ontario’sthirty-six public health units (PHUs) collected H1N1immunization data using this type of system, whereby 2011 Pereira et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative CommonsAttribution License (, which permits unrestricted use, distribution, and reproduction inany medium, provided the original work is properly cited.

Pereira et al. BMC Health Services Research 2011, 0client demographic, medical history and vaccinationadministration data were immediately inputted electronically, facilitating their use for program planning andreporting. Hybrid systems were implemented acrossOntario’s remaining PHUs, comprised of mainly paperbased data collection, with use of an electronic component at client registration or during post-vaccinationtransfer of data from paper.In a qualitative study involving interviews with pandemic planners across Canada prior to the vaccinationcampaign, cost was identified as a primary perceivedbarrier to implementing an electronic system to collectclient-level immunization data [2]. To provide valuableinformation to decision-makers planning mass vaccination campaigns, our study objective was to compare thecosts to Ontario PHUs associated with implementingelectronic and hybrid immunization data collection systems, during both pandemic and seasonal influenza vaccination campaigns.MethodsAnalysisWe performed a costing analysis comparing electronicand hybrid approaches to immunization data collectionused in Ontario during the H1N1 vaccination campaign,from the perspective of the PHU; only direct equipmentand staffing costs were included.Electronic SystemThe fully electronic immunization data collection system considered in this study is the Protocol for Electronic Clinic Systems (PECS), developed by NiagaraRegion, Ontario, Canada. Clients register with a clerkequipped with a laptop who records their demographicinformation (by swiping their driver’s license or healthcard) and medical history data. An immunization nursethen retrieves each individual’s record from the networked database, reviews medical history before administering the vaccine, and enters data regarding thevaccine given (injection site, vaccine name/lot number/expiry date) into the system. Proof of vaccine is printeddirectly from the client record and provided to each client before they exit the clinic. The individual vaccinerecords created at each immunization clinic areuploaded into a regional database from which reportsare produced.Hybrid SystemThe hybrid system considered in this study combinesboth electronic and paper-based components. At registration, all client demographic and medical history dataare recorded on paper by the client and confirmed by theclerk. After the immunization nurse reviews the medicalhistory and administers the vaccine, the nurse writes outa paper record of the vaccination information as well as aproof of vaccine and provides the latter to the client. AllPage 2 of 10client data are later inputted into an electronic databaseby data entry clerks.Given that a five-year time-frame includes equipmentpurchase and utilization until replacement is required,the primary outcome was overall five-year costs in 2009Canadian dollars, per data collection system. Reportedresults have been discounted at 5% [3].ModelWe developed a provincial model in Microsoft Excel tocompare the equipment and staffing costs associated withimplementing data collection systems during influenzavaccination campaigns. Two scenarios were tested: inScenario 1, Year 1 was modeled as a pandemic campaigndirectly based on the costing data provided, while Years2-5 were assumed to be seasonal influenza vaccinationcampaigns; in Scenario 2, all five years were modeled asseasonal campaigns. It was assumed that the system wouldbe implemented for the first time in a given jurisdiction inYear 1 (requiring purchase of equipment), electronicequipment would be replaced every five years only, andthat costs for Years 2-5 would therefore be mainly for support and maintenance. The model predicted the totalfive-year costs as well as the cost per capita per PHU associated with both systems in each scenario.The model was based on costing and resource utilization data received from participating Ontario PHUs, andstandardized for populations of 100,000 (small), 500,000(medium), and 1,000,000 (large).DataParameters describing: i) the costs associated with the datacollection systems employed during the pandemic (H1N1)2009 vaccination campaign; and ii) comparative resourceuse during seasonal influenza campaigns, were collectedthrough questionnaires and telephone interviews withmanagers and information technology (IT) staff from aconvenience sample of six Ontario PHUs that participatedin a related efficiency study [4]. Each of these PHUs utilized a fully electronic system for the pandemic campaign,but had used paper-based or hybrid data collection systems for recent seasonal campaigns.Equipment and salary costs were compiled (Table 1).The base case consisted of mid-range cost values for allparameters, unless otherwise stated. Equipment costswere mainly associated with the electronic system,although laptops were also required for post-vaccinationdata entry for the hybrid system. Software fees for theelectronic system were included in the analysis; however,because they were provided by the vendor on the condition that they would not be divulged, we also conducteda threshold analysis to determine the total five-year software cost at which the hybrid and electronic systemscosts were equal.

Pereira et al. BMC Health Services Research 2011, 0Page 3 of 10Table 1 Equipment/Staffing Costs*Unit CostUnit Cost RangeServerEquipment 1,300 1,052 - 1,5495 yearsRouter 173 76 - 2695 yearsSwitchesExpected Lifetime 323N/A5 years 1,144 747 - 1,5405 yearsPrinter 874 236 - 1,5115 yearsCables 134N/A5 yearsLaptopPrinting**Miscellaneous items***StaffNurseIT personnelRegistration/data entry clerk 6,230- 3,000 - 9,460Annual cost 10,869- 10,075 - 11,663Annual cost Hourly WageHourly Wage Range 32 - 50 46 40 40 23.75 22 - 25.50* Source: Convenience sample of Ontario PHUs (n 6)** Includes carbon copies, consent forms, vaccine administration forms, proof of vaccination printing, toner etc.Listed costs are total annual costs, rather than unit costs.*** Includes magnetic swipe-card readers, laptop cases, air cards, battery packs, USB keys, storage cases, laptop locks etc. Only a portion of H1N1 miscellaneous equipment costs is expected to be incurred during seasonal campaigns. Base case wage was chosen because it was consistently reported by the sources.Assumptions for Scenario 2 and Years 2-5 of Scenario1 were derived based on input from the participatingPHUs who estimated their immunization data collectionresource use during previous seasonal campaigns relative to the pandemic campaign (Table 2). Since a smaller proportion of a PHU’s population is vaccinated bypublic health during seasonal campaigns compared to apandemic campaign, less equipment and staffingresources are required. The proportion of H1N1 staffinghours used during seasonal campaigns (15%) was chosenbased on: i) input from three PHUs in the conveniencesample; and ii) the ratio of assumed H1N1 public healthvaccination rate (18%) to the assumed seasonal influenzapublic health vaccination rate (2.5%) (unpublished data).Assumptions for the hybrid system resource use incomparison to the electronic system were based on estimates from the convenience sample of Ontario PHUs aswell as peer-reviewed literature (Table 2). Given that themajority of miscellaneous costs are equipment-related,and that hybrid systems use minimal equipment, weTable 2 Model AssumptionsParameterCoverage Rate*Paper costsMiscellaneous costsSeasonal Influenza Vaccination Campaign AssumptionsSource2.5%Ontario PHUs1/3 of costs during H1N1 campaign1/3 of costs during H1N1 campaignOntario PHUsOntario PHUsNursing hours15% of hours during H1N1 campaignOntario PHUsClerk hours15% of hours during H1N1 campaignOntario PHUsIT hours15% of hours during H1N1 campaignOntario PHUsNurse training hours**15% of hours during H1N1 campaignOntario PHUsIT training hours**15% of hours during H1N1 campaignOntario PHUsParameterHybrid System AssumptionsSourceLaptop (n)Paper costs1 laptop per 10,000 vaccinees200% of electronic systemUnpublished dataAssumptionMiscellaneous costsNursing hoursClerk hoursIT hoursData sorting timePost-clinic data entry hours10% of electronic systemAssumption133% of hours needed for electronic systemOntario PHUsEqual to electronic systemOntario PHUsNone requiredOntario PHUs7 hours per 500 recordsBillittier et al. [6]62 seconds per vaccine recordQuach et al. [5]*Percent of PHU population vaccinated by public health** The model assumes that all IT and nursing staff involved with a PHU’s mass vaccination campaign will be trained annually.

Pereira et al. BMC Health Services Research 2011, 0assumed that the miscellaneous cost of the hybrid system would be 10% that of the electronic system. Similarly, as the hybrid system utilizes paper forms for themajority of data collection tasks, whereas only the proofof vaccination administration is printed on paper for theelectronic system, we conservatively assumed that thepaper costs of the hybrid system would be twice that ofthe electronic system.It was assumed that post-vaccination data entry for thehybrid system would require one laptop per 10,000 clientsvaccinated, based on interviews with public health contacts in a separate province (unpublished data). Theassumption that the hybrid system would require onethird more nurse hours than the electronic system wasbased on staffing formulas provided by two of the PHUs,indicating that a nurse could see 20-30 clients per hourwhen collecting data using a hybrid system, compared toover 40 clients per hour with an electronic system. Additionally, in a sub-analysis of data from our time andmotion study examining immunization data collectionduring the pandemic vaccination campaign in Canada [5],we compared nursing time per client for PHUs implementing PECS versus organizations that: i) used a hybridsystem, and ii) collected a similar number of data elementsfrom clients as PECS users; we found that nursing timeper client for the organizations using the electronic systemwas almost 40% shorter than for jurisdictions using ahybrid system. For the model, we chose to use the conservative 33% value as the base case, and to test a largerrange in the sensitivity analysis.The duties of the registration clerks were consideredwhen developing related assumptions: for the electronicsystem, the clerk uses swipe-card technology to populatethe client’s demographic data fields and manually entersother necessary information, while for the hybrid system,the clerk either completes the client’s form or reviewswhat the client has completed before they proceed to theimmunization nurse. It was therefore expected that thesystems would require an equal number of registrationclerk hours.Compared to the electronic system, the hybrid systemrequires resources for two additional data processing steps:counting and sorting client forms, and subsequent dataentry into an electronic database. Billittier et al. found thatseven person-hours were required to sort and alphabetizepaper forms for 500 influenza vaccinations; this value wasused in our model [6]. We calculated expected hours fordata entry clerks by multiplying the number of vaccinatedindividuals within a PHU (public health administration rate population size) with time required to enter one client’srecord; this was assumed to be 62 seconds based on theresults of our time and motion study [5].IT support was assumed to be required for the electronic system only and included time spent setting upPage 4 of 10the electronic database, pre-testing the system and providing clinic support.Sensitivity AnalysisData uncertainty and jurisdictional variation in parameterswere explored with one-way sensitivity analyses. Severalclinic parameters (number of nurses, number of IT personnel, etc.) and economic parameters (equipment costs,salary wages, etc.) were modified within a pre-specifiedrange based on data from public health contacts, publishedliterature and best/worst case assumptions.Research Ethics Board ApprovalThis study received approval from the Research EthicsBoard at the University of Toronto, Toronto, Canada.ResultsBase CaseScenario 1 (1st year pandemic 4 years of seasonal influenza vaccination campaigns): Assuming a pandemic inYear 1 only, the five-year costs per capita for the electronic system is less than the hybrid system for all PHUpopulation sizes: 4.33 vs. 4.34 [100,000], 4.17 vs. 4.34[500,000], 4.12 vs. 4.34 [1,000,000]. As the electronicsystem’s software fees are based on a sliding scale that isdependent on PHU size, there is a decrease in cost percapita with larger PHUs. Of total costs, staffing comprises83-87% for electronic systems and 98% for hybridsystems (Table 3).Scenario 2 (5 years of seasonal influenza vaccinationcampaigns): For the scenario reflecting five regular influenza seasons, the five-year costs per capita of the electronic system are also less than the hybrid system for allpopulation sizes. Similar to Scenario 1, the betweensystem cost difference increases as the population sizeincreases ( 1.93 vs. 1.95 [100,000], 1.91 vs. 1.94[500,000], 1.87 vs. 1.94 [1,000,000]). For PHUs usingfully electronic systems, staffing-related expenses comprise83-85% of electronic system total costs and 98% ofhybrid system total costs (Table 4).Sensitivity AnalysisDeterministic one-way sensitivity analysis was performedon various model parameters for both scenarios. For several parameters, variation led to changes in the magnitudeand/or direction of the total cost differences between electronic and hybrid systems.Scenario 1 (1st year pandemic 4 years of seasonalinfluenza vaccination campaigns)For each PHU population size, several parameter changesresulted in the hybrid system becoming less expensivethan the electronic system by fairly small margins(Figures 1, 2 and 3). One parameter change - decreasingnurse hours for the hybrid system from 133% to 100%

Pereira et al. BMC Health Services Research 2011, 0Page 5 of 10Table 3 Total Five-Year Cost Comparison of Electronic and Hybrid Immunization Data Collection Systems for Scenario1 (1st year pandemic 4 years of seasonal influenza vaccination campaigns)*PHU population 100,000PHU population 100,000PHU population 500,000ElectronicHybrid 75,200 77,377 7,405 7,774Staffing CostsStaffing Costsu 357,546 373,442TOTAL COSTS(per capitacosts)TOTAL COSTSu(per capita costs) 432,746( 4.33) 450,819( 4.51)Equipment CostsEquipment CostsuPHU population 500,000PHU population 1,000,000PHU population 1,000,000ElectronicHybridElectronicHybrid 296,907 306,887 37,024 38,868 547,447 563,775 74,049 77,735 426,817 445,773 1,787,728 1,867,210 2,134,084 2,228,867 3,575,456 3,734,419 4,268,170 4,457,734 434,222( 4.34) 453,547( 4.54) 2,084,635( 4.17) 2,174,097( 4.35) 2,171,108( 4.34) 2,267,735( 4.54) 4,122,903( 4.12) 4,298,194( 4.30) 4,342,218( 4.34) 4,535,469( 4.54)u 1,476 2,728System CostDifferenceSystem CostDifferenceu 86,473 93,368 219,315 237,275*Total costs have been discounted at 5%.u Undiscountedthat of the electronic system - led to a costing differenceof greater than 50,000. For the smallest, medium-sizedand largest PHUs, the between-system cost differenceswould be 84,533, 343,575, and 649,365, respectively.Results from our threshold analysis demonstrated thatin order for the electronic system to become less costlythan the hybrid system, the total five-year software feesfor small, medium and large-sized PHUs would have toincrease to approximately 29,000, 143,000 and 281,000, respectively.Scenario 2 (5 years of seasonal influenza vaccinationcampaigns)For PHUs with a population of 100,000 (Figure 4), several parameter changes including decreasing nursehours during the seasonal influenza campaign to 5%that of the H1N1 campaign and decreasing seasonalvaccine coverage rate to 1% led to the electronic system bec