Combining Web-Based and Mail Surveys Improves Response Rates: A PBRN Study From PRIME Net ========================================================================================= * Philip J. Kroth * Laurie McPherson * Robert Leverence * Wilson Pace * Elvan Daniels * Robert L. Rhyne * Robert L. Williams ## Abstract **PURPOSE** The advent of Web-based survey tools has provided the investigator with an alternative to paper-based survey methods that in many instances may be less expensive to implement than traditional paper-based surveys. Newer technology, however, does not diminish the importance of obtaining an adequate response rate. **METHODS** We analyzed response rate data obtained from a survey implemented across 3 practice-based research networks (PBRNs) in which the survey was first implemented electronically with 5 rounds of electronic solicitation for an Internet-based questionnaire and then by 2 rounds of a paper-based version mailed only to nonresponders. **RESULTS** Overall, 24% of the total survey responses received were in the paper mode despite intense promotion of the survey in the electronic phase. **CONCLUSIONS** Our results suggest there is still an important role for the use of paper-based methods in PBRN survey research. Both hard copy and electronic survey collection methods may be required to enhance clinician response rates in PBRNs. * Data collection * paper * questionnaires * attitude of health personnel * attitude to computers * electronic mail * time factors * user-computer interface * computer communication networks * health surveys * postal service * health care surveys/methods ## INTRODUCTION Research using surveys for data collection is common in practice-based research networks (PBRNs). The continuing growth of electronic communications, potential for cost savings, and more rapid results has created interest in Internet-based survey designs. Regardless of the method or technology used in performing a survey, it is clear that an adequate response rate is still critical to the scientific validity of survey findings.1 Several investigators have shown that the mode of the survey measurably affects response rates in various biomedical venues.2 Our objective was to ascertain the value of a paper follow-up to an electronically based survey conducted on clinicians across 3 PBRNs. ## METHODS We performed a secondary, retrospective analysis on data obtained from a completed survey on clinician attitudes toward the screening and treatment of hepatitis C. The survey was conducted in the Primary Care Multi-Ethnic Research Network (PRIME Net).3 At the time of this study, PRIME Net was a collaboration of 3 PBRNs: the Research Involving Outpatient Settings Network (RIOS Net),4 the Colorado Research Network (CaReNet),5 and the South Eastern Regional Clinicians Network (SERCN).6 The survey instrument7 consisted of 28 questions exploring clinicians’ approaches to hepatitis C in their practices and available resources for care of patients infected with the hepatitis C virus. Invitations to participate in the survey were sent to all 805 active clinician members of the 3 PRIME Net PBRNs for whom a valid e-mail and postal address was available. Before the survey began, clinicians received 1 or more advance e-mail messages from their home PBRN describing the purpose of the survey and requesting participation. At the start of the survey, each clinician received a personalized e-mail solicitation from his or her home PBRN with a link to the Web-based version of the survey. The customized link had the target’s unique identifying code embedded to provide automatic log-in and response identification. We repeated the e-mail solicitation to nonresponders at intervals of 1 to 2 weeks for a total of 5 e-mail solicitations. The Web page for the survey also contained a link to a PDF version of the questionnaire that could be printed out and faxed or mailed back. We mailed the first round of the paper survey questionnaire to nonresponders when the fourth e-mail solicitation was sent. The postal survey packet contained a personalized cover letter with a Web address that allowed the respondent the option to complete the questionnaire through the Web. A stamped, addressed envelope was provided for return mail. We sent 2 complete paper mailings and 1 postcard reminder to nonresponders only. Continuing medical education credit, links to additional free hepatitis C continuing medical education opportunities, and periodic drawings for $75 gift certificates were offered as incentives. Procedures for both the electronic and paper formats assured respondent anonymity while allowing for targeted follow-up solicitations to nonrespondents. After the survey began, ad hoc general e-mail announcements were sent through individual network list-serves to all clinicians regardless of whether they had responded. (The purpose of these messages was to remind everyone to complete the questionnaire they had already received.) ## RESULTS A total of 301 electronic and 97 paper responses (97 postal mail and 3 faxes) yielded a valid overall PBRN response rate of 61% (Table 1⇓). Despite a total of 5 rounds of e-mail solicitations, almost one-quarter of the total responses received were in paper form (Table 1⇓). View this table: [Table 1.](http://www.annfammed.org/content/7/3/245/T1) Table 1. **Network Solicitation and Response Rates** Response times were several days faster for Web-based electronic questionnaires than for postal-based paper questionnaires (Figure 1⇓). Each e-mail solicitation had an immediate (ie, within a few days) yet diminishing response. The paper-based solicitations appeared to have a more delayed (ie, 1 week or so) response, as similarly reported in the literature (Figure 1⇓).8–10 ![Figure 1.](http://www.annfammed.org/https://www.annfammed.org/content/annalsfm/7/3/245/F1.medium.gif) [Figure 1.](http://www.annfammed.org/content/7/3/245/F1) Figure 1. **Practice-based research network survey responses timeline.** The estimated total costs to create and operate the Web site (technology support) and to mail the paper questionnaires (postage, envelopes, and stationery) were $4,260 and $6,230, respectively, for a cost of $14.20 and $62.30 per usable response for the electronic and paper questionnaires, respectively. ## DISCUSSION Despite multiple solicitations for the electronic mode, 24% of total responses were in paper form. This finding suggests that paper is still an important survey mode in PBRN research. We suspect that had the paper option been provided earlier in the process, the proportion of paper responses might have been even higher. Our experience in the PBRN realm is perhaps not surprising in light of the increasing number of randomized trials that also report an added value of paper in optimizing survey response rates.2,9,11–16 Only 2 respondents chose to print the PDF version of the survey and fax it back (as reported similarly).10 Perhaps the most touted feature of Web-based surveys is their potential for cost savings compared with paper.6,17 If the costs of developing and hosting the Web-based survey are taken into account, the differences in cost per usable survey response may not be as large as anticipated. Interestingly, there is an economy of scale dependent on sequencing. Electronic surveys have higher initial fixed costs but limited marginal costs. Postal surveys have lower fixed costs but higher marginal costs.8 Reversing the sequence in which electronic- and paper-based questionnaires were offered would substantially increase the total cost. The mixed-mode design of this survey was an attempt to reduce overall cost while maintaining adequate response rate. The higher cost of paper in this setting may be justifiable because it is possible we received paper responses from clinicians who might have responded only to paper regardless of the number of electronic solicitations. On the other hand, possibly some paper responders did so because they might have believed the paper questionnaire was their last opportunity to respond. With either case, those performing surveys of PBRN clinicians may do well to consider using both electronic and paper methods to enhance the total response rate. In our study, the response rate to the electronic version of the survey alone was insufficient to support the internal validity of the survey results. The conclusions that can be drawn from this analysis are somewhat limited because of the lack of a control group. Nevertheless, this study provides important data about surveys conducted in the context of clinicians in a PBRN.11–17 Results reported here suggest that in PBRN research surveys, using a mixed-mode research method that includes paper may yield a greater response rate than electronic alone. ## Footnotes * *Conflicts of interest: none reported* * **Funding support:** This project has been funded in whole or in part with grants from the National Institutes of Health, under contract No. HHSN268200425211C, “Re-Engineering the Clinical Research Enterprise.” * Received for publication March 6, 2008. * Revision received August 4, 2008. * Accepted for publication August 11, 2008. * © 2009 Annals of Family Medicine, Inc. ## REFERENCES 1. Dillman DA. *Mail and Internet Surveys: The Tailored Design Method.* New York, NY: John Wiley; 2000. 2. Beebe TJ, Locke GR III, Barnes SA, Davern ME, Anderson KJ. Mixing web and mail methods in a survey of physicians. Health Serv Res. 2007;42(3 Pt 1):1219–1234. [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1111/j.1475-6773.2006.00652.x&link_type=DOI) [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=17489911&link_type=MED&atom=%2Fannalsfm%2F7%2F3%2F245.atom) [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000246201400017&link_type=ISI) 3. Primary Care Research Network (PRIME Net). Web site. 2008. [http://hsc.unm.edu/som/primenet/](http://hsc.unm.edu/som/primenet/). Accessed Jan 8, 2008. 4. Research Involving Outpatient Settings (RIOS Net). Web site. 2008. [http://hsc.unm.edu/rios/](http://hsc.unm.edu/rios/). Accessed Jan 8, 2008. 5. The Colorado Research Network (CaRe Net). Web site. 2008 [http://fammed.uchsc.edu/carenet/](http://fammed.uchsc.edu/carenet/). Accessed Jun 9, 2008. 6. Southern Primary-care Urban Research Network (SPUR-Net). Web site. 2008. [http://www.spurnetwork.org/](http://www.spurnetwork.org/). Accessed Jan 8, 2008. 7. Kroth PJ, McPherson L, Leverence R, et al. PRIME Net Hepatitis-C Survey. Web page. 2008. [http://hdl.handle.net/1928/3620](http://hdl.handle.net/1928/3620). Accessed Jan 8, 2008. 8. Griffis SE, Goldsby TJ, Cooper M. Web-based and mail surveys: a comparison of response, data, and cost. J Bus Logistics. 2003 2003;24(2):237–58. 9. Truell AD, Bartlett JE II, Alexander MW. Response rate, speed, and completeness: a comparison of Internet-based and mail surveys. Behav Res Methods Instrum Comput. 2002;34(1):46–49. [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=12060989&link_type=MED&atom=%2Fannalsfm%2F7%2F3%2F245.atom) [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000175223200005&link_type=ISI) 10. Cobanoglu C, Warde B, Moreo PJ. A comparison of mail, fax, and Web-based survey methods. Int J Mark Res. 2001;43(4):441–442. 11. Raziano DB, Jayadevappa R, Valenzula D, Weiner M, Lavizzo-Mourey R. E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist. 2001;41(6):799–804. [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZ2Vyb250IjtzOjU6InJlc2lkIjtzOjg6IjQxLzYvNzk5IjtzOjQ6ImF0b20iO3M6MjI6Ii9hbm5hbHNmbS83LzMvMjQ1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 12. Rodriguez HP, von Glahn T, Rogers WH, Chang H, Fanjiang G, Safran DG. Evaluating patients’ experiences with individual physicians: a randomized trial of mail, internet, and interactive voice response telephone administration of surveys. Med Care. 2006;44(2):167–174. [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1097/01.mlr.0000196961.00933.8e&link_type=DOI) [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=16434916&link_type=MED&atom=%2Fannalsfm%2F7%2F3%2F245.atom) [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000234950400010&link_type=ISI) 13. Seguin R, Godwin M, MacDonald S, McCall M. E-mail or snail mail? Randomized controlled trial on which works better for surveys. Can Fam Physician. 2004;50:414–419. [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjg6IjUwLzMvNDE0IjtzOjQ6ImF0b20iO3M6MjI6Ii9hbm5hbHNmbS83LzMvMjQ1LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 14. VanDenKerkhof EG, Parlow JL, Goldstein DH, Milne B. In Canada, anesthesiologists are less likely to respond to an electronic, compared to a paper questionnaire. Can J Anaesth. 2004;51(5):449–454. [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15128630&link_type=MED&atom=%2Fannalsfm%2F7%2F3%2F245.atom) [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000221430800008&link_type=ISI) 15. Mavis BE, Brocato JJ. Postal surveys versus electronic mail surveys. The tortoise and the hare revisited. Eval Health Prof. 1998;21(3):395–408. [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BlaHAiO3M6NToicmVzaWQiO3M6ODoiMjEvMy8zOTUiO3M6NDoiYXRvbSI7czoyMjoiL2FubmFsc2ZtLzcvMy8yNDUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 16. Akl EA, Maroun N, Klocke RA, Montori V, Schunemann HJ. Electronic mail was not better than postal mail for surveying residents and faculty. J Clin Epidemiol. 2005;58(4):425–429. [CrossRef](http://www.annfammed.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2004.10.006&link_type=DOI) [PubMed](http://www.annfammed.org/lookup/external-ref?access_num=15862729&link_type=MED&atom=%2Fannalsfm%2F7%2F3%2F245.atom) [Web of Science](http://www.annfammed.org/lookup/external-ref?access_num=000228055200014&link_type=ISI) 17. McMahon SR, Iwamoto M, Massoudi MS, et al. Comparison of e-mail, fax, and postal surveys of pediatricians. Pediatrics. 2003;111(4 Pt 1):e299–e303. [Abstract/FREE Full Text](http://www.annfammed.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjExMS80L2UyOTkiO3M6NDoiYXRvbSI7czoyMjoiL2FubmFsc2ZtLzcvMy8yNDUuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9)