Panel Retention Research Paper

2. Factors Associated with Panel Retention

Retaining panel respondents is generally believed to depend on three factors. First is the respondent’s willingness to participate in the study at future waves. Even when respondents are located for follow-up, keeping them interested and engaged in participation is a challenge. Second, the respondent’s continued availability to participate depends on factors like illness or death. The third, and most important factor in panel maintenance, is the ability to locate respondents for interview at later waves.

Failure to find a respondent at later waves is predominantly due to residential mobility. Based on data from the 1998 Current Population Survey, during which respondents were asked whether they had lived at the same residence one year before the interview, it was estimated that 42.5 million Americans moved (Faber 2000). The majority (64 percent) moved within the same county, 19 percent moved to a different county in the same state, and 16 percent moved to a different state. Age is correlated with moving; the highest rates of residential mobility are found among those aged 20–29 and they decline as age increases. Rates of residential mobility also vary with race and ethnicity and by region of the country. They are lowest for non-Hispanic whites (14.5 percent), higher for Blacks and Asians (18.9 percent) and highest among Hispanics (21.2 percent). People who rent their homes are more likely to relocate than those who own their residences are. Nearly one-third of people living in rental units reported having moved in the previous year compared with less than 10 percent of homeowners. Rates of residential relocation vary by region as well. They are highest in the West (19.4 percent), lower in the South (17.2 percent), Midwest (14.7 percent) and Northeast (11.5 percent), respectively. Understanding these correlates of residential relocation will help investigators plan for the level of tracking they will need to do during follow-up interviews. For example, a panel of Hispanics in California would be likely to have higher attrition than a panel of white householders in Maine. Hence more resources would be required for respondent retention in the California sample than in the Maine sample. Including these variables as indicators at the baseline wave of data collection is important for postsurvey attrition assessment and weighting to adjust for it.

3. Study Design Features to Minimize Panel Bias

There are three main factors that affect attrition: recruiting the respondent to the study, locating the respondent for subsequent interviews, and maintaining the respondent’s commitment to the panel.

3.1 Recruiting Respondents to the Panel

Plans for recruitment and retention should be developed at the time the study is designed. They are as important as any other feature of the study design, but often do not get the serious attention (or budget allocation) they deserve. Later, we discuss the specific recruitment and relocation strategies that might be built into these plans.

3.1.1 Preplanning.

Preplanning will ensure not only that there are sufficient resources to successfully recruit and retain the respondents, but also that the information needed to track respondents is collected during the baseline interview. It will also guide the selection of the sample size since this must allow for attrition if locating respondents is anticipated to be a serious problem. It may also affect decisions about the mode of interview because certain segments of the population may not be available by telephone or computer. For example, telephone coverage is related to socioeconomic status and to ethnicity (Belinfante 2000), and computer access is even more strongly so.

3.1.2 Study logo.

Part of recruitment and retention is creating a recognizable identity for the study. A study logo and identifiable study title that is used in all correspondence (written and oral) and that appears on self-administered questionnaires, where applicable, have also been suggested as a way to create an identifiable image of the study for the respondent (Given et al. 1990).

3.1.3 Ad ance letters.

Depending on the sample frame (list, random-digit dial generated, or blocklisted sample), an advance letter should be sent to respondents at baseline that explains the purpose of the study, and where they can go for further information (a phone number or study web site, depending on the population). Advance letters are most commonly used in studies where the mode of interview administration is face-to-face. Advance letters have also been used in dual frame telephone surveys where telephone interviewing is combined with face-to-face or mail on the theory that they will increase response rates, although the evidence is mixed (Parsons et al. 1999, Traugott et al. 1987). Clearly, however, advance letters require that the investigator have a current address for the respondent. Even if addresses are not possible at the baseline interview wave, they should still be obtained at the baseline interview for follow-up purposes.

3.1.4 Response rates.

The most important aspect of recruitment is the baseline survey where respondents have their initial contact with the study. Obtaining an acceptable rate of cooperation to the interview will affect the overall data quality. If the baseline survey is biased due to nonresponse, then the internal and external validity of the panel will always be problematic. Four methods are typically used for ensuring high contact and recruitment rates (Groves and Couper 1998): (a) budget for additional contact attempts with those sample units that cannot be immediately contacted; (b) schedule these contact attempts (whether by telephone or face-to-face) at various time of the day, evenings, and weekends to maximize the opportunity to reach someone at home; (c) plan for lengthier data collection during the follow-up waves to allow for respondent vacations and for locating hard-to-find respondents; and (d) allow interviewers to use local sources, such as neighbors, for information on the respondent or household. These techniques are standard in cross-sectional surveys but are especially important for panel studies to improve response rates.

3.2 Locating and Tracking Respondents for Follow-up Wa es

Once respondents are recruited into the panel, several methods can be used to increase the likelihood of contact at designated follow-up waves.

3.2.1 Advance letters.

Regardless of whether advance letters were used to recruit subjects, at every subsequent data collection wave, advance letters should be sent before interviewing begins. These letters notify the respondent of the upcoming visit or call and alert the investigator to a possible address change. The ‘address correction requested’ service available through the post office should be used with each letter.

3.2.2 Secondary sources.

During the baseline interview, information should be obtained that will help locate respondents later. One invaluable resource is the secondary contact source. Secondary sources are usually one or two persons, not residing with the respondent, who will know how to contact the respondent at any point in time. At a minimum the relationship of the secondary source to the respondent and that person’s telephone number should be collected.

In addition, during the baseline interview all respondents should be asked to provide their dates of birth, social security numbers (for use with credit-based tracking programs), and the names under which the respondents’ telephones are listed. Depending on the population, respondents’ e-mail addresses should also be collected as they can be used as an inexpensive means of tracking respondents.

3.2.3 In-person location strategies.

Even the most conscientiously designed study will not prevent respondents from being lost to follow-up. If all secondary source information does not lead to the whereabouts of the respondent, there are several sophisticated tracking devices that can be used. At the very least, attempts at locating the respondent should persist until a final disposition of the case can be ascertained for the purposes of calculating response rates.

In face-to-face studies, an excellent source of information about respondents comes from people who are connected in some way with the respondent. Roommates, neighbors, property managers, mail carriers, family, and friends might be available to assist in searching for, and contacting, a hard-to-find study participant. When finding the respondent through direct contact does not lead to complete or solid information, personal informants can sometimes provide essential information about how to reach the respondent, what kind of schedules they keep, and whether they have moved.

3.2.4 Mail location strategies.

In addition to ‘address correction requested’ stamped on advance letter envelopes, a number of other useful services are available through the US Postal Service (USPS) to improve mailing addresses and locate respondents who have moved (United States Postal Service 1999). Through the Address List Correction Service, business and resident mailing lists can be submitted for correction. Services provided include cross-listing of names to which mail cannot be delivered or forwarded; correcting ZIP codes, house numbers, rural, or post office box numbers, street names, and suffixes; furnishing new addresses for customers who have moved; and adding building, apartment, suite or room numbers when known. There is a minimum charge for correcting each submitted list, which can be arranged through local post office branches. In addition, the USPS National Change of Address (NCOA) program processes change of address data nationwide. The NCOA service is provided by private sector companies, certified and licensed by the USPS, who charge a fee for the service. These companies receive updated, computerized COA information on a regular basis. To search for lost or unlocatable study respondents, a file of names and addresses is submitted to one of the NCOA vendors, who search for matches in the NCOA. New addresses are provided when an exact match is found. Most other countries will have systems with many of these attributes.

3.2.5 Haines criss-cross directories.

In the USA, the Haines Directory alphabetically cross-lists every street in a given community with the associated names, address, and (officially listed) telephone numbers of the people who live on these streets. A second section of the directory lists all published telephone numbers in sequence and cross-references them with the corresponding address and names of the residents. Haines directories can thus be used when only the telephone number or only the address is available. They can be found in local public libraries, which also house the directories for several years in the past.

3.2.6 Credit bureaus and ital statistics agencies.

Another source for tracking panel respondents is credit bureau records. Software packages can be purchased from credit bureaus that allow investigators to search the credit bureau databases directly (for updated name and address information). By entering the last known address of the respondent, the file will provide data on whether the respondent has moved and, if so, the new address will be provided. These systems are a valuable investment if you are able to obtain the respondents social security numbers, as credit records are tied to that identifier. However, the social security number is not useful for locating all populations in the USA. For example, low-income or homeless respondents will be unlikely have a credit history that can be referenced or tracked. There are also a variety of Internet search sites that can be used for tracking. While they are free of charge, they are also not very comprehensive or up-to-date. They are often linked to the respondents’ e-mail address and if the respondent does not use the Internet, they may not be found via this source.

In other situations, accessing public records is necessary for obtaining missing information about respondents. Public records contain data on births, deaths, marriages, criminal activity, military service, motor vehicle registration, and employment. Of particular use in panel studies that span a long period of time is the National Death Index, maintained by the National Center for Health Statistics. For a fee, the index will match respondents with their file of deceased persons using the name and date of birth. The likelihood of a match increases if the social security number or middle initial is available. In some states it is possible to search a web site from the department of corrections to ascertain if any study participants have been incarcerated. (For searching public records in the USA, see Gunderson 1996, Hoyer and McCann 1988, and Parco 1994.)

3.3 Retaining Respondents Between Inter iew Waves

Being a member of a panel is burdensome for the respondent. Thus, it is critical to the successful maintenance of the sample that the relationship be reinforced between contacts.

3.3.1 The importance of contact.

Regardless of the duration of time between data collection waves, it is advisable to maintain contact with respondents every 3–6 months as resources allow (Hunt and White 1998). Regular contact keeps the respondents engaged and reminds them of the value placed by the investigator on their participation. It also helps the investigator keep track of the respondent. The extent of the contact need not be more than the advance letter or a thank you note. Besides a friendly greeting and an expression of thanks for the respondents’ participation, the content should include a reminder of the time of the next follow-up survey and a request for new contact information if the respondent has moved or will be moving. Investigators can also personalize the contact between interviewing waves by using birthday cards, holiday greetings, or other forms of written correspondence (Walton et al. 1998). These formats reinforce that the study staff is interested in and cares about the participants.

3.3.2 Incentives.

Incentives are also critical to effective panel maintenance. Overall, the literature favors the use of incentives to increase response rates. Not only do they represent a concrete expression of appreciation to the respondent, but they also have the effect of keeping the respondent interested and involved. Anecdotal evidence also suggests that respondents are more likely to notify investigators of new addresses when they have moved if they expect an incentive.

Incentives typically take the form of money (cash or checks) or something with cash value (such as gift certificates or gifts), and can be presented to the respondent in advance of the interview, at the time the interview is conducted, or postinterview. The benefits of incentives have been empirically demonstrated in a meta-analysis of their effects on response to mail surveys (Church 1993). This analysis demonstrated that prepaid incentives yielded higher response rates than when the incentive was promised upon return of the survey form, and that cash was more successful than gifts in increasing response rates. Similarly, Singer et al. (1999) reported the results of their study of the effects of incentives in telephone and face-toface studies and concluded that monetary incentives have a demonstrable effect on response rates. As did Church, they also found that cash incentives are more effective than gifts or tokens. Moreover, the authors found incentives to be effective regardless of the degree of burden on the respondent (e.g., 20 minutes vs. an hour interview).

The work reported by Singer et al. (1999) is also applicable to panel respondents. It suggests that informing the respondent at the start of the study that they will be offered an incentive for participation increases cooperation at later waves. Thus, at the conclusion of each interview, and before collecting the data that will help track the respondent at subsequent waves, it is useful to inform respondents that they will be paid an incentive for their future participation. However, it is important to emphasize that the respondent is not being paid for the data just provided. Rather, the incentive is a token of appreciation for having provided something that is invaluable and therefore cannot be purchased.

4. Summary

Panel retention is an under-appreciated but very important part of longitudinal studies that involve reinterviewing a cohort of respondents over time. While participation in a panel is a burden for the respondent, maintaining an unbiased panel is a challenge for the investigator. Thus, before such a project is undertaken planning is critical and the level of effort and associated costs need to be considered from the outset.

A key aspect of a successful panel is the study design. The sample, questionnaire, and associated budget must all accommodate the need to recruit an unbiased panel and maintain their participation over a lengthy period of time. Recruitment involves attracting individual respondents to agree to participate in the baseline and to be followed for some period of time afterwards. Locating respondents is critical to ensuring the continued quality of the panel data. Finally, recruiting and maintaining the panel will be facilitated if respondents develop a commitment to it. Ensuring that the panel is an unbiased sample of the population of interest requires effort to not only convince individuals to participate but to provide sufficient information to enable the panel to continue.

Achieving a high level of cooperation to the baseline is critical and obtaining cooperation must be done in a manner that does not alienate the respondent. It must also be recognized that overall trends in survey research response rates indicate that collecting any survey data collection is going to become increasingly difficult. In the USA, there has been sufficient decline in response rates to social surveys since the 1990s that the leading statistical agencies and professional associations that rely on survey data convened a special conference devoted to the topic (the International Conference on Survey Nonresponse in Portland, Oregon, October 1999).

In sum, plans for follow-up must be established at the onset of the study so that all the relevant information is collected from the respondents as necessary resources are allocated. Without sufficient data on respondents, when they change residence or telephone, they are likely to be lost to follow-up, introducing bias into the data. Convincing respondents that they are important to the success of the panel and that the investigator values their participation ultimately helps reduce the cost of locating and following the respondents. Overall, participation in research is a process of exchange. Thus, it is important to show respect and consideration for the sources of the data.

Bibliography:

1. Belinfante A2000 Telephone Subscribership in the United States. Federal Communications ommission, Washington, DC
2. Church A H 1993 Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly 57: 62–79
3. Cook T D, Campbell D 1979 Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton-Mifflin Company, Geneva, IL
4. Cottler L B, Compton W M, Ben-Abdallah A, Horne M, Claverie D 1996 Achieving a 96.6 percent follow-up rate in a longitudinal study of drug abusers. Drug and Alcohol Dependence 41(3): 208–17
5. Faber C S 2000 Geographic Mobility: March 1997 to March 1998. Current Population Reports P20-520. US Census Bureau, Washington, DC
6. Given B A, Keilman L J, Collins C, Given C W 1990 Strategies to minimize attrition in longitudinal studies. Nursing Research 39(3): 184–6
7. Groves R, Couper M 1998 Nonresponse in Household Inter iew Sur eys. John Wiley and Sons, New York
8. Gunderson T 1996 How to Locate Anyone Without Lea ing Home. Plume, New York
9. Hoyer F, McCann J 1988 Find Them Fast, Find Them Now. Citadel Press, Secaucus, NJ
10. Hunt J R, White E 1998 Retaining and tracking cohort study members. Epidemiologic Reviews 20: 57–70
11. Parco V 1994 Researching Public Records. Citadel Press, New York
12. Parsons J A, Owens L, DeGrush A, Johnson T 1999 Using a dual frame design to increase response rates: Results of two experiments. Paper presented at the. In: 54th Annual Conference of the American Association for Public Opinion Research. St. Petersburg Beach, FL
13. Singer E, Van Howeyk J, Gebler N, Raghunathan T, McGonagle K 1999 The effect of incentives on response rates in interviewer-mediated surveys. Journal of Official Statistics 15(2): 217–30
14. Traugott M W, Groves R M, Lepkowski J M 1987 Using dual frame designs to reduce nonresponse in telephone surveys. Public Opinion Quarterly 51: 48–57
15. United States Post Office 1999 Address Information Systems: Products and Ser ices, Publication 40
16. Walton M A, Ramanathan C S, Reischel T M 1998 Tracking substance abusers in longitudinal research: Understanding follow-up with difficulty. American Journal of Community Psychology 26: 233–53
17. Wright J D, Allen T L, Devine J 1995 Tracking non-traditional populations in longitudinal studies. E aluation and Program Planning 18(3): 267–77