I have a question about surveys, random samples, etc and the validity of survey data. Our group does a pretty well rounded community based social marketing effort on changing transportation behavior - I think we do anyway. We do conduct before and after surveys but the validity of our data has been questioned by folks, mostly in our own bureau and even among our own group - in big part due to the respondents to our surveys being self-selecting. In measuring the results of cbsm efforts what effect does self selection have on survey results? If you agree with the premise that there are folks who are early adopters, some folks on the fence and those who are not at all interested in changing any behavior during your campaign, then your target audience and your biggest opportunity for success is to target those on the fence - correct? If this is the case, then is not your selected target audience already 'self selecting'? Further, if when you intially survey you also ask folks what type of information they would like to have on using various forms of transportation, is this not a form of survey in itself? And also self-selecting? Based on the above information, individualized packets of information are sent to each individual responding. That is to say they get only that information they ask for and if they do not respond after two prompts (and one introductory newsletter) they are left alone. Each transportation mode has it's very own multi-level program designed to identify and overcome barriers, provide opportunities for people to get involved in workshops, discussions, group activities and earn incentives. (Additionally, each mode program is further broken down for different audience nees such as senior citizens, school children and their parents, commuters/worker/employees and employers, and women in order to address specific unique barriers and needs of these groups.) If we want to measure the success of our programs it makes sense that we would survey the participants of those programs. When all households in the targeted area receive the survey, they answer yes or no to receiving materials or particpating in any programs. Those with yes answers are directed to answer questions on the materials received and programs they participated in. By far most respondents to the survey received information and or participated in programs. (25% of the households in our target area participated). Seems pretty much self-selecting. Even if we conducted a random survey calling people in the targeted neighborhoods would it still not be self selecting? I get those survey calls and only twice have I ever agreed to answer questions. Both were topics near and dear to my heart - transportation and healthcare. The heathcare one was arduous! Very long, required a lot of thinking and went on so long I was very late leaving home, did not answer a knock at my door and was highly annoyed and frustrated yet I felt it was important to give my 2 cents on this issue. Isn't my agreeing to respond to those two surveys and saying no to all the others a form of self selction? I would appreciate any comments from those more knowledgeable about survey technique on this topic.
Thanks,
Barbara Plummer
Portland Office of Transportation
Question on Surveying
Sign in or Sign up to comment
...
Self selection to participate in the survey is a problem to the extent that there is a relationship between A) an individual's likelihood of choosing to participate in the survey and B) the likely effect your intervention has had on the individual. If you get questionnaires back primarily from early adopters, this group may also be more affected by your marketing than are others; if you assume that your respondents are representative of the whole population, you will overestimate the effect you've had. On the other hand, the early adopters may already be doing what you recommend; they may respond to your survey saying they have not increased their use of public transportation because it's already all that they use. So, if your survey shows that people who saw your marketing increased their use of public transportation by 5%, you do not know if, on average, everyone who saw it increased usage by 2% or 10%. If you agree with the premise that there are folks who are early adopters, some folks on the fence and those who are not at all interested in changing any behaviour during your campaign, then your target audience and your biggest opportunity for success is to target those on the fence - correct? If this is the case, then is not your selected target audience already 'self selecting'? Yes, your selected target audience is self-selecting. Further, if when you initially survey you also ask folks what type of information they would like to have on using various forms of transportation, is this not a form of survey in itself? And also self-selecting? Yes, people select to respond (or not to respond) to tell you what types of information they want. This is not a problem if you are giving that information to them directly. But if you are using those survey responses to design a public information campaign, you do not know whether the public in general wants the information that your survey respondents say they want. If we want to measure the success of our programs it makes sense that we would survey the participants of those programs. When all households in the targeted area receive the survey, they answer yes or no to receiving materials or participating in any programs. Those with yes answers are directed to answer questions on the materials received and programs they participated in. By far most respondents to the survey received information and or participated in programs. (25% of the households in our target area participated). Seems pretty much self-selecting. If you get responses from all participants in your program, then you have a good measure of the program's impact on its participants and self-selection is not an issue. But you cannot generalize from those results to reach any conclusion about how your program would impact people who did not self-select. If you made a general offer for people to participate in your program, and only early adopters chose to participate, a survey of participants would tell you the effectiveness of the program on self-selectors but it doesn't tell you how it would affect fence sitters. Even if we conducted a random survey calling people in the targeted neighbourhoods would it still not be self selecting? I get those survey calls and only twice have I ever agreed to answer questions. Both were topics near and dear to my heart - transportation and healthcare. The healthcare one was arduous! Very long, required a lot of thinking and went on so long I was very late leaving home, did not answer a knock at my door and was highly annoyed and frustrated yet I felt it was important to give my 2 cents on this issue. Isn't my agreeing to respond to those two surveys and saying no to all the others a form of self selection? This is a serious, and increasing, issue in marketing research. To the extent that people who choose to respond to a survey will also tend to answer the survey questions in a particular way, it is inappropriate to generalize from the survey results. We should not assume that the entire population feels the same way you do about transportation and health care.
Herb Koplowitz, Ph.D.
Terra Firma Management Consulting
Tel: 416-324-9240
Fax: 416-972-1354
email: [email protected]
307 Ontario Street Toronto,
Ontario M5A 2V8 Canada