Advantages and disadvantages of probability and non- probability sampling pdf

Quota Sampling

Quota sampling is the nonprobability equivalent of stratified random sampling. Like stratified random sampling, the population is first divided into strata. For a fixed sample size n, the nh required in each stratum for proportional stratification is determined. A quota is set for each stratum of nh observations, and the researcher continues sampling until the quota for each stratum is filled. For instance, if a population is known to be 70% men and 30% women, a survey of 100 people using quota sampling would ensure that 70 of the interviews were with men and 30 were with women. Subjects for the interviews are selected based on convenience and the judgement of the interviewer.

Quota sampling is generally less desirable than stratified random sampling for two reasons. First, because the selection of sampling units is non-random, the usual sampling error formulas (such as the estimation of variances on our estimated parameters) cannot be applied to the results of quota samples with any confidence. Second, because the observations included in a quota sample are selected nonrandomly, this may introduce bias into the sample that a random sample would not. That is, while a quota sample will be representative of the population on the variables used to define the strata, it may not be on other variables. Randomized samples will most likely be more representative on uncontrolled factors than an equivalent quota sample.

Sampling Methods

Sampling, the process of choosing the subset of people who will represent your population of users, is a complex topic that is described only briefly here. The two major types of sampling are probability and nonprobability (Bailey, 1994; Levy & Lemeshow, 1999; Robson, 2002). In probability sampling, the probability of selection of each participant is known. In nonprobability sampling, the interviewer does not know the probability that a person will be chosen from the population. Probability sampling is expensive and time-consuming and may not even be possible because there is no complete list of everyone in a population. For many interview studies, you are likely to be dealing with nonprobability samples where you can use one or a combination of the following approaches (Bailey, 1994; Robson, 2002):

Quota sampling. You try to obtain participants in relative proportion to their presence in the population. You might, for example, try to get participants in proportion to a distribution of age ranges.

Dimensional sampling. You try to include participants who fit the critical dimensions of your study (time spent as an architect or engineer, time using a particular product, experience with a set of software tools).

Convenience sampling. You choose anyone who meets some basic screening criteria. Many samples in UCD are convenience samples that can be biased in subtle ways. For example, the easiest people to find might be users from favorite companies that are generally evangelists of your product. You might end up with a “positivity bias” if you use participants from your favorite companies.

Purposive sampling. You choose people by interest, qualifications, or typicality (they fit a general profile of the types of participants who would be typical users of a product). Samples that meet the specific goals of the study are sought out. For example, if you are trying to understand how experts in a particular field work on complex projects, you might seek out the “best of the best” and use them for your interviews.

Snowball sampling. You identify one good participant (based on your user profile or persona) who is then asked to name other potential participants, and so on. Snowball sample is useful when there is some difficulty in identifying members of a population. For example, if you are looking for cosmologists who use complex visualization tools, you might find one and then ask him or her about any friends or colleagues in the field who might want to be interviewed.

Extreme samples. You want people who are nontraditional or who have some exceptional knowledge that will provide an extreme or out-of-the-box perspective.

Extreme Input Can Be Useful

The use of “extremes” in user research can provide inspiration (Jansen et al., 2013) and help you understand the limits of a system. In addition to extreme samples of users, you can also explore extreme data sets that are large and dirty (something that usability research often ignores in small-scale testing) and extreme scenarios that highlight risks and rare, but critical, usage patterns.

Heterogeneous samples. You select the widest range of people possible on the dimensions of greatest interest (e.g., you might choose people from many industries, countries, genders, and experience ranges).

For any type of user research, it is important to be explicit about your sampling method and its limitations and biases.

Web-Survey Typology

Web-Survey Formats

Examining the different Web-survey formats is also enlightening. While Web surveys involve many different topics, there are only two main formats for presenting a Web survey: interactively or passively. These two formats are aesthetically different and have distinct advantages and disadvantages.

Figure 1 contains an example of a typical interactive survey. As illustrated in this figure, interactive surveys are presented screen-by-screen. By clicking on a button, like the “to continue” button in Fig. 1, respondents can go to a new question on a new screen. This allows the data from the question to immediately be electronically transmitted to the surveyor ensuring that data from partially completed surveys is maintained. However, this format may make it difficult for respondents to review and correct their answers. Interactive surveys can also automatically skip questions which are determined to be irrelevant to the respondents based on how they answer previous questions. For example, if respondents indicate that they do not have children, all subsequent questions related to children can be automatically skipped. A drawback of this design is that respondents do not see the survey in its entirety and therefore cannot easily determine its length. To compensate, a progress indicator can be used to inform the respondent how much of the survey remains to be completed. Another difficulty with interactive surveys is that they may require special software, such as Java, potentially making it difficult for respondents with older, less powerful computers and Web browsers to respond.

Figure 1. Example of an interactive style Web survey.

Passive survey designs involve presenting the entire survey at once. Figure 2 displays the first part of a passive survey. The bar on the right-hand side of this figure indicates that respondents can scroll down on the page to view the rest of the survey. The data from passive surveys is transmitted once the respondent has completed all questions and clicked a submit button. An advantage of passive surveys is that respondents can easily browse through questions and review their responses before submitting. These types of Web surveys are also easy to produce and easy to access so technical difficulties are less likely.

Figure 2. Example of a passive style Web survey.

In addition to these two basic formats, the appearance of a Web-based survey also depends on how question response options are presented. There are four distinctly different ways to present response options: drop-down boxes, radio dials, check boxes, and open-ended boxes. Figure 1 contains an example of radio dials while Fig. 2 illustrates both open-ended boxes and drop-down boxes. Drop-down boxes appear in the questionnaire as a box with a downward pointing arrow—clicking on this arrow displays the list of response options and allows respondents to select from the list provided. Drop-down boxes are convenient for long lists of items since the response options are hidden. Radio dials, on the other hand, display all the responses options and require the respondent to click in the circle corresponding to their choice. Both drop-down boxes and radio dials are usually used when only one response must be selected from among the options provided. When respondents are allowed to select more than one option from a list, check boxes are the appropriate question format—respondents click on all the boxes that correspond to their answers. Finally, open-ended boxes allow respondents to type their responses in the space provided. As with other survey formats, open-ended question boxes can be useful when the associated question does not have responses that can be conveniently listed.

Probability Versus Nonprobability Sampling

As with any other research method, you may want or need to outsource the recruiting for your study. In the case of surveys, you would use a survey panel to collect data. These are individuals who are recruited by a vendor based on the desired user characteristics to complete your survey. Perhaps you are not interested in existing customers and therefore have no way to contact your desired population. There are many survey vendors available (e.g., SSI, YouGov, GfK, Gallup), and they vary in cost, speed, and access to respondents. The key thing you need to know is whether they are using probability or nonprobability sampling. Probability sampling might happen via a customer list (if your population is composed of those customers), intercept surveys (e.g., randomly selected individuals who visit your site invited via a link or pop-up), random digit dialing (RDD), and address-based sampling (e.g., addresses are selected at random from your region of interest). In this case, the vendor would not have a panel ready to pull from but would need to recruit fresh for each survey. Nonprobability sampling involves panels of respondents owned by the vendor that are typically incentivized per survey completed. The same respondents often sign up for multiple-survey or market research panels. The vendors will weight the data to try to make the sample look more like your desired population. For example, if your desired population of business travelers is composed of 70% economy, 20% business, and 10% first-class flyers in the real world but the sample from the panel has only 5% business and 1% first-class flyers, the vendor will multiply the responses to increase the representation of the business and first-class flyers while decreasing the representation of the economy flyers. Unfortunately, this does not actually improve the accuracy of the results when compared with those from a probability-based sample (Callegaro et al., 2014). In addition, nonprobability-based samples are more interested in taking surveys and are experts at it.

Suggested Resources for Additional Reading

To learn more about probability sampling versus nonprobability sampling, check out the publications below. The authors have deeply researched this topic and cite years’ worth of data.

Blair, E. A., & Blair, J. E. (2015). Applied survey sampling. Los Angeles: Sage.

Callegaro, M., Baker, R. P., Bethlehem, J., Göritz, A. S., Krosnick, J. A., & Lavrakas, P. J. (Eds.). (2014). Online panel research: A data quality perspective. John Wiley & Sons.

Online Services

There are many vendors available to conduct studies on your online product. A simple web search for “online usability testing” will highlight those vendors. They can conduct research methods like surveys, card sorting, and usability evaluations online. Most provide their own panel of participants for your study. These are most likely nonprobability-based samples and may be rife with professional participants (refer to Chapter 10, “Probability Versus Nonprobability Sampling” section, page 271). Some allow you to conduct the research yourself using their tools and your own participants (e.g., from your customer database), so be sure to ask.

If they are recruiting, you must indicate your desired user profile and provide a script for the participants to follow. They also may require you to provide a link to your site/product, upload a prototype, or provide other content (e.g., names of your features in your product for card sorting). The vendor then e-mails study invitations to members of their panel that meet your user profile. Within hours, you can get dozens of completed responses, sometimes with videos of the participants’ think-aloud commentary. Be aware that, although participants may be under an NDA, there is a greater risk of confidential product details being leaked. If this is not a concern for you, the online data collection is an excellent way to get feedback quickly from a large sample across the country.

2.7.2 Sampling techniques

There are two main types of samples: probability and nonprobability samples. Nonprobability samples are cases where you do not know of every unique member of the population in question (i.e., the entire user group in our case). Another way to describe it is when every member of the population does not have an equal chance of being invited to participate. Probability samples are when you do know of every unique member of the population and therefore each has a probabilistic chance of being invited for the sample (e.g., 100 users of a product, each has a 1/100 chance of being invited). Here's a taste of a couple of common nonprobability sampling techniques.

Convenience sampling. This is the most common nonprobability sample. You might send invitations to people in your company, students from a school you're affiliated with, the city you live in, and so on. It's referred to as “convenience” sampling because unless the targeted user group is truly limited to those people, it is likely introducing some bias to recruit just a particular slice of the population.

Snowball sampling. This is a type of convenience sampling in which those participants invited invite other participants and so on to create a pyramid effect.

With probability sampling, you can choose a more scientific way to sample because you know the number and characteristics of the true population. For instance, a particular product has a small contingent of users. Some of the common techniques include the following.

Simple random sampling. This is a known population of users from which you take a random sample via some means, such as a program or application, an Excel formula, or a simple “pick out of a hat” lottery.

Stratified sampling. You assign everyone in the population to a specific (but meaningful) group and then take another probability sample within each category. The users are therefore chosen randomly, but there is a representation from each “strata.” Note that the nonprobability sampling method that correlates to stratified sampling is called quota sampling. It means that despite not knowing the true population, you divide the users you know about into groups and still try to get some representation from that group (usually via convenience sampling).

Systematic random sampling (also known as just systematic sampling). The idea here is that you list all of the users in no particular order (in fact, it should not be in any logical order) and then pick every Nth user. You define what N is; for instance, if there are 1000 users total and the goal is to invite 500 to participate, you'd simply take every second person from the list and invite them.

Multistage sampling. This takes different samples in different ways to eliminate bias. For example, you may do a random sample, then stratified, and so on.

Chances are that the number and characteristics of users for a product you're testing may not be entirely known, especially for Web sites, and still further for Web sites that don't require users to register or create an account (and thus can't be tracked). If using a generic participant panel, it's likely a convenience sample and is not necessary to worry about probability sampling techniques. However, in cases where you're providing a recruiting service with part of a customer or user list from which to invite people, you may want to use one of the probability sampling techniques discussed. For example, if there are 10,000 customers on a customer list and it needs to be whittled down to a 1000 person sample for study invitations, you might use a simple random or stratified sampling method to get a representative sample of customers.

If recruiting by posting on the Web via message boards, forums, or in newspaper ads, rather than using email invitations, it's likely a snowball sample as people might forward on the information to others that they know. Just be aware that this comes with a self-selection bias. This type of bias is where the participants who choose to participate may have certain outstanding and overrepresentative characteristics, such as being the ones who particularly love and/or hate the Web site enough to participate. One way to minimize this bias is to use a phased launching strategy (breaking up the study launch into multiple groups).

TIP

Check out http://stattrek.com/Reading/Sampling.aspx for references to some good books on sampling methods and the biases associated with them.

Suggested methodology and research design for adapting the model to local contexts

As catalysts in providing lifelong learning in information literacy and cultural heritage, museums, archives and libraries need to converge in the adaptation of the model for suitable and effective application to their environments and countries, and to the needs of their users/visitors. Where possible, each of these institutions should be represented, but where there are local limitations – such as only libraries and museums existing in a town or city, but no archives, or only archives and museums, but no libraries – the convergence could include two of the three institutional types. The ideal is for all three to converge, since each domain has a unique perspective that adds value to the learning experience of cultural heritage and information literacy.

It is recommended that working groups are convened, with representatives from museums, archives and libraries who have expertise in working with users and providing instruction to their users, and also who are knowledgeable about their collections and the resources available. The working group should appoint a convenor and chair who will coordinate the overall process, and different tasks can be allocated to sub-groups which focus on a particular aspect, before bringing it all together in one cohesive plan.

Intrinsic components of a research design and methodology to adapt the model to local contexts include:

Formulation of a description of what data is required, and research questions. The working group should identify what particular questions need to be asked in their context, and clearly present these, with a research plan to explore the questions further using the various social science research methodologies available.

Literature review and environmental scan. The overview provided in this book can be referred to, but it would be necessary to expand the literature review and environmental scan further in the context of the particular country concerned for a more country-specific contextual focus of existing research and initiatives. As Babbie and Mouton noted, literature reviews should not be too extensive, not mentioning every single study in the field, but rather should highlight the main trends, arguments and disagreements (Babbie and Mouton, 2009: 566).

Sampling decisions. Before further research can continue, the working group would need to make decisions as to what type of sampling method should be used for surveys of their user and visitor populations at each of the participating sites. There is extensive literature available on sampling methods and techniques, but the most well-known, as described by Uys and Puttergill (2005), are probability sampling where the sample needs to adequately represent variations in the population (ibid.: 109) and non-probability sampling where it is not possible to obtain a representative sample from existing records, thus sampling error cannot be calculated (ibid.: 112). Uys and Puttergill highlight the fact that sampling can be a particular challenge in developing countries, and especially in rural areas where there are often no central records of citizens and users (ibid.: 115). To overcome this, they note that creativity and enterprise are required of the researcher.

Survey questionnaires of existing users. Survey questionnaires can be helpful instruments to obtain more information on the user profiles of visitors to museums, archives and libraries. It would also be useful to survey potential instructors among staff in each of the institutions (curators, archivists and librarians) in order to determine any skills gaps that need to be addressed before staff themselves can provide training. The questionnaires should be designed to provide data on variables such as ethnic or cultural background, gender, age, education level, religion and home language. They should be designed to be as unobtrusive as possible, although this is never completely achievable. Bookstein referred to the two factors which reduce the reliability of data from questionnaires, namely failure of the respondent to understand the question(s) and response decisions which can be influenced by a number of factors (Bookstein, 1985: 25–7). For this reason, qualitative investigation also needs to be undertaken. An example of a questionnaire which can be adapted and modified to be more relevant to the data required in the given context is included in Appendix 1.

Follow-up interviews. For more qualitative data, and as a reliability check against possible flaws in the questionnaire process that inevitably occur due to the limitations of applying only one research instrument, it is recommended that follow-up interviews take place with selected respondents, to explore responses to questions in more depth. It is important that a schedule is drawn up within a time frame, and that interviewers are identified and scheduled to conduct interviews over a specified period of time. An example of an interview schedule is included in Appendix 2.

2.1 Ad Hoc Panel Method

The use of ad hoc panels or groups is a widely-employed organizational method for consensus decision-making. Such panels are composed of experts convened in a face-to-face meeting to make a specific decision. The method can be either unstructured or structured. In an unstructured procedure, a nonrepresentative (or nonprobability) convenience sample of experts is gathered for one session to address a single, global question such as, ‘Should the swine flu vaccine be produced and distributed?’ Such convenience panels are often neither representative of the range of beliefs nor independent of the agency or organization requesting the consensus decision (Neustadt and Fineberg 1978).

In a structured procedure, an effort is made to create a panel containing members representing all the views on the topic. This involves an extended search and nomination process, and often open public comment sessions that include the press as well. Rather than one global issue, the decision is also broken down into a set of subsidiary questions or issues with a procedure for addressing them. Typically, subgroups are formed to address these issues. They can either be members of the panel or another group of experts who are not part of the group and thus not subject to its potential biases (e.g., groupthink, the dominance of prestigious panel members, etc.).

The swine flu decision, as reviewed at government request (Neustadt and Fineberg 1978), illustrates both the strengths and weaknesses of this method. In early 1976 four cases of swine flu were reported among US Army recruits at Fort Dix. The Center for Disease Control (CDC) convened an ad hoc panel to determine if a vaccine should be produced and distributed to prevent a pandemic that fall. The panel was composed of members of four Federal and State agencies plus CDC staff. A month later this same group reconvened in a joint meeting with a standing CDC advisory committee appointed and staffed by the Director. Only one of the acknowledged elder statesmen in the field of virology participated in this meeting, and he strongly believed that the country was approaching the end of an 11-year cycle when a new pandemic would occur. At the end of a day's discussion a consensus emerged that ‘the possibility of a pandemic existed’ and that everyone was at risk. Therefore, the panel recommended that enough vaccine be produced to inoculate the entire US population.

A similar biased consensus unanimously endorsing a swine flu vaccine program also resulted from a third panel appointed by President Ford one month later. He wanted the advice of a ‘representative group’ of experts, but also had to rely largely on the CDC to select them. Although some famous scientists were added, there were still notable exceptions. Most of the panelists felt that the decision had already been ‘programmed.’ President Ford publicly announced the Federal program later that day. A series of disasters then followed including the deaths of three high-risk elderly people in Pittsburgh early that fall shortly after being vaccinated, and soon thereafter numerous reports of vaccine-related paralysis. In December, upon the recommendation of CDC, President Ford suspended the program.

A month later an outbreak of Victoria flu erupted in a Florida nursing home. The CDC advisory panel recommended ‘limited resumption of the swine flu program’ so that the Victoria vaccine that was included in the bivalent doses would be available. The new US Secretary of Health, Education, and Welfare decided to convene a new, fourth ‘advisory group’ that was more representative and independent than the first three—two CDC and President Ford's—panels. The ad hoc group was to be chaired by two of the ‘nation's most distinguished scientists’ who were not part of the ‘flu establishment’ and the meeting was open to the public and the press. The ‘improvised’ consensus procedure led to a recommendation to lift the ban on the bivalent vaccine for ‘high-risk’ groups such as the elderly. It was considered ‘a great success.’

What are the advantages and disadvantages of probability and non

What are the advantages and disadvantages of probability sampling techniques?

What is the advantage of following a probability and non

What are some advantages of probability sampling?