Marketing survey research best practices: evidence and recommendations from a review of JAMS articles

In conducting their topical review of publications in JMR, Huber et al. (2014) show evidence that the incidence of survey work has declined, particularly as new editors more skeptical of the survey method have emerged. They conclude (p. 88)—in looking at the results of their correspondence analysis—that survey research is more of a peripheral than a core topic in marketing. This perspective seems to be more prevalent in JMR than in JM and JAMS, as we note above.

A copy of the coding scheme used is available from the first author.

Several studies used more than one mode.

Traditionally, commercial researchers used phone as their primary collection mode. Today, 60% of commercial studies are conducted online (CASRO 2015), growing at a rate of roughly 8% per year.

This is close to the number of studies in which an explicit sampling frame was employed, which makes sense (i.e., one would not expect a check for non-response bias when a convenience sample is used).

It is interesting to note that Cote and Buckley examined the extent of CMV present in papers published across a variety of disciplines, and found that CMV was lowest for marketing (16%) and highest for the field of education (> 30%). This does not mean, however, that marketers do a consistently good job of accounting for CMV.

In practice, these items need to be conceptually related yet empirically distinct from one another. Using minor variations of the same basic item just to have multiple items does not result in the advantages described here.

In general, the use of PLS (which is usually employed when the measurement model is formative or mixed) was uncommon in our review, so it appears that most studies focused on using reflective measures.

Most of the studies discussing discriminant validity used the approach proposed by Fornell and Larcker (1981). A recent paper by Voorhees et al. (2016) suggests use of two approaches to determining discriminant validity: (1) the Fornell and Larcker test and (2) a new approach proposed by Henseler et al. (2015).

This solution is not a universal panacea. For example, Kammeyer-Mueller et al. (2010) show using simulated data that under some conditions using distinct data sources can distort estimation. Their point, however, is that the researcher must think carefully about this issue and resist using easy one-size-fits-all solutions.

Podsakoff et al. (2003) also mention two other techniques—the correlated uniqueness model and the direct product model—but do not recommend their use. Only very limited use of either technique has been made in marketing, so we do not discuss them further in this paper.

These techniques are described more extensively in Podsakoff et al. (2003), and contrasted to one another. Figure 1 (p. 898) and Table 4 (p. 891) in their paper are particularly helpful in understanding the differences across approaches.

It is unclear why the procedure is called the Harman test, because Harman never proposed the test and it is unlikely that he would be pleased to have his name associated with it. Greene and Organ (1973) are sometimes cited as an early application of the Harman test (they specifically mention “Harman’s test of the single-factor model,” p. 99), but they in turn refer to an article by Brewer et al. (1970), in which Harman’s one-factor test is mentioned. Brewer et al. (1970) argued that before testing the partial correlation between two variables controlling for a third variable, researchers should test whether a single-factor model can account for the correlations between the three variables, and they mentioned that one can use “a simple algebraic solution for extraction of a single factor (Harman 1960: 122).” If measurement error is present, three measures of the same underlying factor will not be perfectly correlated, and if a single-factor model is consistent with the data, there is no need to consider a multi-factor model (which is implied by the use of partial correlations). It is clear that the article by Brewer et al. does not say anything about systematic method variance, and although Greene and Organ talk about an “artifact due to measurement error” (p. 99), they do not specifically mention systematic measurement error. Schriesheim (1979), another early application of Harman’s test, describes a factor analysis of 14 variables, citing Harman as a general factor-analytic reference, and concludes, “no general factor was apparent, suggesting a lack of substantial method variance to confound the interpretation of results” (p. 350). It appears that Schriesheim was the first to conflate Harman and testing for common method variance, although Harman was only cited as background for deciding how many factors to extract. Several years later, Podsakoff and Organ (1986) described Harman’s one-factor test as a post-hoc method to check for the presence of common method variance (pp. 536–537), although they also mention “some problems inherent in its use” (p. 536). In sum, it appears that starting with Schriesheim, the one-factor test was interpreted as a check for the presence of common method variance, although labeling the test Harman’s one-factor test seems entirely unjustified.