Book Review: Important Guidance for Conducting Multinational Social Surveys: A Review of Advances in Comparative Survey Methods: Multinational, Multiregional, and Multicultural Contexts (3MC).

 

This book contains what I think is a comprehensive review of the issues faced by survey researchers who conduct multinational survey research. The book’s subtitle refers to multinational, multiregional, and multicultural contexts (3MC). I will use 3MC throughout the review but for the most part, my comments will refer primarily to multinational survey research. This book is most useful for ANPOR members who conduct cross-country survey research, but all readers can benefit from learning about 3MC methods.

I first learned about the book when I was planning to conduct a multinational survey research project. A colleague, Timothy Johnson, recommended the book.^[1] Before reading this review, I suggest that you look at the description of the book.^[2] The book has 48 chapters on a wide variety of topics related to 3MC research. The complexity and the length of the book (1136 pages) does not allow for a careful review of every chapter.

Therefore, this review focuses on selected chapters and skims over some topics. Similar to other edited volumes, many topics are covered in multiple chapters although the focus differs by chapter. Every chapter has a long bibliography that can be used as a source for more readings on the topics. My review is from the perspective of someone new to 3MC surveys who wanted to learn more about topic. This review does not attempt to be critical of the book but is aimed to help readers best use the book. The book’s editors and chapter authors are internationally known and respected scholars, so the overall high quality of the book’s contents is assumed.

The book may be best used as a reference to learn more about specific topics in 3MC survey research. Many chapters are not relevant to every researcher. The two predominant topics in the book chapters are issues related to question translations and cultural differences in interpreting and responding to survey questions. Those two topics are most appropriate for 3MC research as they are major challenges to equivalence across countries.

My experience with the book was part of my preparation for a survey that would be conducted in the US and South Korea. The book provided enormously helpful information that helped me think through issues I was mildly familiar with but not nearly as knowledgeable as I needed to be to conduct the research. My experience with the book indicates that it can be very useful for new 3MC researchers. Experienced 3MC researchers can use the book to be certain that they are current on issues facing 3MC researchers.

Let me suggest how you might approach the book. To get an overview of the issues and current research on 3MC surveys, start with Chapters 1, 48, and 2. In particular, Chapter 2 is a comprehensive review of the challenges to conducting 3MC surveys. The chapter uses the Total Survey Error paradigm (TSE) to anchor the multiple challenges to survey data quality in 3MC surveys. The chapter demonstrates how the types of survey error are increased when surveys are conducted in multiple countries. Survey researchers typically think about TSE when designing surveys but few would completely understand the complexity of possible accumulated errors when conducting surveys in multiple countries. TSE across multiple surveys can be even larger, and comparison error can be even greater than TSE for each country’s survey. While eliminating all TSE is impossible, the chapter does describe techniques for detecting challenges to TSE. Figure 2 (p. 16) has a very detailed chart of all components of TSE.

The chapter also describes an approach to understanding the 3MC survey process through a strong sociological imagination. To reduce errors, survey researchers need to understand the societal structure, contexts, cultures, and languages of each country. In addition, they need to understand possible differences across ethnicity, class, and status in each country when designing survey procedures. For example, if similar measures of education are used in two countries, the effect of different levels of education on question understanding and response may differ in each country.

Each chapter from 3–48 has a specific focus. Chapters 4 and 5 describe some sampling procedures and are useful for developing a sampling plan for specific types of surveys. A public opinion researcher interested in research methods would find the chapters informative.

Much of the important research relevant to 3MC researchers focuses on two interrelated topics – translations and cultural differences in responses. Chapters 3, 6, 7, 8, and 14–17 cover these issues in detail. The chapters cover the challenges of translation, cultural differences in response patterns, multinational questionnaire design, and documentation of all the procedures needed to take a set of questions from a researcher and turn them into a questionnaire that can be used in multiple countries while limiting TSE.

The chapters mentioned in the previous paragraph are especially helpful for thinking about all the issues that make translations across languages and cultures so difficult. The authors provide examples of their research and the challenges they faced. Some issues were mostly resolved but some problems with translations were not fully resolved. In addition, some were not detected until the field work began.

Chapter 7 examines the challenges created by differences in sensitive topics across countries. In some countries, questions may seem normal but in others, the same questions can be sensitive. Sensitive questions could include questions such as voting behavior which generate socially desirable responses. They can also include questions about topics such as mental health that may seem normal in some cultures but very private in others. Survey researchers know that sensitive questions require different procedures so when questions differ in sensitivity, the procedures need to be different across countries, and the desired level of equivalence is not quite achieved.

The authors also point out some items that might seem simple, such as demographic items, but are not always as easy as expected. Some demographic items might differ because of the racial or ethnic compositions of the countries, making comparisons challenging. Education is assumed to measure some amount of learning, but educational systems and access to education can differ across countries, making it difficult to interpret the impact of education in multiple countries. In Chapter 22, the researchers found that acquiescence response styles were not affected by demographic characteristics but extreme response styles differed by demographic characteristics across countries.

The chapters provide sufficient guidance on translations such that researchers who follow their recommended steps for translations – Translation, Review, Adjudication, Pretesting, and Documentation (TRAPD) – can assume they have done as much as possible to reach equivalence. At the same time, the TRAPD process describes the many steps where errors can be made. In Chapter 1, the editors make a point about the possible problems when using English as the base language for question translations and chapter 6 also includes a section on the problems that can arrive with the use of English as the base language. For 3MC research in Asia, English appears to be the common language that researchers use to communicate and collaborate across countries. The chapters on translations should be read when developing questions based on English as the base language.

Much experimental survey methodology has been conducted in the United States, and the findings may not be similar in other countries and cultures. Chapter 8 is especially important for questionnaire design. In the chapter, the authors describe some 3MC question experiments that tested design effects on responses across multiple countries. The research found surprising differences that became apparent after the experiments were conducted. Some questions had interpretations that may have been the opposite of what was expected. This chapter provides more evidence that to attain equivalence across countries, questions and questionnaire designs must reflect the countries’ population characteristics.

Obviously, this review of the book is not thorough, complete, detailed, or critical. The chapters not reviewed include methods for improving response rate, mode choice, and country-specific procedures for conducting surveys. However, if you have read this far, you have a fair understanding of the value of the book and how it can help researchers reduce the known challenges to 3MC survey research.

Biographical Note

John Kennedy directed the Indiana University Center for Survey Research for 24 years. He also directed the University of Hartford Institute of Social Research for two years and was employed at the US Census Bureau for four years. He earned a PhD in sociology from the Pennsylvania State University. He has been involved in the development of a number of professional journals and was the founding editor of Survey Practice, an e-journal published by the American Association for Public Opinion Research. He has also been actively involved in professional research ethics including chairing Indiana University Social Behavioral IRB for 12 years and he served on two committees that revised the American Sociological Associations Code of Ethics. He teaches a graduate course in Survey Design.

He can be reached at Smith Research Center 123, 2805 E 10^th St, Indiana University, Bloomington, IN 47408, USA or by e-mail at kennedyj@indiana.edu.

Correspondence

All correspondence concerning this article should be addressed to John Kennedy, Smith Research Center 123, 2805 E 10th St., Indiana University, Bloomington, IN 47408, USA or by e-mail at kennedyj@indiana.edu.