Total quality management system (TQM) has been widely studied as a strategy capable of providing a competitive advantage for companies. One of the main focuses in quality management is the improvement of the system. Attention to processes, products, and information technology is a form of quality improvement. However, attention to these aspects may not be a key variable in quality improvement. The human resources (HR) factor takes into account improving the organization’s quality (Bowen & Lawler, 1992). The human factor received greater attention, recognizing that teamwork, cooperation, and motivation cannot be underestimated (Cowling & Newman, 1995). Promoting TQM practices may not necessarily improve performance, but it is ultimately people who make quality happen (Prajogo & Cooper, 2010).

Human resource practices are essential in the implementation of this strategic initiative. Ironically, many HR professionals today are involved in a tug-of-war with their company’s quality efforts (partly because HRM and TQM are often seen as peripheral and separate functions). In the past, TQM implementation could be led by outside consultants who (without staff involvement) developed training programs and organizational changes that are indeed the scope of the HRM department. That change is often problematic (Hart & Schlesinger, 1991). However, at present, the responsibility of the human resource management (HRM) department is to ensure quality throughout the organization. Failure of HRM to communicate TQM messages can result in ineffective TQM initiatives, as it is the responsibility of HRM to educate members of the organization (Sarwar et al., 2016; Shahraki et al., 2011).

TQM has a special relationship with HRM in terms of systems for continuous improvement. HRM is practiced as part of quality planning at the enterprise level and is thus directed at the needs of internal consumers (Izvercian et al., 2014). TQM requires that job designs provide long-term benefits for multiple beneficiaries (Simmons et al., 1995). Therefore, the challenge is to improve the quality of the system and improve employee performance. TQM-oriented HRM has become one of the major innovations of the last few decades (Izvercian et al., 2014).

However, some literature on TQM and HRM does not offer an integrated framework with empirical support that identifies the role of TQM-oriented HRM practices in organizational performance improvement and strategic capability development (Gutierrez-Gutierrez et al., 2018). Another conflict is over which human resource practices are most useful in the context of TQM. For example, some previous studies claim that goal setting and incentive compensation systems do not work in companies that practice TQM (Gutierrez-Gutierrez et al., 2018; Shahraki et al., 2011), while others claim that incentive compensation is an integral part of the implementation process (Chandler & Mcevoy, 2000; Hwang, 2015; Ooi et al., 2012; Patro, 2013; Yue et al., 2011)

Our study will focus on this issue but with significant differences from previous studies. First, most existing research is theoretical or based on case studies. Our study will advance research into HRM and TQM using a methodology that will allow generalizability. This study is needed to fill the gap in the literature by empirically analyzing the relationship between HRM and TQM practices and organizational performance. Our study will incorporate the previous framework within which the HRM model for TQM will be tested.

Literature Review

TQM-Oriented HRM

TQM is a holistic management philosophy that strives for continuous improvement in all organizational functions (Gutierrez-Gutierrez et al., 2018). It can also be explained as the relationship between the system and the implementation of quality, closely related to competitiveness and performance. In the TQM literature, two types of dimensions are usually mentioned – one is more technical, and the other is more tacit or intangible. Examples of technical elements of TQM might include statistical process control and Ishikawa’s troubleshooting tools. On the other hand, intangible elements can come from leadership, organizational skills and culture, executive commitment, open organization, participatory team dynamics, and empowerment (Jiménez-Jiménez & Martínez-Costa, 2009).

Concerning HR, TQM and HRM are essential aspects of the business environment due to the considerable impact on individual and organizational performance. In implementing the TQM process in a company, the HR department plays a significant role: namely developing and communicating the TQM vision, preparing organizational details for the implementation of TQM procedures, actual implementation, and providing the necessary support to maintain enthusiasm about TQM (Izvercian et al., 2014).

Previous literature has highlighted the important role of HRM in implementing TQM systems in an organization (Jiménez-Jiménez & Martínez-Costa, 2009). Some literature mentions this using different names, such as people-related TQM (Prajogo & Cooper, 2010), quality-oriented HRM system (Jiménez-Jiménez & Martínez-Costa, 2009), TQM-oriented HRM practice (Bowen & Lawler, 1992), TQM-based human resources management practice (Perdomo-Ortiz et al., 2009), but the emphasis is the same. This study uses the term TQM-oriented HRM. TQM-oriented HRM is an integrated organizational effort designed to improve quality at every level by engaging a set of practices guided by the principles of human development and human relations. TQM-oriented HRM practices can make organizations more sensitive to variations in the environment, become more strategically flexible, better adapt to change, improve communication and information exchange, assign responsibilities to employees, and facilitate autonomy for experimentation, all of which enhance innovative processes (Gutierrez-Gutierrez et al., 2018). Table 1 describes the evolution of TQM in the HR paradigm.

Table 1. The Evolution of a Total Quality HR Paradigm
Human resource characteristics Traditional paradigm Total quality paradigm
Training Job-related skills
Function, technique
Productivity
Broad range of skills
Cross-function
Diagnostic, problem solving Productivity, quality
Empowerment Efficiency
Productivity
Standard procedures
Narrow span of control
Specific job descriptions
Quality
Customization
Innovation
Wide span of control
Autonomous work teams Empowerment
Teamwork Individual goals
Functional teams
Team goals
Emphasize quality and service
Cross-functional teams
Compensation Competition for individual merit increases and benefits Team/group-based rewards
Financial rewards
Financial
and nonfinancial recognition

Note. The information in this Table was compiled from the following sources: Blackburn & Rosen, 1993; Crosby, 1979; Deming, 1986; Perdomo-Ortiz et al., 2009.

Research conducted by Palo and Padhi (2005) investigated the relationship between TQM and HRM effectiveness and found a positive relationship between the two. The study found that the TQM approach emphasizes planned change through HRM. HRM partnership with other departments is necessary to support TQM progress. Because TQM does not limit itself to any functional specifications but spreads throughout its business (Palo & Padhi, 2005), without team direction and a more open approach to managing and engaging staff, TQM would be meaningless (Wilkinson et al., 1991). Some literature has also shown a relationship between HRM practices and TQM. There is agreement that implementing TQM requires changes in HRM policies and practices to conform with TQM principles (Boselie & Van Der Wiele, 2002; Prajogo & Cooper, 2010).

TQM-oriented HRM involves a set of practices guided by the principles of human development and human relations. The evolution of the TQM-oriented HRM model can be explained from two perspectives: (1) a conceptual perspective based on the original teachings of TQM originators (Deming and Crosby); and (2) an operational perspective based on the dimensions and measurement instrument items of the TQM-oriented HRM model. Based on a review of previous research, this study presents a conceptual model of TQM-oriented HRM with training, empowerment, teamwork, and compensation as measurement dimensions (Blackburn & Rosen, 1993; Boselie & Van Der Wiele, 2002; Jiménez-Jiménez & Martínez-Costa, 2009; Ooi et al., 2013; Perdomo-Ortiz et al., 2009; Prajogo & Cooper, 2010; Usrof & Elmorsey, 2016).

This measurement also refers to the TQM originators, namely Deming and Crosby. Deming (1986) articulated the importance of people in managing quality, a 14-point management philosophy on quality needs proposed by Deming and Crosby, more than half of which relate to specific practices related to people and organizations, including institutionalizing on-the-job training, increasing employee empowerment, teamwork by eliminating sectoral egos, and compensation. Similarly, Crosby (1979), in 14 steps of continuous improvement, emphasizes people factors, including training, communication, and reward systems for quality. In essence, this argument suggests that these practices are important in enabling employees to develop and utilize their full potential to achieve organizational goals (Prajogo & Cooper, 2010).

Training and development have been recognized as essential for TQM implementation. One of Deming’s 14 points is that all employees should be trained in quality improvement techniques (Deming, 1986). Companies committed to TQM invest in training. TQM training is not a single effort but must be continuous, planned systematically and objectively, and with a process-oriented methodology. Employee training is fundamental to many TQM programs, such as adopting new quality concepts, setting up and practicing customer satisfaction systems, using statistical quality control, or changing culture or quality control circles. In addition, employees need three primary areas of training: TQM principles, use of TQM tools, and problem-solving techniques (Jiménez-Jiménez & Martínez-Costa, 2009; Vermeulen & Crous, 2000).

Training plays an essential role in raising awareness, developing a supportive culture, building quality skills, encouraging team building, and commitment to quality policies and strategies (Gutierrez-Gutierrez et al., 2018; Palo & Padhi, 2005). TQM requires a change in the skills that employees and managers need. In the context of more special teams and functional integration, employees must be equipped with a broad skill base spanning several different jobs (Simmons et al., 1995). In addition, the training structure should be top-down, starting with the top team and flowing down the organization. This is necessary to demonstrate management commitment and to ensure managers fully understand the principles and methods of TQM (Vermeulen & Crous, 2000).

Extensive training may provide employees with the skills needed to implement productivity and quality initiatives effectively. Therefore, training must be organized around an organized, strategic drive to be most effective, which is the case in TQM implementation (Chandler & Mcevoy, 2000).

Empowerment provides an environment where workers can unleash, develop, and utilize skills and knowledge to their full potential for the organization’s well-being and themselves (Prajogo & Cooper, 2010). TQM should promote the empowerment of front-line employees, give them more responsibility and information, and thus undermine the traditional role of managers in implementing and monitoring top management instructions. So, for quality initiatives to be successful, all employees must have adequate participation and communication in quality decisions, as they can improve organizational processes. In addition, job descriptions require customer orientation, innovation, and continuous improvement, which allows for employee autonomy, responsibility, and flexibility (Jiménez-Jiménez & Martínez-Costa, 2009).

Empowerment for employees will motivate employees, facilitate decentralization, and generate new ideas and opportunities for innovation (Gutierrez-Gutierrez et al., 2018). Most people still assume that total quality is the responsibility of quality specialists. However, this belief is flawed and detrimental to realizing total quality. Therefore, it is crucial to involve and empower employees (Palo & Padhi, 2005).

Training gives employees the skills and confidence to achieve high-quality results from their work. Training also reflects the organization’s commitment to helping employees develop their skills, which will allow them to advance in their career development (Jaleel & Ahamed, 2018; Prajogo & Cooper, 2010). In the context of TQM, teamwork is an important outcome and condition for continuous improvement. It facilitates collaborative efforts to solve quality problems, places overall responsibility for quality with the team while reducing the potential for individual blame, allows for better information sharing within workgroups, and facilitates greater collaboration. (Jiménez-Jiménez & Martínez-Costa, 2009). Teamwork will develop employee ideas, encourage employees to exchange knowledge, and generate different orientations, thereby creating wider choices. In addition, teamwork breaks down barriers to improve communication and information exchange, both of which contribute to innovation (Gutierrez-Gutierrez et al., 2018).

It is evident from research conducted by Palo and Padhi (2005) that pioneering organizations emphasize the value of people working together in teams. This is not surprising as many theories and studies show that people are motivated and perform better when part of a team. Teams can also achieve more through concerted effort and problem-solving (Palo & Padhi, 2005).

Compensation is comprised of all types of payments, either direct or indirect, in the form of material/money and awards given by the company as a reward to its workers/laborers through employment relationships within an organization (Riyadi, 2015). In the context of the relationship with TQM, alignment between reward systems and TQM is required. Consequently, rewards should encourage collaboration, employee engagement, and teamwork, not emphasizing individual-oriented compensation (Obeidat et al., 2018). This incentive system may be based on quality criteria, emphasize equality among employees, and be oriented towards improvement targets (Jiménez-Jiménez & Martínez-Costa, 2009). It is worth noting that there is a solid direct conflict between these traditional practices and TQM’s emphasis on collective responsibility, horizontal relationships, and horizontal learning. The first conflict is in the area of job description and salary based on job value. This approach suggests that an individual’s accountability and responsibility is limited to the tasks they perform regularly. A job description defines what a person should and cannot be responsible for. In contrast, much of the TQM literature emphasizes flexibility and responsibility for the entire production or service process.

Individual incentives can hinder cooperation, teamwork, and promote internal discord (Ooi et al., 2012). This, of course, contradicts quality management. Alignment between reward systems and TQM is required. In addition, TQM requires the elimination of output-related payment systems and a greater emphasis on personal development and training. Finally, compensation should include both financial and non-financial rewards, promoting ongoing recognition (Shahraki et al., 2011).

In conclusion, TQM orientation requires a set of internal HRM practices, which are more compatible with TQM (see Table 2).

Table 2. HRM Practices Congruent with a Quality Strategy
Practice Characteristics
Training Extensive and continuous at all levels
Long-term orientation
Polyvalence and broad variety of competences
Group orientation
Job rotation
Career path definition
Horizontal movement based
Qualitative criteria promotion (changes openness, flexibility, innovation capacity, etc.)
Employee career path feedback
Empowerment Formal planning of job task
Broad job descriptions
Flexibility
Autonomy
High participation
High communication
Teamwork Teamwork job orientation
High team autonomy
High responsibility for their tasks
Cross-functional teams
Compensation Employee competences, abilities, and flexibility determines salary
Use of incentives
Group orientation
Equilibrium between financial and non-financial compensation

Note. The information in this Table was compiled from the following sources: Blackburn & Rosen, 1993; Jiménez-Jiménez & Martínez-Costa, 2009; Perdomo-Ortiz et al., 2009.

TQM-Oriented HRM and Employee Performance

Employee performance results from a person’s work in carrying out the tasks assigned to them and how much the employee contributes to the organization (Rizal et al., 2014). Mangkunegara and Miftahuddin’s research explains that when employees do quality work, it will have a relationship with good performance as well. Performance variable indicators refer to research conducted by Mangkunegara and Miftahuddin (Mangkunegara & Miftahuddin, 2016). In terms of TQM-oriented HRM, performance improvements have been intensified by the struggle to manage quality in the workplace. The TQM-oriented HRM approach continuously seeks to improve performance at every level of operations and in every functional area of the organization, using all available human and capital resources (Sarwar et al., 2016). TQM-oriented HRM can lead to better organizational results (Hwang, 2015). Alignment of HR and quality policies, such as creating and communicating a TQM vision, prepares organizations and employees for TQM implementation and generates quality awareness among employees at all levels, functions, departments, and ultimately contributes to the improvement of employee performance (Patro, 2013; Shahraki et al., 2011).

Increased competition, international trade, and globalization have led companies to focus on the concept of quality in recent decades. The increasing role and importance of quality in business have led many organizations to conclude that effective quality management can enhance their competitive capabilities and provide a strategic advantage in the marketplace. Organizations that wish to achieve their institutional vision, mission, and goals, and effectively implement quality management initiatives need to adopt new management philosophies and strategies to enable their leadership and employees to improve organizational command, control, and communication. More recently, the goal of achieving quality covers all kinds of industries (Akdere, 2009).

Previous research has shown that TQM-oriented HRM affects employee performance and even improves business performance (Patro, 2013; Pramuka & Adawiyah, 2012). The TQM-oriented HRM approach is important because its implementation has a positive effect on the organization and its employees in terms of increased customer satisfaction, increased employee job satisfaction and job-related attitudes, decreased production costs and higher productivity, increased knowledge management efforts, and better employee performance (Hwang, 2015; Patro, 2013; Yue et al., 2011). This argument is in line with Prajogo and Cooper’s view that considers TQM-oriented HRM as a vehicle for creating an organizational climate that will provide a supportive environment for high-performance workplaces. This can provide strategic focus and objectives for HRM policies, namely to achieve high-quality performance (Izvercian et al., 2014; Prajogo & Cooper, 2010).

Previous research has shown a positive relationship between TQM-oriented HRM and employee performance (Aeknarajindawat et al., 2020; Hwang, 2015; Jiménez-Jiménez & Martínez-Costa, 2009; Lam & Schaubroeck, 1999; Ooi et al., 2012; Patro, 2013; Perdomo-Ortiz et al., 2009; Prajogo & Cooper, 2010; Sihotang & Zebedeus, 2013; Yue et al., 2011).

H1: TQM-oriented HRM can be operationalized as a second-order latent construct.

H2: TQM-oriented HRM has a positive effect on employee performance.

Method

The research was conducted in Jakarta, which is Indonesia’s capital city and the world’s second largest urban agglomeration. The research was conducted from May 2021-August 2021. This study used a survey as quantitative approach and structural equation modeling (SEM).

Survey data was collected by distributing questionnaires to private employees of the manufacturing sector in Jakarta using Google Forms due to restrictions issued by the Indonesian government due to the COVID-19 pandemic, which prevented in-person interviews. Manufacturing sector employees are often managed using TQM, ensuring that they make good quality products. This study examined the relationship between HRM and TQM on employee performance. These results were then analyzed using the program Lisrel 8.5. The sampling technique used was non-probability purposive sampling.

As required for testing the hypotheses, a hierarchical, second-order factor model was used to represent the construct of TQM-oriented HRM. Our hierarchical model represented the first-order factors as a set of latent dimensions that reflected a second-order factor. The estimation used is maximum likelihood and has a multivariate normal data distribution, so a sample size of 100-200 is good, because if it is less than 100, this test tends to accept H0 (Hair et al., 2014; Yamin & Kurniawan, 2009). The number of participants in this study was 100. To measure the items, a five-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree was used (Sekaran & Bougie, 2016). Based on the suggestions of Hair et al. (2014) and Yamin and Kurniawan (2009), H0 would be rejected if the t-value was greater than 1.96 or not rejected if the t-value is less than 1.96.

Results

Respondents

Table 3. Profile of Respondents
Profile Frequency Percent
Sex Male 63 63
Female 37 37
Age <26 5 5
26–30 28 28
31–35 22 22
36–40 11 11
>41 34 34
Level of education completed High school 3 3
Associate’s degree 6 6
Bachelor’s degree 60 60
Master’s degree 29 29
Doctorate 2 2
Marital status Unmarried 26 26
Married 72 72
Divorced 2 2

A total of 100 questionnaires were collected. Information about the respondents is shown in Table 3. Regarding respondents’ education, a majority had a bachelor’s degree (60%). In addition, a majority (72%) of respondents were married. Those who work in the manufacturing industry in the Jakarta area are mostly in the age group range of 26-30 and 31-35. The age distribution in our survey matches the age distribution of those working in the manufacturing industry in Indonesia (Kemenperin, 2019).

The Measurement Model

Confirmatory factor analyses (CFA) were first conducted to evaluate the distinctiveness of the critical variables before testing the main hypotheses. Furthermore, the goodness of fit analysis was carried out to estimate the initial measurement model (CFA) and evaluate the measurement model (Hair et al., 2014; Latan, 2012).

The results of the goodness of fit test shown in Table 4 indicate that this model has a good fit with ꭓ2 = 231.69 (df = 99); the p value for RMSEA <.05 (cut-off value >.05, table = .00) is a close fit. Parsimonious Normed Fit Index (PNFI) > .6 (cut-off value > .6, table = 0.73) is a good fit. Parsimonious Normed Fit Index (PNFI) > .9 (cut-off value > .9, table = 0.93) is a good fit. Incremental Fit Index (IFI) > 0.85 (cut-off value > 0.9, table = 0.93) is a good fit.

Table 4. Construct Measurement Summary: Confirmatory Factor Analysis
First-order latent variables Indicator (Likert Scale 1-5) t-value Standardized loading factors (path coefficient) Adapted from
Employee performance I can complete the work given to me well. 1.96 0.89 (Mangkunegara & Miftahuddin, 2016)
I can complete work according to company targets. 10.83 0.88
I obey company rules. 8.25 0.72
I have good initiative towards work. 6.20 0.58
Training Resources are available for quality-related training in the company. 1.96 0.80 (Prajogo & Cooper, 2010)
Quality-related training is given to all employees in the company. 7.25 0.73
Training is given in the “total quality and continuous improvement” concepts throughout the company. 8.22 0.83
Empowerment Employees are encouraged to fix problems they find. 1.96 0.77 (Gutierrez-Gutierrez et al., 2018; Obeidat et al., 2018)
All employees are involved in developing plans to improve quality. 7.79 0.79
The company expects the participation of all employees to improve quality and performance. 7.64 0.77
Teamwork Employees are involved in quality improvement teams. 1.96 0.70 (Jiménez-Jiménez & Martínez-Costa, 2009)
Cross-functional teams are established for solving quality problems. 7.87 0.93
Resources are available for supporting quality-related teams. 7.23 0.79
Compensation The company's compensation system encourages employees to achieve company goals. 1.96 0.89 (Obeidat et al., 2018)
Compensation process for employees is based on evaluation principles and criteria. 7.68 0.74
The compensation and rewards offered by the company are in line with employee expectations. 7.74 0.74
Second-order latent variables First-order latent variables t-value Standardized loading factors (path coefficient)
TQM–oriented HRM Training 7.04 0.82
Empowerment 7.88 0.95
Teamwork 5.86 0.75
Compensation 6.19 0.67

Note. Fit statistics for measurement model: ꭓ2 = 231.69 (df = 99); RMSEA = 0.00; PNFI = .73; CFI = 0.93; IFI = 0.93

The goodness of fit results generated in this model is accepted based on the measurement results above, which indicate that it is in a suitable category. The results are used to estimate and evaluate the initial measurement model (CFA) simultaneously.

The results of testing Hypothesis 1 show that TQM-oriented HRM can be operationalized as a second-order latent construct. This can be seen from the t-value above 1.96 and the standardized solution value above 0.5 (Hair et al., 2014; Wijanto, 2008; Yamin & Kurniawan, 2009). The values were as follows: training (β = 0.82, t-value = 7.04); empowerment (β = 0.95, t-value = 7.88); teamwork (β = 0.75, t-value = 5.86); compensation (β = 0.67, t-value = 6.19) (see Table 4).

The results of Hypothesis 2 testing show that TQM-oriented HRM has a positive effect on employee performance (β = 0.51, t-value = 4.62) (see Table 5). The magnitude of the impact of TQM-oriented HRM on employee performance is 0.47. The results of the study also supported the hypothesis, as the relationship was positive and significant. The results of this study also fill the gaps of previous research conducted qualitatively.

Table 5. Structural Modeling Results
Exogenous variable Endogenous variable Path t-value Standardized loading factors (Path Coefficient) Remarks
TQM – Oriented HRM Employee Performance TOH → EP 4.62 0.51 Supported

Note. TOH = TQM-oriented HRM; EP = Employee Performance

Figure 1. SEM with TQM-Oriented HRM Practices as a Second-Order Latent Construct

Discussion

Overall, the empirical results show that TQM-oriented HRM has an effect on employee performance. TQM-oriented HRM plays a key role for companies to achieve a competitive advantage through improving employee performance and contributing to sustainable excellence through organizational learning and innovation (Shahraki et al., 2011). Useful collaboration between HRM and TQM can result in better organizational performance and is a significant component of success for all managers and employees (Vouzas, 2007).

The results of this study also support previous research (Aeknarajindawat et al., 2020; Lam & Schaubroeck, 1999; Perdomo-Ortiz et al., 2009; Sihotang & Zebedeus, 2013). There is an important line of empirical research that has found a positive relationship between TQM-oriented HRM and employee performance. They explain that these resources, appropriately articulated and coordinated, will promote strategic capabilities for innovation, the basis of competitive advantage and better performance.

In some organizations, HR managers champion TQM-oriented HRM by sponsoring educational initiatives, communicating successes, and bringing in outside consultants to redesign work processes (Palo & Padhi, 2005). HR itself has a role in the pillars of the Baldrige Award, which is a guide for companies to gain performance excellence through an integrated management practice framework (Uysal, 2012). Alignment of HR and quality policies is important in improving employee performance (Jiménez-Jiménez & Martínez-Costa, 2009).

From an HR perspective, this suggests two implications. First, TQM-oriented HRM can help organizations build a platform for their HR policies aimed at improving employee performance. This argument is in line with the view that TQM is a vehicle for creating an organizational climate that will provide an enabling environment for high-performance workplaces. Second, TQM-oriented HRM can provide strategic focus and objectives for HR policies to achieve high-quality employee performance.

Conclusion

The main finding of this study is that TQM-oriented HRM has a positive effect on employee performance. The results of this study support the hypothesis that was built and fill the gaps of previous research conducted qualitatively. The results of this study contradict previous studies which state that HRM and TQM are peripheral and separate functions. However, this study supports other studies which explain that the TQM model associated with HRM has a positive effect on performance. The implication of this research is that companies can overcome the dynamics of environmental change by using HRM and TQM as new approaches to respond to these changes effectively. Quality efforts should be based on a long-term perspective and be part of the overall business strategy, including HR-related training, employee empowerment, compensation, and teamwork.

Limitations and Future Research

Our study has several limitations. Our sample is limited to companies located in Indonesia. The second limitation is the sampling method due to restrictions from the COVID-19 pandemic. Other researchers can strengthen or refute our conclusions by using samples of firms of different sizes or from other geographic contexts. The third limitation is the cross-sectional design of this study. Thus, even if the structural equation method is used, the interpretation of causality between constructs must be treated with caution. Finally, there is a need to confirm findings later when it is possible to conduct a survey in a different way.