The Differences Between Mystery Shopping and Marketing Research

Published in Quirk's Marketing Research Review, January 2001

By Mark L. Michelson

Editor’s Note: Mark Michelson is President/CEO Michelson & Associates, Inc., an Atlanta, Georgia based marketing research firm that provides mystery shopping services and past president of the Mystery Shopping Provider’s Association. He can be reached at 516-576-1188 or at solutions@serviceevaluation.com.


Both mystery shopping and marketing research are long-established research services that help businesses and organizations operate more effectively. These research services share common goals in providing businesses with information critical to their success. However, mystery shopping and marketing research vary widely in technique and process. With this in mind, mystery shopping should not be used to replace marketing research, but rather to compliment an organization’s marketing and operational knowledge.

This essay will attempt to compare and contrast mystery shopping and marketing research services and offer some insights into how mystery shopping can be used effectively to compliment marketing research efforts. First, let’s start with some basic definitions of these services:

Mystery Shopping is a long-established research technique that uses shoppers who are given guidelines to anonymously evaluate and monitor customer service, operations, employee integrity, merchandising, and product quality. Mystery shopping fills in a gap of critical information between operations and marketing. Mystery shopping is used on the front line to collect data that helps determine what happens to customers and prospects when they visit or call your company.

Marketing Research is the process of obtaining knowledge and gaining an understanding about what people think, feel and do in relationship to meeting their needs, desires and preferences related to buying products and services. Marketing research is used to identify and define marketing opportunities and problems; generate, refine, and evaluate marketing actions; monitor marketing; and improve understanding of marketing as a process. In plain English, it is determining what real customers, real prospects and other specific groups of people think about companies, services, products, and marketing communications.

Though many marketing research firms conduct mystery shopping, technically, mystery shopping is not marketing research. It is research, but it is not marketing research. It is more closely related to operations research, Mystery shopping compliments marketing research, but it is different in critical ways. If mystery shopping data is used for marketing research purposes - then certain rules would apply, such as the guidelines established by ESOMAR (see below).


How is Mystery Shopping Different From Marketing Research?

Mystery shoppers must follow specific guidelines on what to do during an evaluation and shop at specified locations they may not normally visit. Marketing research study participants are not given evaluation guidelines in advance

Mystery shopping is typically more operational in nature than marketing research and is most often used for quality control, training and incentive purposes. Marketing research is used most often to determine real customer and prospect opinions, perceptions, needs, and wants

Mystery shoppers are recruited based on specific profiles that closely match a companies’ real customers. Marketing research study participants are sampled at random from a qualified population to represent a larger population

Mystery shoppers are asked to be objective and explain observations. Marketing research study participants are encouraged give their subjective opinions freely.

Mystery shopping reports on specific visits or calls – each evaluation can be used independently to make improvements to operations and training. Mystery shopping is not predictive of every customer’s experience unless sufficient samples are taken and data analyzed in aggregate.

Mystery shopping should not be used alone to determine customer satisfaction - it can compliment, but not replace traditional customer satisfaction research. You can't predict or measure customer satisfaction using mystery shopping because customer satisfaction is a subjective topic based on what real customers think. Mystery shoppers are not real customers - they know what to evaluate before entering the store and they may not typically visit the store they are evaluating.


Types of Mystery Shopping Methods

As with marketing research, there are many different types of data collection methods for mystery shopping. Some of the common mystery shopping data collection methods include:

Questionnaires for mystery shopping evaluations should be designed to provide objective, observational feedback with a system to allow for checks and balances. Criteria to be evaluated must be objective rather than subjective. Typical retail mystery shopping questionnaires cover: greeting, customer service, facility cleanliness and orderliness, speed of service, product quality, and employee product knowledge.

Unlike marketing research questionnaires that employ likert scales for ratings, mystery shopping questionnaires ideally use only binary ("yes" and "no") questions. For certain questions, shoppers may be required to provide open-ended narratives for clarification of observations. Multiple response questions are used to allow shoppers to check off the features and benefits that are mentioned during the shop. Most shopping questionnaires include a "general comments" section that encourages shoppers to remark on anything they find significant or interesting during the shop.

For mystery shopping questionnaires, some questions may be more important than others - a point/scoring system for questions can emphasize the most important issues. If using a scoring system, which is often recommended, appropriate weighting of questions is critical. Some questions may not need to have points allocated to them at all, but may be necessary for background of the shop experience. Shoppers’ evaluations may be questioned and/or appealed once the facility knows that a mystery shop has occurred.


What Are The Benefits of a Mystery Shopping Program?

With a mystery shopping program, companies can establish customer service guidelines, monitor and reward excellent performance. As management guru Tom Peters says, "What gets measured gets done."

Once shopper reports are compiled, sharing those results with operations, training and other key personnel is the important next step in a program’s success. Make it a positive, motivating experience that rewards people for a job well done while identifying areas where training may improve customer service and sales.

Mystery Shopping can be used as a marketing and training tool to help ensure a company’s communications, service, and operational objectives are being carried out on the front line. An established, ongoing program, where employees know that any customer may be the mystery shopper, is more effective and objective than sporadic audits.

Use a mystery shopping provider that has experience in designing and managing mystery shopping programs. Many different kinds of companies provide mystery shopping services including: Mystery Shopping Specialists, Marketing Research Firms, Private Investigators, Merchandising Companies, Training Companies, Advertising/Promotion Agencies and others.

For more information on effective uses of mystery shopping, please visit the Mystery Shopping Provider’s Association website: www.mysteryshop.org or contact any of the members of this professional trade association.

ESOMAR Guideline
Mystery Shopping Requirements

ESOMAR expects researchers to conform to the following Requirements when carrying out Mystery Shopping research:

1. Mystery Shopping studies must be designed and carried out in ways which avoid unreasonably wasting the time and money or abusing the goodwill of the organizations and individuals being researched. Researchers must take great care to minimize the risk of any disruption to the normal working of the organization being researched.

2. Individual members of staff must not be identifiable in the report on a Mystery Shopping study (this issue is normally unlikely to arise in the case of 'competitive' Mystery Shopping). Similarly reporting should not be at individual outlet/branch level since in many cases this would implicitly identify specific individuals (e.g. because there is only one relevant staff member at a given location): data should be reported on only at a higher, aggregated level.

3. The interviews must not be electronically recorded unless respondents have agreed to this in advance. Electronical recording of interviews is not permitted if this could endanger the anonymity of respondents.

4. If for any research purposes (eg. for fieldwork quality checking or further follow-up research) individuals or individual outlets/branches are to be identified respondents must have agreed to this in advance. Any such agreement must be restricted to the use of individual information for research purposes only; any other use is not permissible. The identity of respondents must not be revealed to the client but to other researchers only.

5. Mystery Shopping calls on the client's own organization. The client should be made aware of any time and other operational costs to the organization of the calls involved and agree to these in advance. In addition, in order to minimize any staff concerns about such research:

(a) it is good practice (and in some countries, a legislative requirement) to inform staff - and also any relevant staff association, works council etc. - if the organization proposes to carry out Mystery Shopping studies (but not necessarily the timing or precise details of these). Staff should be told the objectives and general nature of such research; and given reassurances that individuals and individual outlets/branches will not be identified in the reports (but see 4b below) and that no disciplinary or similar action will be taken vis-à-vis individuals as a result of the research.

(b) where staff remuneration to any extent depends on commission or bonuses, consideration may need to be given to making good any losses of salary as a result of time spent in dealing with Mystery Shopping calls.

6. Mystery Shopping calls on non-client organizations. Occasionally there will be agreement (not necessarily a formal one) within a given industry to accept 'competitive' Mystery Shopping calls in the interests of general quality improvement. Where no such agreement exists, it is even more important that the time and other demands created by such calls are kept to a minimum (and generally-acceptable) level. What this level is likely to be will vary with the nature of the calls (e.g. the proportion of observation to interviewing time), by industry and possibly by country:

(a) simple observational checks of shopper/staff behavior are unlikely to create problems of this kind provided that there is no interference with the normal working of the organization (although it may be necessary to deal with possible management objections).

(b) similarly with calls where the interviewing of staff members lasts only 2-3 minutes in total, or calls where a purchase is made the value of which is commensurate with the time taken up by the call.

(c) in other cases the acceptable length of time spent with members of staff may be determined by local codes of practice. Where these do not exist it is recommended that, unless there is some strong technical reason to the contrary, such time should normally not exceed:

- 10 minutes in manufacturing and retail businesses (other than automotive)

- 15-20 minutes in other service industries and businesses

(d) if the project is one where part of the evaluation involves some follow-up paperwork by the organization called on (e.g. provision of a brochure etc.) this must also be kept to a minimum.

(e) if Mystery Shopping calls are made on self-employed or professional people, etc., where time spent on an interview may literally cost them (lost) money, consideration should be given to reimbursing the individuals involved at an appropriate professional rate.

7. Where there would be difficulty in conforming to the preceding recommendations the activity should not be regarded as a form of market research and should not be carried out by, or under the name of, a market research organization.


Copyright © 1999 ESOMAR
ESOMAR Central Secretariat
E-Mail : email@esomar.nl

Home | About Us | Services | Focus Groups | Survey Research
Mystery Shopping | Request A Quote | Downloads | Clients | Links