The Differences Between Mystery Shopping and Marketing Research
Published in Quirk's Marketing Research Review, January 2001
By Mark L. Michelson
Editors Note: Mark Michelson is President/CEO Michelson & Associates, Inc., an Atlanta, Georgia based marketing research firm that provides mystery shopping services and past president of the Mystery Shopping Providers Association. He can be reached at 516-576-1188 or at email@example.com.
This essay will attempt to compare and contrast mystery shopping and marketing research services and offer some insights into how mystery shopping can be used effectively to compliment marketing research efforts. First, lets start with some basic definitions of these services:
Mystery Shopping is a long-established research technique that uses shoppers who are given guidelines to anonymously evaluate and monitor customer service, operations, employee integrity, merchandising, and product quality. Mystery shopping fills in a gap of critical information between operations and marketing. Mystery shopping is used on the front line to collect data that helps determine what happens to customers and prospects when they visit or call your company.
Marketing Research is the process of obtaining knowledge and gaining an understanding about what people think, feel and do in relationship to meeting their needs, desires and preferences related to buying products and services. Marketing research is used to identify and define marketing opportunities and problems; generate, refine, and evaluate marketing actions; monitor marketing; and improve understanding of marketing as a process. In plain English, it is determining what real customers, real prospects and other specific groups of people think about companies, services, products, and marketing communications.
Though many marketing research firms conduct mystery shopping, technically, mystery shopping is not marketing research. It is research, but it is not marketing research. It is more closely related to operations research, Mystery shopping compliments marketing research, but it is different in critical ways. If mystery shopping data is used for marketing research purposes - then certain rules would apply, such as the guidelines established by ESOMAR (see below).
Mystery shoppers must follow specific guidelines on what to do during an evaluation and shop at specified locations they may not normally visit. Marketing research study participants are not given evaluation guidelines in advance
Mystery shopping is typically more operational in nature than marketing research and is most often used for quality control, training and incentive purposes. Marketing research is used most often to determine real customer and prospect opinions, perceptions, needs, and wants
Mystery shoppers are recruited based on specific profiles that closely match a companies real customers. Marketing research study participants are sampled at random from a qualified population to represent a larger population
Mystery shoppers are asked to be objective and explain observations. Marketing research study participants are encouraged give their subjective opinions freely.
Mystery shopping reports on specific visits or calls each evaluation can be used independently to make improvements to operations and training. Mystery shopping is not predictive of every customers experience unless sufficient samples are taken and data analyzed in aggregate.
Mystery shopping should not be used alone to determine customer satisfaction - it can compliment, but not replace traditional customer satisfaction research. You can't predict or measure customer satisfaction using mystery shopping because customer satisfaction is a subjective topic based on what real customers think. Mystery shoppers are not real customers - they know what to evaluate before entering the store and they may not typically visit the store they are evaluating.
As with marketing research, there are many different types of data collection methods for mystery shopping. Some of the common mystery shopping data collection methods include:
Designing Mystery Shopping Questionnaires/Evaluation Forms
Questionnaires for mystery shopping evaluations should be designed to provide objective, observational feedback with a system to allow for checks and balances. Criteria to be evaluated must be objective rather than subjective. Typical retail mystery shopping questionnaires cover: greeting, customer service, facility cleanliness and orderliness, speed of service, product quality, and employee product knowledge.
Unlike marketing research questionnaires that employ likert scales for ratings, mystery shopping questionnaires ideally use only binary ("yes" and "no") questions. For certain questions, shoppers may be required to provide open-ended narratives for clarification of observations. Multiple response questions are used to allow shoppers to check off the features and benefits that are mentioned during the shop. Most shopping questionnaires include a "general comments" section that encourages shoppers to remark on anything they find significant or interesting during the shop.
For mystery shopping questionnaires, some questions may be more important than others - a point/scoring system for questions can emphasize the most important issues. If using a scoring system, which is often recommended, appropriate weighting of questions is critical. Some questions may not need to have points allocated to them at all, but may be necessary for background of the shop experience. Shoppers evaluations may be questioned and/or appealed once the facility knows that a mystery shop has occurred.
How to Make The Most of A Mystery Shopping Program
With a mystery shopping program, companies can establish customer service guidelines, monitor and reward excellent performance. As management guru Tom Peters says, "What gets measured gets done."
Once shopper reports are compiled, sharing those results with operations, training and other key personnel is the important next step in a programs success. Make it a positive, motivating experience that rewards people for a job well done while identifying areas where training may improve customer service and sales.
Mystery Shopping can be used as a marketing and training tool to help ensure a companys communications, service, and operational objectives are being carried out on the front line. An established, ongoing program, where employees know that any customer may be the mystery shopper, is more effective and objective than sporadic audits.
Use a mystery shopping provider that has experience in designing and managing mystery shopping programs. Many different kinds of companies provide mystery shopping services including: Mystery Shopping Specialists, Marketing Research Firms, Private Investigators, Merchandising Companies, Training Companies, Advertising/Promotion Agencies and others.
For more information on effective uses of mystery shopping, please visit the Mystery Shopping Providers Association website: www.mysteryshop.org or contact any of the members of this professional trade association.
ESOMAR expects researchers to conform to the following Requirements when carrying out Mystery Shopping research:
1. Mystery Shopping studies must be designed and carried out in ways which avoid unreasonably wasting the time and money or abusing the goodwill of the organizations and individuals being researched. Researchers must take great care to minimize the risk of any disruption to the normal working of the organization being researched.
2. Individual members of staff must not be identifiable in the report on a Mystery Shopping study (this issue is normally unlikely to arise in the case of 'competitive' Mystery Shopping). Similarly reporting should not be at individual outlet/branch level since in many cases this would implicitly identify specific individuals (e.g. because there is only one relevant staff member at a given location): data should be reported on only at a higher, aggregated level.
3. The interviews must not be electronically recorded unless respondents have agreed to this in advance. Electronical recording of interviews is not permitted if this could endanger the anonymity of respondents.
4. If for any research purposes (eg. for fieldwork quality checking or further follow-up research) individuals or individual outlets/branches are to be identified respondents must have agreed to this in advance. Any such agreement must be restricted to the use of individual information for research purposes only; any other use is not permissible. The identity of respondents must not be revealed to the client but to other researchers only.
5. Mystery Shopping calls on the client's own organization. The client should be made aware of any time and other operational costs to the organization of the calls involved and agree to these in advance. In addition, in order to minimize any staff concerns about such research:
6. Mystery Shopping calls on non-client organizations. Occasionally there will be agreement (not necessarily a formal one) within a given industry to accept 'competitive' Mystery Shopping calls in the interests of general quality improvement. Where no such agreement exists, it is even more important that the time and other demands created by such calls are kept to a minimum (and generally-acceptable) level. What this level is likely to be will vary with the nature of the calls (e.g. the proportion of observation to interviewing time), by industry and possibly by country:
7. Where there would be difficulty in conforming to the preceding recommendations the activity should not be regarded as a form of market research and should not be carried out by, or under the name of, a market research organization.