How to Test Creative Concepts
What is Concept Testing?
Concept testing is the process of sharing creative concepts with the intended audience to get their feedback and identify the best idea before designing materials (See the How to Develop a Creative Concept guide). Results from creative concept testing help the creative team to revise concepts, drop ones that do not resonate well with the audience and identify the ones the audience likes best. Sometimes none of the concepts is appropriate for the audience. In these cases, the creative team returns to brainstorming, keeping in mind lessons learned from the concept tests.
Keep in mind that for social and behavior change communication (SBCC) campaigns and materials to be most effective, they should be tested at several stages of development. In the SBCC process, four types of testing are typically conducted: concept testing, stakeholder reviews, pretesting and field testing. The graphic below demonstrates the relationship between the four types of testing. This guide covers concept testing.
The purpose of creative concept testing differs from that of materials pretesting. With creative concept testing, the intention is to seek audience input into an overall creative concept or idea. Materials pretesting, on the other hand, seeks to assess audiences’ reactions to creative materials before they are finalized. Since the purpose of concept testing is not to pretest the actual copy and visuals of a creative material, concept test materials need not include final copy or visuals. Concept tests only need to communicate the creative idea.
Why Test Creative Concepts?
Testing creative concepts helps identify the best way to meet campaign objectives. Understanding what works—or does not work—saves time and money. It helps ensure that communication materials are developed based on an effective creative concept that connects with the intended audience. Concept testing allows teams to evaluate big ideas side-by-side and select from among alternative versions. Testing also reveals:
- Confusing words, phrases or ideas that should be clarified
- Preferred language used by the intended audience
- Preferred visual styles
- Weak concepts that should be cut
- New ideas
Who Should Conduct Concept Testing?
Ideally, researchers trained in concept testing methodologies – either an external research firm or internal research staff – should conduct concept testing. Sometimes, the hired creative agency conducts the concept tests. It is important that whoever conducts the testing be objective and able to ask questions without biasing or leading participants to a certain decision. The campaign implementation team, including stakeholders, can observe the testing or read through transcripts of interviews and focus group discussions to understand audience reactions and suggestions.
When Should Concept Testing be Conducted?
After completing the activities in the concept testing guide, the reader will:
- Select the most appropriate study participants and method for concept testing, based on available resources, timing and objectives
- Prepare tools for the concept test
- Refine a creative concept based on what has been learned during the testing process
Estimated Time Needed
Concept testing typically takes between one to two months depending on the testing method used and the number of revisions necessary. If creative concepts require a complete rework, it could take longer.
Step 1: Determine Which Elements to Test
- Attention. Does the idea attract audience attention? This is often measured as a person’s ability to remember an idea, message or image.
- Comprehension. Is the idea clearly understood?
- Motivation. Does the idea inspire the audience to take a desired action?
- Personal relevance. Can the audience connect with the idea? Does it take their point of view into consideration?
- Cultural appropriateness. Is the idea consistent with the values, attitudes, beliefs, traditions and history shared by the intended audience?
Step 2: Identify Testing Audience
The best way to get a sense of how the audience will react to a concept is to test it with them. Review the audience profile created for the primary and influencing audiences. These descriptions can help determine where to find audience representatives, as well as the methods that will be used for concept testing. Some examples include recruiting pregnant women from antenatal care clinics, female sex workers from known hot spots, like night clubs, or mobile men from truck stops.
Based on the audience profiles, create a list of characteristics that concept test participants must have in order to participate. These will become part of the screening questionnaire in Step 5 that establishes criteria for participation. Consider demographic (age, sex, relationship status, number of children), psychographic (needs approval from friends, values intimacy), behavioral (never used family planning, do not want more children) and geographic (rural, urban, density) characteristics.
Also consider getting feedback from people who influence the intended audience, such as health care providers, teachers, religious leaders or spouses. Testing concepts with these influencers is a good way to learn about cultural appropriateness and to gauge their support for the campaign.
Step 3: Select the Best Testing Method
Decide which method(s) will be used to test the concepts: focus group discussions, in-depth interviews, intercept interviews or self-administered questionnaires. The method selected will depend on the depth of feedback required, characteristics of the audience representatives, time and budget.
|Test Method||Sample Size||Appropriate For|
Focus Group Discussions (FGDs)
A discussion on the concept guided by an experienced moderator. Can be conducted online or face-to-face.
|Six to ten people; Usually several FGDs/ concept.||Good for generating new ideas and gauging emotional impact. Work best when it is fairly easy to bring together a group of people.|
One-on-one discussion between an interviewer and a participant. Can be conducted in person or over the phone.
|At least ten||Work best when testing concepts among people from several different geographic locations, or when the creative content is sensitive, such as reproductive and sexual health, making the audience feel uneasy discussing it in a group.|
Trained interviewer shows concept materials to the intended audiece in a place they frequently visit (markets, clinics) and conducts a quick survey with them.
|60-300||Work best when many audience representatives frequent the intercept location, when testing for one element, such as attention or cultural relevance, across alternative concepts. They help reduce recruiting costs.|
Questionnaires of mostly close-ended questions are completed by respondents. Can be done in-person, via email or mobile, or online.
|20-200||Good for large numbers of respondents with a short timeframe. Work well when testing creative concepts for websites, mobile applications or communication campaigns intended for audiences who come from diverse geographic locations, such as from across countries or regions of one country.|
Step 4: Develop Testing Methods
Once the testing method(s) have been chosen, the team will need to develop mockups or visual presentations of each alternative creative concept that can be shared during concept testing. How these materials are presented will depend on the method of testing, the nature of each concept and the characteristics of the people who will participate in the testing exercise. Often, the best way to present creative concepts is through a visual presentation that includes samples of various materials that could be used. This helps to minimize the effort and costs of preparing full sets of campaign materials for each creative concept. Presentations can be projected electronically, shared online and as poster boards, or printed as handouts. It is important that mockups of each creative concept are comparable in quality to minimize bias.
|When using intercept interviews, it might be best to present each concept via a poster to minimize on time and eliminate the need for technology. When using in-depth interviews or focus group discussions, an electronic presentation, such as Powerpoint® or Prezi, might work well since they are more interactive and there is sufficient time to explore. When pretesting concepts for a video or film, storyboards work well, since they visually show a sequence of events. When testing concepts for a multi-channel campaign, an electronic presentation allows audience representatives to see mock-ups of a variety of media. On the other hand, when testing concepts for radio or TV spots among rural residents, where electric power is not readily available, dramatic readings work better than electronic presentations.|
Several creative concepts were developed and tested for the Safe Male Circumcision Campaign Uganda. A Powerpoint® presentation was printed as poster boards and shared during focus group discussions with uncircumcised men to test the creative concept for a national campaign promoting voluntary medical male circumcision in Uganda. See the Samples section for the full presentation.
Step 5: Prepare Testing Tools
Develop the tools that will be used during concept testing. The tools needed depend on the testing methods to be used. Four types of tools may be required: recruitment screener, consent form, moderator’s guides and questionnaires.
- Recruitment Screener. A short questionnaire that is administered to potential participants—either in-person, online or by telephone—to ensure that they meet the criteria for participating in the concept testing, as described in Step Two. It should be used to guide the selection of testing participants, regardless of testing method used (see the Samples section for sample Screeners).
- Consent Form. A brief form that describes the research and what a participant should expect, including the risks involved, and allows a participants to voluntarily consent to participate. The form can be read to participants if they are not literate (see the Samples section for sample form).
- Moderator’s Guide. An important tool for focus groups and in-depth interviews, the moderator’s guide serves as an outline to help keep the moderator on track during discussions. The guide contains topics, questions and activities to spark discussion. Normally, when testing more than one creative concept, testing involves sharing each concept individually, followed by a few questions. After all alternative creative concepts have been shared and discussed, the moderator asks participants which of the concepts they prefer and why. In developing the guide, determine the key questions to ask and probes for obtaining answers in a conversational manner. A typical guide contains the following (see the Samples section for examples of moderator’s guides):
- Objectives and elements to be tested
- Brief respondent profile, including the number of interviews or discussions to complete
- Where and when the interviews are to take place
- An introduction script to be read to participants
- Questions and activities
- Closing script to be read to participants
- Questionnaire. A standard set of questions and response options that may be completed by the respondents themselves or by interviewers (see the Samples section for sample questionnaires). In general, questionnaire questions should:
- Contain only one idea or question
- Define the scope to consider, such as the time period or activities that are relevant to the question
- Be written with neutral language to avoid leading the respondent to a specific answer
- Use language that enables respondents to easily understand the question
- Contain response options that are simple, clear, consistent and include the full range of responses that might occur
- Provide mutually exclusive response options so that a respondent can pick only one option
Here are some sample questions that can be asked after showing each individual concept. These questions are designed to test concepts for each of the five elements listed in Step One.
Provide pre-coded questions
These are open-ended questions that will be coded during analysis
Provide pre-coded questions
Provide pre-coded questions
Use pre-coded questions to gauge degree of appropriateness; use open ended questions to probe how a concept is inappropriate
Ask after all concepts have been shared and tested
Step 6: Conduct the Testing
Proceed with testing as detailed in the moderator’s guide or survey instrument. At the start of each focus group, interview or survey, be sure to welcome the participants and provide a brief overview on the purpose of the testing. Reinforce that there are no “right” or “wrong” answers, and encourage everyone to provide their honest views. Make sure to obtain permission from participants if planning to video or audio record the sessions. Take notes and/or record the feedback on an audio or video recorder, as appropriate.
Step 7: Analyze Testing Outcomes and Summarize Results
Review all notes and recordings from the testing exercise and write a report outlining the process and the findings. The report should have the following sections:
- Background. What was tested? What were the objectives of testing? Who participated in the testing? How was the testing done?
- Highlights. Summarize the main points that came up during the testing.
- Findings. Present a complete report on the findings. Where appropriate, describe the participants’ reactions, incorporate key quotes, describe which creative ideas and concepts worked the best versus those that were not appealing or effective, and share suggestions given by participants.
- Conclusions. Describe the key learnings that came up and/or the major differences that were observed across individuals and/or groups.
- Recommendations. Suggest and prioritize revisions for the tested creative concepts based on the findings and conclusions.
Step 8: Apply Testing Results
Share the testing report with the campaign implementation team and apply the recommendations to sharpen and hone in on one creative concept. Use the research to inform decisions and help develop improved concepts and creative materials. Sometimes the testing will identify one creative concept that is clearly preferable and the results can be used to improve that concept. In other cases, there will be no clearly preferred creative concept, and the results can be used to inform new concepts.
|Be aware of the limitations of testing. Keep these things in mind to avoid using the results incorrectly.|
Tips & Recommendations
- Be sure to test the concept test tools/resources developed for each method before using them with the audience
- Emphasize at the beginning of the concept test that these are only rough drafts/prototypes, and that they should evaluate based on the big idea, not the specifics. Colors, fonts, models and details will be further refined at a later point.
- Do not try to merge concepts following the concept test, picking the most well liked pieces from different concepts and combining them into one. This dilutes and confuses the entire campaign.
- When asked to compare creative concepts, respondents sometimes get distracted by the way the concepts are presented, and rate a concept based on the quality of artwork, color schemes or typeface used. A way around this is to test each creative concept on its own merits and not ask respondents to rate concepts against each other. Being consistent with the quality of visual materials also helps. Moderators also need to be well trained in how to steer the conversation away from this level of feedback and focus it on the bigger picture.
- Sometimes, the order in which creative concepts are introduced creates bias. This sort of bias can be reduced by varying the order in which the concepts are presented in each interview or focus group discussion.
- Concept testing is useful not only to test creative concepts for a campaign, but also to get audience feedback on design features of communication products. For example, when designing websites, it is useful to allow prospective users to interact with the site during the design stage to test the layout, features and look. Testing design features can save time and money over the long run by preventing costly redesign.
- Often, the differences between creative concepts is so slight that audience representatives cannot distinguish them. When testing more than one creative concept at a time, it is important that each concept is unique from one another and there is no ambiguity.
- The location where focus group discussions or in-depth interviews take place can influence results. Focus groups and interviews that take place in an office often yield different results from those that take place in communities where the respondents are from.
- Too many concepts produce concept fatigue in the respondent. The maximum number of concepts to test in a focus group or interview is three.
Glossary & Concepts
- Audience analysis: A description of the defining characteristics of the people who are targeted by a health communication campaign or product.
- Concept testing: seeks feedback about general ideas, concepts and creative concepts; typically done before materials are developed.
- Creative brief: A short document outlining the audience, objective, promise, key message content and the communication channels that will be used; a succinct account of a campaign strategy.
- Creative concept: The overarching artistic theme that ties together all elements of an SBCC campaign. It is an imaginative plan for capturing the audience’s heart and mind.
- Field testing: allows practitioners to observe whether the SBCC materials are used effectively in their intended settings and contexts, usually through observation and focus group discussions. It determines whether the material meets the intended purpose.
- Pretesting: is the process of bringing together members of the priority audience to react to the components of a communication campaign before they are produced in final form.
- Stakeholder reviews: are input from technical experts, partners and decision-makers prior to finalizing materials. These reviews do not replace pretesting with the priority audience and can be done before or after pretesting.
- Dramatic reading: is a public reading or recitation of a work of literature with an interpretative or dramatic use of the voice and often of gestures.
- Pre-coded questions: are a type of question that is asked by interviewers as though they are open-ended questions, but they have pre-coded responses that interviewers use to match (code) respondents' answers, rather than copy down the verbatim response.
Sources & Citations
- Making Health Communication Programs Work: A Planner's Guide. December 2001. Washington, DC: U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health. Office of Cancer Communications, National Cancer Institute. NIH Publication No. 02-5145. Pp 53 – 89. www.cancer.gov/pinkbook
- Concept Testing in Qualitative Market Research, http://www.focusgrouptips.com/concept-testing.html
- How to Test Advertising Effectiveness, http://www.fusionb2b.com/blog/2011/11/how-to-test-advertising-effectiveness/
Banner photo: © 2005 Jennifer Orkis, Courtesy of Photoshare