class: title-slide, right, top background-image: url(https://www.cityu.edu.hk/sites/g/files/asqsls3821/themes/site/cityu/images/logo-coloured-2019.svg) background-position: 10% 90% background-size: 15%, cover .right-column[ # Public Communication Campaigns <br> .small[COM 5110] ### Week 10: Evaluating public communication campaigns **Ye Sun**<br> 22/03/2023 ] --- background-color: var(--my-blue) class: inverse # This week -- .left-column[ ### Two readings ] -- .right-column[ Valente & Kwan (2013) (Chapter 6 in Rice & Atkin): > _Evaluating Communication Campaigns_ <br> Berkowitz et al. (2008): > _Overview of Formative, Process, and Outcome Evaluation Methods Used in the VERB™ Campaign_ ] --- ## Evaluation Defined -- > Evaluation is the systematic application of research procedures to understand the conceptualization, design, implementation, and utility of interventions (here, communication campaigns). -- - Evaluation research determines *whether* a program was effective, *how* it did or did not achieve its goals, and *the efficiency* with which it achieved them. -- - Evaluation contributes to the knowledge base of how programs reach and influence their intended audiences so that researchers can learn lessons from these experiences and implement more effective programs in the future. --- ## Three _Functions_ of Evaluation -- ---- .left-col[ The first function of an evaluation is to .red[determine the expected impacts and outcomes] of the program. ] ??? For example, if a campaign was designed to raise the awareness of the dangers of substance abuse, the evaluation proposal should state the percentage increase expected in this awareness -- .center-col[ The second function of a campaign evaluation is to help planners and scholars .red[understand how or why] a particular campaign worked or did not work.] ??? Knowing how or why a program succeeded or failed—that is, the theoretical and causal as well as implementation reasons—increases the likelihood that successes can be repeated and failures avoided in future behavioral promotion programs. -- .right-col[ The third function of evaluation is to provide information relevant for .red[planning future activities]. ] ??? Evaluation results can indicate what behaviors or which audiences should be addressed in the next round of activities. In sum, we conduct evaluation (particularly process and summative evaluation) to know whether the program worked, how and why it worked, and how to make future programs better --- ## Three _Phases_ of Evaluation ### Distinguished by their .blue[timing] and .blue[main purpose] ---- -- .left-col[ #### .red[Formative Evaluation] Conducted _before_ or _in the early stages_ of a campaign, guides the development of campaign materials and techniques that would appeal to the target audience ] -- .center-col[ #### .red[Process Evaluation] Assesses campaign _implementation_ _during_ the campaign (also known as _monitoring_) ] -- .right-col[ #### .red[Summative/Outcome Evaluation] Assesses the _short-term_ and _long-term changes_ that result from campaign activities. ] ??? used to gain insight into the subject the product deals with and helps in problem identification and solving Formative evaluations tell you what is not usable about an existing product, or, what is broken. The purpose is to discover insights and shape the designs direction. Formative evaluation research inputs can enhance campaign effectiveness by guiding the development of sophisticated strategies and effective messages. This preliminary phase of research is useful for determining which approaches are most promising and for revealing whether certain components are ineffective or even counterproductive. summative research - compared to a benchmark - to evaluate the effectiveness & impact of the campaign - more rigorous, systematic research for verification and validation ] --- ## Three _Phases_ of Evalution .panelset[ .panel[.panel-name[Formative Evaluation] .pull-left[ #### .my-yellow[Goals] - define the scope of the problem - gather data on possible intervention strategies - learn about the intended audience (beliefs, motivations, perceptions, etc) - investigate possible factors that might limit program implementation - also used to test message strategy - test the effectiveness of possible communication channels ] .pull-right[ #### .my-yellow[Methods] - mostly qualitative research methods such as focus group discussions , in-depth interviews, and ethnographic observations - though initial surveys are useful as well. ] ] <!----> .panel[.panel-name[Process Evaluation] .pull-left[ #### .my-yellow[Goals] - to measure the degree of program implementation to determine whether the program was delivered as it was intended. - to establish if a project is advancing towards its goals. - to identify problems or barriers in the project, pointing to needed corrections or modifications ] .pull-right[ #### .my-yellow[Methods] - usually by collecting data on when, where, and for how long the campaign is implemented. - monitoring the media activities, conduct content analyses of media coverage or reports, or collect usage data such as various digital metrics ] ] <!----> .panel[.panel-name[Summative/Outcome Evaluation] .pull-left[ #### .my-yellow[Goals] - to measure the program's impact and the lessons learned from the study and to disseminate research findings ] .pull-right[ #### .my-yellow[Methods] - usually conducted by analyzing quantitative data collected before, during, and after the campaign, depending on the research design - an interactive and iterative process in which preliminary findings are shared with program planners and other stakeholders prior to widespread dissemination ] ] <!----> ] <!--end of panelset--> ??? Summative research is an interactive and iterative process in which preliminary findings are shared with program planners and other stakeholders prior to widespread dissemination. The stakeholders provide insight into the interpretation of data and can help set the agenda for specific data analyses. Once the summative research is completed, the findings can be disseminated. The evaluation framework in Figure 6.1 delineates the steps in the evaluation process, but it is theory concerning human behavior that informs it. >>> ## Exercise: Identify which phase of evaluation each of these questions belongs to? ### Get on <www.menti.com> [The Menti Page](https://www.mentimeter.com/app/presentation/alqqouy79f6mht7p3c6defbqs22zcjn6/s7gp1gkpxaow/edit) ??? #### Has there been any change in beliefs, attitudes, behaviors, social norms? #### What are the vision and values that guide the campaigners? #### How many materials have been put out? What has been the campaign’s reach? How many people have been reached? #### How does the campaign’s target audience think about the issue? What messages work with what audiences? #### Have any policies changed? Has there been any systems-level change? --- class: middle ## More from Valente & Kwan (2013) on the evaluation processes... --- ### The Importance of Theoretical Frameworks -- To guide the .my-gold[**design**] of the program and inform its .my-gold[**evaluations**] e.g., **Theory of Planned Behavior** .center[ <img src="https://amactraining.co.uk/wp-content/uploads/2014/05/61.png" width = "65%"> ] --- ### The Importance of Theoretical Frameworks .pull-left[ <img src="https://amactraining.co.uk/wp-content/uploads/2014/05/61.png" width = "100%"> ] .pull-right[ #### Important to evaluate/measure the intermediate steps that your campaign is designed to focus on. .large[ - Formative evaluation - Process evaluation - Summative evaluation #### The congruence between theory and program implementation and evaluation is very important. ] ] ??? If, for example, self-efficacy is theorized to be an important influence on whether individuals use condoms during sex, then the program may try to increase self-efficacy among the intended audience. The evaluation therefore needs to measure self-efficacy before, during, and after the campaign to determine whether the program did indeed deliver self-efficacy messages or training (process evaluation), and change self-efficacy levels in the appropriate communities (summative evaluation). ### Study Designs .center[ <img src="design.png" width = "100%"> ] --- ## Process Evaluation .panelset[ .panel[.panel-name[Campaign Exposure] Process evaluation can help identify how much of the planned exposure or treatment is actually engaged by particular audiences. Campaign exposure is the degree to which audience members have access to, recall, or recognize the intervention. ] <!----> .panel[.panel-name[Interpersonal Communication] Campaigns often purposively stimulate interpersonal communication, but the impact of this media-generated communication will not be apparent until the information percolates through interpersonal networks and individuals have time to share their attitudes, experiences, and opinions with one another .pull-left[ <img src="w.png" width = "100%"> ] .pull-right[ <img src="w1.png" width = "100%"> ] ] <!----> ] <!--end of panelset--> ??? https://www.nielsen.com/insights/2012/consumer-trust-in-online-social-and-mobile-advertising-grows/ --- ## Summative Evaluation .panelset[ .panel[.panel-name[Measuring Impact] Through statistical analyses: - determine whether outcomes have changed between baseline and follow-up surveys. - determine whether change was different for different subgroups (e.g., demographic and socioeconomic). - assess the relative effects of different factors ] <!----> .panel[.panel-name[Dissemination of Findings] sometimes referred to as _feedback_ or _feed-forwarding_ Five main ways of dissemination: - 1) via scheduled meetings with the programmers - 2) at scientific conferences relevant to the topic - 3) through key findings or technical reports - 4) through academic papers published in refereed journals - 5) online channels/platforms ] <!----> ] <!--end of panelset--> ??? Dissemination (sometimes referred to as feedback or feed-forwarding) is often neglected in evaluation plans because researchers are preoccupied with planning and conducting the study and often feel that the results will speak for themselves. Moreover, it is difficult to anticipate the appropriate audiences and channels for dissemination before the findings are known. Most campaigns not documented in evaluation reports are often forgotten within a year or two. --- background-color: var(--PaleGrey) class: inverse ## .my-blue[Evaluation Framework] .center[ <img src="f6.png" width = "65%"> ] --- background-image: url(verb.jpg) background-size: 100% cover background-position: 80% 20% class: bottom, left # .blue[**the VERB™ Campaign**] ### .blue[A multiethnic media campaign with a goal to increase and maintain physical activity among tweens (children aged 9 to 13 years)] ??? The first year of the campaign consisted of national advertising, plus extra marketing activities in nine CDC-selected communities. The CDC based its selection of the nine communities on a variety of factors, including the size of the media market, racial and ethnic diversity, geographic diversity across the United States, existing infrastructure, and population size. Six of the nine communities received even more local advertising so that the CDC could evaluate whether the added media made a measurable difference in behavioral outcomes. These six evaluated communities were called “high-dose” communities --- ### .blue[VERB] .center[ <img src="v1.png" width = "70%"> ] --- ### .blue[VERB] .center[ <img src="v0.png" width = "70%"> ] --- ### Evaluation Methods Used in the VERB™ Campaign <br> .center[ <img src="abstract2.png" width = "120%"> ] --- ## .blue[**VERB**]: Formative Evaluation ### .my-yellow[Develop the brand] .panelset[ .panel[.panel-name[overarching message strategy] .pull-left[ #### Data ---- - 48 interviews with triads of tweens - 6 focus groups with parents of tweens - 2 focus groups with adults who work with tweens (i.e., influencers) - 8 in-depth interviews with industry professionals ] .pull-right[ #### Main discoveries ---- The campaign should focus on helping tweens enhance their **self-esteem and self-confidence** by participating in **fun** activities where they **can achieve success**. ] ] <!----> .panel[.panel-name[gaining insights] .pull-left[ #### Goals ---- to understand: .small[ - (1) the **cultural, ethnic, and economic dynamics** that _unify and differentiate tweens_; - (2) the **motivators for and barriers** to _tweens’ participation_ in physical activities; - (3) the **motivators for and barriers** to _parental support_ of tweens’ participation in physical activities.] ] .pull-right[ #### Data ---- - 25 in-depth interviews with parents and tweens - diaries kept by 25 parents; follow-up interviews - 33 additional focus groups with ethnic minorities ] ] <!----> .panel[.panel-name[eliciting ideas] .pull-left[ #### Data ---- - Brainstorming for `positionings`: developing brand concept ideas - .small[more focus groups and interviews with tweens/parents/influencers] - Getting feedback on the 12 positioning statements generated - .small[more focus groups] ] .pull-right[ #### Concept development ---- - Generate the `brand essence` (a one-word description of the core message of the campaign): .blue[**_Lift_**] - Develop full concepts: .blue["**_VERB. It's what you do._**"] ] ] <!----> ] <!--end of panelset--> ??? The positioning that proved most motivational was lift, suggestive of the act of rising or elevating and the exhilaration of flying or expanding ones’ views. --- ## .blue[**VERB**]: Formative Evaluation ### .my-yellow[Develop the VERB advertising] .panelset[ .panel[.panel-name[Overview] .left-col[ #### Overview To test the content, tone, and execution of VERB advertisements and promotions A three-step process, drawing on qualitative methods ] .center-col[ #### Methods For each step - focus groups (held in person or by telephone) - individual interviews or dyadic interviews. ] .right-col[ #### Participants - (1) children aged 9–13 - (2) parents - (3) adult influencers ] ] <!----> .panel[.panel-name[Step 1 ] ### .blue[Creating Communication Strategy] .pull-left-narrow[ To learn what motivates tweens, what would appeal to tweens ] .pull-right-wide[Results revealed that **motivators** for tweens to be physically active include being with friends, engaging in self-discovery, and participating in events with an element of competition. **Barriers** to being physically active included fear of failure, lack of time, family responsibilities, and competing interests.] ] <!----> .panel[.panel-name[Step 2 ] ### .blue[Concept Testing] .pull-left-narrow[ To validate the communication strategy and aid in selecting concepts ] .pull-right-wide[Participants were presented with up to three concepts for an advertising campaign and asked to respond to a set of questions about the advertisements themselves. Rough versions of the advertisements consisted of print and video storyboards, sketch drawings, and video animations. ] ] <!----> .panel[.panel-name[Step 3 ] ### .blue[Message Testing] .pull-left-narrow[To assess whether the audience interpreted the main message as intended and identify what's confusing or controversial] .pull-right-wide[ Participants were shown almost completed versions of the final advertisements and were asked for their initial reactions, their likes and dislikes, their perception of the advertisement’s relevance to VERB’s main message, and their perception of the advertisement’s ability to motivate tweens to be physically active. ] ] <!----> .panel[.panel-name[Main findings] Messages should promote the behaviors sought by a physical activity campaign for tweens by .my-yellow[using images and audio cues] that .red[convey happiness, excitement, and fun] about being physically active. _Mothers_ responded favorably to messages about physical activity’s .red[contribution to a child’s success] in .my-yellow[other areas of life]. ---- For example, [CDC's press release](https://www.cdc.gov/media/pressrel/r020717.htm) see [this VERB ad](https://youtu.be/47cHf1YqE2A) ] <!----> ] <!--end of panelset--> --- ### .blue[**VERB**] .center[ <img src="v2.png" width = "70%"> ] --- ## .blue[**VERB**]: Process Evaluation .panelset[ .panel[.panel-name[Overview] The overarching goal for the process evaluation was - to gather .red[interim and ongoing] feedback about the .blue[execution and evolution] of VERB rather than about the program’s effects on participants. ---- included activities - to measure reach > `reach`: the percentage of the target audience who heard or saw the campaign’s messages - to monitor implementation of promotions and experiential marketing activities. ] <!----> .panel[.panel-name[Methods] .center[ <img src="table1.png" width = "100%"> ] .pull-left[ Monitoring of .small[ - campaign-related paid television advertising, television news coverage, school-based television coverage, radio news coverage, and paid radio advertising - print media coverage through clipping services - VERB websites (VERBnow.com, VERBparents.com, and www.cdc.gov/verb) ] ] .pull-right[ Interviewing community partners Evaluating VERB-sponsored events for tweens on-site Conducting a continuous tracking survey ] ] <!----> ] <!--end of panelset--> --- ## .blue[**VERB**]: Outcome Evaluation .panelset[ .panel[.panel-name[Overview] - assessed the changes in the target audiences’ awareness, knowledge, attitudes, and behaviors related to physical activity - determined whether these changes could be attributed the VERB campaign. ] <!----> .panel[.panel-name[Methods] .pull-left[ #### Youth Media Campaign Longitudinal Survey - an annual telephone survey of parent–child dyads - longitudinal cohort design with a national sample of children aged 9–13. - Conducted annually in the spring 2002 - 2006 with random-digit-dialing and computer-assisted telephone interviewing methods ] .pull-right[ #### A secondary analysis - to examine the impact of additional marketing activities and advertising in six communities. - In 2002, 2003, and 2004, data were collected on a sample of child–parent dyads in high-dose communities ] ] <!----> ] <!--end of panelset--> ??? high-dose communities- (i.e., communities where the level of marketing activities and advertising was deliberately higher than throughout the rest of the nation). --- ### .blue[VERB] Effectiveness .center[ <img src="v3.png" width = "70%"> ] --- ### .blue[VERB] Effectiveness .center[ <img src="v4.png" width = "70%"> ] ??? background-color: var(--my-blue) class: inverse, middle ### Next: ### Group Exercise: ### For the campaign you have chosen, could you tell what kind of evaluations were conducted? Use the provided worksheet as a guideline. What evaluation plans would you propose to add to a future campaign like that?