ASSESSMENT DETAILS
Summary
Interviews with workshop participants
Product analysis
Newsroom surveys
Selecting organizations to assess
Summary
Between May 2002 and July 2003, Professor William Damon – one of the original architects of the Traveling Curriculum program – and his associates interviewed 126 print journalists from seven newspapers that participated in Curriculum workshops about their experiences. What did they remember? Did they find anything particularly valuable? Were they able to implement any training lessons into their daily routines? Did they believe their newsrooms had implemented training lessons in a noticeable way? What could their newsrooms, and they as individuals, do to keep alive a spirit of striving for excellence in their journalism? These interviews became the foundation for a formal assessment of the impact of the Traveling Curriculum program.
The formal assessment began in July 2003, after a fulltime assessment associate was added to the CCJ staff. The assessment team decided to focus on three measurement tools to determine what, if any, impact the Traveling Curriculum workshops were having on the individual journalists and newsrooms that participated. These three measurement tools included: 1) One-on-one interviews with journalists who participated in workshops. 2) Analysis of staff surveys administered before and after workshop participation. And 3) analysis of samples of “product” collected before and after workshop participation.
Between July 2003 and September 2005, the assessment team traveled to 34 more news organizations that had participated in Traveling Curriculum workshops to conduct interviews and collect samples of news product. Of these 34 news organizations, 17 were newspapers, 15 were broadcast stations, one was an online organization, and one was a student newspaper. The assessment team interviewed 549 more former workshop participants and collected and analyzed almost 1,800 staff surveys. The team also collected and analyzed nearly 2,100 newspaper stories and 300 local television newscasts.
A more detailed description of the three measurement tools, the rationale for their selection, details about how they were utilized, and a description of the process for selecting organizations to assess follows:
[top]
1. Interviews with workshop participants
Interviews with journalists who’ve participated in CCJ workshops are the primary component of the assessment model. The goal was to interview at least 15-20 former workshop participants at each organization assessed. As noted above, some 675 former workshop participants have been interviewed across Damon’s preliminary assessment visits and the “formal” assessment period.
Interviews were conducted in on-on-one settings and typically lasted 30 minutes. An effort was made to interview a representative group of staff from each newsroom – a mix of participants from different job positions, hierarchical levels, ages, genders, ethnic backgrounds, etc. The top newsroom manager was interviewed in nearly every newsroom, as was the individual whose decision it was to invite CCJ in for training, if that person wasn’t the newsroom manager.
Interviews followed a protocol that was specific to the training modules to which each newsroom was exposed. For example, if a newsroom received the accuracy and verification module during its workshop, interviewees were asked what they remembered about the workshop in general, and then whether they remembered and implemented any lessons or tools from that specific module. Each battery of questions related to a particular module included open and closed-ended questions.
Interview responses were transcribed and loaded into a database for further analysis. These responses were content coded for consistent themes and ideas. Then a detailed, customized report was prepared for the staff of each newsroom that focused on what workshop participants said they learned from the CCJ training and what they felt like their newsrooms could still do to improve. These reports relied heavily on direct quotations from interviews and the assessment team’s observations of each organization’s culture, communication systems, and staff morale. In some cases, a confidential memo was prepared for the newsroom manager when it seemed important to share information from interviews deemed too inflammatory or contentious to include in the staff report.
Interviews were chosen as the primary assessment tool because they allowed us to gather rich qualitative data and, through open-ended questions, get a sense of the universe of responses and reactions to the Curriculum program. One-on-one interviews conducted “on-location” allowed CCJ assessment staff to process not only whether training lessons did or didn’t “stick,” but also how the specific circumstances of each newsroom environment impacted the saliency of training lessons for individuals and entire newsrooms.
In some cases, the opportunity to observe the culture and communication systems in a newsroom while on an assessment visit proved as valuable in terms of understanding CCJ’s impact, or lack thereof, as what interview respondents had to say about their workshop experiences. The importance of being in a newsroom while conducting interviews was reinforced at an early assessment visit to a large Midwest television station. Interviews revealed that participants had high recall of the Traveling Curriculum workshop in which they participated, but few of the suggestions put forth in the post-workshop report had been implemented. Many of the interviews were tinged with negativity toward the newsroom manager. The CCJ assessment team observed editorial meetings and the way staff interacted with each other and the manager. The manager seemed gruff and intimidating during the meetings observed. CCJ assessment staff pointed out to the manager that his management style seemed abrasive – something newsroom staff and middle managers weren’t comfortable doing themselves. We, as concerned third-parties, were able to give the manager advice in a non-threatening way, and received insight into why training suggestions hadn’t been implemented – staff were overwhelmed by the low morale caused by negative feelings toward their manager.
As the example above demonstrates, assessment interviews have been invaluable in not only providing data about which CCJ workshop tools and lessons were most memorable and useful, but also offering context and explanations for variances in impact from newsroom to newsroom.
[top]
2. Product analysis
We collected pre and post workshop “product” from 15 newspapers, 11 local broadcast stations, and one Internet newsroom.
We collected two types of “product” from the newspaper and Internet newsrooms we assessed. In order to get a sense of how a newspaper might have been impacted broadly by a CCJ workshop, we collected and analyzed four weeks of “Local/Metro” section front stories – two weeks from right before the CCJ workshop and two weeks from somewhere between six and 18 months after the workshop. The Local/Metro section of most papers we studied contained mostly original content – not a lot of wire stories contributed by reporters outside of the assessed newsrooms. We reasoned that national and international events drive much of a newspaper’s “Front” page content. But Local/Metro section stories generally are driven by reporter enterprise and the newspaper’s identification of salient local issues and interests, making the content in this section more likely to be visibly affected by the many workshop lessons aimed at making stories relevant and interesting to one’s community.
The other type of product collected from newspapers was stories from individual reporters who had participated in a CCJ workshop. One such reporter was selected from each newspaper assessed. We asked the editor of each newspaper assessed to identify a reporter who, from her perspective, seemed to get the most out of the CCJ workshop. Then we collected 20 of that reporter’s stories for analysis – ten from before the workshop and ten from six to 18 months after. For two reporters whose work we decided to use in case studies (see the “Impact on quality of news product” section), we collected an additional 20 pre and 20 post workshop stories. We applied the same coding scheme to these stories as we did to the section front stories we analyzed. The goal here was to determine whether lessons from the workshop became salient enough to measurably affect the quality of the work produced by individuals exposed CCJ training.
For the lone Internet newsroom we trained and assessed, WEB-1, the product collection process was similar. Instead of a “section front,” we collected the first five stories to appear in, next to, or below the website’s homepage “target box” on a given day. As with the newspapers we assessed, we identified an individual reporter who appeared to have been especially impacted by the CCJ workshop and collected a sample of his pre and post workshop stories.
The assessment team collected and analyzed 2,208 print and Internet stories. The pre and post workshop product samples were nearly identical in size – 1,098 stories from before the training and 1,110 from after. Of the 2,208 stories collected, 384 were from the individual reporters we identified from each newsroom. The rest were from Local/Metro and Front section stories from various authors.
Broadcast station product was a bit trickier to obtain than newspaper and Internet product. The ideal broadcast product to collect is video of newscasts, since that is what viewers actually see and experience. Unfortunately, we learned that few local television stations have the capacity to store video copies of their newscasts for long periods of time in such a way that they’re easily and affordably accessible.
We experimented with collecting newscast “director scripts” which include the dialogue that newscast anchors read and details about the visuals that accompany stories, but a single half-hour newscast script can run more than 100 pages long. It wasn’t practical to ask newsrooms to print out weeks of these scripts – the few times we asked for them it took hours to print them out, which tied up employees’ work time and depleted paper supplies and stations’ goodwill.
We settled on collecting newscast “rundowns” of each assessed station’s “broadcast of record.” A “rundown” is a condensed list of the stories that appear in a newscast with information about the order in which they ran, the manner in which they were reported – commonly referred to as their “treatment” – and how long they ran, among other things. A rundown does not contain the “text” of a story, but does include a brief phrase describing the content of each story.
We collected rundowns from 11 local broadcast stations. As with print and Internet, the sampling period we aimed for was two weeks of product from before the workshop, and two weeks from six to 18 months after the workshop. The 44 weeks of rundowns collected consist of 295 newscasts containing more than 5,000 stories. We did not collect individual reporter stories from the broadcast stations we assessed.
[top]
3. Newsroom surveys
Starting with the Traveling Curriculum’s first training visit to PR-3 in February 2001, Curriculum coordinators have administered staff surveys in advance of their workshops. The initial aim of these surveys was to provide trainers with an idea of a given staff’s morale, perceived job challenges, strengths, weaknesses, and routines. Curriculum staff used survey responses to customize workshops to the needs and aptitudes of each organization.
Importantly, these surveys also initiated contact with the staffs of each organization to get them thinking reflectively about their routines and product quality. They introduced CCJ and informally primed staff for the training they would soon receive.
Initially, staff surveys were administered in paper form and respondents could mail or fax them back to the Traveling Curriculum Coordinator for analysis. This process proved cumbersome and time-consuming, so the surveys were translated into an online format that is still used today for print workshops.
Broadcast Traveling Curriculum Director Wally Dean found that, even today, many local television staff – primarily photographers, field staff, and studio staff – have limited Internet access. So he continues to administer and collect paper surveys from the stations at which he coordinates and conducts workshops.
The wealth of information contained in the responses to these pre-workshop surveys has been invaluable to workshop trainers. Having a system already in place for surveying newsrooms also created a natural research opportunity for the assessment team. We re-administered similar versions of these surveys in advance of our assessment visits. Survey responses were then used to aid in determining how newsroom attitudes and practices may have changed in the year or so after a CCJ workshop.
We analyzed the pre and post workshop survey responses of the newsrooms we assessed. This survey sample included 695 pre-workshop surveys and 437 post-workshop surveys from 14 newspapers. It also included 382 pre-workshop surveys and 261 post-workshop surveys from 13 local television stations. In all, we analyzed 1,775 surveys.
[top]
4. Selecting organizations to assess
The earliest newsrooms assessed had participated in a workshop within the past 16 months. We tried to vary the time lapse between initial workshops and assessment visits within a six to 16-month range, hoping to anecdotally determine how much of a factor time played in the “freshness” of training lessons. The earliest we returned to a news organization for an assessment visit was six months after the initial workshop. The longest gap between an initial workshop and an assessment visit was 32 months.
The best time for an assessment visit seemed to be approximately a year after the initial workshop. By that point, enough time had passed that the initial excitement from the workshop had worn off. Interview respondents were left with the lessons, examples, and tools that were most salient for them personally. These things were, by that time, more likely to be a part of their routines and thinking. Enough time had passed that if suggested changes were being implemented on the newsroom-level, interview respondents would be aware of them and able to make a judgment about their effectiveness. We also reasoned that measurable changes in product quality that could be directly or indirectly attributed to the Traveling Curriculum workshop lessons were likely to engrain themselves in a year’s time.
Also, we found it was important to provide adequate time between administering our pre and post-workshop staff surveys. Some organizations we assessed hesitated to allow a re-surveying of their staffs within a year, citing “survey fatigue” and a drain on their workers’ time. Plus, administering an almost identical survey within too short a time period after an initial survey introduces concerns about a “Hawthorn affect” – where respondents tell surveyors what they think they want to hear because they become more aware that they’re part of an experiment or study.
Besides timing, the next most important factor in determining which newsrooms we assessed was trying to include newsrooms with a mix of staff size, audience size, geographical location, and ownership characteristics. We also hoped to include newsrooms that had been exposed to a variety of Curriculum modules.
The 41 newsrooms that ended up in the assessment sample are diverse in these areas. Of the 24 newspapers assessed, seven have staffs over 1,000, four have staffs between 500 and 1,000, eight have staffs between 200 and 500, and the remaining five have staffs under 200. Of these newspapers, seven have daily circulations over 200,000, five have circulations between 100,000 and 200,000, five have circulations between 50,000 and 100,000, and the remaining seven have circulations under 50,000.
Geographic diversity was attained to some degree in selecting newspapers to assess as well. The geographic breakdown of assessed newspapers went as follows: Eight newspapers from the Western U.S., seven from the Midwest, five from the Southeast, three from the Northeast, and one from the Southwest.
In terms of ownership, the assessed newspapers break down this way: Five newspapers are owned by local companies or trusts, four are owned by regional chains, and 15 are owned by national chains. Of the 15 owned by national chains, one is owned by Gannett, three were owned by Knight Ridder, four are owned by McClatchey, one is owned by Media News Group, one is owned by the New York Times Regional Newspaper Group, two are owned by Scripps Howard, two are owned by the Tribune Corp., and one is owned by the Washington Post Company.
Similar efforts at diversity were made in selecting local television stations to assess. Of the 15 broadcast stations in the assessment sample, five are in one of the largest 35 markets in the country, six are in markets that fall between the 36th and 75th largest markets in the country, and the remaining four are between the 95th and 120th largest markets in the country. Local television staff sizes don’t differ significantly from large to small markets. The largest station assessed had only 40 or so more newsroom staff than did the smallest station assessed.
Geographically, the local television stations assessed break down as follows: Six stations from the Midwest, six stations from the Southeast, two stations form the Southwest, and one station from the Northeast.
In terms of ownership, the assessed local television stations break down this way: Local companies own five, and national chains own the remaining ten. Of the ten chain-owned stations, one is owned by Gannett, one is owned by Granite Broadcasting, two are owned by Hearst-Argyle, two are owned by Media General, two are owned by the New York Times, one was owned by Raycom Media, and one is owned by Viacom.
[top]
A summary of the findings from an intensive three-year assessment of the impacts of CCJ's Traveling Curriculum newsroom training program. This assessment was conducted by Dr. William Damon and Brett Mueller.
Evolution of the Traveling Curriculum
Defining Features
Training Statistics
How We Assessed Training Outcomes
TRAINING IMPACTS/OUTCOMES
Characteristics of Successful Training Programs
About the Researchers
In 2001, with support from the Pew Charitable Trusts, the Committee of Concerned Journalists launched a program of intense training workshops designed to bring new learning opportunities to journalists working in the field. The John S. and James L. Knight Foundation joined in support of the program’s implementation and assessment in 2003.
The CCJ training takes place in journalists’ own newsrooms: for this reason, the program is called the “Traveling Curriculum.” Because the workshops “travel” to news organizations, entire newsrooms can experience the workshops together. All levels of staff, from managerial to reportorial, participate as peers in the workshop discussions. Indeed, it is a requirement of the CCJ training that all staff levels must participate.
The workshops consist largely of small-group exercises and problem-solving, with occasional formal presentations that convey the essential tools and methods of good journalism. The curriculum covers core ethical standards, but it is not limited to ethics; it covers the entire spectrum of good journalism practices, such as how to write interesting stories, how to ensure accuracy of reported information, how to expand a newsroom’s target audience, how to uncover information for an investigative report, and how to plan a feature series.
For print newsrooms, the workshop program takes a day-and-a-half. During that time, the program delivers three “modules” that have been pre-selected by the newsroom from a set of 12. Module topics for print newsrooms include verification, bias, conflict of interest, engagement (or making news stories interesting), proportionality, independence, and watchdog. For broadcast newsrooms, two abbreviated modules (news judgment and engagement) are offered in a compressed, half-day workshop.
The workshop format of small-group exercises and frequent interactive dialogues was chosen because it is known to foster critical thinking and deep understanding. The workshop exercises are implemented in a Socratic manner, with trainers asking thought-provoking questions and guiding the discussions towards solutions that emphasize the importance of key standards and demonstrate useful new strategies.
For a more detailed description of the Traveling Curriculum and the pedagogy that supports it, click here.
Defining Features of the Traveling Curriculum
The defining features of the CCJ workshop program are: 1) its location in the home community of the newsroom, 2) its inclusion of all levels of newsroom staff, 3) its emphasis on reflection and critical thinking, 4) the intensity of the discussions that it promotes, and 5) its emphasis on active participation in its training experience. This particular combination of features is unique in journalism education.
For more information on the defining features of the Traveling Curriculum, click here.
[top]
Training Statistics
The CCJ Traveling Curriculum program was launched in February 2001. As of June 2006, the program had conducted over 300 sessions at more than 120 print, broadcast, radio and Internet newsrooms, reaching over 7,300 journalists who work in these newsrooms. In addition, workshops have been offered at the National Press Club and other professional associations; to journalists in Europe, Asia, and Latin America; and to student newspapers in U.S. colleges and universities.
For a more detailed breakdown of Traveling Curriculum training statistics, click here.
[top]
How We Assessed Training Outcomes
A formal assessment of the impact of the CCJ Traveling Curriculum began in July 2003. The main assessment instruments were: 1) One-on-one interviews with those who participated in the workshops, 2) staff surveys administered before and after workshop participation, and 3) samples of “product” (news stories, broadcast segments) collected before and after workshop participation. Between July 2003 and September 2005, the assessment team traveled to 34 of the newsrooms that had participated in the workshops to conduct interviews and collect surveys and samples of news product. Of these 34 newsrooms, 17 were newspapers, 15 were local television stations, one was an online organization, and one was a student newspaper. During these visits, the assessment team interviewed 549 workshop participants and collected over 1,800 staff surveys. The team also collected and analyzed nearly 2,100 newspaper stories and 300 local television newscasts.
For more details about the assessment methodology and demographics of the organizations assessed, click here.
[top]
TRAINING IMPACTS/OUTCOMES
The assessment established the following outcomes, many of which are directly attributable to the unique features of CCJ program:
1. An extremely high level of positive feeling regarding participation in the workshops: In post-workshop evaluations, over 95% of participants gave the workshops the highest available ratings. This indicates that most participants found the training to be engaging and believed that it was a worthwhile use of their time. The small-group format, the opportunity to discuss their journalistic mission and standards, and the chance to work collaboratively with newsroom colleagues, all are enjoyable, meaningful, and all-too-rare experiences for today’s journalists.
For more details about participants high level of positive feeling about workshop participation, click here.
2. A strong and positive impact on participants’ sense of purpose and morale regarding their journalism: Participants reported having a clearer sense of their mission after the workshops and found that to be a morale-booster. Mission and purpose are central themes of the CCJ curriculum: every module offers participants multiple occasions for discussing the reasons why they chose to become journalists, which is a naturally morale-enhancing experience.
For more details about increases in participants' sense of purpose and morale, click here.
3. Marked effects on the willingness and ability of participants to use critical thinking in their journalism: The intense group discussions foster critical thinking skills by forcing participants to reflectively examine every aspect of their practice and to think skeptically about their habits of researching, sourcing, verifying, editing, and writing stories.
For more details about participants' increased willingness to engage in critical thinking about their journalism, click here.
4. Improvements in quality of news product: Several measures of news story quality in both print and broadcast newsrooms showed strong improvement after the training. This finding is likely due to a closer attention to journalistic mission and an increased tendency to think critically and deeply about routine habits of practice, all of which the training demonstrably fostered.
For more details about improvements in product quality, click here.
5. Learning tools of good journalism: Participants reported learning a number of journalistic tools imparted by the workshops, including tools of verification, accuracy, investigative reporting, engaging and proportional story telling, and balance. The substantive core of the CCJ curriculum consists of the tools and other elements of good journalism. Participants in the training engaged in exercises that require mastery of the tools and listened to distinguished journalists present state-of-the-art strategies for researching and reporting the news.
For more details about participants learning the tools of good journalism, click here.
6. Improved communication within the newsroom: Three-quarters of all participants reported that inter-staff communications within their newsrooms had improved due to the CCJ workshops. Because the CCJ training visits newsrooms and engages staff at all levels in intense discussions, it offers participants multiple opportunities to become more familiar with one another (necessary for many workers who have felt isolated) and to correct malfunctioning communications patterns that often prevail in many newsrooms. Moreover, the continuing consulting that CCJ provides participating newsrooms after the workshops offers a source of ideas about methods that a newsroom can put in place to facilitate more productive and clearer communications among staff.
For more details improvements in newsroom communications, click here.
7. A proliferation of newsroom innovations dedicated to fostering better journalism practices: The CCJ training left behind a legacy of self-improvement manifested by multiple innovations within many newsrooms visited. Because CCJ training focuses on the particular needs and mission of each newsroom that it visits, it can provide direct feedback regarding changes that could advance that particular newsroom’s purposes. Regular follow-up contacts, either through second visits, e-mails, or phone calls, offer opportunities for such feedback, providing rich contexts for the generation of many innovative solutions.
For more details about newsroom innovations, click here.
[top]
Characteristics of Successful Training Programs
In addition to these findings regarding the impact of the CCJ training program on journalists and their newsrooms, the assessment pointed to ways that mid-career training can be made most effective. The principles underlying the CCJ program were:
Training must address issues that the participants themselves recognize as important for doing their jobs well.
Training must actively engage the participants in collaborative exercises.
Training must be vigorously supported by the leadership of the organization.
Training must challenge the participants with difficult, cutting-edge problems.
Each of these principles was confirmed by the Traveling Curriculum’s successful reception in the field. In addition, the CCJ field experience revealed the importance of having lead trainers who themselves had been distinguished journalists, for the sake of adding credibility and texture to the training.
Moreover, the value of regular follow-ups by trainers and CCJ staff after the initial visits was demonstrated in multiple instances. The follow-up provided occasions to reinforce the workshop message, to guide the staff towards needed newsroom innovations, and to keep alive the spirit of self-improvement fostered by the workshops.
[top]
About the Researchers:
Dr. William Damon is Director of the Stanford Center on Adolescence and Professor of Education at Stanford University. He is co-author of Good Work: When Excellence and Ethics Meet (2001), with Howard Gardner and Mihaly Csikszentmihalyi. They collaborated on the Good Work Project (GWP), which studies exemplary leaders and practitioners known in their professions for doing work that is both successful in the usual terms of the field as well as highly ethical. Damon applied findings from this project in consulting with CCJ Founders Bill Kovach and Tom Rosenstiel on the creation of the Traveling Curriculum.
Brett Mueller managed the collection, organization, and analysis of assessment data.
Paul Hitlin vetted the newspaper and broadcast product codebooks, trained student coders, and ran the statistical analyses.
LaToya Drake, Mahvish Khan, Judy Mayka, and Ashley Spillane coded newspaper stories, television broadcasts, and staff survey responses.
Click here for a link to a more detailed version of this report.
[top]
Details from a comprehensive report outlining the findings from a three-year assessment of the impacts of CCJ's Traveling Curriculum newsroom training program.
Engagement in workshop experience
Purpose and mission
Reflectiveness and critical thinking
Quality of news product
Use of tools related to particular elements of journalism
Communication
Innovations
Engagement in workshop experience
We expected that participation in the workshops would be viewed as a positive experience, since it provided an opportunity to revisit the fundamental purposes that draw most journalists to this line of work. Most newsrooms never break away from their daily routines long enough to reflect on the central issues in journalism, and mid-career training in this field has been available only sporadically to a chosen few. For this reason alone, serious efforts to bring training opportunities to newsrooms are likely to meet with widespread staff approval, and this is exactly what we found.
Consistently, in the vast majority of print and broadcast newsrooms that we visited, participants reported that their experience of engaging in the workshops was strongly positive. In post-session evaluations, when participants were asked to rate the overall workshop experience from 1 (outstanding) to 5 (poor), over 95% of respondents gave the workshops the highest ratings. This is an extraordinarily high proportion of high ratings. Even considering the inevitable “Hawthorn effect” referred to above, the 95% figure reflects an extremely positive overall response to the program.
Generally, participants responded that they found the workshops to be stimulating, informative, worthwhile, memorable, enjoyable, and thought-provoking. Typical is a comment of a reporter from a mid-sized newspaper: “Interesting and more useful than any other conference I have attended.” One journalist from a larger newsroom wrote: “The discussions of issues...were lively, engaging, and struck at the heart of what we are or need to be thinking about as we do our jobs.”
[top]
Purpose and mission
The most consistent theme of all the workshop modules was the importance of always keeping the public mission of journalism in mind; and all the workshops offered participants multiple occasions for reflecting on the most fundamental purposes of their work. For this reason, we expected marked impacts on participants’ awareness of the newsroom mission and their own journalistic purposes.
An analysis of interview data revealed strong and consistent effects on participants’ sense of journalistic mission and on their inclinations to care about the broader purposes of their work. These outcomes are connected to the Curriculum’s first goal, reinforcing the dedication of newsrooms and those who work in them to journalism’s traditional public mission. Accomplishing this mission requires an understanding of how journalism serves the public in ways that help them make informed choices about their lives and their roles as citizens in a democratic society.
71% of respondents, aggregated across all newsrooms, reported having a clearer sense of journalistic mission after the workshops. The print and broadcast newsrooms reported virtually the same levels of effect (71% of participants for print, 74% for broadcast), with only the Internet newsrooms reporting a somewhat lower effect (49%).
[top]
Reflectiveness and critical thinking
Another primary goal of the curriculum was to impart to participants the importance of critically examining their sources of information, the nature of their own biases, and, ultimately, the quality and validity of their news judgment. For this reason, we expected marked effects on the critical thinking capacities of participants. Critical thinking capacities are essential in participants’ abilities to make sound news judgments.
The interview data revealed clear effects on the willingness and ability of participants to use critical thinking in their journalism. The majority of respondents reported that they, or their newsrooms, had become more reflective about their work as a consequence of the workshops. They reported a greater tendency to apply critical thinking to their stories, in both the research and the writing phases. 58% of all respondents reported this change. Overall, aggregating responses for all varieties of newsrooms, 27% of participants said that bolstering their critical thinking and reflectiveness was the most valuable benefit of the workshops for them.
Print participants reported the critical thinking effect at higher levels than broadcast participants and at markedly higher levels than Internet participants: the numbers were, respectively, print 70%, broadcast 51%, and Internet 26%. Although broadcast newsrooms showed a bit less impact than print newsrooms in their critical thinking dispositions, they showed a strong effect in their capacities to make sound news judgments. 44% of broadcast participants reported making better news judgments subsequent to their workshop experiences. News judgment certainly includes the capacity for critical thinking, but it also refers to priority-setting in assigning, airing, and ordering stories, decisions about how much context to provide viewers, and a host of other choices regarding emphasis, proportional coverage, controversial content, and balance. Among broadcast participants, 64% said that the ability to make sounder news judgments was their most valuable takeaway from the workshops. Many participants from broadcast news attributed their improvements in news judgment to the “story stacking” exercise that played a central role in the broadcast training (see description of this exercise above). At every station, someone mentioned the particular value of this method, and over 40% of the broadcast participants spontaneously noted during their interviews that it was the most memorable part of their experience.
[top]
Impact on quality of news product
The most important consequence of clear purpose, critical thinking, sound news judgment, and high morale is the quality of the news stories that journalists and newsrooms actually produce. We expected to find improved quality in the actual reporting of the newsrooms that we visited. As noted above in the methods section, in order to assess quality of reporting, we combined two methodological strategies. First, we interviewed participants and managers about their perceptions of quality before and after the workshops. Second, we examined the actual “product” – print stories and broadcast rundowns – of the newsrooms before and after the CCJ visits. This kind of product analysis has rarely, if ever, been done in assessing the impact of journalism training programs. Results from both methodological approaches confirmed that the reporting product of participating newsrooms improved markedly after the CCJ visits.
1. Interview assessments of product quality
The most evident consequence of strong mission, good news judgment, and high morale is the quality of the news stories that journalists and newsrooms actually produce. The section on product analysis below reports direct assessments of newsroom products before and after the CCJ visits. The interview data, in contrast, offer indirect indicators, consisting of participants’ perceptions of changes in product quality due to the workshops. In all newsrooms combined, 31% of workshop participants reported improved quality in their newsroom’s news products. The effect was largest for medium-sized print newsrooms, where 52% of participants stated that the quality of their newsroom’s reporting had markedly improved due to the workshops. In those newsrooms, unlike in the larger ones, most or all staff members were able to attend the workshops; and, unlike in the smaller papers, there were enough staff to arrange multiple versions of small-group discussion sessions. Unlike in the broadcast training, the print newsrooms were offered full day-and-a-half workshops that were able to explore three modular journalistic topics in depth.
2. Direct assessments of product quality
The analyses of the product collections referred to above yielded a number of direct indications of improvements in the quality of news stories after the CCJ visits. For the print newsrooms, a sample of work done by individuals who participated in the workshop training and were designated by their newsroom managers as highly engaged in the workshop experience showed clear and consistent improvement trends. This designation did not necessarily set them apart from most other staff who participated in the training since, as noted above, over 95% of total participants in the CCJ workshops gave the workshops top scores on their overall ratings for value and interest.
Print product analysis
At eleven of the newspapers where we assessed work of individual reporters, the quality of work improved significantly subsequent to the CCJ workshops. Breaking down the quality assessments into sub-components, the work of the individual reporters whose work was analyzed showed improvements in the following areas: explaining the future impact and context of the news event; writing articles without inserting personal opinion; writing articles that gave the reader multiple perspectives on the conflict described by the story; and explaining in the story what is not known as well as what is known. For each of these quality components, no individual work declined in score from pre to post-test – all stories showed either positive change or none at all. Regarding the careful and limited use of anonymous sources, and the use of transparency in communicating story background to readers, there also was net improvement subsequent to the workshops. At three of the papers, on the other hand, participants’ work showed little or no improvement; and at two of the papers the assessed quality of work was scored lower by a small degree after the workshop experience. The work of individual reporters at one newsroom (not the same newsroom for the two instances) showed negative rather than positive change on the sub-components of quality noted above.
As could be expected, the print newsrooms taken as a whole showed no clear pre/post improvement trends, no doubt because of the inclusion in the product analysis of many stories written by reporters who had not attended the workshops. But overall, the analyses of individual reporter’s quality of work in the print newsrooms assessed yielded more direct indicators of pre/post improvement than has been reported in most prior evaluation studies.
We examined the work of two individual reporters as case illustrations of these general trends. For the sake of this report, we shall refer to these as PR-16 and PR-17. Both reporters work for mid-sized newspapers, one on the East Coast and one on the West Coast.
PR-16 is an enterprise reporter who covers a wide range of news and feature topics, including political, crime, environmental, and human interest stories. She is a talented writer who composes stories in an extremely engaging and literate manner. The stores that we collected from the periods prior to and after the CCJ visits all reflected her highly engaging writing style, with no noticeable change in this dimension subsequent to the visits. But on other product quality indexes, PR-16’s stories showed marked improvement after participating in the CCJ workshops. Although her stories predating the CCJ visits received relatively strong quality scores on most of the indexes, subsequent to the visits they were scored significantly higher.
For example, PR-16’s sourcing practices directly reflected the discussions from the accuracy and verification module that CCJ presented at her newspaper. After the CCJ visit, PR-16’s use of anonymous sources declined significantly. Prior to the visit, she used anonymous sources in 36% of her stories, a moderate amount; but after the visits, she used anonymous sources in only 10% of her stories, a very low figure. Moreover, her sourcing showed greater “transparency” – that is, she more clearly identified not only the source but the source’s relationship to the events in the stories and the source’s capacity to provide reliable information. In a post-visit crime story, for example, she offered full titles and backgrounds of every government source and noted that the defendant’s attorney, also quoted, was state-appointed. In a science story, she provided titles and backgrounds for all scientists quoted and also pointed out that they had authored a paper that had taken a position on the issue that they were commenting about.
In her interview with us during our assessment visit, PR-16 said, “I’m definitely more thoughtful about sourcing and verification issues.” She also said that she now had “More awareness of our [newspaper’s] credibility all around.” These reflections are consistent with the improvements noted above. Interestingly, she also said that one of her most valuable takeaways from the CCJ visit was learning about “making stories more positive, more entertaining, fun and easier to read, more relevant to readers’ lives.” In fact, as we noted, PR-16’s pre-visit stories were already so high on these dimensions that it was impossible to detect any post-visit improvement. It may be that reporters with special strengths frequently look for ways to build further on these strengths. When this is the case, training became more than a matter of filling in deficits in skill – it becomes an opportunity to move further toward excellence in craft.
Finally, PR-16’s post-visit stories showed modest increases in conveying the potential impact of stories on readers’ lives and discussing their implications to the broader community. Her stories also showed a small (and not statistically significant) increase in their tendencies to include a diverse group of subjects, including more frequent mentions of children, young adults, senior citizens, minorities, and people with multiple religious faiths.
PR-17 is a metro and general features reporter from a mid-sized Western newspaper. His pre-CCJ visit work reflected a straightforward, “no frills” approach to news writing. He generally used simple declarative sentences that conveyed the necessary information, but were not particularly colorful.
The quality of PR-17’s work showed significant post-visit improvement in the area of telling readers “what is not known” about the topic or event being covered in a story. This improvement hints at a greater effort to be transparent with readers, a key idea in the conscience and communication in the newsroom module to which this reporter was exposed. “I feel comfortable explaining myself and see value in it,” this respondent said during his assessment interview. This statement came in an interview response in which he made the case that his newsroom could do a better job of explaining how the editorial process works and how decisions are made. His improvement in relating what is not yet known to readers shows he found a tangible way to translate that feeling into his work.
This reporter also showed marked, although not statistically significant, improvements in using fewer anonymous sources, telling readers about the potential implications of topics covered in his stories, and presenting multiple viewpoints in stories in which there was conflict.
In general, PR-17’s writing was noticeably “snappier” and more interesting to read subsequent to his participation in the CCJ workshop as well. The pre-workshop sample of writing from PR-17 can be fairly characterized as rather formulaic, reliant on institutional sources, and lacking in creativity – especially in the story leads. Several of the stories failed to move beyond isolated instances to connect to broader themes and trends in the community, state, region, or country. They weren’t “poorly-written” stories by any stretch. But the writing in most stories was not particularly compelling, detailed, or deep; and the author seemed to stop short in places where the typical reader may have wanted or needed more information to judge why she should care about the story.
The most striking feature of PR-17’s pre-workshop product sample is his over-reliance on a specific single source for information in several stories. Six stories on four different topics contained quotes from the same local elected official.
PR-17’s post-workshop stories displayed more interesting leads, a wider diversity of sources, and efforts to present multiple viewpoints on the central conflict (where there was one) of stories. Additionally, the “nut graf” in many of these stories was more visible and more thoughtfully written.
“When I do in-depth stories in particular, I reference the training,” PR-17 said during an assessment interview one year after the CCJ workshop. “Especially on the concept of ‘stakeholders.’ Am I getting them? How does this story affect people? I try to do extra fact checking and get back to sources and make sure the tone [of the story] is appropriate.” This conscious effort to make stories more relevant and thorough, among other things, appears to have helped PR-17 improve the overall quality of his writing.
Broadcast product analysis
In the broadcast newsrooms assessed, the analyses found improvements in quality and treatment of first block stories. The average length of first block stories increased significantly – a noteworthy finding, since prior research has established a strong positive correlation between length of story segment and quality in broadcast news. In addition, subsequent to the CCJ workshops, there was a greater tendency for broadcast news stations to present stories – and, in particular, their first-block stories – in edited packages. This was noteworthy because edited packages reflect deeper and more contextualized treatments of information offered in the stories. Moreover, we found a greater tendency for broadcast stations to present campaign election stories in edited packages as well as serious lifestyle stories involving the lives and deaths of political leaders. In some of the newsrooms, less time was devoted to live crime reporting and more to live reporting of civic and political news.
Improvements in quality and treatment of first block stories were most pronounced in the broadcast stations that we examined for case examples, due to the seriousness with which these particular stations engaged in the training. In addition to the improvements noted above, we found in some of these stations an increase in live shots and packages in stories related to civic and political issues and a decrease in live shots and packages in stories related to crime. Both of these changes imply improvements in the quality and seriousness of the stations’ new coverage.
In general, our assessments of news product quality before and after CCJ training showed a marked impact on individual reporters in print newsrooms but not much effect on entire print newsrooms. In the broadcast training, in contrast, we found a number of positive effects at the whole-newsroom level. Part of this difference may be attributable to a methodological issue: the print product that we collected at the whole-newsroom level necessarily included many stories that were written by reporters who had not received the training; whereas in broadcast, where virtually everyone participated the training, the product that we collected almost always was done by people who had been trained.
In addition to this methodological artifact, it may be that whole-newsroom effects are easier to obtain in broadcast than in print (see the section “Impact on newsroom communications” below, where we report a similar whole-newsroom impact favoring broadcast over print). This difference might stem from a number of causes, including the more entrenched practices of print journalism, which evolved over many decades before broadcast news had been in existence. The collaborative nature of creating broadcast product could further magnify this difference.
One reporter, for example, might remember a fragment of a training lesson and refer to it while discussing with colleagues problems in reporting a complex story properly. This fragment can spark memories of the shared training experience among the photojournalist, the producer, the tape editor, and others who are working on the same story; and together they may retrieve enough of the training idea to solve the story problem. This scenario is more common in broadcast than in print, where reporters tend to work through story problems that they encounter without much consultation with colleagues.
BR-11 is a moderate sized Midwestern news station, and BR-3 is a strong broadcast station in a highly populated Southern community. Analysis of the newscasts aired by BR-3 and BR-11 before and after their CCJ workshops revealed that these stations had improved in several important areas after the CCJ workshop.
We coded these stations’ newscast rundowns for story treatment – a measure of the manner in which the story was produced and appeared to viewers. In order to do this, we created a four-code treatment “hierarchy” in which those story treatments that required the most reporter effort were at the top of the hierarchy and those that required the least were at the bottom.
An “edited package” typically requires the most reporter effort. This treatment requires a reporter and photojournalist to spend time in the field capturing video and sound, generally produces a longer piece, and can require extensive editing time to allow for all of the relevant footage to be thoughtfully organized into a coherent story.
Next in the hierarchy came the “SOT,” short for “sound on tape.” A reporter and photojournalist spending time in the field capturing video and recording sound, usually at a single location, characterize this treatment. This type of story is typically shorter than an edited package and requires considerably less editing.
The third treatment in the hierarchy was the “VO (short for ‘voice over’) with moving images.” This treatment requires a photojournalist to capture video in the field, but does not require the capturing of sound from the field. Generally, a reporter will not accompany a photojournalist into the field for this type of story. A reporter can record sound in a studio in the newsroom to play over the video, or a desk anchor can speak while the video is playing during the newscast.
The final treatment in the hierarchy was the “reader without moving images.” This treatment requires little or no field reporting and appears during the newscast as the anchor briefly telling the viewers the story without any accompanying video.
The newscasts of BR-3 showed marked improvement in a number of important areas. The proportion of total newsroom stories with edited packages more than doubled, a dramatic increase directly affecting the quality of presentation. The proportion of BR-3’s newsroom stories that employed live shots almost tripled, another direct indication of greater effort at quality improvement. Moreover, there was a noticeable shift in BR-3’s focus of coverage, from entertaining and sensational topics to more weighty civic matters. For example, BR-3 decreased the proportion of time devoted to accidents, disasters, and sports and increased its coverage of governmental issues in first-block stories by almost 50%. In general, BR-3’s first-block story treatments reflected many of the suggestions imparted during the CCJ workshops: there was a dramatic increase in average length of first-block stories (from an average of 35.5 seconds to an average 57.25 seconds); and the proportion of first-block stories that were presented as edited packages more than doubled (from 13.3% to 32.7%). As we have noted above, both types of changes have been shown to be highly correlated with a broad range of quality indicators.
As noted above, BR-11 is a moderate sized Midwestern news station. The percentage of BR-11’s post-workshop newscast stories that were SOTs increased from 8% to 22%. The percentage of post-workshop stories that were “readers without moving images” fell from 32% to 26%. This is a positive development, as stories with more intensive treatments are generally more substantive and of higher quality.
BR-11’s post-workshop newscasts featured less time devoted to crime stories, particularly during the all-important first block of newscasts. BR-11 devoted 15% less time to first-block crime stories during its post-workshop newscasts. It devoted 6% less time to crime stories across its entire newscasts.
Despite the decreased amount of time devoted to crime stories in BR-11’s first block stories, the average length of all of its first block stories increased by nine seconds after the CCJ workshop. In We Interrupt This Newscast, Dean and colleagues found the average local television newscast story to run between 60 and 65 seconds, which makes nine seconds a comparatively sizable change.
BR-11’s post-workshop newscasts contained almost 14% fewer live shots on stories related to crime. A respondent from BR-11 noted that the BR-11 news director “put more pressure on producers to understand the stories [in their newscasts]. They [were not permitted] to just go ‘live for the sake of live’ anymore,” said this respondent.
This “live for the sake of live” comment hits on a key lesson shared during broadcast workshops. Dean and his co-trainers teach that live shots can be invaluable for offering viewers a sense of immediacy and “place.” However, they also warn that live shots can be misused to place a reporter in front of a dark building where something happened hours ago in the misguided belief that a sense of immediacy can be artificially recreated.
As noted above, BR-3 used of live shots in almost three times as many stories after its CCJ workshop as before. BR-3 posted 21% increases in its use of live shots on both government and election-related stories. While we can’t infer as to the quality of these live shots (due to the limitations of our product analysis, we were not able to see the live shots and thus make any judgment about their quality), the mere fact that they did so many more on topics that typically aren’t “live shot-friendly” offers hope that the station was using its live shots to add creativity and meaning to the telling of those stories.
While BR-3 was increasing its use of live shots all-around, BR-11 halved its use of live shots. And, as noted above, the station used live shots in 14% fewer crime stories than before the workshop. As the quote above suggests, going “live for the sake of live” may have been an issue in this newsroom before the CCJ workshop. The more sparing use of live shots, especially on crime stories – which have a greater likelihood of acquiring a “live from in front of a dark building where nothing is happening” look to them – may have been a compensatory effort to use resources more responsibly and thoughtfully after a period of misuse in service of live shots.
CCJ teaches that the live shot technique is not inherently good or bad – live shots with a thoughtful purpose and high level of execution that clearly add substantively to the telling of stories can be good regardless of story topic.
[top]
Impacts on particular elements of journalism
1. Making important stories interesting
One of the most popular modules that newspapers selected for their CCJ workshop is the engagement and proportionality module. This module proposes that a key task of journalism is to make the important interesting. Participants exposed to this module are asked to debate the supposed dichotomy between what readers want to know and what they need to know. Is there an inevitable conflict or can journalists do both? Participants are asked to review sections of their own newspaper and consider how what is written reflects the community. Who is included in their stories? Who is left out? The module offers ideas on how to make stories more relevant and pushes participants to engage in dialogue about the community they serve.
There are elements of this module in the broadcast workshops as well. Instead of reviewing sections of a newspaper, participants break into groups and must collaboratively decide how to best organize a mix of nine hypothetical stories into a newscast. These hypothetical stories are characteristic of what one might see in a typical local television newscast – an anthrax scare at the local post office, a bomb hoax at the airport, a cut in street repair funding, a live announcement by the mayor on security, a cut in education funds, a dead body found in an “exclusive” neighborhood, the threat of war in Pakistan, a medical story on coping with pain, and a runaway elephant at the zoo. The groups are tasked with “stacking” these stories in a way that reflects their significance and potential interest to viewers - this is the broadcast “stacking exercise” referred to above.
Interview respondents from newspapers, local broadcast stations, and the Internet newsroom exposed to this module consistently cited lessons from this module as key workshop takeaways.
A full 65% of the print and Internet interview respondents exposed to the engagement and proportionality module, and at least one respondent from each newsroom that requested it, said lessons from that module impacted them or their newsroom in some way. These impacts ranged from provoking staff and managers to interact with their readers more frequently; to devising systems for capturing reader comments; to writing, editing, and laying-out stories in ways that make them more relevant, creative, and interesting.
Interview respondents from case example newspapers PR-16 and PR-17 reported high degrees of impact from their exposures to this module. 88% of PR-16 respondents and 70% of PR-17 respondents said lessons from this module left lasting impressions in their newsrooms. One PR-16 interview respondent described the impact of this module in the following way: “‘Who are we impacting?’ I am now asking myself this with every story. I try to cover all the angles [and] get all the points of view.” Another PR-16 respondent said there’d been a big improvement since the CCJ workshop in the area of writing stories from a more reader-oriented and less “institutional” perspective. “[Our writing] is more engaging [and] very conversational,” this respondent said. “It makes you want to read on.” Other respondents noted that “Why should anyone care” has become a common phrase in the newsroom. One respondent said a line has actually been added to the story budget template to focus reporters’ attention on that question from the outset of their reporting.
Respondents from PR-17 suggested that some fundamental changes in the way stories were conceived and reported after the CCJ workshop grew out of the engagement and proportionality module. One PR-17 respondent described how lessons from this module changed the way stories are planned in his section. “[Now] planning involves the reporter getting an idea and working with the editor to develop a reporting plan [and] set up reporting dates and a coverage strategy. Reporters are encouraged to think about how to make ideas matter to people beyond a single story.” Other respondents described how the newsroom instituted new systems for engaging with readers, including an “email Rolodex” of community members that can be queried for story ideas and feedback and a new weekly “Community Extra” section, the purpose of which one respondent said was “to embrace the ideal of being responsive to readers [and] reflect them in the paper.” One respondent said the messages from this module caused him to “think more about getting out into the community to get the best sense I can of how people view things. I do more info gathering in the community now than I used to.”
37% of broadcast respondents cited impacts related to the parts of the broadcast curriculum dealing with engagement. Specific impacts reported include: providing “beat days” to reporters to allow them to cultivate new sources, meet with old sources, and dig up more diverse and interesting story ideas; holding “reporter meetings” for staff to share reporting techniques; and working harder to avoid cliché statements in reporting.
Station BR-3 started providing beat days to reporters and running promotions at the close of newscasts encouraging viewers to email or call the station with story ideas and feedback. “I call new people,” said one BR-3 respondent. “Now I’m consciously trying to diversify my contacts [and] get to different places.” Another respondent added the station is “attacking topics that most stations wouldn’t because they’re not very ‘sexy.’” This respondent added that in the broadcast environment, taking those kinds of risks can pay off. “If a story works out, others will follow us and run it [too].”
28% of print/Internet respondents and 29% of broadcast respondents said engagement and proportionality module lessons were the most valuable takeaway from the CCJ workshops in which they participated. One PR-16 respondent said the story brainstorming exercise and the exercise where groups dissected different sections of their paper for ways to make them more interesting were the highlights of the training for him. “[The exercises] got me so fired up to be more vigilant and less complacent.” A respondent from BR-3 added that the engagement aspects of the broadcast training taught him to not “always assume that what appears to be a big story of the day is the one that appeals to most people. How engagingly stories are written is the key to a focused newscast.”
[top]
2. Accuracy and verification
As noted above, public skepticism about journalists’ abilities and willingness to practice tools of accuracy and verification is one thing currently undermining the media’s credibility and relevance. CCJ’s accuracy and verification module introduces the idea that technology has made information easier to get and transmit, which has had implications on journalists’ traditional roles as “gatekeepers.” The module encourages journalists to examine their methods of reporting and verifying information and presses them to think about whether they are getting at “the truth about the facts,” as the Hutchins Commission put it 50 years ago. It offers concrete suggestions for enhancing credibility with readers and improving accuracy, including checklists and advice on how to handle anonymous sources.
The broadcast curriculum does not include an accuracy and verification module per se, but discussions of accuracy and sourcing are informally threaded into the engagement and news judgment lessons.
74% of respondents exposed to the accuracy and verification module, and at least one respondent in every newsroom where it was taught, cited some impact from its lessons after the workshop. These impacts frequently involved the individual or newsroom-wide use of CCJ-selected tools, including: increased transparency with staff and readers, use of accuracy checklists, use of the “skeptical editing technique,” and use of the “red pencil technique.” 25% of respondents exposed to this module said its lessons were the most memorable part of their workshop experience.
47% of respondents exposed to the accuracy and verification module reported increased efforts at transparency after the workshop. 45% reported individual or newsroom-wide use of accuracy checklists. 36% reported at least occasional use of the skeptical editing technique. And 25% reported individual or newsroom-wide use of the red pencil technique.
Being “transparent” with readers means making it clear to them how you know something – such as why an anonymous source wasn’t identified, how a source came to know some information, and what is still unknown about a story. Respondents often cited the use of editor’s columns or notes, occasional stories on how the editorial process works, and stories about how a controversial story was written as examples of transparency impacts.
“Accuracy checklists” are posted or mental checklists reporters and editors can use to help them focus on what they’ve already made sure is accurate and what may need further verifying. Some respondents said CCJ workshops inspired them to take the checklist idea further to encompass not only accuracy, but the entire reporting process.
“Skeptical editing” involves an editor going through a writer’s story line-by-line with her and playing “devil’s advocate” on any presumption or piece of information that can be questioned. A lack of time kept this from being a tool to be used on every story everyday in most newsrooms. Instead, respondents said it was especially useful on long, controversial, or complex stories.
“The red pencil technique” originated with Tom French, a Pulitzer Prize winning reporter for the St. Petersburg Times. French went through his stories line by line and with a red pencil and put a check mark on every fact and assertion. Every mark meant the fact or assertion was known or he had double-checked it. He didn’t turn his stories in until every fact and assertion had a check. Newsroom PR-23 took to this lesson wholeheartedly. During our assessment visit to this newsroom, we found signs posted in hallways and on bathroom mirrors encouraging staff to “Remember the Red Pencil Technique.”
Newsroom PR-16 modeled the internalization and implementation of accuracy and verification module tools and lessons CCJ hoped to achieve. 94% of assessment interview respondents in this newsroom said they’d changed their verification practices or noticed newsroom changes in verification practices since the CCJ workshop.
The most visible change respondents mentioned was the creation of an “accuracy and credibility group” comprised of newsroom staffers charged with analyzing and addressing corrections issues and innovating ways to keep colleagues focused on accuracy. The group developed a comprehensive staff manual that included not only accuracy checklists for each newsroom job position, but also a stylebook customized for PR-16’s specific beats and communities, a description of all newsroom policies, a “how to” guide on everything from writing obituaries to operating the newsroom computer system, and an emergency contact list and protocol.
[top]
3. Bias
The goals of the bias module are to aid participants in identifying predispositions towards certain biases and to offer strategies for identifying and avoiding inappropriate biases. Workshop participants exposed to this module are encouraged to think beyond political biases – the biases most often flagged by audiences as problematic. Workshop discussions move through political biases to general conflict of interest issues and questions of maintaining journalistic independence. The broadcast curriculum does not include a formal bias module, but ideas and lessons about identifying and dealing with biases are sprinkled throughout a typical broadcast workshop.
70% of respondents exposed to the bias module reported individual or newsroom-wide impacts from its lessons and ideas. 43% said exposure to this module was the most valuable part of their workshop experience.
Respondents at PR-16 noted that exposure to the bias module helped them increase their consciousness of subtle biases and implement checks on those biases. “[The workshop] made me really careful. Before the workshop, I didn’t consciously check myself for biases…now I do,” said one respondent, who added that probes about identifying potential biases appear in the newsroom’s new staff manual. Other PR-16 respondents mentioned that the training opened their eyes to the idea that biases can extend beyond politics.
Respondents mentioned controversial national issues such as the intelligent design debate and local issues such as a high-pitched debate over the use of new energy sources in their community as places where they needed to be more conscious of their biases. One astute PR-16 respondent added that he’d been made more conscious of what he called “logistical bias,” or the tendency to “do easier stories that require less work [and allow for the] use of sources who are easily available.”
[top]
4. Covering politics
Only a handful of newspapers have requested the covering politics module since its inception. The module discusses the purpose of political reporting, assesses participants’ own reflexes and routines, and offers a host of tools and ideas for improving political coverage. The concluding exercise invites participants to generate a new plan for covering an upcoming race. Again, there is no formal covering politics module in the broadcast curriculum, although aspects of reporting on elections are covered when relevant.
45% of interview respondents that participated in this module reported newsroom-wide impacts associated with its tools and lessons. These impacts included holding internal meetings to plan election coverage, soliciting election-related story ideas from non-newsroom staff, and trying to tell political stories in engaging ways without resorting to “canned” stories.
Newsroom PR-17 responded especially positively to its experience with the covering politics module. 26% of PR-17 interview respondents called the covering politics module’s lessons and tools the most valuable part of the workshop. “Our election coverage was more issue-focused and reader-oriented,” said one PR-17 respondent. “[The CCJ workshop] reinforced the importance of getting at what readers want to know. We surveyed readers about what was important to them.”
Another respondent added that the CCJ workshops “reinforced that we could plan to do things in a way that hadn’t been done before…be creative. We took things up a notch. Content, appearance, and delivery [of election news] was better than it’s been in the past.” In a nod to the crossover in module lessons, one respondent mentioned that the paper did a better job of “trying to maintain balance on issues and candidates,” and added that there were many fewer complaints about bias in their coverage of the 2004 election than in previous elections.
[top]
5. Watchdog journalism
Newspaper and broadcast assessment interview respondents frequently noted that one goal their managers had for improving content was to report more, and higher quality, investigative stories. The purpose of the watchdog journalism module is to get journalists thinking about subtle but important distinctions in investigative reporting, such as the differences between original investigative work and reporting on the investigations of others, or investigations into consumerism versus those of public institutions. Participants are given ideas on how to manage a major investigative project and asked to brainstorm a list of possible investigative projects. There is no formal watchdog journalism module in the broadcast curriculum; however, aspects of improving investigative journalism appear frequently in broadcast lessons and exercises.
44% of respondents exposed to the watchdog journalism module reported associated impacts. Responses featured digging deeper on stories to generate investigative angles, putting processes for generating and executing investigative story ideas into place, and other initiatives aimed at increasing the amount and quality of investigative content in their newspaper’s pages.
Newsroom PR-11 set the standard among newsrooms exposed to the watchdog journalism module in terms of the way that newsroom prepared for and incorporated module lessons. The executive editor of this newspaper told us bringing CCJ in to present the watchdog journalism module was part of a “distinct, spoken, deliberative objective to make investigative journalism a greater part of the newsroom culture.” The editor credited CCJ with “setting up the framework for the push on investigative journalism,” and remarked that setting up this framework was the most concrete impact of the training. Staff respondents clearly got the message. One remarked that the CCJ training was in line with a “greater emphasis on investigative journalism from the top-down.”
Several respondents remarked that the training reinforced that investigative elements can be incorporated into many different types of stories, and one respondent learned that investigative journalism wasn’t what he thought it was. His editor shared his story. “[One writer on our staff] saw investigative journalism only as ‘gotcha’ journalism. I didn’t know that’s how he looked at it. He learned from the CCJ workshop that that’s not how it has to be.” Importantly, the editor also noted that he learned that he shouldn’t take for granted that his staff defined certain concepts the same way he did. The editor followed up with staff that made comments during the workshop indicating that they weren’t on the same page so he could clarify things for them.
[top]
6. News judgment
Teaching workshop participants to identify the core principles embedded in the journalistic reflexes of their newsrooms and consider how they match up with each individual’s personal conscience about “doing the news” is a main goal of the broadcast curriculum workshops. Workshop participants are asked to define the “purpose” of their stations’ newscasts. Then they participate in the previously mentioned “story-stacking” exercise, where they find out how closely their reflexes match their definition of their purpose as they make decisions about how to prioritize and tell nine hypothetical stories. The idea is to refine the news judgment of each individual, and by extension each station. This module typically is only presented to broadcast audiences, and is presented at every broadcast workshop.
44% of broadcast respondents reported impacts associated with news judgment. These impacts included things such as more thoughtful story selection and placement in newscasts, more staff collaboration in developing story ideas and strategies for reporting them, and more effort to make the substance and tone of stories consistent with the station’s chosen identity.
“[The training] reinforced that telling good stories is the way to get and maintain viewers – people want substance,” said a BR-12 respondent. “[Now] we drop the stories that aren’t great. [We] learned you don’t have to do every [story]…you can just do the good stuff,” this respondent continued. This respondent added that she believed this new approach to news judgment was part of the reason for the station’s recent ratings climb.
“[The training] affected my perceptions of what is newsworthy,” said a BR-7 respondent. “You don’t have to cover every shooting. We’re rethinking viewers’ values…what viewers really want.” Another BR-7 respondent added that since the CCJ workshop there’s been “more focus on substantive stories, thinking about why we pick certain stories, [and thinking about] why they should be played where they are played [in our newscasts].”
Another BR-7 respondent credited the training with bringing about a spirit of more collaborative editorial decision-making. “[The training] made me feel like I have the ammunition to speak up about editorial decisions because [we’re doing] more collaborative decision-making. We reference the story stacking exercise in the morning meeting sometimes [and] talk about what viewers really want.”
A BR-6 respondent had a similar reaction to the training. “[The workshop] opened up opportunities to discuss story treatment decisions. [There have been] lots of ‘remember the way we discussed these things at the workshop?’ type discussions,” this respondent said. “As a team we’re more open to discussion,” said a BR-13 respondent. “We’re more able to talk about why a story is or isn’t a story. We don’t just go do things – [we] think and talk first,” this respondent continued.
One station in particular seemed to feel as though the training helped them match their news judgment to the editorial vision their recently hired news director was trying to instill in the newsroom. “[The workshop] streamlined our focus,” said one BR-11 respondent. “We think about why and how we cover what we cover [and try to] justify why we do certain stories. We look more critically at [our new judgment]. What’s the viewer value? What’s the impact? [The training] has created a momentum of thinking. There’s less following the newspaper and the wires. [We] feel better about the direction we’re taking.”
[top]
Impact on newsroom communications
Although newsroom communication was not a major focus of the CCJ training modules, it was a target of concern and a subject of many workshop conversations. It is well known in the industry that newsroom communications among and across ranks are often inadequate. Because the CCJ workshops offer participants opportunities to collaborate with colleagues on challenging tasks and to discuss jointly – often for the first time – their fundamental purposes, standards, and strategies, we expected that the experience could leave a legacy of improved communication in participating newsrooms. Moreover, since communications shortfalls in newsrooms are not only common but also highly bothersome to many journalists, we expected that participants would use the interactive workshop format as an opportunity to raise this problem and to work out better channels for communication with their colleagues. The assessments confirmed all of these expectations.
Improved communications with coworkers up and down the ranks as a consequence of the workshops was reported by 75% of all participants. This finding was virtually the same for print, broadcast, and Internet newsrooms: 72% of print, 78% of broadcast, and 74% of Internet participants reported this improvement. 55% of all participants across the three media found this to be the most valuable takeaway from their workshop experiences, overwhelming all other benefits other than learning the CCJ tools of good journalism (see section below entitled “Use of CCJ tools”).
What did participants mean by “improved communications”? Breaking down the variable into specific components, 61% of participants identified the main benefit as improved teamwork and information sharing and 24% identified the main benefit as more effectively structured lines of responsibility and, related to this, better-organized meetings. Others mentioned issues such as more frequent interactions with colleagues on off-task matters and a generally more collaborative atmosphere throughout the newsroom.
As noted above, the finding of improved communication was noteworthy because the topic of newsroom communication was not generally the main subject of the workshop modules: it was, rather, more a sub-text than a text, embedded in the way that the small-group discussions and other exercises were conducted. Informal observations by trainers confirmed the pressing need of newsrooms to pay more attention to communications gaps that undermine the working conditions of staffs. For example, in one newsroom that was visited, reporters requested that the trainers distribute nametags so that the participants could become acquainted with one another. This was a reasonable request, certainly; but it seemed somewhat surprising, because the entire newsroom consisted of less than 30 staff members, almost all of whom had been working for this same paper for well over a decade.
In another newsroom, the trainers uncovered a vast pool of discontent about the way the editor approved or disapproved investigative reporting proposals. The staff felt that the editor’s decisions were either arbitrary or based on unjustifiable favoritism. When the trainers spoke with the editor, it became clear that he had worked out an intricate, well-articulated policy on the matter. The policy reflected reasonable judgments about the reportorial experience of his staff, the particular interests of the community the newsroom served, and the newsroom’s niche within that community. The editor was convinced (wrongly) that the staff knew about this policy and understood the principles behind it. He believed that his staff knew about the policy because he had mentioned it to three or four members of his inner management team and thereby assumed that they must have spread the word around to all the others.
In the workshops’ small-group exercises, staff members who work in differing roles and levels and are forced to confront each other’s perspectives, gaining awareness of one another’s agendas and priorities. For example, just as reporters hear from the business manager about what keeps the automobile advertisers happy, the business manager hears from reporters about what keeps the credibility of the paper intact. It is not that such dialogues always lead to agreement – indeed, uniform agreement might not even be desirable – but they do lead to the kinds of mutual understandings that make for good working communication. In the case of the newsroom where the editor had failed to broadly communicate his investigative reporting policy, a follow-up visit revealed an improved sense of morale among the staff that stemmed directly from its increased trust in the fairness and legitimacy of the editor’s decisions.
Both print and broadcast newsrooms implemented improved communication systems as a consequence of the workshop training; but the broadcast communications improvements were more extensive and probably more effective than those put in place by the print newsrooms. This difference likely reflects the training’s greater impact at the whole-newsroom level in broadcast versus print newsrooms. The broadcast training format favors a whole-newsroom effect, since in broadcast stations nearly the entire newsroom eventually participates in the training: the broadcast workshop sessions are only a half-day in duration, and CCJ offers as many sessions as it takes to cycle a high proportion of newsroom staff through a training. It can be expected that this increases the likelihood of newsroom-wide changes resulting from the broadcast training. The print workshops provide intense learning experiences for the 25-40 people who participate in them; but for mid and large-sized papers, this training group may not be a large enough proportion of the newsroom to alter whole-newsroom practices in major ways.
Still, print newsrooms implemented a number of noteworthy improvements in their communication systems. These included:
1. Brown bags – A handful of newsrooms reported doing more brown bag lunches after the CCJ training. Few newsrooms began doing brown bags if they hadn’t already been doing them in the past – they just did more of them and tried to give them more focus.
2. Writer’s groups – Two newsrooms noted that they have a writer’s group. In both instances this group preexisted the CCJ training. The groups’ existence came up because both were said to have discussed relevant ideas from the CCJ training in their meetings.
3. Corrections committee – Some newsrooms started “corrections committees” consisting of rotating newsroom staff who tracked errors and met with consistent offenders to discuss ways to improve their work.
4. “Meet the copy desk” – One newsroom hosted an event so that the daytime newsroom staff could meet the nighttime copy desk staff so they could get to know each other better and develop empathy for each other’s job responsibilities.
5. Teamwork/collaboration – 57% of all print interview respondents said some aspect of teamwork, information sharing, and collaboration improved in their newsrooms after the CCJ workshop. These reports were generally anecdotal comments about the breaking down of communication “barriers” between sections of the paper, a greater appreciation for what colleagues do in their work, and in some cases a greater willingness to initiate brainstorming with colleagues about story ideas or coverage strategies.
The more extensive improved broadcast communication systems that were implemented included:
1. Reality checks – This is the “biggest” communication system change spurred by CCJ training in many broadcast newsrooms. The idea behind a reality check is that reporters and their crews in the field will call the producers of the shows they’ll be appearing in once or several times throughout the day to give updates on story progress. These progress reports ideally touch on both editorial content and logistics. Ideally, a “bridge line” is used for the calls so the assignment desk, managers, promotions staff, and anyone else can listen in and contribute, and the calls are logged in the newsroom computer system so anyone with questions about a story can simply check the log. Several stations implemented reality checks in various forms.
2. Quality control meetings – One problem many stations suffered from before their CCJ workshops was that staff would head for the door after their newscasts without discussing what went right and wrong; hence, recurring, correctable problems would not be addressed. Several stations implemented quality control meetings after newscasts – especially their main ones – where anchors, directors, producers, managers, and others could discuss the highlights and lowlights of the newscast, make plans to purge problems, and reinforce the good things.
3. Editorial meetings (“Morning meeting”) – Several station managers started approaching their morning editorial meetings differently based on suggestions from the CCJ workshop. The biggest changes were in tone, participation, and format. Managers reported trying to take a less visible role in the decision-making, instead turning that function over to assignment managers and collaborative group decisions. Some managers encouraged the participation of photojournalists, tape editors, production staff, business staff, and promotions staff to draw ideas from a more diverse audience. Some newsrooms moved away from a structure in which the manager would “go around the table” asking each participant what they had to contribute, and instead started by running through existing enterprise ideas and daily events that seemed interesting. Then they moved into whether participants had anything to add to those ideas and any new things they’d heard or been thinking about.
4. Pre and Post “Sweeps” socials – At Broadcast Curriculum Director Wally Dean’s urging to “celebrate their successes,” some stations started holding socials before and after Sweeps periods. The Pre Sweeps socials were efforts to boost morale and share a common vision for the coming months. The Post Sweeps socials were efforts to show staff their efforts were appreciated and keep the momentum going from good stories and projects that aired, even if ratings weren’t ideal. These socials also had the effect of improving newsroom collegiality.
5. “Right things done right” meetings – Two stations started doing “Right things done right” meetings at which the newsroom manager shared clips of good work from the previous days, weeks, or months and either explained what was good about them, or invited staff to share what they thought were the highlights. These meetings helped staff better understand the managers’ expectations and visions for good journalism.
6. Town hall meetings – One station started holding regular town hall meetings in all of the counties it covered to expose staff to community members’ story ideas and coverage feedback. Transcripts of these meetings were shared with all newsroom staff, and negative feedback served not only to educate staff about community views, but also to hold them accountable for their work.
7. Reporter meetings – One station’s reporters starting holding regular weekly meetings after the CCJ workshop as a way to share skills, stay on top of relevant community news, develop mentoring relationships, and bring community members into the newsroom to build new relationships and lines of communication.
8. “The easel” – Some stations started using dry erase boards or easels to log story ideas and assignments. This gave staff a simple, visible way of figuring out who would be doing what and kept good story ideas that didn’t fit into newscasts “alive” for the next day.
9. “Blue light” – One station affixed a flashing light to the assignment desk that would be flipped on during breaking news to alert staff to assemble and find out what was going on, how they could help, and how their previous assignments might change.
Both the print and broadcast improvements were promising and, certainly, constructive; but, as noted, the broadcast changes were more sweeping, innovative, and transformative than those that we observed in the print newsrooms.
[top]
Newsroom innovations
Traveling Curriculum workshops offer journalistic tools and the inspiration to think critically and communicate more openly with colleagues. Another encouraging impact discovered during assessment interviews is that workshop participants are using CCJ tools, lessons, and ideas as the foundation for their own initiatives and innovations. Individuals in nearly 80% of all print, broadcast, and Internet newsrooms assessed described innovations in their newsrooms they attributed directly or indirectly to their CCJ training experiences.
At newspapers, the innovations ranged from forming “accuracy and credibility” staff committees, to creating “reader panels” to critique previous work and offer story ideas, to sending letters to sources to ask if they felt like they were quoted accurately and fairly, to initiating newsroom awards programs, to “how I did the story” brown-bag lunches to share reporting techniques.
At local television stations, innovations ranged from inviting non-newsroom staff to editorial meetings to offer story ideas and feedback, to inviting viewers to share digital photos to the stations’ websites and newscasts to make them feel more a part of stories, to structured story filing protocols aimed at increasing accuracy and creating accountability.
Non-medium specific innovations included: reader/viewer databases that can be queried for story ideas, leads, and feedback; community meetings to discuss controversial editorial decisions and make connections between staff and the community; and holding “diversity workshops” where groups of people representing various demographic groups in a community are brought into the newsroom to meet staff and discuss their ideas and perspectives.
Interestingly, the assessment visits spurred ideas for innovations as well. One newspaper started cycling all editorial staff through small-group meetings with the executive editor to give them opportunities to discuss editorial decisions, ethics, and the editor’s visions for the future after the assessment visit. The report sent to the newsroom after the assessment interviews recommended creating more opportunities for staff to interact with upper management and each other. One broadcast station started discussing how they might conduct “pre-show” meetings before newscasts to give directors and producers a chance to go over the details of the newscast and thus avoid preventable errors. This discussion was spurred by the assessment team observing through interviews that preventable errors were a problem and recommending that staff formalize ways to deal with them.
[top]
