What do you get for your money?

AIM THIS PAGE

A vexing question about training in the news business, especially when money is tight, is: What are we getting for our money?

How do you measure the impact of a training program? How can you say how much better an editor is leading or a reporter is writing? The task is difficult but it’s not impossible. We’re used to overcoming obstacles in pursuit of big news stories. We need to overcome the obstacles to proving the value of training.

I took a stab at measuring the impact of two recent API leadership seminars. The results were encouraging, both about the quality of our programs and about the ability to quantify matters that seem to defy measurement.

Evaluations at the end of a program are helpful, but they do not measure impact. They measure enthusiasm for the experience completed, not lasting value of the lessons taught. To measure impact, you wait a while and give people a chance to apply (or forget) the lessons they’ve learned. You wait until they’ve returned to the daily routine, where deadlines and peers push us toward our default settings of “the way we’ve always done it.”

In early 2005 API presented leadership seminars for two groups of middle managers of Ottaway Newspapers Inc., totaling more than 60 leaders from all departments: newsroom, advertising, circulation, production, information technology, finance, online, niche products. After the managers had returned to their jobs (two months later for one group, five months for the other), I surveyed the managers and their publishers, working in cooperation with Sue Glassey director of human resources for Ottaway.

We asked both groups if the managers’ leadership was unchanged since the seminar. If they saw improvement, was it a vague sense of improvement that they couldn’t quite identify? Could they identify a specific way the manager’s leadership had improved? Or multiple ways? All but one publisher returned the survey and majorities of the members of both seminars responded.

The responses demonstrated the value of the seminars:

• Publishers noticed improvement in the leadership of more than 80 percent of the managers attending the seminars.
• Publishers cited specific improvements in about half of the managers.
• Nearly 90 percent of the managers cited specific improvements in their leadership.
• Nearly 60 percent of the managers cited multiple specific improvements.

In addition to the data, our survey provided anecdotal evidence of the specific impact of the seminars. After asking participants for multiple-choice answers we could quantify about the general impact of the seminar, we asked open-ended questions about how, if at all, they were applying the lessons of each session of the seminar.

Those answers might have been even more helpful than the overall data. The comments helped us see how each session contributed to that quantifiable result.

For instance, we opened both sessions with a communication session led by an improvisational comedian, Bob Orvis of ComedySportz in Milwaukee. Everyone had fun with that session and it was a great ice-breaker. But did the participants learn anything to take back to their jobs? Now we know. Yes.

“I think I was already good at communicating well with co-workers, but what this session did was point out some small surprising things I was (or wasn’t) doing. I’ve been able to change those things,” one manager said.

“I have been more effective in communicating with employees during their reviews,” said another.

Other comments:

• "I was amazed to see how two identical situations can be handled two different ways – one with tempers flying and the other with calm conversation.”
• “Orvis’ session was not only entertaining, but an eye-opener. I began looking at my more difficult co-workers a bit differently, watching their body language and listening to key words that helped me realize what was really behind their behavior. Realizing it’s all about power helped me know how to better respond.”
• “I have applied much of what I learned in this two-hour session in countless circumstances since. I have learned to not only listen better, but understand why I – or the person I am dealing with – am reacting the way I am/do.”

Sure, some people just had fun, but clearly this session changed the way many of the leaders communicate with their staff. That’s not a guess or a hope now. We know it’s a fact.

We wondered about the lasting value of another session that received rave views on the initial evaluations: “Leadership Lessons from the Battle of Gettysburg.” We toured the battlefield with a spellbinding discussion leader, Carol Reardon, a military history professor from Penn State. Carol’s thorough knowledge of the battle’s story and her passion for telling it made the tour a fascinating day that drew initial raves from nearly all the participants. But did they learn anything?

Their answers, more than two months later:

• “The lessons I learned from Carol’s presentation possibly had the greatest impact on my leadership here. … So many of the aspects of the battle can be related to business situations, like dragging the heavy cannons around that had such a long range they could not be utilized in actual battle (or putting in place procedures which can not be fully utilized or that a manager can not forecast the benefit from); and again, effective communication of the strategy and visualization of the end results.”
• “Learning about how the communication going up as well as coming down is what makes a team work really opened my eyes to why that type of communication is essential. I have been able to use this lesson to improve the communication that I give to my superiors, to my peers and to my employees.”
• “In strategic planning for 2006, I’ve taken more care in examining the possible results of ‘quick fixes’ vs. more practical long-term applications.”
• “I learned to give specific instructions. I learned it is important to survey the whole picture and know the goal. Also never underestimate anyone.”
• “I have repeated several of Carol’s leadership stories to management and staff. Very valuable to me.”
• “The battlefield was the most enlightening to me. I learned the differences in leadership skills and what impact they had to the results. I was able to identify with some of the generals and how we need to keep a constant communication flow to upline managers and subordinates.”
• “I continue to remind myself to look at the whole picture prior to taking action.”

E-mail me if you would like to receive a copy of my eight-page report to Sue on this survey or the 26-page appendix with comments. If you have found ways to measure the impact of your training programs, I would be pleased to hear about them.

Training and leadership development are too important to guess about their value and impact. We can figure out how to measure the impact and show the value.

sbuttry@americanpressinstitute.org

Post a comment