Measuring Compliance Program Effectiveness

For all the time and money spent designing and building ethics and compliance programs, compliance officers—and regulators, and investors, and many others—still want to know: Does all this effort actually work?

Apparently the answers are still elusive. How companies measure the effectiveness of their compliance programs was one of the questions posed to senior compliance executives as part of the State of Compliance 2011 study conducted by Compliance Week and PwC earlier this year. The majority of respondents said they use numerous tools to measure compliance program effectiveness, including hotline activity (92 percent), employee training data (89 percent), and audits of the compliance program (79 percent).

Those tools are a good start, to be sure. The peril in analyzing these statistics alone, however, is that they only say whether a compliance department is busy, rather than effective.  “Most organizations focus on process measures when they try to measure effectiveness,” says Miles Everson, head of PwC's governance, risk, and compliance services practice. That might help explain why 38 percent of survey respondents admitted they don't measure the success of their compliance programs at all.

The biggest challenge with measuring compliance program effectiveness is the difficulty in interpreting the data, says Colette Simo, director of corporate compliance and risk management at OfficeMax. “For example, does an increase in whistleblower hotline calls mean an increase in compliance effectiveness,” she asks, or merely an increase in misconduct?

That example of hotline calls a long-simmering question among the compliance community. “You know [the hotline] is effective if you find problems … and correct them before anything happens,” says a compliance executive of a large insurance company, who asked not to be named. “It's almost scarier if you don't hear anything.”

But even with an active hotline, the executive adds, “it's difficult to come up with statistics that demonstrate effectiveness.” After all, measuring the change in the volume of calls doesn't tell you much about the quality or content of the complaints.

“The biggest reason why this is such a struggle is that it can come across as a very daunting task, because to identify effectiveness you have to evaluate each and every piece of a compliance program,” says Andrea Falcione, vice president of advisory services and chief ethics officer at SAI Global Compliance. This can be particularly challenging for larger companies, where the compliance program often is spread throughout the organization, rather than owned by one group, she says.

Effective Metrics

The Holy Grail of assessing compliance program effectiveness is quantitative data, Falcione says. It's the ability to answer the question: “What can we measure numerically that we can then analyze from a trend perspective?” she says.

When applied to hotline calls, for example, a company might collect and analyze other data, such as the experience of the employee who called in the complaint. The goal there is to measure whistleblowers' satisfaction with the system—a key metric to have, especially in the Dodd-Frank Act's era of government bounty programs offered to whistleblowers who run straight to them rather than report internally.

“The biggest reason why this is such a struggle is because it can come across as a very daunting task, because to identify effectiveness you have to evaluate each and every piece of a compliance program.”

—Andrea Falcione,

VP of Advisory Services, Chief Ethics Officer,

SAI Global Compliance

“If you're looking at your whistleblower satisfaction rates and those numbers aren't trending well, the organization needs to step up and change its investigatory procedures, and make sure people aren't afraid to make a report,” Falcione advises. “Ineffective whistleblower programs may well also result in employee reporting to the SEC under the Dodd-Frank whistleblower bounty rules.”

Companies may also want to compare the number of anonymous calls versus those where the callers identify themselves. Calls much more heavily weighted in the anonymous category—or a trend upward on a year-over-year basis—could suggest an ineffective communications campaign around, say, the company's commitment to its anti-retaliation policy. Or even worse, Falcione says, a spike in anonymity could imply “an ineffective policy itself, leading to a culture of fear and, most likely, under-reporting.”

Employee training data also can be measured quantitatively by looking at the percentage of employees who meet their certification and training requirements, based on their performance evaluations. Companies can then compare year-over-year results to indicate whether that number is trending up or down.

Responsiveness is another useful way to assess how fast the company reacts to issues and detecting problems. For example, how long does it take to find a defect in a product? Did a customer find it after the product was on the shelf for a year, or did the company find it before it came off the assembly line? “Those would be important indicators of performance, and whether the compliance function is actually performing well,” says Scott Mitchell, CEO of the Open Compliance and Ethics Group.

Looking at the outcomes of data to identify situations that might warrant further explanation is the best way to get at the crux of whether a compliance program is effective, Everson says.

Compliance Ownership

Another common question posed by many compliance officers: Who is supposed to be the person putting the compliance program into place and ensuring that it does work? The compliance department itself? Internal audit (perhaps assisted by an external consultant)? The legal department? Business unit managers? Someone else?

In many companies, the person in charge of compliance is responsible for writing the policy, and then enforcing and monitoring compliance with those policies. “They don't actually participate in the act of complying,” Everson says. The compliance function tends to be the second line of defense for an organization, after the business unit leaders; that means the compliance officer is one step removed from direct line management, he explains.

MEASURING COMPLIANCE

The following chart for the CW/PwC State of Compliance Survey gauges methods used by respondents and their frequency of measuring compliance program effectiveness.

Source: Source: Compliance Week & PwC “State of Compliance Survey.”

For this reason, the individuals responsible for validating compliance effectiveness should be the ones in charge of the core operations, where the risks are most likely to arise. Anti-corruption risks, for example, may be best managed by executives in global sales, because “it's in their processes where these risks originate,” Mitchell says.

But you don't want senior management completely removed from the process, either. “You want your chief executive, general counsel, and compliance officer to be out and about and speaking publicly—internally and externally—about the importance of corporate integrity and compliance with the law and doing business with ethics,” Falcione says.

Also in the State of Compliance 2011 Survey, participants were asked how frequently they assess their compliance program's effectiveness. Many said their departments “constantly” track complaints received, cases resolved, and employee training. For those companies that do audit the effectiveness of their compliance programs, 30 percent do so continuously, 3 percent audit quarterly or semi-annually, 21 percent annually, and 46 percent use some other interval.

Experts favor the continuous approach. “You can't wait two years. You can't even wait one year to reassess your program,” says Keith Darcy, executive director of the Ethics & Compliance Officer Association. “Every day you wake up there is a new risk being exposed on a grand scale.”

Everson says compliance officers would benefit by collecting data to answer the question: “How is the compliance program being adapted to match with the level of change occurring in the company and in the business environment in which the company operates?”

“The rate of change is continuing to accelerate,” Everson says. “The compliance program needs to keep pace with the rate, or it can quickly become an ineffective program.”