For many years, ethics and compliance officers have mulled how to measure the effectiveness of their programs. Some have said that finding some quintessential measurement for program effectiveness is essentially the holy grail of compliance programs. We all continue to believe that if we can simply identify that perfect measurement our programs will have definitive direction and sustainable justification.

Unfortunately, like most epic journeys, while the goal is worthy, attaining it is proving to be far more difficult than first imagined.

We face two fundamental challenges in identifying a definitive set of meaningful metrics to measure and assess compliance program effectiveness. First, the identification requires a set of well-articulated program objectives and goals that are common to companies within and across various sector groups. Second, it requires a universally accepted set of tangible, concrete, and significant standards that can be used to measure and compare program effectiveness.

Both of these challenges can be addressed, but first we need to back up the effectiveness discussion so the ethics and compliance industry can establish these essential foundational elements.

Let's also note a distinction exists between programs themselves and the substantive component activities that comprise the programs. For example, an ethics program usually has activities focused on hotlines, investigations, codes of conduct, training, and communication. In the past several years, the industry has done an admirable job defining those various program components and benchmarking best practices around their operation. Still, that's different than identifying whether the sum of these components equals overall program effectiveness.

Goals and Objectives

An ethics and compliance program is effective when it produces the desired and intended results. Therefore, prior to assessing effectiveness, a program's goals and objectives must be defined and articulated in a manner that can be measured and evaluated. Complicating this discussion are two facts: First, ethics and compliance programs have myriad and often overlapping goals and objectives, each seeking a distinguishable, desired result; and second, the industry has not settled on a common set of program goals and objectives allowing for meaningful benchmarking.

The most universally accepted objective for a compliance program is the program should manage the risk of an adverse regulatory action. Put another way, the program should institute controls that meet or exceed regulatory requirements and expectations.

Of course, most companies are subject to regulations from several regulators (state and federal) and the regulatory expectations don't always coincide. Moreover, regulations have an increasing tendency to be framed in terms of subjective, risk-based principles rather than objective and definitive rules making articulation of the objective more complicated.

Not only do individual companies have multiple complex objectives that their ethics and compliance programs must meet; these objectives are often unique to the company and not sufficiently universal to allow for a comparative analysis.

Regulators and government prosecutors provide ethics programs with even less direction. The U.S. Sentencing Guidelines provide a good general direction but do not contain (nor should they contain) sufficient detail for companies to use as a set of program objectives.

Of course, meeting regulatory expectations is only one of a program's many objectives. Most programs also seek to influence company leaders at all levels of an organization, in a manner that both inspires ethical leadership and drives compliant decisions. The ability to influence requires a tailored organizational structure; having and maintaining a broad set of credible hard and soft skills; and a clear alignment around the scope of the program's mandate for exercising this influence. Not only is this objective difficult to articulate for an individual company, it may be close to impossible for an entire industry to agree on the articulation of a common objective.

Going one step further, an often unstated, but fairly universal, objective for an ethics and compliance program is to help the company's business areas meet their long- and short-term strategic objectives. While the two objectives stated above clearly contribute to this third objective, there are additional expectations. Company risk-management programs look to ethics and compliance programs to contribute strategic prioritization of risk mitigation efforts, as well as to the identification of opportunities for the strategic acceptance of risk.

These are only three of the many stated and unstated broad, high-level objectives that most ethics and compliance programs seek to achieve. There are often many others. They range from internal objectives focused on cultivating the professional excellence of the compliance resources to finding ways the company can use adherence to its ethical and compliance standards as a market differentiator.

Bottom line: Not only do individual companies have multiple complex objectives that their ethics and compliance programs must meet; these objectives are often unique to the company and not sufficiently universal to allow for a comparative analysis. The question of assessing program effectiveness requires establishing numerous underlying assumptions, including (but not limited to!) the complicated task of identifying the very objectives that are the subject of the measurement. Consequently, before engaging in an industry-wide discussion on effectiveness, we need to first engage in a discussion around objectives.

No Standard Measures

Even if the industry could agree on a general set of universal objectives, the concept of effectiveness implies a standard of comparison for measurement purposes. Right now we're still limited in our ways to measure the effectiveness of various program elements, let alone standards for assessing broad program objectives.

For example, if the objective is to meet regulatory requirements and expectations, few, if any, truly meaningful ways exist to measure effectiveness. Most compliance officers and regulators agree that examination results rarely provide an indicator of current, overall program effectiveness. Examinations are almost always retrospective in their approach, providing little insight into current practices.  Moreover, companies with industry-leading compliance programs have received hefty fines for historically accepted activity that regulators one day decide to deem problematic (unclaimed property, for example). In addition, examination results often fail to create a distinct difference between findings that indicate a systemic control weakness and those focused on an isolated event.

The above only touches on the problems associated with using the results of regulatory examinations as the measure of program effectiveness. Unfortunately, even though exam results are of questionable value from a program assessment perspective, they're often the only measure we have for assessing the important objective of meeting regulatory expectations.

When ethics and compliance program objectives become even more company-centric (“influence ethical decision making at all levels of the company,” for example), the problem of measuring effectiveness magnifies. Recently, there has been an increased interest in the use of ethical culture surveys as a means of measuring how the ethical climate in a company changes over time and compares to industry peers. Still, the question remains as to whether these surveys actually get to the application of ethical considerations in making decisions.

These surveys often ask about the observation of misconduct (that is, non-compliant behavior). However, it is one thing to say a company has a strong ethical culture when employees make a clear decision between acting in a compliant versus non-compliant manner. It's something altogether different when the choice is between two compliant options that require a refined assessment of whether a proposed action furthers a company's ethical values. If the objective is simply to drive more compliant behavior, the surveys may provide beneficial information. On the other hand, if the objective is to drive more ethical behavior, the surveys may be limited.

Even using objective data as the measure creates a challenge. The industry has recently engaged in a significant amount of discussion around the measure of hotline usage and investigations. Most practitioners are aligned around the fact that an increase in calls to a hotline may, in fact, indicate an increase in program effectiveness (better training and communication). As a follow-up, however, one could reasonably ask when the increase in hotline calls would indicate the program lacks effectiveness.

Where Do We Go From Here?

I would submit that if the industry is serious about identifying meaningful ways to assess program effectiveness, we start by taking on the difficult task of articulating program objectives and goals. Industry surveys, meetings, and publications could provide excellent forums for the discussion around program objectives. For those goals and objectives common to a majority of industry programs, we can then begin to discuss what constitutes “effective” and how it can be measured.

Our search for the key metrics that will help identify effective programs should not distract us from continuously improving our program activities. No doubt, an element of overall effectiveness will be the quality of our underlying program activities. Most ethics and compliance officers, however, understand that another level of analysis can help determine whether our programs are achieving the results expected by our companies, regulators, and the industry in general.

The quest for identifying a means to assess ethics and compliance program effectiveness is laudable but not easily attainable. What is imminently attainable and worthy of our collective effort is the continued dialogue around this topic.