How Much Testing Is Enough? Cos. Struggle With Sampling

Section 404 of The Sarbanes-Oxley Act of 2002 requires that companies document and test their internal control over financial reporting. And, of course, the standard that auditors use to gauge the success of those efforts is known as Audit Standard Number 2, issued by the Public Company Accounting Oversight Board in June 2004.

But the PCAOB's AS No 2., officially called "An Audit of Internal Control Over Financial Reporting Conducted in Conjunction With an Audit of Financial Statements," does not specify exactly how much testing is enough.

Though in some cases auditors are directed to test controls over a "large portion" of the company's operations and

financial position, the PCAOB has preferred not to provide details on what that means. Some auditors, for example, have assumed that a "large portion" means 60 percent or 75 percent of all controls, but the Board does not agree. "Auditing Standard No. 2 does not establish specific percentages that would achieve this level of testing," wrote the PCAOB in answer to one of many "frequently asked questions" published last year.

Instead of prescribing specific numbers, the Board decided to use a "principles-based" approach to the issue. "Therefore," continued the PCAOB FAQ, "Auditing Standard No. 2 leaves to the auditor's judgment the determination of what exactly constitutes a 'large portion.'"

The PCAOB notes that accountants—especially those engaged in audits at large enterprises with multiple locations and business units—may satisfy the testing requirement by testing a representative sample of the company's locations or business units. "The auditor may select the representative sample either statistically or nonstatistically," says the Board FAQ. "However, the locations or business units should be selected in such a way that the sample is expected to be representative of the entire population.

"Statistically Significant"

That’s a lot of leeway, and it's one that many still consider "up in the air." In fact, some registrants may suspect that their auditor has moved in, given the sheer volume of on-site tests that have been performed.

So when will they be satisfied and go home? The dreaded answer is: it depends; that is, it depends on the current state of the company's information system and its controls, as well as the size and complexity of the underlying business.

Gazzaway

Indeed, the audit firms seem to have differing opinions on "statistically significant" samplings for the purpose of testing.

While the PCAOB's FAQ noted that some auditors looked to test 75 percent of controls, others believe that 32 "samples" of a process that is performed daily is considered statistically significant; others prefer 33.

But no standards exist. “No matter how much research I do, I couldn’t give you a number,” says Trent Gazzaway, managing partner, corporate governance at Grant Thornton in Charlotte, N.C.

For most companies, the "testing" process can't start until the company has completed an assessment of the key business processes. Unfortunately, many companies often don’t know how their systems are designed. For example, if a firm is relying in part on legacy components, changes and redundancies over the years may obscure the core process controls. “I know one organization that has identified 17,000 controls,” says Daniel Langer, global internal audit and controls solutions director at Jefferson Wells in Brookfield, Wis.

Kolins

Once the controls are mapped out, tests need to be designed. But the type of test typically depends on the type of control. “Automated controls may only have to be tested once," says Wayne Kolins, national director of assurance at BDO Seidman in New York. But manual controls, which are common and ubiquitous, "may have to be done 25, 45, 60 times, depending on the extent to which we can rely on the client’s own testing,” adds Kolins.

In addition, additional tests can proliferate quickly based on the results of initial tire-kicking. “If I have a control in my system that says I can only enter valid, unique invoice numbers, I only have to do one test,” says Dennis Keglovits, national director of risk management services for RSM McGladrey in Kansas City, Mo. But if that test fails, others would be triggered. For example, the auditor might enter a duplicate number to see if the system accepts it. If it does, then the tests multiply.

The test process is so far-reaching, says Kolins, that BDO Seidman designed its own tool—including a spreadsheet checklist—to guide staffers through the testing process. “We thought audits were detailed,” Kolins says. “They pale in comparison to 404 work papers.”

PCAOB AS No. 2 includes a framework for developing testing procedures at companies with many locations. In an illustration included in the standard (see box at left), the Board presents a sample flow chart for creating a testing plan. The hypothetical company has 150 locations, 90 of which need some kind of testing or evaluation.

Year One Lessons

According to experts, new and non-accelerated filers may be able to avoid some of the pain suffered by those who survived the first year by understanding their systems and their controls.

“If you have a lot of manual controls, try to design them so they can automated,” says Langer at Jefferson Wells. He recommends that companies consider finding shared services opportunities for systems that are not part of their core competencies, putting the onus on the outsourcing provider. (The filing company still has to ensure that the vendor’s controls are adequate.)

Laura Phillips, associate chief auditor at the Public Company Accounting Oversight Board, thinks that the wide range of control system design and test practices will narrow. "I would expect to see convergence as companies and auditors gain more experience with the process," she says. But given that the PCAOB does not begin the inspection process for AS No. 2 until later this year, that point is still several months away.

“Ultimately, what you are doing is a risk assessment of the organization, to find out what’s inside and outside of materiality,” says Keglovits at RSM McGladrey.

Those vagaries created more work than most IT managers expected, says Al Schmidt, CIO of Arch Chemicals in Norwalk, Conn., and chairman of the Fairfield/Westchester Chapter of the Society for Information Management. “It was a series of surprises as we went forward, and I’ve heard the same thing from my colleagues,” he says.

Schmidt estimates that Sarbanes-Oxley testing took up about half of his time last year, but the planning paid off. “By the time the external auditors came around and started looking at stuff, we were able to get through it OK,” he says. And even though Schmidt says it was somewhat "laborious" to explain and demonstrate control effectiveness to auditors, he notes that Arch had an easier time than many firms because the documentation was in place. “A lot of companies have legacy code that has been built up over the years, and the documentation is extremely poor.”