Compliance Program Gap Analysis vs. Effectiveness Evaluation

Comparisons, Cost Benefits, and Considerations

As compliance officers and organizations try to determine when and under what circumstances to have their compliance programs undergo outside review, they must also determine what the review process should look like. Undergoing a review is a must and should not be considered optional. OIG guidance calls for periodic independent reviews of compliance programs, and the US Sentencing Commission calls for companies to “evaluate periodically the effectiveness of the organization’s compliance and ethics program.” This standard highlights that compliance is just like other management programs that should be checked to verify that they are actually working as designed. Further, in February 2017 the DOJ Fraud Section released its “Evaluation of Corporate Compliance Programs,” which puts compliance officers on notice about what the DOJ expects to see with compliance programs.

There are many types of compliance program reviews with different scopes of work and costs, performed by consultants with varying degrees of competence. To further complicate matters, there exists a plethora of ill-defined terminology relating to this subject that is often used interchangeably, such as compliance program “review,” “effectiveness evaluation,” “gap analysis,” “assessment,” “appraisal,” and “examination,” to name a few. By and large, these terms fall into two general categories of reviews: “compliance program gap analysis” and “compliance program effectiveness evaluations.” There is a huge difference between them, both in terms of cost and quality of results. It can be confusing to request a proposal only to find a huge differential in costs, and that the requested reviews have been a complete waste of expenditures in terms of value received. It is important to be very specific as to what one wants from the engagement, and to avoid mixing proposals or deciding upon price without knowing whether the engagement is for a compliance program effectiveness evaluation or a compliance program gap analysis.

Compliance Program Gap Analysis

Suzanne Castaldo, JD, explains the approach used for a compliance program gap analysis is primarily a desk review that follows a checklist tied to the seven standard elements of a compliance program. It is most useful for newer compliance programs lacking maturity; or for a new compliance officer that wants, in effect, an inventory audit to know the status of the program at the time he or she assumes responsibility of it. It is a very limited project that consists of conducting a document review of charters, the Code of Conduct, compliance-related policies, the hotline log, minutes of compliance oversight committees, the compliance training program, compliance audit plans, and related reports. The review may be supplemented with limited telephone interviews with the compliance officer to obtain additional background, clarification on documents provided, and to identify missing pieces of supporting documentation. The benefit of the review is that it can provide milestones as to what is in place, as well as what is needed to strengthen the program structure. The resulting observations and recommendations can relate to the completeness and clarity of the documents, but won’t tell you how effective these elements are in the operation of the compliance program, which the OIG stresses in its compliance guidance documents. There is no way to evaluate effectiveness without going beyond the documents for in-depth examination and evaluation of the program’s operations. Depending on the scope of work and approach followed by the individuals conducting the review, the resulting report typically will range between 20 and 30 pages and include identification of gaps, along with recommendations as to how to address them.

Compliance Program Effectiveness Evaluation

Carrie Kusserow employs teams of compliance experts that perform full-fledged compliance program effectiveness evaluations. She explains that these reviews provide a far more detailed, in-depth approach than a gap analysis and require a high degree of expertise to provide qualitative and useful results. This approach not only includes verifying program development but also, more importantly, examines how effectively the program is operating. This type of review is designed for measuring outcome rather than output. It does not focus solely on what documents support the program, but how well they achieve the desired goals. This in-depth review is best for maturing programs that are seeking ways and opportunities to improve or enhance effectiveness. Results from this work are many times more valuable in terms of identifying best practices, needs for improvement, and overall strength of the program. The benefits include providing independent evidence of the advancement of the compliance program for executive leadership, Boards, and government agencies.

The compliance program effectiveness evaluation focuses on how the elements of the program are working in advancing the goals. Analyzing documents is just one part of the evaluation, but is far more than a checklist review. It involves a much more critical examination of the content of the written documents to lay the foundation and planning for the onsite review. The major portion of the work is done onsite, where a team of experts engages in the testing and evaluation of program operations which includes, but is not limited to: (a) observing firsthand operations; (b) reviewing hotline logs and how complaints are managed; (c) conducting extensive interviews with key members of the executive team, Board members, program managers, and selected staff on their understanding of the program; (d) reviewing how successfully written guidance has been implemented and whether it is achieving the desired outcomes; (e) testing processes for credentialing and sanction screening of covered persons; (f) critically examining the content of compliance education and training programs; (g) reviewing the ongoing monitoring of high risk areas by program managers; and (h) reviewing the ongoing auditing processes that verify and validate the effectiveness of monitoring. Depending on the scope of work and expertise level of the evaluators, the resulting report may range from 35 to 50 pages in length. The report should include findings of program weaknesses and opportunities for improvement, along with specific recommendations and suggestions for remedial actions.

Deciding on Approach

The relative costs for a gap analysis and effectiveness review can vary significantly depending on scope of work, complexity of the organization operations, and maturity of the program. However, generally speaking, a compliance program gap analysis will cost only about a third to half of a full compliance program effectiveness evaluation, but with limited results. For either type, it is critical that those performing the work are highly experienced and experts in the field. Even in a checklist review approach, having the experience to be able to qualitatively evaluate documentation and identify weaknesses is very important to obtaining valuable results. It is therefore advisable to only engage a firm that has frequently performed compliance program reviews. It is not unreasonable to expect a qualified firm to have conducted 50 or 100 reviews. Equally important is identifying the qualifications and experience of the individuals assigned to perform the review.

Need Help Choosing Between a Gap Analysis or an Effectiveness Evaluation?

Strategic Management Services has compliance experts with years of experience evaluating the effectiveness of compliance programs. If you have questions regarding how to best get your compliance program reviewed, give us a call at (703) 683-9600 or contact us online.