Publication

Confidence and Precision in Claims Audits: Quality of the Estimate

Cornelia Dorfschmid | December 2011

The contractor reform in health care brought a consolidation of Medicare contractors and new contractors, as exemplified by the Medicare Administrative Contractors (MACs), Medicaid Integrity Contractors (MICs), Medicare Recovery Audit Contractors (RACs), and Zone Program Integrity Contractors (ZPICs). These government contractors have different objectives, some are more fraud oriented (e.g., ZPIC), and others are focused on detecting payment errors (e.g., MACs, RACs). They conduct pre- and post-payment audits. However, no matter what their charge and CMS-assigned tasks are, these contractors have aggressively been monitoring and auditing claims that were paid to health care organizations by the federal and state health care programs. In their claims audits, the contractors typically assess whether there were inappropriate payments received by a health care organization and, if so, they determine the recovery amount.

Oftentimes the totality of cases (e.g., charts, claims, line item of claims, beneficiaries, or whatever the unit of observation may be), which may potentially be affected by a suspected billing error, cannot be reviewed. Time and cost constraints and benefit/cost considerations make a sample a much more viable alternative. If the sample is a statistically valid random sample (SVRS), such as a “probability sample” as set forth in the Centers for Medicare & Medicaid Services (CMS) Medicare Program Integrity Manual (PIM), then the contractor may draw conclusions from the sample to the universe (total number) of cases. Simply put, one can estimate the total overpayment in the total number of cases by projecting overpayments from a relatively small sample to the universe at large.

Similar considerations, which weigh the possibility of using the universe of cases affected by a potential payment error pattern versus a sample with appropriate projection, are increasingly also part of many providers’ internal auditing and monitoring strategies. So what does it take to develop a good estimate? Three aspects can be considered.

• Correct interpretation of the projected estimate.To begin with, it requires that the estimate is projected from a random sample that was based on the correct interpretation and application of the various medical documentation requirements and payer coverage rules. If the medical review, the application of coverage criteria, and case-by-case review findings can be challenged in an appeal or a quality assurance process, the overpayment estimate derived from the sample would not be tenable.
• Statistically valid random sampleAnother aspect of a good estimate is that it must be generated from a statistically valid random sample that was selected. If there is no statistically valid sample, then the validity of the projection of the total overpayment estimate is difficult to defend.
• Confidence and precisionIf each sampled case was reviewed correctly and the sample was a statistically valid random sample, acceptable confidence (i.e, degree of certainty that the sample correctly depicts the universe) and precision (i.e., range of accuracy) are the third piece needed for a quality estimate of the total overpayment in the universe.

Precision and confidence trade–off

The quality of the estimate depends on the precision and confidence levels reached in the estimation process. Both high confidence and high precision are desirable qualities in an overpayment estimate. They are expressed in ranges or percentages. There are one- and two-sided confidence intervals that are reported at various levels of precision. One-sided confidence intervals deal with whether the true value for the universe (e.g., total overpayment) is greater or smaller than the estimate. Two-sided confidence intervals deal with whether the true value for the universe is between two given numbers (lower and upper bounds), i.e., a bounded range.

In overpayment extrapolations, the quality of the estimate of total overpayment in the universe can be described as a range or a percentage for the so-called “point estimate.” The point estimate is simply the average overpayment in the sample inflated by the universe size. Using the point estimate views the universe as just a larger version of the sample. The observed confidence and precision levels are statistical measures calculated from the data of a particular sample and qualify that view by reporting the uncertainty that is associated with the particular point estimate. Namely, confidence and precision allow us to give a range of uncertainty around the point estimate. This range implies that the random sample that was drawn is not an exact miniature version of the universe. Each sample may render a somewhat different picture, hence a range.

For example, if an auditor reports the overpayment estimate of $10,000 with a two-sided 90% confidence interval and 5% precision level, this means that the auditor is 90% certain that the true overpayment value for the universe is $10,000 +/- $500, (i.e., is between $9,500 and $10,500). The $500 is the precision amount and the precision percentage is 5%. Clearly, if we could raise the confidence level to 95% or even 99% for that same precision level of 5%, that would render a higher quality estimate. Similarly, if we could tighten the precision range and thereby make the estimate more precise at 90% confidence level (e.g., make it +/- 3% or equivalent to a $600 range), that would also be preferable.

For any given sample, the auditor can always raise the certainty (i.e., raise the confidence level) by making the statement about the estimate in relation to the true value in the universe less precise. The extreme position may be illustrative. For example, one can always say for the point estimate that one is 100% confident that the true value is between minus infinity and plus infinity (i.e., pairing complete imprecision with complete certainty/confidence). Any other more meaningful combinations of confidence/precision levels for the same point estimate that was generated from one-and-the-same sample are just re-statements of an inherent tradeoff. This tradeoff renders the same statistical information. To conclude, whenever the sample size is fixed and the point estimate is calculated, there is a tradeoff between confidence and precision levels. One cannot improve both any more.

Another valid method of stating confidence and precision levels that is sometimes used for point estimates in overpayment cases is using the one-sided confidence interval. In this approach, the auditor typically states the confidence level of the point estimate at a lower limit. Rather than reporting a bounded range around the estimate, the range becomes open-ended. For example, the auditor may report a point estimate of $10,000 for a one-sided 90% confidence interval with a $9,650 lower limit. This means the auditor reports a point estimate of $10,000 for which he/she is 90% confident that the true overpayment in the universe is no less than $9,650. No upper bound is reported for this one-sided confidence interval; the interval is open-ended upwards.

Sample size affects precision and confidence

Generally speaking and all else being equal, the higher the confidence and precision levels of a point estimate, the closer one is to the true overpayment in the universe. High precision and confidence are therefore indications of a quality estimate. The larger the sample size that underlies the overpayment extrapolation, the better the confidence and precision levels (i.e., narrower ranges around the point estimate) one may expect. These basic concepts, however, are not always fully understood and therefore, sample size and validity of the estimate can be confused in this context.

One of the first issues, which I often see raised by providers and/or their attorneys in appeals that involve extrapolated overpayments in claims, is that the sample size was too small and hence the extrapolation not valid. However, the validity of the sample does not depend on the sample size. A small but statistically valid sample can always generate a valid overpayment estimate. Validity is only a necessary condition, but not a sufficient condition for a quality estimate. The quality of the estimate is, however, affected by the sample size because it affects confidence and precision levels. The latter are statistical concepts that describe the quality of the estimate. They can also be set as thresholds or targets and then used in the decision process to determine when valid overpayment estimates are actually “good enough” (i.e., of acceptable quality).

OIG and CMS on confidence and precision

Once the auditor has selected the sample, the size is set and claims get reviewed. The precision percentage for the point estimate derived from the particular sample can then be reported at 80%, 90%, 95%, or even 99% confidence level. It is a matter of choice how it is reported. The higher the confidence level used to report the point estimate, the lower the precision percentage will be. This trade–off between confidence and precision is simply driven by the data audited and laws of statistics. Without more observations one cannot improve both. But, what should be done? How should it be reported? How do we compare apples to apples?

Government contractors and agencies set different objectives that describe the minimum standards or targets levels for the quality of the point estimate (i.e., precision and confidence levels of the total overpayment estimate). A 90% confidence interval is most frequently used in government audits and self-disclosures. Understanding the basic concepts above and nuances in the government agencies and contractors’ expectations are critical to success in:

• appealing a claims review result or recovery amount demand, and
• applying government auditing standards and best practice claims review methods proactively in provider-internal monitoring and auditing projects.

OIG (Office of Inspector General of the US Department of Health and Human Services) has put forward target levels for:

• providers that self-report overpayments to the OIG, and
• Independent Review Organizations (IROs) that review claims of certain health care entities under Corporate Integrity Agreements (CIAs).^[1]

OIG requires reporting the overpayment point estimate for the universe at 90% confidence level (two-sided) and requires that it must reach a precision of +/- 25%. If the precision of the point estimate that is reached is worse and exceeds this percentage threshold, the point estimate is not considered acceptable. One way to cure this deficiency would be to increase the sample size and re-project. CMS is less clear in its requirements on confidence and precision levels for the point estimate of overpayments. They allow the point estimate for recovery amount demanded, but only if the precision is high. The CMS Medicare Program Integrity Manual(PIM) states the following:

In most situations the lower limit of a one-sided 90 percent confidence interval shall be used as the amount of overpayment to be demanded for recovery from the provider or supplier. The details of the calculation of this lower limit involve subtracting some multiple of the estimated standard error from the point estimate, thus yielding a lower figure. This procedure, which, through confidence interval estimation, incorporates the uncertainty inherent in the sample design, is a conservative method that works to the financial advan- tage of the provider or supplier. That is, it yields a demand amount for recovery that is very likely less than the true amount of overpayment, and it allows a reasonable recovery without requiring the tight precision that might be needed to support a demand for the point estimate. However, the PSC [Program Safeguard Contractor] or ZPIC BI [Benefit Integrity] unit or the contractor MR [Medical Review] unit is not precluded from demanding the point estimate where high precision has been achieved.^[2]

The PIM does not define what exactly “high precision” means in claims audits and recovery situations. There is no clear CMS standard for acceptable precision in overpayment audits. The ambiguity has led to much debate and can be an issue raised in the appeals process. CMS contractors routinely use and demand the lower limit of the one-sided 90% confidence interval of the point estimate as the recovery amount. The problem with this is that technically, one can always calculate the lower limit of a one-sided 90% confidence interval for any point estimate, regardless how precise it may be. What remains unclear is when a low precision is just too low to render a point estimate that is still meaningful. Without a meaningful point estimate, however, there may also not be a good reason in calculating and using a lower bound either. OIG, in that respect, defined a much clearer standard for the health care industry.

This author believes the OIG’s standard of 90% confidence/25% precision is one that is clear and worth referring to as a minimum standard in any claims audit that involves overpayment extrapolation using the point estimate. Providers and suppliers may want to consult this standard in any payment disputes.

[1]

HHS Office of Inspector General, Publication of the OIG’s Provider Self-Disclosure Protocol (1998). Available at http://oig.hhs.gov/authorities/docs/selfdisclosure.pdf

[2]

See, CMS Medicare Program Integrity Manual, Chapter 8-Administrative Actions and Statistical Sampling for Overpayment Estimate, 8.4.5.1 – The point estimate. (Note, the PIM was recently rearranged and sections of Chapter 3 moved to Chapter 8.) Available at http://www.cms.gov/manuals/downloads/pim83c08.pdf; http://www.cms.gov/Transmittals/Downloads/R377PI.pdf