EVALUATION OF THE DEMONSTRATION. 103. Submission of Draft Evaluation Design. The state shall submit to CMS for approval a draft Evaluation Design for an overall evaluation of the demonstration within 120 days of CMS approval of the demonstration. At a minimum, the draft design must include a discussion of the goals, objectives, and specific hypotheses that are being tested, and identify outcomes measures that shall be used to evaluate the demonstration’s impact. It shall discuss the data sources, including the use of Medicaid encounter data, and sampling methodology for assessing these outcomes. The draft Evaluation Design must include a detailed analysis plan that describes how the effects of the demonstration shall be isolated from other initiatives occurring in the state. The draft design must describe the state’s process to contract with an independent evaluator.
EVALUATION OF THE DEMONSTRATION. The evaluation design must include a discussion of the goals and objectives set forth in Section II of these STCs, and develop evaluation questions specific to the changes implemented in the demonstration during this extension period.
EVALUATION OF THE DEMONSTRATION. 74. The evaluation design must include a discussion of the goals and objectives set forth in Section II of these STCs, and develop evaluation questions specific to the changes implemented in the Demonstration during this extension period.
a) The evaluation questions should include, but are not limited to:
i. To what extent has the provision of continuous eligibility affected the stability and continuity of coverage and care to adults? How has the implementation of the Statewide Enrollment Center impacted “churning” by Demonstration participants?
ii. A quantitative and qualitative assessment of the effectiveness of the provider and enrollee education and outreach efforts, as well as plan oversight and compliance monitoring, in minimizing the impact of the transition of individuals living with HIV into mandatory Medicaid managed care.
iii. To what extent has the mandatory enrollment of individuals living with HIV into MMC impacted their perceptions of care (FFS v. SNP v. mainstream)?
iv. Has the required enrollment of individuals living with HIV into Medicaid managed care (either mainstream plans or HIV SNPs) impacted quality outcomes, which in earlier studies showed that these individuals enrolled in managed care on a voluntary basis received better quality care than in FFS?
v. An assessment of the successes and failures, along with recommendations for improvement, of the HIV SNP program.
vi. Has the State‟s H-MH Demonstration resulted in demonstrable improvements in the quality of care received by Demonstration participants?
vii. To what extent has the H-MH demonstration produced replicable residency program design features that enhance training in medical home concepts?
viii. How has the H-MH demonstration helped the selected facilities improve both their systemic and quality performance under each initiative implemented by the selected facilities?
ix. How have the results of the PPR demonstration program informed changes in reimbursement policies that provide incentives to help people stay out of the hospital?
x. How has the PPR demonstration program improved quality and cost savings at selected facilities? To what extent are the interventions tested both replicable and sustainable?
xi. How has the additional funding provided under the Clinic Uncompensated Care program increased the use of patient-centered medical homes and electronic medical records?
xii. How the results of the family planning expansion program expanded access to family planning service...
EVALUATION OF THE DEMONSTRATION. 72. Submission of Draft Evaluation Design. The State should submit a draft evaluation plan to CMS no later than 120 days after the award of the demonstration. When developing the evaluation plan, the state should consider and include the following:
a) The specific research questions and hypotheses that are being tested. The research questions should focus on the programmatic goals and objectives of the demonstration and their potential impacts, particularly as they relate to CMS’ Three Part Aim of improving access to and experience of care, improving quality of health care and decreasing per capita costs.
b) A description of any experimental study design employed (e.g., cohort, controlled before-and-after studies, interrupted time series, case-control, etc.) including a proposed baseline and/or control comparison groups.
c) Quantitative and qualitative process improvement and outcome measures with corresponding specifications that will be used in evaluating the impact of the Demonstration, particularly as it relates to the Three Part Aim of improving access to and experience of care, improving quality of health care and decreasing per capita costs. The evaluation plan should ensure that all outcomes selected have a clear description and the numerator and denominator should be defined clearly.
d) Data sources and collection frequency.
e) The population being studied (consider the target population of the demonstration), including the sampling methodology for selecting the population being included in your analysis.
f) A detailed analysis plan that describes the statistical methods that will be employed, particularly those that will allow for the effects of the Demonstration to be isolated from other initiatives occurring in the State. The level of analysis might be at the beneficiary, provider, and aggregate program level, as appropriate, and may include population stratifications to the extent feasible, for further depth. Qualitative analysis methods should also be described.
g) The timelines for evaluation related deliverables.
EVALUATION OF THE DEMONSTRATION. 92. Submission of a Draft Evaluation Design Update. The Commonwealth must submit to CMS for approval a draft evaluation design update no later than 120 calendar days after CMS’ approval date of the renewal. The draft evaluation design update must build and improve upon the evaluation design that was approved by CMS for demonstration period ending on June 30, 2014. At a minimum, the draft evaluation design must include a discussion of the goals, objectives, and evaluation questions specific to the entire health care reform demonstration set forth in section II of these STCs. The draft design must discuss the outcome measures that will be used in evaluating the impact of the demonstration during the period of approval, particularly among the target population, specific testable hypothesis, including those that focus on target populations for the demonstration and more generally on beneficiaries, providers, plans, market areas and public expenditures. The updated design should be described in sufficient detail to determine that it is scientifically rigorous. The data strategy must be thoroughly documented It must discuss the data sources, including the use of Medicaid encounter data, and sampling methodology for assessing these outcomes. The draft evaluation design must include a detailed analysis plan that describes how the effects of the demonstration shall be isolated from other initiatives occurring in the Commonwealth. The draft design must identify whether the Commonwealth will conduct the evaluation, or select an outside contractor for the evaluation. The design should describe how the evaluation and reporting will develop and be maintained to assure its scientific rigor and completion. In summary, the demonstration evaluation will meet all standards of leading academic institutions and academic journal peer review, as appropriate for each aspect of the evaluation, including standards for the evaluation design, conduct, and interpretation and reporting of findings. Among the characteristics of rigor that will be met are the use of best available data; controls for and reporting of the limitations of data and their effects on results; and the generalizability of results. Information from the external quality review organization (EQRO) may be considered for the purposes of evaluation, as appropriate. The updated design must describe the state’s process to contract with an independent evaluator, ensuring no conflict of interest.
EVALUATION OF THE DEMONSTRATION. Submission of a Draft Evaluation Design. The Commonwealth must submit to CMS for approval a draft evaluation design no later than 120 days after CMS’ approval of the demonstration. At a minimum, the draft evaluation design must include a discussion of the goals, objectives, and evaluation questions specific to the entire health care reform demonstration set forth in section II of these STCs. The draft design must discuss the outcome measures that will be used in evaluating the impact of the demonstration during the period of approval, particularly among the target population. It must discuss the data sources, including the use of Medicaid encounter data, and sampling methodology for assessing these outcomes. The draft evaluation design must include a detailed analysis plan that describes how the effects of the demonstration shall be isolated from other initiatives occurring in the Commonwealth. The draft design must identify whether the Commonwealth will conduct the evaluation, or select an outside contractor for the evaluation.
EVALUATION OF THE DEMONSTRATION. State Must Separately Evaluate Components of the Demonstration. As outlined in subparagraph (a), the outcomes from each evaluation component must be integrated into one programmatic summary that describes whether the State met the Demonstration goal, with recommendations for future efforts regarding all programs in the Demonstration. The State must submit to CMS for approval a draft evaluation design no later than October 1, 2011.
EVALUATION OF THE DEMONSTRATION
