Statistical Sampling and Validity

Statistical Methodology Design and Critique

Given that statistics is sometimes as much art as science, statisticians often disagree on the most appropriate method to use to analyze data, even if the desired results are precisely the same.  In order to resolve such disputes, Precision Consulting often provides detailed third party reports on optimal methodology design, including comments about the validity and reliability of the existing model. To do this effectively, it is insufficient to simply have a statistical understanding of the topic at hand, and rather requires an intuitive understanding as well.  In this way, we can help bridge the gap between statistics and real-life, and the methodology critique will reflect not only statistical shortfalls, but also intuitive ones.

Once the basic structure and general aims are finalized, we can perform tests on standard model reliability and statistical power features.  Importantly, a good statistical methodology will have broad reliability and validity, be robust under changes in underlying assumptions, and have high statistical power without sacrificing confidence level.  Precision Consulting will then be able to provide the client with a full report about any shortcomings of the model, give the client suggestions for potential improvements, and defend this opinion in court.

Subsample Selection Criteria and Validity

Our team has helped dozens of firms with settling or mediating statistical disputes over subsample validity, and also with pre-emptive subsample selection (to minimize the likelihood of dispute). Since it is rarely possible to gather and test data on the full population of desired samples, it is necessary to both correctly specify the subsample to be tested, and to be able to defend that methodology in court. 

One of the issues that we see repeatedly in our expert testimony work is the issue of minimum required sample size. In order to estimate the needed sample size for a prospective study, we would conduct a statistical power analysis to determine a minimum sample size, and then would randomly select the sample from the population to meet the criteria determined in the power analysis (whether with or without stratification).  There are several other major considerations we consider when doing a power analysis for sample size determination, including:

  • The general approach to determining sample size assumes that a simple random sample is the sampling design. More complex designs, e.g., stratified random samples, must take into account the variances of subpopulations, strata, or clusters before an estimate of the variability in the population as a whole can be made.
  • The sample size should be appropriate for the analysis that is planned. In addition, an adjustment in the sample size may be needed to accommodate a comparative analysis of subgroups. Also, skewed distributions can result in serious departures from normality even for moderate size samples, causing the need for a larger sample.

In the event that the data analysis has already been run, but there is a dispute about the validity of the subsample, a similar analysis is done, but in reverse.  While the prior analysis was an ‘A Priori’ Power Analysis, the subsequent one is a ‘Post-Hoc’ Power Analysis.  The sample demographic and descriptive characteristics are rigorously compared to the population to check for discrepancies, and the effective ‘power’ of the results is computed.

Confidentiality

We understand the importance and necessity of confidentiality, and provide all of our prospective clients with a Non-Disclosure Agreement immediately upon contact. This ensures you that your information will not be shared with any third parties, and your consultation with us is completely confidential.

Please click here to read our Data Security Policy.