The flawed statistical methods of charlatans promoting charters

Today, I take a path away from the local situation in Bellevue and examine a specific report that puts the New York charter schools way ahead of the New York public schools (pdf). The report comes from Dr. Caroline Hoxby, a proponent of charters who has used statistically incorrect methods - according to the Educational Policy Institute (EPI) (pdf) - to exaggerate some limited measured success of charter schools.

This is certainly not the first report from Dr. Hoxby who remains a firm advocate in using whatever means available - statistically correct or not, backed by science or otherwise -  to mislead and misdirect the debate on charters. She has persisted in rebutting the statistical methods of CREDO-Standford in their much cited study that finds that charter schools lead to no significant improvement. These rebuttals have been rebutted repeatedly by CREDO. While the CREDO web site carries the full arguments (look under CREDO-Hoxby Debate), Dr Hoxby's web site, in the manner suited to charlatans, conveniently only carries her version of the story. Data can increasingly be used in this unethical manner to distort the issues and present the lie as the truth. These efforts however, strike a body blow at the corporate deform movement (you may have heard this being referred to as "reform" by some mistake) that uses data as the final and only yardstick to evaluate learning and teaching.

It needs to be repeatedly and insistently voiced that the public does not rely on the unethically manufactured conclusions using data in provably incorrect manner. Just as the public has no faith in the provably incorrect representation of data by the charter proponents, it has no faith in data driven conclusions made by people cut from the same cloth - those that worship at the altar of neoliberal "free" markets - to misguide the education process for the children and youth of this nation.

EPI finds a statistical flaw in Dr. Hoxby's report that the experiment "destroys the randomization of the sample". The main argument is that comparing scores at each grade level to later make a cumulative comparison destroys the randomization present at the very beginning. EPI agrees with Dr. Hoxby that the subjects are completely randomized at the beginning due to the lottery effect making sure that students from largely similar backgrounds are indeed measured in both lotteried-in and lotteried-out groups.

In fact, there is more to this than that. The EPI rebuttal - as solid as it is - misses the crucial variable of the peer group. The student group of the charter school undeniably has a better motivated, more educationally capable mix than the local public school. This is true even if students are lotteried, as it is the case that economically disadvantaged parents are less likely to apply for the lottery. Also, any student who cannot make it in the strict regime of the charter school will be summarily dismissed and re-enrolled in the public school. I have explained the complete reasoning on the process of privatization here. Thus Hoxby's study attempts to fix all variables except the school (charter vs public) but crucially cannot fix the peer group variable. Thus whatever measurements made will reflect the effect of two variables - the school and the peer group. Therefore the study loses all scientific validity.

One notable study that demonstrates the peer group effect is the work of Dave Orbits from "Where's the Math". Dave was kind enough to share some of his extensive research with me. Please see the following graph that plots the pass rate of various student groups with the poverty level on the X axis. The trend lines for both low income and high income students show a drop, and there is strong reason to believe that peer group is indeed the reason for this, as that is the main factor not being accounted for.

It should be revealing how subtle these issues seem when attempts are made to analyze them outside the social context. Insights into these issues can only be developed as we gain a deeper understanding of the basic injustice and in-equity of the American education system. Researchers and leading scholars on matters of education policy must discard the dead-weight of "disinterested" scholarship and undertake a serious attempt to challenge the powers that shape education policy with an empathy towards those who have no voice in the supposedly "free" market of neoliberal America. As the concerned public, we are fully aware of the dangers the academia face as they take upon themselves this grave task and we would support such efforts in any way we can.