Cognitive psychologists like to use visual illusions to make the point that while we may be very intelligent, rational beings, our brains rely on heuristics, or mental shortcuts to gather clues about our surroundings and leap to conclusions based on them. Although we tend to benefit greatly from this setup, very often those conclusions are severely flawed. It is easy to see the shortcomings visually, but recognizing when we make cognitive mistakes requires greater effort, not to mention a serious dose of humility.
The Problem with Flow Info On Friday, I spoke at Drobny Global’s annual macro conference in Santa Monica. Afterwards, I sat and chatted with an allocator from one of the large government pension funds who asked, “Don’t you think flows matter?” Being a reader of Seeds for some time now, we were both aware that she already knew my answer, however it didn’t mean that her question was a rhetorical one. It simply meant I hadn’t yet convinced her. There are numerous reasons why I cut myself off from flow information long ago. All of them being statistical in nature. I will do my best to chip away at its value by attacking but one of its legs with a related story.
The Gates Foundation is widely accepted as the leader in education reform. The organization is of course backed by multi-billionaire Bill Gates, viewed by most as one of the great minds of our time and an exceptional business leader. As it is with many similar organizations, the belief is that by applying the rigorous analytical tools that led to Microsoft’s incredible success in software, the same can be achieved with the many social causes to which his foundation turns its attention. None of this is contentious, and in fact, the foundation carries so much weight, that the mere mention of its backing for a non-profit or research finding will virtually guarantee a veritable avalanche of additional support from others, as well.
In the United States, one of the great frustrations is that K-12 (kindergarten through 12th grade) education, once the gold standard for the rest of the world, has fallen well behind, despite nearly $650 billion being spent on it every year. Not only are public funds thrown at it, but it is a favorite pet project of many who come from the business world, armed with massive war chests, rivaled only by the confidence they have in their ability to effect positive change. Theories abound as to how we got here and what should be done to fix the problem. School lunches, higher pay for teachers, and computers in every classroom are but a few of the solutions that have been proposed, received tremendous funding, and been implemented.
In the late 1990’s, the argument that smaller schools produced better results was gaining steam, and powerful adherents. Research, such as that based on test results from the Pennsylvania System of School Assessments, had made a compelling argument in favor of smaller schools. Being based on cold, raw data, it was the kind of result that appeals to great business minds like Bill Gates, and the evidence was compelling.
They’d gathered scores from 3rd, 5th, 8th and 11th grade math and reading tests, plus writing scores for 6th, 9th and 11th grade. In analyzing the data provided by 1,662 separate schools, they found that of the 50 top-scoring schools (the top 3%), 6 of them were among the 50 smallest (the smallest 3%). If the size of the school were unrelated to performance, the smallest schools should have represented just 3% of the top 50, but according to the data they actually represented 12% (6 out of 50). That’s an overrepresentation by a factor of four!
The Gates Foundation was sold. They began pouring money into programs designed to support small schools, nationwide. By 2001, they provided roughly $1.7 billion in grants to education projects and were quickly joined by the upper echelon of not-for-profits, including the Annenberg Foundation, Carnegie Corporation, Harvard’s Change Leadership Group, the Pew Charitable Trusts and the U.S. Department of Education’s Smaller Learning Communities Program. As Wainer and Zwerling so accurately stated in their follow up research, “The availability of such large amounts of money to implement a smaller-schools policy yielded a concomitant increase in the pressure to do so, with programs to splinter large schools into smaller ones being proposed and implemented broadly in New York, Los Angeles, Chicago and Seattle.”
In market parlance, the smart money flows became evident, and had become a powerful force in the direction of education reform in their own right. Lives were uprooted and impacted for years to come. Westlake Terrace High School, a suburban school in Seattle with 1,800 students was broken into 5 smaller schools, enabled by a Gates grant of nearly $1 million. It was just one of many such changes being carried out across the country. School boards were taking action, not based on the underlying data that had created the flows, but on the flow itself. Articles were written about the flows, who was behind it and the action being taken as a result. Politicians jumped on board ideologically, and financially.
Then, in 2005, the Gates Foundation made a stunning announcement. They were moving away from converting large schools into smaller ones. They decided that improving classroom instruction was more important to improving schools than breaking down the size. What led them to stop out after leading the charge, and seemingly mid-course? Turns out their initial analysis was severely flawed, and both it and the follow up data made a very strong case that not only were smaller schools not better, they may actually be worse.
The researchers who made the argument that smaller schools were better than larger ones had shown an exaggerated faith in small samples, a very common problem in research of all kinds, even today. Recent advances in cognitive psychology have taught us that we pay more attention to the content of messages than to information about their reliability. As Daniel Kahneman points out in Thinking, Fast and Slow, “Many facts are due to chance, including accidents of sampling. Causal explanations of chance events are inevitably wrong.”
That’s what happened in the case of the findings re: smaller schools. You might look at the fact that the data included 1,662 schools and think that’s a sufficient sample size from which to draw conclusions, but that isn’t the problem. The error occurred in not recognizing that each of the small schools represented a far smaller sample size than those of the big schools. What you should expect when dealing with small sample sizes versus large ones, is a higher degree of variability. As an example, the chances of tossing a fair coin 5 times and having it land on heads every time is far greater than if you tossed it 500 times. The researchers, Gates, Harvard, Pew and the others that had reviewed the data focused on only one side of the results. Had they gone one extra step and looked at the opposite end of the spectrum exclusively, they would have arrived at the exact opposite conclusion.
Going back to the Pennsylvania System of School Assessments data, among the 50 worst performing schools, you will find that 9 of them were among the 50 smallest schools. In others words, there was an overrepresentation by a factor of 6! Far bigger even than what they found on the positive side. Truth is, drawing conclusions based solely on that information is just as flawed as the other way around. All it proved is that smaller samples generate data with greater variance. When a regression was done of all the data, it showed no relationship between results and school size. However, when applied to those of high school students, the regression line showed a significant positive slope. In other words, the larger the high school, the better the scores.
The Gates Foundation recognized the error and stopped out of the position. Harvard, Pew, Carnegie and the others followed suit shortly thereafter. Unfortunately, the students, teachers and communities were left holding the position with no one around to help them put the schools back together. No new funding to fix the problems. Principals at the larger schools that had been broken up were held accountable for the diminishing returns on their students. School boards had to answer for the inefficiencies that resulted from the greater overhead of administration per student required for the smaller schools. In other words, those who had invested based on the smart money flows were left holding the bag. If instead, they had done their own research, dug into the data themselves and recognized where their time and effort would have been better spent, they would have been years ahead of everyone else, being held up as the standard by which all others would aspire to.
So when you ask me why it is that I don’t want to know about trading flow information, my simple answer is that it is more noise than signal. Whatever limited selection of flow info that might make its way to me is but a small sample of the total, leading to the very same problem experienced by the Gates Foundation. I’d prefer to depend upon fundamental data, which if you think about it, is also flow data, representing behaviors that are affecting price action, but from a much larger sample set, and therefore worthy of much greater confidence.
Besides, if I were to generate poor returns and an investor or client came to me after the fact to ask why, I never want my answer to be, “Well, everyone else was doing it.” Further Evidence of Historic Levels of Risk Aversion In my speech at the Drobny Global conference, I made the argument that markets are exhibiting extreme risk aversion, to which I received some serious pushback. Audience members suggested (quite vehemently at times) that the rally in equity markets, and particularly the compressed spreads serve as evidence of quite the opposite of what I was proposing. I will go into why these arguments are flawed in the next edition of Seeds, but I thought it worth noting that immediately following my talk, and the counterarguments, the conference hosts posed a number of questions to the audience related to positioning. Nearly every single one of them showed an almost perfect bell shaped curve result, prompting Andres to remark that he’d rarely seen results so symmetrical, with the preponderance of audience members selecting No Position, and a diminishing distribution as you approached either Very Overweight and Very Underweight. In other words, the very people who were arguing that the markets are not exhibiting indications of extreme risk aversion had just provided clear evidence of exactly that in their own behaviors.
"While everyone else is scrambling to answer who, what, where and when, Duneier is focused on explaining the 'why'."
About the Author For nearly thirty years, Stephen Duneier has applied cognitive science to investment and business management. The result has been the turnaround of numerous institutional trading businesses, career best returns for experienced portfolio managers who have adopted his methods, the development of a $1.25 billion dollar hedge fund and 20.3% average annualized returns as a global macro portfolio manager.
Mr. Duneier teaches graduate courses on Decision Analysis in the College of Engineering, as well as Behavioral Investing, at the University of California.
Through Bija Advisors' coaching, workshops and publications, he helps the world's most successful and experienced investment managers improve performance by applying proven, proprietary decision-making methods to their own processes.
Stephen Duneier was formerly Global Head of Currency Option Trading at Bank of America, Managing Director in charge of Emerging Markets at AIG International and founding partner of award winning hedge funds, Grant Capital Partners and Bija Capital Management. As a speaker, Stephen has delivered informative and inspirational talks to audiences around the world for more than 20 years on topics including global macro economic themes, how cognitive science can improve performance and the keys to living a more deliberate life. Each is delivered via highly entertaining stories that inevitably lead to further conversation, and ultimately, better results.
His artwork has been featured in international publications and on television programs around the world, is represented by the renowned gallery, Sullivan Goss and earned him more than 50,000 followers across social media. As Commissioner of the League of Professional Educators, Duneier is using cognitive science to alter the landscape of American K-12 education. He received his master's degree in finance and economics from New York University's Stern School of Business.
Bija Advisors LLC In publishing research, Bija Advisors LLC is not soliciting any action based upon it. Bija Advisors LLC’s publications contain material based upon publicly available information, obtained from sources that we consider reliable. However, Bija Advisors LLC does not represent that it is accurate and it should not be relied on as such. Opinions expressed are current opinions as of the date appearing on Bija Advisors LLC’s publications only. All forecasts and statements about the future, even if presented as fact, should be treated as judgments, and neither Bija Advisors LLC nor its partners can be held responsible for any failure of those judgments to prove accurate. It should be assumed that, from time to time, Bija Advisors LLC and its partners will hold investments in securities and other positions, in equity, bond, currency and commodities markets, from which they will benefit if the forecasts and judgments about the future presented in this document do prove to be accurate. Bija Advisors LLC is not liable for any loss or damage resulting from the use of its product.
Learn how Bija's proven, proprietary approach to decision making helps the world's top institutional investors generate better results.