In December 2014 I wrote a post titled Finding the elusive big wisdom in big data. We have made big advances since then in storing, processing and generally handling large volumes of data in a digital context, but humans still have trouble when it comes to reliably finding relevant nuggets that can improve business outcomes. I’m revisiting the idea to see how far we’ve come and how far we still have to go.
Baseball is one of the best examples of how data can impact a business. The book and movie “Moneyball” showed how Oakland Athletics’ GM Billy Beane changed the sport by introducing advanced statistical analysis instead of simply relying on reports from human scouts. Today, baseball is ruled by analysts as much as experienced baseball professionals — but could there be too much data?
Alex Spier writing in the Boston Globe’s Sunday Baseball Notes column recently pointed out that Boston Red Sox Manager Alex Cora will have 11 coaches on staff this year for 26 players. Compare that with Terry Francona in 2011, who had six for 25, and you can see the number has nearly doubled.
Spier attributes this in part to the growing amount of data that teams are collecting, which requires more people to observe, interpret, and implement a plan to leverage. As Spier wrote, “The result? Several teams now feature three hitting coaches, and staffs keep growing in an effort to distill the mountains of information into a digestible form for the 26 players on a roster.”
“Startups that analyze and extract insights from fast data have garnered extremely high public and private market investor interest.” Deepak Jeevankumer
Baseball is like a laboratory for advanced statistical analysis, and businesses could learn a lot from watching how the sport deals with expanding datasets.
Companies shovel data into data lakes, tinker with machine learning models, and use this information to make decisions, but they still face fundamental decisions every day that involve their ever-growing mountain of information. Sure, machines and software can help sort it out — indeed, they are getting better at it — but anyone who has dealt with poorly targeted advertising or email marketing knows there’s much work to be done.
Deepak Jeevankumer, managing director at Dell Technologies Capital, says the difference in play now compared to 2014 is that companies now are focused on getting insights to the people who need it faster. “Big Data has become less relevant than ‘fast data.’ E-commerce shoppers, streaming media consumers, gamers, stock/crypto traders and enterprise marketeers are demanding fast insights and fast knowledge,” Jeevankumer told me.
He believes data needs to be analyzed at its source while streaming, and in real-time after a query is created, and that startups that create solutions that allow companies (and baseball teams) to do this will be more successful in the long run.