The author reflects on a symposium panel discussion on “Swimming in the Ocean of Big Data: National Security in the Age of Unlimited Information” that occurred before the Snowden disclosures. He analyzes the panel discussion in context of the time at which it occurred and compares it to what has become known since June 2013. The article then focuses on the path to reform, specifically by focusing attention on the strengths and weaknesses of data collection by both the public and private sectors.
In this article, the author argues that the premise of “big data” lies not in the amount of data that we can generate, collect or store, but in the ability to use data to make informed decisions. He explores both the needs of data analysis projects and how to apply these types of projects to real-world circumstances. The article concludes that whatever data systems are ultimately put in place, privacy and civil liberties should be a primary concern and not simply an afterthought; the author proposes methods that can be implemented during the design of big data programs in order to curtail privacy violations.
The author explores the origins of “big data” and how the phenomenon was able to spread to a wider audience through parallel technological advances in computer hardware and the open source movement. The article focuses specifically on the progress made by Google related to a series of papers published from 2003 to 2006 that were instrumental in the development of other major tech companies that were able to create provide technical support, making the process of deciphering big data more accessible. The paper then examines how this “democratization” has affected the national security community by providing better intelligence at a lower cost.