With this book, managers and decision makers are given the tools to make more informed decisions about big data purchasing initiatives. Big Data Analytics: A Practical Guide for Managers not only supplies descriptions of common tools, but also surveys the various products and vendors that supply the big data market.
Comparing and contrasting the different types of analysis commonly conducted with big data, this accessible reference presents clear-cut explanations of the general workings of big data tools. Instead of spending time on HOW to install specific packages, it focuses on the reasons WHY readers would install a given package.
The book provides authoritative guidance on a range of tools, including open source and proprietary systems. It details the strengths and weaknesses of incorporating big data analysis into decision-making and explains how to leverage the strengths while mitigating the weaknesses.
The book further explores basic statistical concepts that, when misapplied, can be the source of errors. Time and again, big data is treated as an oracle that discovers results nobody would have imagined. While big data can serve this valuable function, all too often these results are incorrect, yet are still reported unquestioningly. The probability of having erroneous results increases as a larger number of variables are compared unless preventative measures are taken.
The approach taken by the authors is to explain these concepts so managers can ask better questions of their analysts and vendors as to the appropriateness of the methods used to arrive at a conclusion. Because the world of science and medicine has been grappling with similar issues in the publication of studies, the authors draw on their efforts and apply them to big data.
So What Is Big Data?
Growing Interest in Decision Making
What This Book Addresses
The Conversation about Big Data
Technological Change as a Driver of Big Data
The Central Question: So What?
Our Goals as Authors
The Mother of Invention’s Triplets: Moore’s Law, the Proliferation of Data, and Data Storage Technology
Parallel Computing, Between and Within Machines
Recap of Growth in Computing Power
Storage, Storage Everywhere
Grist for the Mill: Data Used and Unused
Marketing in the Physical World
Asset Reliability and Efficiency
Process Tracking and Automation
Toward a Definition of Big Data
Putting Big Data in Context
Key Concepts of Big Data and Their Consequences
Power through Distribution
Cost Effectiveness of Hadoop
Not Every Problem Is a Nail
Some Technical Aspects
Hadoop File System
Pig and Hive
Current Hadoop Ecosystem
Amazon Web Services (AWS)
To Run Pig Latin Using Powershell
HBase and Other Big Data Databases
Evolution from Flat File to the Three V’s
Transition to Big Data Databases
What Is Different bbout HBase?
What Is Bigtable?
What Is MapReduce?
What Are the Various Modalities for Big Data Databases?
How Does a Graph Database Work?
What is the Performance of a Graph Database?
Machine Learning Basics
Classifying with Nearest Neighbors
Support Vector Machines
Improving Classification with Adaptive Boosting
Principal Component Analysis (PCA)
Singular Value Decomposition
Big Data and MapReduce
Statistics, Statistics Everywhere
Digging into the Data
Standard Deviation: The Standard Measure of Dispersion
The Power of Shapes: Distributions
Distributions: Gaussian Curve
Distributions: Why Be Normal?
Distributions: The Long Arm of the Power Law
The Upshot? Statistics Are not Bloodless
Fooling Ourselves: Seeing What We Want to See in the Data
We Can Learn Much from an Octopus
Hypothesis Testing: Seeking a Verdict
Hypothesis Testing: A Broad Field
Moving on to Specific Hypothesis Tests
Regression and Correlation
p Value in Hypothesis Testing: A Successful Gatekeeper?
Specious Correlations and Overfitting the Data
A Sample of Common Statistical Software Packages
Big Data Analytics
Big Data Giants
Google Product Offerings
Advertising and Campaign Performance
Analysis and Testing
Non-United States Social Media
Ranking Network Sites
Negative Issues with Social Networks
Some Final Words
Geographic Information Systems (GIS)
A GIS Example
Faceted Search versus Strict Taxonomy
First Key Ability: Breaking Down Barriers
Second Key Ability: Flexible Search and Navigation
Know Thy Data and Thyself
Structured, Unstructured, and Semistructured Data
Data Inconsistency: An Example from This Book
The Black Swan and Incomplete Data
How Data Can Fool Us
Aging of Data or Variables
Missing Variables May Change the Meaning
Inconsistent Use of Units and Terminology
Data as a Video, Not a Snapshot: Different Viewpoints as a Noise Filter
What Is My Toolkit for Improving My Data?
Force Field Analysis
Troubleshooting Queries from Source Data
Troubleshooting Data Quality beyond the Source System
Using Our Hidden Resources
Converting Data Dreck to Usefulness
Geographical Information Systems
New York City
Tucson, Arizona, University of Arizona, and COPLINK
General Educational Data
Grades and other Indicators
Addresses, Phone Numbers, and More
Part Two: Basic Principles of National Application
Collection Limitation Principle
Data Quality Principle
Purpose Specification Principle
Use Limitation Principle
Security Safeguards Principle
Individual Participation Principle
Affirming the Consequent
Denying the Antecedent
Consistency and Hindsight Biases
Von Restorff Effect
Converting Data Dreck to Usefulness Sales
Making Yourself Harder to Track
Michael Porter’s Five Forces Model
Bargaining Power of Customers
Bargaining Power of Suppliers
Threat of New Entrants
The OODA Loop
Implementing Big Data
Nonlinear, Qualitative Thinking