Big data requires new technologies

The phrase “big data” will likely run its course at some point – thanks in no small part to the increasingly vague way it is now used to describe even simple data analytics. But that doesn’t mean that the technology behind the catchphrase will suffer the same fate. Big data in its original sense (which is how it is to be understood here) describes "data sets so large or complex that traditional data processing applications are inadequate". 

In other words, big data requires new technologies. Generally speaking, the benefit of big data lies in crunching large sets of data from the (recent) past to draw conclusions that can be used to make the best possible decisions for the future (sometimes in real time).

The underlying theory is that wherever rules or mathematical formulas alone fail to yield useful forecasts, a combined analysis of all possible marginal parameters will make it possible to identify certain dependencies and patterns that allow more precise statements about the future.

When is data big data?

For example: Knowledge of engineering makes it possible to determine the maximum number of hours a jet engine can be in service until maintenance is required. But it takes the real-time analysis of an array of sensors in the engines measuring the tiniest irregularities during operation (temperature distributions, imbalances, etc.) to make an accurate prediction of when maintenance is actually needed.

So, at what point is big data actually “big”? When is data analysis by conventional methods sufficient, and when are big data methods really called for? In our jet engine example above, the sensor readings on one flight of a Boeing 787 would generate an astounding 500 GB. On one flight. That is big data. What would that mean in a supply chain? Think about it – how much data do you think your supply chain generates?