Leveraging inherent structure to work with 90% less data

Structure in real-world sensor data enables data reduction while preserving information

The amount of sensor data to be collected is dictated by the Shannon-Nyquist sampling theorem, which was proposed in the 1950s. It says that the sampling frequency must be at-least 2x of the highest frequency information present in the signal. For example, if your data has an acoustic signal of 22,000 Hz, then the sampling rate must be at-least 44,000 Hz.

However, in recent times, new techniques and insights have proven we can collect a much smaller fraction of raw data without losing information. When and how does this work?

The general idea is to be able to sample according to the information present in a signal. Less information → collect less data, more information → collect more data. Real world signals generally have some ‘structure’, indicative of the causal reason. This enables collapsing high dimensional signals into efficient low rank representations (less information). On the other hand, random noise lacks this structure and has the most amount of information.

Let’s compare a signal and random noise to understand the relation between structure in the data and the amount of information they contain / their compressibility. As can be seen in the figure below, a structure exists in signals, whereas no such structure exists in random noise. Generally speaking, ‘structure’ means patterns associated with the phenomena being represented by this data.

Signal → structure exists → NOT a lot of information / high redundancy

Noise → no structure exists → lot of information / no redundancy

In the above figure, we are seeing the structure in time-domain itself. However, there are better ways to represent this data, where the structures are much more evident. If we look at the Fourier domain below, the signal can be represented very efficiently (by just 2 parameters), whereas the noise cannot. Signals thus have high compressibility if looked at in appropriate domains, whereas noise does not.

Now, the structure does not have to exist in the Fourier or a fixed domain. If we use learning techniques, we can let the parameters of machine learning / deep learning models learn this structure, ultimately letting us collect less data without losing information.

Signal → structure → less information → collect 90% less data upfront

Noise → no structure → most information → collect more data upfront

We’re leveraging this insight at Lightscline to design novel architectures that can exploit the patterns in data and can be trained to collect 90% less data upfront. This has significant implications or the design of physical / embedded AI systems across space, air, land, and sea. You can learn more about Lightscline here.