An artificial intelligence algorithm allows a supercomputer to process data far beyond its theoretical capacity

2023-09-22 15:15:49

A machine learning algorithm (a type of artificial intelligence) has shown that it can provide the computer where it is installed with the capacity to process data even if it vastly exceeds the computer’s available memory. The algorithm identifies the key characteristics of a massive set of data and divides the set into manageable batches that do not drown computer hardware.

Developed at the Los Alamos National Laboratory, the algorithm broke a world record by processing huge data sets during a test carried out on Summit, the fifth fastest supercomputer in the world, installed at the Oak Ridge National Laboratory, United States.

Equally efficient on a supercomputer and a laptop, the algorithm solves hardware bottlenecks that prevent processing information from data-intensive applications in fields such as cancer research, image analysis captured from satellite, large-scale social media data processing, earthquake study, and national security, to name just a few.

The algorithm has been developed by Ismael Boureima’s team, from the Los Alamos National Laboratory in the United States.

“Traditional data analysis requires data to fit within memory limitations. Our approach challenges this notion,” said Manish Bhattarai, a machine learning specialist at Los Alamos National Laboratory and a member of the research and development team. “When the volume of data exceeds the available memory, our algorithm divides the data set into smaller segments. It processes these segments one at a time, moving each segment into and out of memory as appropriate. This technique gives us a unique ability to efficiently manage and analyze extremely large data sets.”

The Summit supercomputer seen partially from one of its corners. It occupies an area as large as an industrial warehouse. (Photo: Oak Ridge National Laboratory)

In the record run of the algorithm, a 340 terabyte dense matrix and an 11 exabyte sparse matrix were processed using only 25,000 GPUs.

Boureima, Bhattarai and their colleagues lay out the technical details of their algorithm and what it does in the academic journal The Journal of Supercomputing, under the title “Distributed Out-of-Memory NMF on CPU/GPU Architectures.” (Source: NCYT from Amazings)