MIT researchers have developed a new method for computing directly on these devices that significantly reduces latency. Their method moves the steps of running a machine-learning model that use a lot of memory to a central server, where model parts are written onto light waves.
The waves are transmitted to a connected device via fiber optics, which allows massive amounts of data to be sent in real time across a network. The receiver then uses a simple optical device that uses the parts of a model carried by the light waves to quickly do calculations.
When compared to other methods, this technique improves energy efficiency by more than a hundredfold. It may also make things safer because a user's information is not sent to a central place to be processed.
This method could allow a self-driving car to make decisions in real time while consuming only a fraction of the energy currently consumed by power-hungry computers. It could also be used for live video processing over cellular networks or even high-speed image classification on a spacecraft millions of miles away from Earth.
"You have to run the program every time you want to run a neural network, and how fast you can run the program depends on how fast you can pipe the program into memory." Our pipe is enormous—it is equivalent to sending a full-length movie over the internet every millisecond or so. That is how quickly data enters our system. "It can compute at that speed," says senior author Dirk Englund, an associate professor in the Department of Electrical Engineering and Computer Science (EECS) and a member of the MIT Research Laboratory of Electronics.
The paper's co-authors include EECS graduate student Alexander Sludds, EECS graduate student Saumil Bandyopadhyay, Research Scientist Ryan Hamerly, and others from MIT, the MIT Lincoln Laboratory, and Nokia Corporation. The study was published in Science today.
Reducing the workload
Neural networks are machine-learning models that use layers of interconnected nodes, or neurons, to recognize patterns in datasets and perform tasks such as image classification and speech recognition. However, these models can have billions of weight parameters, which are numerical values that transform input data as it is processed. These weights must be remembered. At the same time, the process of transforming data involves billions of algebraic calculations, which take a lot of processing power.
The process of retrieving data (the weights of the neural network in this case) from memory and moving it to the parts of a computer that do the actual computation, according to Sludds, is one of the most significant limiting factors to speed and energy efficiency.
So we thought, why don't we take all that heavy lifting — the process of retrieving billions of weights from memory — and move it away from the edge device and put it somewhere where we have abundant access to power and memory, giving us the ability to retrieve those weights quickly?" he says.
Netcast, the neural network architecture they created, involves storing weights on a central server that is linked to a novel piece of hardware known as a smart transceiver. This smart transceiver is a chip about the size of your thumb that can receive and send data. It uses silicon photonics technology to get trillions of weights out of memory every second.
Weights are received as electrical signals and imprinted onto light waves. Because the weight data is encoded as bits (1s and 0s), the transceiver converts them by switching lasers; a laser is turned on for a 1 and a laser is turned off for a 0. It puts these light waves together and sends them over a fiber optic network on a regular basis. This means that a client device doesn't have to ask the server for them.
"Optics is fantastic because there are so many different ways to carry data within optics." "For example, you can put data on different colors of light, which allows for much higher data throughput and bandwidth than with electronics," Bandyopadhyay explains.
Trillions of dollars per second
When the light waves reach the client device, they are used by a simple optical component known as a broadband "Mach-Zehnder" modulator to perform super-fast analog computation. This entails encoding device input data, such as sensor information, onto the weights. Then, it sends each wavelength to a receiver. The receiver picks up the light and figures out what the result is.
Researchers came up with a way to use this modulator to do trillions of multiplications per second. This greatly speeds up the device's ability to do calculations while only using a small amount of power.
"In order to make something faster, it must be more energy efficient." However, there is a cost. We've created a system that can run on a milliwatt of power while still performing trillions of multiplications per second. "That is an order of magnitude improvement in terms of both speed and energy efficiency," Sludds says.
They put this architecture through its paces by sending weights across an 86-kilometer fiber that connects their lab to MIT Lincoln Laboratory. Netcast enabled machine learning at high speeds with high accuracy—98.7 percent for image classification and 98.8 percent for digit recognition.
"We had to do some calibration, but I was surprised at how little work was required to achieve such high accuracy right out of the box." "We were able to achieve commercially relevant accuracy," Hamerly adds.
Moving forward, the researchers hope to improve the performance of the smart transceiver chip. They also want to shrink the receiver, which is currently the size of a shoe box, to the size of a single chip, so that it can fit on a smart device like a cell phone.
"Using photonics and light as a computing platform is an extremely exciting area of research with potentially enormous implications on the speed and efficiency of our information technology landscape," says Euan Allen, a Royal Academy of Engineering Research Fellow at the University of Bath who was not involved in this study."Sludds et al.'s is an exciting step toward real-world implementations of such devices, introducing a new and practical edge-computing scheme while also exploring some of the fundamental limitations of computation at very low (single-photon) light levels."
NTT Research, the National Science Foundation, the Air Force Office of Scientific Research, the Air Force Research Laboratory, and the Army Research Office have all contributed to the research.