Computing with Gyroscopes
There is a growing number of startups exploring alternative compute paradigms; i.e. companies not making GPUs/CPUs/TPUs (such as Mythic, Vaire, Ludwig Computing, Extropic, Normal Computing, Unconventional, and many more). Each company is approaching the problem of computing in slightly different ways, and in this blog we will look at one of the more recent entrants, Unconventional. They recently stormed into the scene with a large $500 million "seed" round (at $4.5 billion valuation). They recently started putting public information out in the form of technical blogs, one of which included specific results which we will be looking into today: https://unconv.ai/blog/machine-learning-with-dynamics/. Note that after writing this blog, I saw they released a more advanced and large scale oscillator based algorithm, we will dive into that in another blog.
The basic idea of the "learning with dynamics" direction they propose is to hybridize compute by offloading some of the work to an alternative computation platform (which runs more efficiently, either in time or energy, than the GPU/CPU). This, naturally, is bound by Amdahl's law, but could potentially yield practical speedups for early versions of alternative chips that cannot run full end to end workflows. The chip proposed here is a physical one in which you have a system of nonlinear coupled gyroscopes. Computation is done by integrating this system (ordinary differential equation) over time. The input data is fed through a transformation, which projects it to the dimensionality of the input oscillators and controls a force applied to the \(x\) and \( y\) directions of the gyroscope. The gyroscopes are left to evolve for some period of time (sufficient time for information to propagate through the couplings to reach the output nodes) after which the position and velocity of the output gyroscopes are then fed into a decoder linear layer which outputs the final prediction. This setup is generally agnostic to the type problem, you could use this for classification, or generation, or anything, since it is just a parameterized function. The original blog uses it for time series classification of handwritten digits (i.e. the lines of MNIST digits being drawn, see below).
The dataset itself is only 8 dimensional, but in the blog is referenced as 100-step sequence, which I understood as being linearly interpolated between the 8 data points. This figure has 20 interpolated points (which I found to be more stable than 100 during training).
To do inference on this data point (i.e. to attempt to classify it as a digit), we feed each point into the input linear, apply the forces, integrate the system, measure the output, feed into the decoder, and take the prediction. Since the force function is not continuous in time (for the \(N\) points, the force jumps to the force at that point instantaneously), numerically we integrate from the beginning time of the force to the end time for each force (they are applied sequentially). Once we have the prediction, we need to train the model. The gradients of the input and output linear layers are trivial, and the ODE integrator (Tsit5) is differentiable as well, which makes the system amenable to end to end automatic differentiation aka backprop.
With this setup in hand we can train the system. Like many systems that rely on numerical integration, it was quite annoying to get reliably stable, but with some bounding on the values, and adaptive stepsizing the integrator, I got it to reliably be stable.
And it works! And we can plot the logits of the results too (I say logits, as the mass is not actually negative, but goes through a softplus function to keep it bounded above 0, so this is kind of the "logit" of the mass). I don't really have a meaningful interpretation of these, but they look kind of cool.
Of course the real question is, why would we do this? Presumably, we could make a chip of gyroscope-like objects that is very fast or low energy. However, we need to be careful not to fall into the trap of hybrid computing, common with e.g. variational quantum computing, in which the "hybrid" computer doesn't really add anything to the algorithm. So if we take the exact same setup and remove the gyroscopes, you would expect to see very poor results if the gyroscopes were meaningfully contributing to the algorithm. Alas that is not the case, as we plot below without any gyroscopes.
One might argue that this requires the full data at once (as opposed to iteratively feeding in the data and applying forces), and thus isn't really time series in the same way. And while that is true, the time series element isn't really an important part of the gyroscope computing, as you can see from their plots of prediction versus integration time, the correct prediction doesn't surface until the very last moment. So it's not like partial data (before every time step is in) matters much here anyway.
As always, the (non LLM) code is available at: https://codeberg.org/lockwo/machine_learning
Additionally, I have a video on the topic here: https://www.youtube.com/watch?v=0ZWUsuqtYxM
P.S. Maybe this is just what the world is now, but the original blog is super AI generated. I like to think better things are possible.