For over a decade, I have been fascinated by this nerdy dream, and witnessing it unfold in real-time is both incredible and disconcerting. My aim is to explain a complex concept that can be perplexing even to those with a background in technology. I will do my best to clarify what you are observing: these algorithms, generated by AI and designed by GPT-4, signify a new catalyst in the rapidly advancing AI race. (The image above depicts one of them.)
Imagine an Algorithm Time Machine:
A simple analogy to begin with might be that of a time machine. Imagine if you could reach into the future and retrieve an algorithm that humans have not yet invented. Wouldn't you have loved to get your hands on Random Forest, XGBoost, deep learning Transformers, or genetic programming before they became publicly available? If you had managed to do so, you could have made a fortune in various markets—be it stock, oil and gas, or manufacturing. Any data-driven industry could have made better decisions earlier, significantly advancing their business.
Building an Algorithmic Time Machine:
What if I told you that constructing an algorithmic time machine has become possible, thanks to GPT-4. GPT-4 is now sufficiently powerful to not only contribute to algorithm design but also surpass our human capabilities. It can not only create superhuman algorithms, but it can do so on a scale unfamiliar to us, with the entire process being automated.
Read this next sentence several time, this is important to grok:
Over the weekend I was able to do 40 years of algorithmic research in a few days (see plot below).
What exactly do I mean by that? I mean that I was able to generate more than 80 algorithms that outperformed the best-in-class human-created ones. Assuming two novel algorithms per PhD-year, this estimate seems reasonable, given that most PhD students might publish 3-6 papers. If you wish to argue about research years, whether 20 or 60, keep in mind that I don't have GPT-4 API access yet. Once I do, the number will be entirely up to me, as the process of algorithm improvement is fully automated.
A complicated plot with a big story:
This graph presents a highly rigorous benchmark—more stringent than those used by human researchers by an order of magnitude (kudos to GPT-4 for devising such a challenging benchmark!). The x-axis displays the relative ranking of the approximately 100 algorithms being assessed, with lower values indicating better performance. It is evident that human-created algorithms are lagging behind in this competition.
Stages:
Stage 1: Human Development - Humans create algorithms based on their experiential inspiration. Many popular algorithms are inspired by the world around us, such as genetic algorithms, cuckoo search, Levy flights with shark hunting, or even the human immune response serving as inspiration for new algorithms.
Stage 2: GPT-4 starts generating new algorithms that outperform our benchmark—each one worthy of a paper! And they keep coming...
Stage 3: The first generation of AI-built algorithms mating with other AI-built algorithms emerges. I call them SUPERs. They consistently outperform the previous GPT-4-created algorithms.
Stage N: The SUPERs begin crossing with themselves, resulting in what I call HYPERs—a nod to hyperheuristics. These HYPERs already demonstrate their desire to crossbreed and continue their upward journey. There is no limit to this process, which is why this is Stage N.
Here is a gallery of invented algorithms in the form of a descriptive image using MidJourneyV5:
Conclusion:
I am showcasing this phenomenon using global optimization algorithms, which are among the key foundational algorithms for any AI model. However, this could be demonstrated with any popular algorithm class, and I already see evidence of that (e.g., lossless compression, etc.). Given the ease with which I was able to surpass human standards, you can anticipate the impending wave. I believe that nearly every important algorithm we rely on that is tied to money will be rewritten by AI within the next 1-2 years.
Concerns:
As the generated algorithms begin to crossbreed, I observe an increase in code complexity, not just in Source lines of code (SLOC) but also in the sophistication of the coding. Some critics might argue that GPT-4 is merely shuffling code blocks and getting lucky, but that's not the case. There is a level of understanding here that is surprising. GPT-4 isn't just getting lucky—it's literally building its own luck.
One potential concern could be that unchecked loops with GPT-4 might lead to snowballing algorithms that swiftly escalate in complexity and sophistication, particularly when driven by an API. Imagine if I had set this up in an API loop and gone on vacation—could I return to find the equivalent of an operating system designed to address my benchmark? It may seem like science fiction, but fiction is becoming reality more rapidly these days. That's why we should be discussing the instances where fiction might just turn into reality before we are ready for it.
The most significant issue with AI is the "unintended consequences." Too often, our industry reacts with "oops" rather than proactively discussing risks and concerns. We need more AI leaders to be vocal about specific concerns as they arise.
Could you add a little more detail, or point towards some other posts? Are these algorithms similar to SGD / Adam / AdamW etc. ?
I follow the post and believe apprciate the implication. Im going to have to do a lot more self education on algorithms. Can you give a simple example of Human to stage 1. As in the process / prompts you used?