Evolutionary Model Merge

Evolutionary Model Merge is a way to create a new AI model by automatically mixing several already-trained models, without retraining anything from scratch.

Imagine several chefs who each excel at different dishes. One is great at Japanese, another is great at solving math problems. Instead of a person manually trying to blend the two chefs' recipes, this method uses a program that mimics natural evolution: over hundreds of generations, it mixes the recipes (the numerical values inside the models) in different ratios and orders, keeps only the combinations that turn out well, and passes them on to the next generation. The combination that survives at the end becomes the new model.

The key point is that there's no retraining from scratch involved. It's recombining finished dishes rather than learning to cook anew, so it uses far fewer computing resources while still producing a model with new capabilities. Sakana AI used this method to combine a Japanese-language model with a math model, producing a model that could solve math problems well in Japanese.

In articles, you'll see it used like: "Evolutionary Model Merge, released by Sakana AI in March 2024, combines multiple existing open models to create a new one." A common misunderstanding is that this is the same as fine-tuning, which retrains a model — but Evolutionary Model Merge is different in that it runs no gradient-descent-based training at all, only combining already-trained weights.