AI news and explainers at 7 AM weekdays, plus a Sunday weekly at 8Get it in your inbox

METAL LAB

marin-community/marin

★ 1,970+214 todayPython

Marin is an open-source platform that publishes the entire process of building large language models, not just the final model

Marin publishes the full pipeline for building large language models, from data curation through training to evaluation, as open code and recorded experiments. It treats failed experiments as part of the public record, not just successes. Stanford CRFM and Open Athena are its core collaborating organizations.

What it does

  1. Marin is currently pretraining from scratch and posttraining a large mixture-of-experts model (a design that routes computation to a subset of specialized sub-models instead of one big dense model) sized at 5e24 model-FLOPs with over 500 billion total parameters, aimed at tasks useful to scientists and researchers.
  2. It released Delphi, a scaling suite spanning compute budgets from 3e18 to 1e23 FLOPs: a recipe mapping compute budget to model configuration, a set of models trained on that recipe using the Google TPU Research Cloud, and a scaling law that predicts larger models from the smaller ones.
  3. All supporting artifacts are public: training-mixture pipelines that reproduce the data mix from Nemotron-CC, StarCoderData, and ProofPile 2, the forkable recipe code, a documented development methodology, and plot-ready data with source links for every figure on Hugging Face.
  4. Previously, an 8-billion-parameter model trained with Marin outperformed Llama 3.1 8B on Marin's own base-model benchmark suite, and the team also trained a 32-billion-parameter model called Marin 32B.
  5. Experiments are structured as dependency-ordered steps, similar to a Makefile — for example, a model-training step can depend on a tokenization step and will run only after it completes.

Why it matters

Most of the practical know-how behind building large language models — data pipelines, training infrastructure, experiment logs — usually stays private, but Marin makes that entire process reproducible and inspectable for other researchers. Its step-based framework is also reused as a library for non-text domains like audio-text, DNA, and protein modeling, which matters for anyone trying to build similar training pipelines beyond plain language models.

Terms in this repo

  • foundation model · a large AI model pretrained on broad data that can be adapted to many downstream tasks
  • mixture-of-experts · a model design where only a subset of specialized sub-networks is used per input instead of one dense network
  • model-FLOPs · a unit measuring the total floating-point computation used to train a model, indicating scale
  • scaling law · a mathematical relationship between model/data size and resulting performance
  • posttraining · additional training applied after pretraining to specialize a model for particular goals
  • tokenization · the process of splitting text into small units (tokens) that a model can process

Repository description (English)

Open-source framework for the research and development of foundation models.

Open on GitHub

Trending repos

All repos →

Latest from METAL LAB