Crumb

hf.co/crumb
pronouns: she / xe / it / fae.

featured

reasoning pretraining

This is my baby right now. I am creating a series of models called CLMR, causal language modeling with reasoning. They use a GAN-like RL self-play loop with a generator and discriminator. Models both complete text and detect when completions are generated, the detector's p(real) becomes the generator's reward, enabling reasoning across all domains represented in whatever corpus I feed it. I've been chipping away at these algorithms and data for >1yr. Starting with a reasoning generator with a discriminator head trained to output softmax labels. I took a detour exploring very deep bidirectional discriminators on top of the generator policy's hidden states, abandoned this line because getting it stable was a pain that required many iterations of reward and loss augmentations. I then created an inverse reasoning-from-input-output pipeline to create reasoning data for the discriminator, and to create higher quality data for the generator with ground truth prompts and completions from a pretraining dataset. Right now it's playing and training in the adversarial RL trainer. I scale to 9B with serious weeks-long training runs with plans to go to 27B.

date	link	desc
02-26-2026	blog hf	a prototype w/o reasoning discriminator
eta:may 3rd	soon.	a fully reasoning 9B CLMR

autoencoders

Models trained to act as encoders and decoders based on latent representations of text.

date	link	desc
03-17-2026	essence-3b-v2 blog	12k-step multi-objective training (reconstruction + span corruption + MLM) on SmolLM3-3B-base. Variable-length latents up to 256 tokens; best reconstruction/interpolation tradeoff at ~12 latent tokens for ~700-token sequences. Generalizes beyond 2k training context with degradation.
soon.	soon.	A type-conditioned autoencoder for automating algorithm discovery, a candidate for a self-contained self-improvement loop.

catalog

datasets

date	link	desc
2024-2025	semantic-corruption-t5-v1_1- base small large	datasets where we semantically corrupt a text with T5 providing a plausible but generated span to replace the original, for small base and large T5 models
02-27-2024	askmistral-pile-2-15 model	4B tokens filtered by LLM-as-judge; surprisingly effective for data curation at 111M scale
03-13-2025	basic_fim_set	65.5k fill-in-the-middle samples
04-2026	functions-656k functions-139k functions-53k	datasets of well-typed functions with variables anonymized, contains all dependency functions and imports

some new ones coming soon

model/arch experiments

date	link	desc
08-26-2024	glort2	transformer w/ transformer layers replacing all linear layers, 205m params
08-03-2025	doc2desc-3b-gguf	qwen2.5-3b tuned to generate document descriptions for control-vector training data
07-19-2024	utf8-gelu-dec utf8-relu-dec	8.5m param utf-8 decoder models, 10kb context

misc

date	link	desc
12-14-2022	notebook	adversarial attacks to 'ai art' tools
01-31-2022	notebook	minimal demonstration of hinton's forward-forward learning algorithm
07-19-2023	hf. x. colab.	Mo' Lora V1
09-04-2023	hf.	Mo' Lora V2 collection.. i cant find the inference code
04-05-2023 to 7-15-2023	eval results w/ links	"Gerbil" models, decoder transformers trained on a UL2-like objective
05-10-2023	notebook	distillation on pythia models

previous contracts @ arcee ai, lexica art, minerva marketing. very into taking self-play RL very far right now.
high iteration velocity + willing to work on a problem for years at a time.
strengths: training custom architectures, weird information flow.

other interests: dr michael levin's work on the platonic space, deep time & deep space / cosmology
kinlist: twilight sparkle, bob belcher
contact me on bsky (@crumb.bsky.social) or on x (@cephaloform)