Skip to content

TriadicGPT

End-to-end prime factorization in a generative language model.

PyPI Python License HF triadic-gpt-40m HF triadic-gpt2-medium DOI Paper

Relationship to the Engine

The Triadic Neurosymbolic Engine provides the mathematical foundation: LSH-to-prime projection as a post-hoc tool applied to pre-trained embeddings.

TriadicGPT takes the next step: a language model that learns prime-factor signatures during training, as a side effect of language modeling. A lightweight triadic projection head (49K params for GPT-2) maps hidden states to binary vectors encoding as prime composites, achieving algebraically verifiable semantics at zero cost to language quality.

King  = 2 x 3 x 5        (Royalty x Male x Authority)
Queen = 2 x 5 x 7        (Royalty x Authority x Female)

Shared:     gcd  --> {2, 5}    Royalty, Authority
Difference: div  --> {3} vs {7}   Male vs Female
Analogy:    factor transfer   king:queen :: man:woman

Key Results

Metric Value
Language cost Zero (PPL 7.69 vs 7.56 ablation, +1.7%)
Analogy verification 98% (51 analogies)
Subsumption accuracy 98.3% held-out (158 supervised anchors)
Domain separation 1.21 mean (12 domains)
GPT-2 transfer gap closure 48% toward Engine PCA upper bound
Signature uniqueness 100% across all evaluated concepts
Scale crossover Semantic ordering emerges at ~20M params

Architecture

Text --> BPE (4096 vocab) --> TriadicGPT (12L / 512D / 8H)
                                    |
                              +-----+-----+
                              |           |
                         LM Head    Triadic Head
                              |           |
                        next-token   tanh(Wx) --> bits --> phi(x) = prod(p_i)
                              |           |
                              +-----+-----+
                                    |
                         L = L_lang + alpha * L_triadic

The triadic loss combines four components: diversity (bits fire ~50%), contrastive (sequences differ), entropy (no dead bits), and embedding alignment (triadic similarity tracks embedding similarity).

Model Scales

Scale Layers Dim Heads Bits Params
small 4 128 4 16 1.3M
base 6 256 8 32 5.8M
large 8 384 8 48 15.9M
xl 12 512 8 64 40M

The triadic-head Package

The standalone PyPI package provides a drop-in triadic projection head for any HuggingFace transformer:

from triadic_head import TriadicWrapper

model = TriadicWrapper("gpt2", n_bits=64, align_mode="infonce")
sigs = model.encode(["king", "queen", "dog"])
result = model.compare("king", "queen")

Supports: GPT-2, LLaMA, Mistral, Phi, Qwen, GPT-Neo, OPT, Falcon.

What's Next