pretraining

13 projects share this GitHub topic

Lines connect members that are measurably related to each other. Dot size reflects stars.

🧬 Members

LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

General technology for enabling AI capabilities w/ LLMs and MLLMs

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and…

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

[NeurIPS 2025 Spotlight] TPA: Tensor ProducT ATTenTion Transformer (https://arxiv.org/abs/2501.06425)

BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

data_management_LLM

Collection of training data management explorations for large language models

A Chinese Open-Domain Dialogue System

LLMInternSkill: LLM internship resume and job-search Codex Skill for resume polish, JD tailoring, evidence…

🔗 Related families

instruction-tuning 28 language-model 113 dialogue 10 pre-training 11 chinese 70 attention-mechanism 54 from-scratch 7 tokenizer 5

Measured from GitHub topics shared by both projects, weighted by how rare each topic is.