Research and Development
Experiments in model architecture, training, evaluation, and efficient inference
All research
From Single-GPU to Distributed Training: A Framework for Making the Right Call
Conscious Engines
What actually fills GPU memory during finetuning, and a decision framework for picking DDP, ZeRO/FSDP, tensor, or pipeline parallelism.
Distributed Data Parallel: How It Actually Works
Conscious Engines
The mechanics behind DDP's near-linear throughput scaling, and the misconfigurations that silently turn a 1.9x speedup into 1.3x.
Tensor Parallelism and Sequence Parallelism
Conscious Engines
Splitting weight matrices across GPUs instead of replicating layers — why it wins without NVLink, and how LoRA complicates sequence parallelism.
Pipeline Parallelism: How It Actually Works
Conscious Engines
How pipeline parallelism splits model layers across GPUs, where the pipeline bubble comes from, and the 1F1B schedules that shrink it.
ZeRO and FSDP: Model Sharding
Conscious Engines
How ZeRO shards weights, gradients, and optimizer state across GPUs to fit models that don't fit on one card, and what each FSDP knob does.