Notes on LLM scaling, distributed training, and building the research environment and teams.https://blog.szha.ai/https://blog.szha.ai/blog/research-problems-in-llm-pretraining/https://blog.szha.ai/blog/research-problems-in-llm-pretraining/A practitioner's map of the pretraining landscape: scaling laws, principled parameterization, optimizer frontiers, and the open mysteries that keep the field honest.Wed, 03 Jun 2026 00:00:00 GMTpretrainingscaling-lawsoptimizationmuPresearchhttps://blog.szha.ai/blog/badly-tuned-training-run/https://blog.szha.ai/blog/badly-tuned-training-run/AI raised individual throughput but coordination overhead stayed fixed. For many product-engineering orgs, the bottleneck flipped from compute-bound to communication-bound.Wed, 15 Apr 2026 00:00:00 GMTmanagementaiscalingorganizationshttps://blog.szha.ai/blog/on-assessing-the-value-of-a-project/https://blog.szha.ai/blog/on-assessing-the-value-of-a-project/A simple framework for picking the most impactful work — with practical tips for estimating chance of success, effect size, and applicable scenarios.Tue, 20 May 2025 00:00:00 GMTresearchdecision-makinghttps://blog.szha.ai/blog/model-size-and-training-horizon-scaling-laws/https://blog.szha.ai/blog/model-size-and-training-horizon-scaling-laws/A framework for optimizing model size and training token count from scaling-law coefficients — accounting for inference cost, data repetition, and system efficiency.Mon, 02 Dec 2024 00:00:00 GMTscalingtrainingcompute-optimalchinchillahttps://blog.szha.ai/blog/gluonnlp-deep-learning-toolkit-for-nlp/https://blog.szha.ai/blog/gluonnlp-deep-learning-toolkit-for-nlp/Introducing GluonNLP, an open-source toolkit that tackles the reproducibility crisis in NLP research with stable APIs, reusable components, and centralized resources.Tue, 24 Jul 2018 00:00:00 GMTnlpdeep-learningmxnetopen-source