Filip Makraduli

Filip Makraduli works across transformer optimization, CUDA implementation, and the infrastructure needed to test modified open models. He co-authored FlashNorm: Fast Normalization for Transformers with Nils Graef, whom he credits as the technique’s originator, and describes implementing its parallel execution and deploying transformed checkpoints. His public LinkedIn profile lists Superlinked, while his account of the work establishes his research and implementation participation without specifying an employment title. Research attribution

His FlashNorm work addresses a useful distinction in inference engineering: a layer’s arithmetic cost can be small while its wall-time cost is substantial. RMSNorm can run repeatedly during a decode step, accumulating kernel-launch, memory-movement, and waiting costs. The optimization folds the learned normalization gain into the following projection’s weights offline, preserving the gain’s effect while removing a separate runtime multiplication. It also moves the input-dependent scalar RMS division after the projection, allowing matrix multiplication and RMS calculation to proceed independently before their results are combined. In applicable double-normalization configurations, scale invariance permits redundant pre-normalization to be removed; this is architecture-dependent rather than a general license to collapse arbitrary RMSNorm layers. Mechanisms FlashNorm paper

Makraduli distinguishes changing a checkpoint from changing its execution. He reports that weight folding through Transformer Tricks produces a checkpoint compatible with torch.compile and quantized models, whereas realizing deferred normalization’s overlap requires kernel work. His CUDA implementation placed matrix multiplication on tensor cores and RMS reductions and related operations on CUDA cores. Long generation then exposed repetition and a one-step lag that passing unit tests and apparently normal perplexity had missed: post-scaling could consume a stale matrix-multiplication buffer because the streams’ join was implicit. He repaired the dependency by explicitly marking both computations complete and making post-scaling wait for both current results. This experience connects algebraic correctness with the separate responsibility to enforce runtime synchronization. CUDA implementation and failure Synchronization repair Checkpoint compatibility

His deployment perspective follows the same distinction: access to open weights permits checkpoint transformations, while control over inference permits kernel changes. He describes using Superlinked’s inference infrastructure and PSI to deploy custom Hugging Face checkpoints and test research in larger workflows. At recording time, he presented model switching and queuing for smaller models sharing GPUs, API control of model and cluster configurations, and support for embedding and re-ranking models. These are capabilities he discussed using or presenting, rather than evidence that he authored PSI or Transformer Tricks. The practical idea is to give an experiment enough control over both weights and execution to test it at scale without rebuilding all the surrounding deployment machinery. Deployment approach Small-model infrastructure