Ideas, made visible. 让知识， 变得可见。

PyTorchtorch.compileTorchInductor

From torch.compile to generated kernels: capture, guards and fusion从 torch.compile 到生成内核：图捕获、守卫与融合

TorchDynamo captures tensor operations into FX graphs and guards the assumptions used to specialize them. AOTAutograd prepares differentiation graphs; TorchInductor lowers and schedules computation, fusing compatible operations to avoid materialized intermediates. Cached code is reused only when its guards hold.TorchDynamo 将张量操作捕获为 FX 图，并用守卫检查特化所依赖的条件。AOTAutograd 准备求导图，TorchInductor 将图转换为低层中间表示并调度计算，将兼容操作融合以减少中间张量的物化。只有守卫满足时，缓存代码才能复用。

Published发布 2026.09.2715 min read15 分钟阅读

PyTorchAutogradComputation graph

Autograd nodes and the backward engine: how gradients meetAutograd 节点与反向引擎：梯度如何汇合

Backward nodes apply vector–Jacobian products. The autograd engine accumulates incoming contributions in input buffers and schedules a node when its dependencies are satisfied. AccumulateGrad then writes or adds the resulting gradient to a leaf tensor’s .grad.反向节点计算向量–雅可比积。Autograd 引擎用输入缓冲区汇总梯度贡献，在依赖满足后调度节点；AccumulateGrad 随后将结果写入或累加到叶子张量的 .grad。

Published发布 2026.09.2613 min read13 分钟阅读

Published发布 2026.09.2616 min read16 分钟阅读

CUDA streams, kernels and timing: when is the GPU actually done?CUDA 流、内核与计时：GPU 到底什么时候算完？

CUDA streams order submitted work; events connect dependencies across streams. Host return, event completion and tensor storage lifetime are different boundaries. Timing a matrix multiplication requires choosing which boundary to measure and accounting for warm-up, precision and synchronization.CUDA 流为提交的操作建立顺序，事件连接跨流依赖。主机调用返回、事件完成和张量存储生命周期是不同的边界。测量矩阵乘法前，需要明确计时边界，并控制预热、计算精度与同步方式。

PyTorchCUDAStreams

INTERACTIVE / 05 LABSInteractive essay交互长文

SNNDynamical systems动力系统Interactive lab交互实验

How do spikes learn to compute?脉冲如何学会计算？

LIF describes membrane integration, threshold firing and reset. BPTT propagates gradients through time, STDP updates synapses from spike timing, and EventProp differentiates continuous event trajectories. UltraLIF modifies the forward model to support differentiable training.LIF 描述膜电位积分、阈值发放与重置。BPTT 沿时间展开图传播梯度，STDP 根据脉冲时序更新突触，EventProp 对连续时间的事件轨迹求导。UltraLIF 则调整前向模型以支持可微训练。

Published发布 2026.09.2330 min read30 分钟阅读

Published发布 2026.09.2414 min read14 分钟阅读

Operator registration and dispatch: how PyTorch selects a kernel算子注册与调度：PyTorch 如何选择内核

PyTorch separates operator schemas from implementations. The dispatcher combines tensor dispatch keys with thread-local state, selects an operator-specific table entry, and lets wrappers redispatch to lower layers. Autograd dispatch and gradient recording are separate decisions.PyTorch 将算子接口与实现分开注册。调度器合并张量的调度键与线程局部状态，从算子专属的调度表中选择入口；包装层再通过 redispatch 进入下层实现。进入 Autograd 调度层与记录反向图是两个不同的判断。

PyTorchDispatcherATen

LANGUAGE / COMPARISON

Published发布 2026.09.2322 min read22 分钟阅读

Rust vs C++: two paths to performanceRust vs C++：高性能的两条路

Safe Rust uses ownership and borrowing rules to restrict invalid references and unsynchronized shared mutation. C++ uses RAII to manage resource lifetimes, while reference validity depends on API contracts and program design. Performance in either language depends on data layout, algorithms and generated code.Safe Rust 通过所有权和借用规则约束无效引用与未同步的共享修改。C++ 用 RAII 管理资源生命周期，引用有效性仍依赖接口约定和程序设计。两者的性能都取决于数据布局、算法与生成的机器码。

RustC++Memory safety内存安全

PyTorchTensorMemory layout

Tensor, strides, and storage: why a transpose can be freeTensor、步幅与存储：转置为什么可以不搬数据

For a dense strided tensor, strides and storage_offset map logical indices to storage locations. Transpose shares the original storage by swapping sizes and strides; making a noncontiguous result contiguous requires copying its data.对稠密步幅张量，strides 与 storage_offset 决定逻辑索引对应的存储位置。转置通过交换形状和步幅共享原存储；将非连续结果转为连续布局则需要复制数据。

Published发布 2026.09.2412 min read12 分钟阅读

COMPUTE / GPU KERNEL

Published发布 2026.09.2313 min read13 分钟阅读

CUDA Rust: from threads to tilesCUDA Rust：从线程到 Tile，理解两条 Kernel 路线

SIMT kernels assign scalar work to individual threads; tile kernels express operations over data blocks and leave thread mapping to the compiler. Correct output partitioning and boundary handling prevent duplicate writes and omissions, while memory access and data reuse shape performance.SIMT 内核为单个线程分配标量计算，Tile 内核则描述数据块上的运算，由编译器组织线程映射。正确的输出分区与边界处理避免重复写入和遗漏，访存方式与数据复用影响性能表现。

CUDARustSIMT

Published发布 2026.09.2321 min read21 分钟阅读

What is PyTorch? Inside torch.matmulPyTorch 到底是什么？跟踪一次 torch.matmul

For dense 2-D inputs in eager mode, torch.matmul calls mm and dispatches to a CPU or CUDA implementation. With gradient recording enabled and inputs requiring gradients, autograd records dependencies for backward to apply the chain rule and accumulate leaf gradients.在即时执行模式下，稠密二维张量的 torch.matmul 调用 mm，并分派到 CPU 或 CUDA 实现。开启梯度记录且输入需要梯度时，autograd 记录依赖关系，供反向传播应用链式法则并累积叶子张量的梯度。

PyTorchATenAutograd

The notebook知识索引THE NOTEBOOK

Small steps toward a deeper understanding.每一篇，都是理解世界的一小步。

TRAVEL / NOTES

Travel & Journals旅行与见闻

Jiuzhaigou, in the depths of blue九寨沟，水色深处

A turquoise lake holds the mountains in reflection; streams wind through the meadows. Colour first draws the eye in Jiuzhaigou, then small details make it linger.一湖青碧收着山影，几湾流水绕过草甸。九寨沟的山水，初看是颜色，久看是让人舍不得移开目光的细处。

Jiuzhaigou九寨沟Travel journal游记Landscapes山水

Published发布 2026.10.016 min read6 分钟阅读

LIFE & NEUROSCIENCE / NOTESInteractive essay交互长文

Life & Neuroscience生命与神经科学

Human vision: from retinal biology to optical limits and engineering人的视觉系统：从生物现象、光学约束到工程实现

Vision视觉Retina视网膜Neuroscience神经科学

Published发布 2026.09.2723 min read23 分钟阅读

PyTorchtorch.compileTorchInductor

From torch.compile to generated kernels: capture, guards and fusion从 torch.compile 到生成内核：图捕获、守卫与融合

Published发布 2026.09.2715 min read15 分钟阅读

PyTorchAutogradComputation graph

Autograd nodes and the backward engine: how gradients meetAutograd 节点与反向引擎：梯度如何汇合

Published发布 2026.09.2613 min read13 分钟阅读

Published发布 2026.09.2616 min read16 分钟阅读

CUDA streams, kernels and timing: when is the GPU actually done?CUDA 流、内核与计时：GPU 到底什么时候算完？

PyTorchCUDAStreams

Published发布 2026.09.2414 min read14 分钟阅读

Operator registration and dispatch: how PyTorch selects a kernel算子注册与调度：PyTorch 如何选择内核

PyTorchDispatcherATen

PyTorchTensorMemory layout

Tensor, strides, and storage: why a transpose can be freeTensor、步幅与存储：转置为什么可以不搬数据

Published发布 2026.09.2412 min read12 分钟阅读

INTERACTIVE / 05 LABSInteractive essay交互长文

Benchmark基准测试Performance性能优化JavaScript

How much faster? Benchmark with evidence快了多少？从一次测量到可信的基准测试

Warm-up reduces initial JIT compilation effects, batched timing amortizes timer overhead, and repeated samples reveal runtime variability. Performance comparisons require consistent inputs and equivalent results; medians and quartiles describe the measured runtime distribution.预热减轻首次 JIT 编译的影响，批量计时摊薄计时开销，重复采样揭示耗时波动。性能比较以输入一致、结果等价为前提，再用中位数和四分位数描述实测耗时的分布。

Published发布 2026.09.233 min read3 分钟阅读

PHYSICS / NOTES

Physics & Models物理与模型

Waves, phase & superposition波、相位与叠加：让公式动起来

In a linear model, waves superpose by adding their displacements. For two sinusoidal waves with equal amplitude, frequency and wave number, phase difference sets the resultant amplitude: zero phase difference doubles it, while π produces complete cancellation.在线性模型中，波的叠加等于位移相加。两列振幅、频率和波数相同的正弦波，其合成振幅由相位差决定：同相时振幅加倍，相差 π 时完全相消。

Waves波动Phase相位Interactive lab交互实验

Published发布 2026.09.233 min read3 分钟阅读

Memory layout存储布局Matrix computation矩阵计算

Band Storage GAXPY带状矩阵的紧凑存储

Band storage keeps only diagonals within the lower and upper bandwidths, allowing GAXPY to skip entries outside the band. For an n×n narrow-band matrix with stored band width w, storage and arithmetic work scale as O(nw) rather than O(n²).带状存储只保留上下带宽以内的对角线，使 GAXPY 跳过带外元素。对于 n×n 窄带矩阵，若存储带宽为 w，存储量与运算量均由 O(n²) 降为 O(nw)。

Published发布 2021.07.195 min read5 分钟阅读

Memory layout存储布局Linear algebra线性代数

Symmetric Storage GAXPY对称矩阵的压缩存储

A real symmetric matrix can store its lower triangle in n(n+1)/2 entries. In GAXPY, each off-diagonal entry contributes to two output components, so storage is nearly halved while arithmetic work remains O(n²).实对称矩阵只需保存下三角的 n(n+1)/2 个元素。GAXPY 中，每个非对角元素都要贡献到两个输出分量，因此存储量接近减半，运算量仍为 O(n²)。

Published发布 2021.07.195 min read5 分钟阅读

MATHEMATICS / NOTES

Published发布 2019.01.126 min read6 分钟阅读

Fast Matrix–Vector Products快速矩阵向量乘法

Radix-2 FFT recursively splits a DFT into transforms over even- and odd-indexed samples. Butterfly operations reuse these results, reducing arithmetic from O(n²) to O(n log n) while computing the same discrete Fourier transform.基 2 FFT 将 DFT 递归分解为偶数位置与奇数位置样本的变换，再通过蝶形运算复用子问题的结果。它计算的仍是同一个离散傅里叶变换，运算量却从 O(n²) 降为 O(n log n)。

FFTLinear algebra线性代数

BLASMatrix computation矩阵计算Performance性能优化

Matrix Multiplication矩阵乘法：结构与效率

Loop order determines memory strides and opportunities for data reuse. Contiguous accesses use cache lines efficiently, while blocking reuses submatrices in cache to reduce data movement without changing the arithmetic work.矩阵乘法的循环顺序决定访存步长与数据复用机会。连续访问有利于利用缓存行，分块则反复使用缓存中的子矩阵，在算术量不变的情况下减少数据搬运。

Published发布 2019.01.079 min read9 分钟阅读

MATHEMATICS / NOTES