Autograd nodes and the backward engine: how gradients meetAutograd 节点与反向引擎：梯度如何汇合

A shared value receives two gradient contributions同一个中间值，收到两路梯度

Part 3 followed a forward call through the Autograd wrapper. Now run backward through a graph with a shared intermediate. If u = x*x feeds both 2*u and 3*u, computing its derivative after only one branch would lose a contribution. The engine must gather both before applying the derivative of the square.第三期沿前向调用经过了 Autograd 包装层。现在看带共享中间值的反向图：u = x*x 同时供给 2*u 和 3*u。如果只收到一路贡献就对平方求导，会漏掉另一路。引擎需要先汇合两路梯度，再应用平方的导数。

import torch

x = torch.tensor(2., requires_grad=True)
u = x * x
u.retain_grad()
a, b = 2 * u, 3 * u
loss = a + b
loss.backward()
print("values:", u.item(), a.item(), b.item(), loss.item())
print("gradients:", u.grad.item(), x.grad.item())

values: 4.0 8.0 12.0 20.0
gradients: 5.0 20.0

u.retain_grad() makes the non-leaf gradient observable in u.grad; it is not required for the chain rule to reach x. The Tensor.retain_grad API concerns retaining a gradient, not retaining the graph’s saved tensors.u.retain_grad() 让非叶子张量的梯度保留在 u.grad，便于检查；即使没有这行代码，链式法则仍能把梯度传到 x。Tensor.retain_grad API 保留的是梯度，不是计算图中为求导保存的张量。

Baseline: PyTorch 2.10.0, commit 449b17684101. Python examples below were executed with the matching 2.10.0+cpu build. Scope: eager reverse-mode differentiation of real CPU tensors, without hooks that modify gradients, distributed execution, or compiled autograd. Node class names and private fields are implementation details of this version.基线：PyTorch 2.10.0，提交 449b17684101。下文 Python 示例已在对应的 2.10.0+cpu 构建中执行。本文讨论实数 CPU 张量的 eager 反向自动微分，不涉及修改梯度的 hook、分布式执行或 compiled autograd。节点类名和私有字段属于当前版本的实现细节。

The backward graph contains nodes and input slots反向图由节点与输入槽连接

A tensor’s grad_fn leads to a backward Node. Its apply() consumes gradients of forward outputs and produces contributions for forward inputs. An Edge identifies the destination node and its backward input slot, input_nr. That slot corresponds to a forward output; it is not the position of an argument in the original Python call.张量的 grad_fn 指向反向 Node。节点的 apply() 接收前向输出的梯度，计算前向输入的梯度贡献。Edge 指定目标节点及其反向输入槽 input_nr。这个槽对应前向输出，不是原 Python 调用中的参数位置。

function.h: Node, next_edges and backward input metadatafunction.h：Node、next_edges 与反向输入元信息

edge.h: destination Node and input_nredge.h：目标 Node 与 input_nr

import torch

x = torch.tensor(2., requires_grad=True)
u = x * x
a, b = 2 * u, 3 * u
print("shared node:", a.grad_fn.next_functions[0][0] is u.grad_fn,
      b.grad_fn.next_functions[0][0] is u.grad_fn)
edges = u.grad_fn.next_functions
print("leaf edges:", [(type(node).__name__, slot) for node, slot in edges])
print("same leaf:", edges[0][0] is edges[1][0])

shared node: True True
leaf edges: [('AccumulateGrad', 0), ('AccumulateGrad', 0)]
same leaf: True

Both branches point to the same square node. The square has two edges to the same leaf accumulator because x occurs in both operand positions of x*x. Each contributes 5*x = 10. Counting unique destination nodes instead of edges would miss this multiplicity.两条分支指向同一个平方节点。平方节点又有两条边指向同一个叶子累积节点，因为 x 在 x*x 中占据两个操作数位置。每条边贡献 5*x = 10。如果只数不同的目标节点、不数边，就会漏掉这种重复依赖。

A backward call propagates a weighted derivative反向调用传播的是加权导数

For a real vector-valued function y = f(x), reverse mode propagates an output cotangent v as Jᵀv, usually called a vector–Jacobian product (VJP), with vectors written as columns here. It need not construct the full Jacobian. A scalar loss has the implicit seed 1; a nonscalar output needs matching gradient weights unless a scalar reduction supplies them.对于实数向量函数 y = f(x)，反向模式将输出端的余切向量 v 传播为 Jᵀv，通常称为向量–雅可比积（VJP）；这里采用列向量记法。它不必构造完整雅可比矩阵。标量损失的默认种子为 1；非标量输出需要提供匹配的梯度权重，或先归约成标量。

import torch

x = torch.tensor([2., 3.], requires_grad=True)
y = torch.stack((x[0] ** 2, x[0] * x[1]))
(g,) = torch.autograd.grad(y, x, grad_outputs=torch.tensor([1., 4.]))
print("VJP:", g.tolist(), "x.grad:", x.grad)

VJP: [16.0, 8.0] x.grad: None

torch.autograd.grad returns the requested gradients. In this example it leaves x.grad untouched. By contrast, torch.autograd.backward normally accumulates results into leaf gradients. These are different result destinations, not different differentiation formulas.torch.autograd.grad 返回所请求的梯度，本例中不会写入 x.grad。相对地，torch.autograd.backward 通常把结果累积到叶子梯度中。两者的区别是结果去向，不是求导公式。

Wait for both branches, then execute the shared node等两路贡献到齐，再执行共享节点

Read the graph from the loss down to the leaf: the arrows carry backward gradient contributions, while the formulas inside nodes identify the forward operations. The full result is visible initially. The animation separates arrival, readiness, and execution; its timing and branch order are explanatory, not a trace of a particular CPU scheduler.从损失向下读到叶子：箭头携带反向梯度贡献，节点内的公式标识对应的前向运算。默认直接显示完整结果。动画将贡献到达、依赖就绪与节点执行分开呈现；时间间隔和分支先后用于讲解，不是某次 CPU 调度的运行轨迹。

The middle status box records the received total for explanation. In the real engine, an InputBuffer is consumed when its node runs; it is not the same storage as the tensor’s persistent .grad. Constants 2 and 3 have no gradient edges here. The scalar seed is shown conceptually: the engine may optimize away an explicit GraphRoot for a single root.中间状态格保留“已收到的总和”以便讲解。真实引擎中的 InputBuffer 会在节点执行时被消费，不是张量持久保存的 .grad。常数 2 和 3 在本例中没有梯度边。标量种子按概念展示；对于单个根节点，引擎可能省去显式的 GraphRoot。

Read the engine at three handoff points从三个交接点阅读引擎源码

The Python entry helper _engine_run_backward forwards the request to the C++ engine binding. PythonEngine::execute delegates to the base engine for ordinary eager execution. Engine::execute sets up a GraphTask; the task owns dependency counts and buffers for nodes not yet ready, rather than storing one global backward state for every call.Python 入口辅助函数 _engine_run_backward 将请求交给 C++ 引擎绑定。在普通 eager 执行中，PythonEngine::execute 委托给基础引擎。Engine::execute 建立 GraphTask，由任务持有依赖计数和未就绪节点的缓冲区，而不是让所有 backward 调用共用一份全局状态。

graph.py: _engine_run_backwardgraph.py：_engine_run_backward

python_engine.cpp: PythonEngine::executepython_engine.cpp：PythonEngine::execute

graph_task.h: GraphTask::dependencies_ and not_ready_graph_task.h：GraphTask::dependencies_ 与 not_ready_

For this complete scalar backward graph, compute_dependencies counts incoming edges. evaluate_function calls a ready node, then visits its valid outgoing edges. Each delivery decreases the destination’s pending count and adds its contribution to the destination slot. At zero, the accumulated inputs move into a queued NodeTask. A branch order may vary while respecting these dependencies.对于本例完整的标量反向图，compute_dependencies 统计入边。evaluate_function 执行就绪节点，再遍历其有效出边。每次交付都会减少目标节点的待到达计数，并将贡献累加到目标输入槽；计数归零后，汇总的输入随 NodeTask 入队。只要满足这些依赖，分支先后顺序可以不同。

dependencies[next_ptr] += 1;

engine.cpp: Engine::compute_dependenciesengine.cpp：Engine::compute_dependencies

engine.cpp: readiness, input accumulation and queue insertionengine.cpp：依赖就绪、输入累加与任务入队

InputBuffer::add handles tensor accumulation, including device and stream considerations. The diagram reduces this to scalar addition on one CPU thread. Restricted gradient requests can prune execution, and undefined gradients are not ordinary numeric zeros; those cases need the engine’s full checks.InputBuffer::add 处理张量累加，也包含设备和流相关的逻辑。图中将它简化为单个 CPU 线程上的标量相加。限定求导目标的请求可以裁剪执行路径，未定义梯度也不是普通的数值零；这些情况需要引擎中的完整判断。

input_buffer.cpp: InputBuffer::addinput_buffer.cpp：InputBuffer::add

The node’s derivative calculations still call tensor operators. For example, matrix-product backward can issue more matrix multiplications, which go through the dispatcher from part 3. The engine chooses when a node is ready; the dispatcher chooses which implementation executes an operator. They solve different scheduling problems.节点的导数计算仍会调用张量算子。例如矩阵乘法的反向计算还会发起矩阵乘法，这些调用会经过第三期介绍的调度器。反向引擎决定节点何时就绪，调度器决定算子由哪个实现执行，两者处理不同层面的调度问题。

derivatives.yaml: mm backward formulasderivatives.yaml：mm 的反向公式

A runnable C++ model of dependency scheduling可运行的 C++ 依赖调度模型

This standalone program implements the graph above with scalar local derivatives. It counts edges, buffers contributions, and runs each node once after its dependencies arrive. The two edges from the square to the leaf remain distinct. Its FIFO queue and printed order are teaching choices, not replicas of PyTorch’s device queues, hooks, locking, or tensor kernels.下面的独立程序用标量局部导数实现上述图：统计边、缓存梯度贡献，并在依赖到齐后执行节点一次。平方节点到叶子的两条边仍然分别计数。这里采用 FIFO 队列，打印顺序用于教学，不是对 PyTorch 设备队列、hook、锁或张量内核的完整复刻。

#include <array>
#include <iostream>
#include <queue>
#include <string>
#include <vector>

struct Edge { int to; double local_derivative; };
struct Node { const char* name; std::vector<Edge> next; };

int main(int argc, char** argv) {
    const double x = argc > 1 ? std::stod(argv[1]) : 2.0;
    // Root, two branches, shared square, leaf accumulator.
    const std::array<Node, 5> graph{{
        {"loss", {{1, 1}, {2, 1}}},
        {"2*u", {{3, 2}}},
        {"3*u", {{3, 3}}},
        {"u=x*x", {{4, x}, {4, x}}},
        {"x", {}}
    }};
    std::array<int, 5> pending{}, calls{};
    std::array<double, 5> buffer{};
    for (const auto& node : graph)
        for (const auto& edge : node.next) ++pending[edge.to];

    std::queue<int> ready;
    buffer[0] = 1;  // Scalar loss seed.
    ready.push(0);
    double x_grad = 0;
    while (!ready.empty()) {
        const int id = ready.front();
        ready.pop();
        ++calls[id];
        const double incoming = buffer[id];
        std::cout << "run " << graph[id].name << ": " << incoming << '\n';
        if (id == 4) x_grad += incoming;
        for (const auto& edge : graph[id].next) {
            buffer[edge.to] += incoming * edge.local_derivative;
            --pending[edge.to];
            if (edge.to == 3)
                std::cout << "u buffer=" << buffer[3]
                          << ", pending=" << pending[3] << '\n';
            if (pending[edge.to] == 0) ready.push(edge.to);
        }
        buffer[id] = 0;  // This node consumed its input buffer.
    }
    std::cout << "x.grad=" << x_grad << '\n';
    for (const int count : calls) if (count != 1) return 1;
}

run loss: 1
run 2*u: 1
u buffer=2, pending=1
run 3*u: 1
u buffer=5, pending=0
run u=x*x: 5
run x: 20
x.grad=20

The square runs with 5, not once with 2 and again with 3. Its two outputs each equal 5*x; their sum reaches the leaf before the leaf runs. Changing the input to 3 gives 30, matching the derivative of 5*x*x. All nodes in this model are reachable from its one root, so counting every listed edge is valid here.平方节点以 5 为输入执行一次，而不是分别以 2 和 3 执行两次。它的两个输出均为 5*x，两者在叶子执行前求和。把输入改为 3，结果就是 30，与 5*x*x 的导数一致。本模型所有节点都能从唯一根节点到达，因此可以统计列出的全部边。

Two kinds of accumulation, two different lifetimes两种累加，对应两种生命周期

Inside one backward task, input buffers combine contributions from graph branches. At a leaf, AccumulateGrad updates the persistent .grad field: initialize it when absent, otherwise add the new result. These are separate operations. A new backward task does not automatically clear a leaf’s existing gradient.一次反向任务内部，输入缓冲区汇合图中各分支的贡献。到达叶子后，AccumulateGrad 更新持久的 .grad 字段：尚无梯度时初始化，已有梯度时加上新结果。这是两种不同的操作；启动新的反向任务不会自动清除叶子已有的梯度。

accumulate_grad.cpp: AccumulateGrad::applyaccumulate_grad.cpp：AccumulateGrad::apply

accumulate_grad.h: initialization and addition of leaf gradientsaccumulate_grad.h：叶子梯度的初始化与累加

import torch

x = torch.tensor(2., requires_grad=True)
for _ in range(2):
    u = x * x  # A fresh forward graph each time.
    (2*u + 3*u).backward()
    print("accumulated:", x.grad.item())
x.grad = None
u = x * x
(2*u + 3*u).backward()
print("after reset:", x.grad.item())

accumulated: 20.0
accumulated: 40.0
after reset: 20.0

In a training loop, clearing gradients before a new optimization step is usually intentional; microbatch accumulation deliberately postpones it. Assigning None makes the next accumulation start without an existing gradient tensor. Setting an existing gradient tensor to zero preserves that tensor. An absent gradient and a zero gradient can also be treated differently by optimizers.训练循环通常在新的优化步骤前主动清理梯度，微批次梯度累积则有意推迟清理。赋值 None 会让下次累积从“没有现有梯度张量”开始；将已有梯度张量清零则保留这个张量。优化器也可能区别处理“没有梯度”和“零梯度”。

Optimizer.zero_grad: set_to_none semanticsOptimizer.zero_grad：set_to_none 的语义

Saved tensors explain two common backward errors从保存的张量理解两类反向报错

The square’s derivative needs the forward value of x. Autograd saves required values or references, not necessarily a fresh copy of every tensor. During ordinary backward, saved intermediates can be released after their node executes. A surviving grad_fn object therefore does not guarantee that its saved data remain available for another traversal.平方的导数需要前向时的 x。Autograd 会保存所需数值或引用，但不一定为每个张量创建独立副本。普通 backward 中，节点执行后可以释放保存的中间数据。因此，grad_fn 对象还存在，不代表再次遍历时需要的数据也仍然可用。

Autograd mechanics: saved tensorsAutograd mechanics：为反向计算保存的张量

import torch

x = torch.tensor(2., requires_grad=True)
y = x * x
y.backward()
try:
    y.backward()
except RuntimeError as error:
    print("reused graph:", type(error).__name__)

x = torch.tensor(2., requires_grad=True)
y = x * x
with torch.no_grad():
    x.add_(1)
try:
    y.backward()
except RuntimeError as error:
    print("modified saved input:", type(error).__name__)

reused graph: RuntimeError
modified saved input: RuntimeError

The first case needs saved data already released. Use a new forward computation, or deliberately retain the graph on an earlier traversal if reuse is required. retain_graph=True keeps needed graph data; it does not clear .grad, and it is not normally needed between training iterations with fresh forwards.第一种情况再次需要的数据已经释放。可以重新前向计算；如果确实需要复用，则在更早的遍历中主动保留图。retain_graph=True 保留所需的图数据，不会清空 .grad，也通常不适用于每次都重新前向的训练迭代。

The second case changes a value needed by the derivative. SavedVariable::unpack checks its version against the saved version and rejects this mismatch in the example. no_grad() prevents recording the mutation as a differentiable operation; it does not suppress the version update or repair the saved value. Do not bypass the check with .data.第二种情况修改了导数需要的值。SavedVariable::unpack 将当前版本与保存版本比较，本例因不匹配而被拒绝。no_grad() 只是不把这次修改记录成可求导运算，不会取消版本更新，更不会恢复已保存的数值。不要用 .data 绕开检查。

saved_variable.cpp: SavedVariable::unpack and version checkingsaved_variable.cpp：SavedVariable::unpack 与版本检查

Recording a derivative is different from retaining a graph记录导数的计算过程，不等于保留旧图

create_graph=True requests a differentiable computation of the derivative, so supported backward operations themselves become part of a graph. For L = 5*x*x, the first derivative is 10*x and the second is 10. Here autograd.grad returns each derivative directly, avoiding persistent leaf accumulation.create_graph=True 请求以可求导方式计算导数，让受支持的反向运算本身也成为图的一部分。对于 L = 5*x*x，一阶导数为 10*x，二阶导数为 10。这里用 autograd.grad 直接返回每阶导数，避免持久叶子梯度累积。

import torch

x = torch.tensor(2., requires_grad=True)
(g,) = torch.autograd.grad(5*x*x, x, create_graph=True)
(h,) = torch.autograd.grad(g, x)
print("derivatives:", g.item(), h.item())

derivatives: 20.0 10.0

The invariant behind the branch example is now explicit: gather contributions at the correct input slot, execute only when dependencies permit it, and treat leaf gradient storage separately from temporary execution buffers. On CUDA the same dependency logic also has to respect stream ordering; CPU queue readiness alone does not describe when GPU work completes.分支示例的核心约束由此明确：贡献必须汇入正确的输入槽，节点只有在依赖允许时才执行，叶子梯度存储与临时执行缓冲区分别管理。在 CUDA 上，这套依赖逻辑还必须满足流顺序；CPU 队列中“已经就绪”并不能完整描述 GPU 工作何时完成。