Tensor, strides, and storage: why a transpose can be freeTensor、步幅与存储：转置为什么可以不搬数据

One buffer, two matrices一块存储，两种矩阵

In part 1, torch.matmul(a, b) passed through dispatch and reached a numerical kernel. Now inspect the operands. A tensor needs both values and a rule for locating those values. For an ordinary dense strided tensor, changing that rule can change its logical shape without moving its elements.第一期中，torch.matmul(a, b) 经过调度进入数值内核。这一期先拆开它的输入：张量既需要数值，也需要定位这些数值的规则。对于普通的稠密步幅张量，改变寻址规则就可能改变逻辑形状，而不必搬动元素。

import torch

a = torch.arange(12, dtype=torch.float32).reshape(3, 4)
b = a.T
print(a.shape, a.stride())  # torch.Size([3, 4]) (4, 1)
print(b.shape, b.stride())  # torch.Size([4, 3]) (1, 4)
print(a.untyped_storage().data_ptr() == b.untyped_storage().data_ptr())
# True

The transpose changes the coordinate system, not the stored sequence. Reading b[2, 1] reaches the same element as a[1, 2]: value 6. The following experiment makes the logical coordinates and physical offsets visible at the same time.转置改变的是坐标解释方式，存储中的数值顺序并没有变化。读取 b[2, 1] 与读取 a[1, 2] 会到达同一个元素：数值 6。下面的实验同时显示逻辑坐标与物理偏移。

Source baseline: PyTorch 2.10.0, commit 449b17684101, matching part 1. Python examples were checked with 2.10.0+cpu. Scope: eager execution, nonempty dense strided tensors with ordinary numeric dtypes. Sparse layouts, tensor subclasses and symbolic shapes have additional rules.源码基线：PyTorch 2.10.0，提交 449b17684101，与第一期一致。Python 示例已在 2.10.0+cpu 中核验。本文讨论 eager 模式下、普通数值类型的非空稠密步幅张量；稀疏布局、张量子类和符号形状另有规则。

Follow an element to memory跟着一个元素找到内存

Choose a layout, then click a matrix entry. The highlighted storage slot and the address calculation refer to that exact entry. “Scan” walks through logical rows: watch whether addresses advance by one, jump, or repeat. It is an address-mapping model in JavaScript, not PyTorch running in the browser.选择布局，再点击矩阵元素。高亮存储格与寻址计算会指向同一个元素。“扫描”按逻辑行依次访问：观察地址是连续递增、跳跃还是重复。这里运行的是 JavaScript 寻址模型，不是在浏览器里运行 PyTorch。

In the contiguous-copy mode, compare values in the new buffer B with their original positions in A. For example, logical [0, 1] contains 4: the transpose reads A[4], while its contiguous copy reads B[1]. The value is unchanged; the storage and stride are different.在连续化复制模式中，对照新存储 B 与原存储 A 中的数值位置。例如逻辑元素 [0, 1] 的值为 4：转置视图读取 A[4]，其连续副本读取 B[1]。数值没有变化，存储对象和步幅发生了变化。

Tensor → TensorImpl → StorageImpl从 Tensor 追到 StorageImpl

The Python tensor wraps a C++ tensor handle. TensorBase holds an intrusive reference-counted pointer to TensorImpl. For the strided path studied here, TensorImpl carries sizes, strides, storage offset, dtype, device and dispatch information; it also connects to autograd metadata. A Storage handle refers to StorageImpl, which manages the data pointer and storage size. These layers separate a tensor’s interpretation from the lifetime of its allocation.Python 张量封装了 C++ 张量句柄。TensorBase 通过侵入式引用计数指针持有 TensorImpl。在本文的步幅布局路径中，TensorImpl 包含形状、步幅、存储偏移、数据类型、设备和调度信息，也关联自动求导元数据。Storage 句柄则指向 StorageImpl，后者管理数据指针与存储大小。这几层结构把张量的解释方式与存储分配的生命周期分开。

Read these declarations in order: TensorBase::impl_ → TensorImpl fields → StorageImpl fields. The diagram describes ownership links, not a byte-accurate C++ object layout. Reference counts allow the shared storage to remain alive after one view is destroyed.按顺序阅读这些声明：TensorBase::impl_ → TensorImpl 字段 → StorageImpl 字段。上图表示持有关系，并非精确的 C++ 对象字节布局。引用计数使一个视图销毁后，共享存储仍可由其他张量继续持有。

// Selected declarations from separate PyTorch classes; not one compilable struct.
// TensorBase
c10::intrusive_ptr<TensorImpl, UndefinedTensorImpl> impl_;

// TensorImpl
Storage storage_;
c10::impl::SizesAndStrides sizes_and_strides_;
int64_t storage_offset_ = 0;

// StorageImpl
DataPtr data_ptr_;
SymInt size_bytes_;

Copying a C++ tensor handle generally shares its TensorImpl. Creating a transpose view gives a different tensor interpretation while sharing storage. Neither operation means “duplicate all numeric elements.” Conversely, a tiny slice can keep a large backing allocation alive.复制 C++ 张量句柄通常会共享 TensorImpl；创建转置视图则得到不同的张量解释方式，同时共享存储。这两种操作都不等于“复制所有数值元素”。反过来，即使一个切片很小，也可能让背后的大块分配一直存活。

Shape tells you the bounds; strides tell you the route形状给出边界，步幅给出路径

Let i be a valid index, o the storage offset, and s the strides. Strides and storage offset are measured in elements, not bytes. Multiply by the dtype’s element size only when converting the element offset into a byte address.设 i 为合法索引，o 为存储偏移，s 为步幅。步幅和存储偏移的单位是元素，不是字节。只有把元素偏移换成字节地址时，才乘以数据类型的元素大小。

For a, moving down one row skips four elements; moving right skips one. After transposition, b[2,1] has element offset 0 + 2×1 + 1×4 = 6. For float32 this is 24 bytes after the storage base. No copy is needed to compute a different address.对 a 而言，向下一行跨过 4 个元素，向右一列跨过 1 个元素。转置后的 b[2,1] 对应元素偏移 0 + 2×1 + 1×4 = 6。对于 float32，它位于存储起点之后 24 字节。通过另一套规则计算地址，并不需要复制数据。

A slice introduces another distinction: data_ptr() points to the tensor’s first element, whereas untyped_storage().data_ptr() points to the storage base. In a[:,1::2], offset is 1 and strides are (4,2); the first element pointer is four bytes after the base. Comparing tensor data pointers alone is therefore not a general alias test.切片还引入一个区别：data_ptr() 指向张量的第一个元素，untyped_storage().data_ptr() 指向存储起点。对于 a[:,1::2]，偏移为 1，步幅为 (4,2)，首元素指针比存储起点晚 4 字节。因此，仅比较张量的 data_ptr 并不能普遍判断是否共享存储。

s = a[:, 1::2]
print(s.stride(), s.storage_offset())  # (4, 2) 1
print(s.data_ptr() - a.data_ptr())     # 4 bytes
print(s.untyped_storage().data_ptr() == a.untyped_storage().data_ptr())
# True

Read the transpose implementation读懂转置的源码路径

In TensorShape.cpp::transpose, separate branches handle sparse and MKLDNN layouts. After those checks, the ordinary strided path swaps the two size entries and the corresponding stride entries, then constructs an as_strided view. The existing storage offset is retained. This is why “transpose is a view” needs a layout qualification.在 TensorShape.cpp::transpose 中，稀疏布局与 MKLDNN 布局先进入各自的分支。通过这些检查后，普通步幅布局路径交换两个维度的大小与对应步幅，再构造 as_strided 视图，保留原有存储偏移。因此，“转置是视图”需要说明所讨论的布局。

std::swap(sizes[dim0], sizes[dim1]);
std::swap(strides[dim0], strides[dim1]);
auto result = self.as_strided_symint(sizes, strides);

These are three selected source lines; creation of the local size/stride vectors and checks is omitted. A standalone C++ model below implements the same address arithmetic without depending on LibTorch. It is deliberately limited to two dimensions and fixed storage; it does not implement dispatch, autograd or PyTorch ownership.以上摘取了三行源码，省略了局部形状／步幅向量的创建及检查。下面的独立 C++ 模型实现同一套寻址计算，不依赖 LibTorch。它仅处理二维和固定存储，没有实现调度、自动求导或 PyTorch 的所有权机制。

#include <array>
#include <cstddef>
#include <iostream>

struct View2D {
    const std::array<float, 12>& storage;
    std::array<std::size_t, 2> shape, stride;
    std::size_t offset;

    float at(std::size_t i, std::size_t j) const {
        if (i >= shape[0] || j >= shape[1]) throw "index out of range";
        return storage.at(offset + i * stride[0] + j * stride[1]);
    }
    View2D transpose() const {
        return {storage, {shape[1], shape[0]},
                {stride[1], stride[0]}, offset};
    }
};

int main() {
    std::array<float, 12> values{};
    for (std::size_t k = 0; k < values.size(); ++k) values[k] = float(k);
    const View2D a{values, {3, 4}, {4, 1}, 0};
    const auto b = a.transpose();
    const View2D s{values, {3, 2}, {4, 2}, 1};
    std::cout << b.at(2, 1) << ' ' << s.at(1, 1) << '\n';
    values[6] = 60;
    std::cout << a.at(1, 2) << ' ' << b.at(2, 1) << '\n';
}

Verified with GCC 14.2 on Godbolt using C++17 and -O2. Standard output:已在 Godbolt 上使用 GCC 14.2、C++17 与 -O2 编译运行。标准输出：

6 7
60 60

view, reshape, and contiguous answer different questionsview、reshape 与 contiguous 回答不同问题

view asks whether the requested shape can be described by new strides over the existing storage. reshape asks for the requested shape and may copy when a view is not possible. contiguous asks for a particular memory format: if already satisfied it returns the input, otherwise it copies. The default format here is torch.contiguous_format.view 判断能否用新的步幅在原存储上表达目标形状；reshape 要求得到目标形状，无法构造视图时可以复制；contiguous 要求指定的内存格式，已满足则返回输入，否则复制。这里讨论的默认格式为 torch.contiguous_format。

The concrete decisions are visible in view_impl (infer shape → compute strides → check → alias), computeStride_impl (match contiguous subspaces), and contiguous (return input or clone). The view API contract explains which adjacent dimensions can be merged.具体判断可沿着 view_impl（推导形状 → 计算步幅 → 检查 → 共享存储）、computeStride_impl（匹配连续子空间）和 contiguous（返回输入或 clone）阅读。view 的 API 约定说明了相邻维度何时可以合并。

For adjacent nonsingleton dimensions that must be merged, this stride relation preserves the traversal order. In a.T, the relation would require 1 = 4×3, so flattening with view(-1) fails. The stepped slice is more interesting: (4,2) with shape (3,2) satisfies 4 = 2×2, so it can flatten into a stride-2 view even though it is not contiguous. Non-contiguous does not mean “view always fails.” Singleton and empty dimensions need the full implementation’s special handling.对于需要合并的相邻非单例维度，这个步幅关系保证遍历顺序不变。a.T 若想合并两维，需要满足 1 = 4×3，因此 view(-1) 失败。间隔切片更有意思：形状 (3,2)、步幅 (4,2) 满足 4 = 2×2，所以它虽然不连续，仍能展平为步幅为 2 的视图。不连续并不意味着 view 一定失败。单例维度与空维度还需要完整实现中的特殊处理。

try:
    b.view(-1)
except RuntimeError:
    print("transpose view(-1): incompatible strides")
print(s.view(-1).stride())      # (2,)
print(b.reshape(-1).tolist())   # a copy for this particular b
# [0., 4., 8., 1., 5., 9., 2., 6., 10., 3., 7., 11.]
c = b.contiguous()
print(c.stride(), c.is_contiguous())  # (3, 1) True
print(c.untyped_storage().data_ptr() == a.untyped_storage().data_ptr())
# False

A transpose is typically cheap to create, but that says little about the cost of the next operation. A backend may consume strided data directly, interpret a transpose flag, or materialize a suitable layout. Calling contiguous() preemptively can add an unnecessary allocation and copy. Measure the complete operation sequence; the scan above visualizes addresses, not cache misses or GPU transactions.转置通常创建成本较低，但这不能直接说明后续算子的成本。后端可能直接处理步幅数据、使用转置标志，也可能物化合适的布局。提前调用 contiguous() 可能反而增加不必要的分配与复制。应测量完整算子序列；上方扫描展示的是地址，不是缓存未命中或 GPU 内存事务。

Shared storage is observable, including during backward共享存储会影响写入，也会影响反向计算

Use a fresh tensor without gradient recording to isolate aliasing. Writing through the transpose changes the original; writing through a contiguous copy does not. The Tensor Views documentation also distinguishes basic indexing (views) from advanced indexing (copies). Do not infer copying just from bracket syntax.先用一个不记录梯度的新张量单独观察别名行为。通过转置视图写入会改变原张量，通过连续副本写入则不会。Tensor Views 文档还区分了基本索引（视图）和高级索引（复制），不能仅凭方括号语法判断是否复制。

a = torch.arange(12, dtype=torch.float32).reshape(3, 4)
b = a.T
c = b.contiguous()
b[2, 1] = 60
print(a[1, 2].item(), c[2, 1].item())  # 60.0 6.0

A view can also remain part of the autograd graph. For a transpose, backward maps the incoming gradient back with the inverse transpose. For an expanded dimension, several output positions depend on one input position, so backward sums their contributions. Sharing storage does not imply sharing a .grad field.视图也可以保留在自动求导图中。对于转置，反向计算用逆转置把传入梯度映射回去；对于扩展维度，多个输出位置依赖同一个输入位置，因此反向计算会将这些位置的梯度相加。共享存储不意味着共享同一个 .grad 字段。

x = torch.arange(6., requires_grad=True)  # leaf
x.reshape(2, 3).T.sum().backward()
print(x.grad.tolist())  # [1., 1., 1., 1., 1., 1.]

v = torch.tensor([[1., 2.]], requires_grad=True)
v.expand(3, 2).sum().backward()
print(v.grad.tolist())  # [[3., 3.]]

In the experiment’s expand mode, stride 0 along the first dimension makes all rows refer to the same four slots. No three-row copy is created. Avoid in-place writes to such overlapping views; clone first if independent writable elements are needed. See the expand contract. Autograd also tracks version counters and rejects certain in-place modifications, especially views of leaf tensors requiring gradients or values saved for backward.实验中的 expand 模式把第一维步幅设为 0，于是所有行都指向同样的四个存储格，没有创建三行副本。应避免向这种重叠视图原地写入；需要独立可写元素时先 clone，参见 expand 约定。自动求导还会追踪版本计数，并拒绝某些原地修改，尤其是需要梯度的叶子张量视图，或反向计算保存过的数值。

The backward rules are declared in derivatives.yaml: transpose and expand: one transposes the gradient, the other reduces it to the input shape.反向规则声明在 derivatives.yaml 的 transpose 条目与 expand 条目中：前者对梯度转置，后者将梯度归约至输入形状。

Predict first, then inspect先预测，再验证

Try three questions before running Python: (1) what are the strides and offset of a[1:, ::2]? (2) does a.contiguous() require a copy? (3) can a transpose followed by another transpose restore the original layout? Explain each answer using addresses rather than printed matrix values.运行 Python 前先回答三个问题：（1）a[1:, ::2] 的步幅和偏移是多少？（2）a.contiguous() 需要复制吗？（3）连续做两次转置能恢复原布局吗？请用寻址规则解释，不要只比较打印出来的矩阵数值。

Check the reasoning查看推导

For a fresh contiguous (3,4) tensor, the slice has shape (2,2), strides (4,2), offset 4, and reads [4,6;8,10]. The original already satisfies the default contiguous format, so its contiguous call returns itself. Swapping the same two axes twice restores both sizes and strides, while retaining the same storage.对于新的连续 (3,4) 张量，切片形状为 (2,2)、步幅为 (4,2)、偏移为 4，读取 [4,6;8,10]。原张量已满足默认连续格式，因此 contiguous 返回自身。对同一对轴交换两次，会恢复形状和步幅，并继续使用同一存储。

For a dense strided tensor, the logical index, strides and storage offset determine which element of the backing storage is accessed. A view changes that mapping while sharing storage; materializing a contiguous copy changes where the values are stored.对于稠密步幅张量，逻辑索引、步幅与存储偏移共同决定访问底层存储中的哪个元素。视图在共享存储的同时改变映射方式；物化为连续副本则改变数值的存放位置。