PyTorch torch.nn.SiLU Function

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

torch.nn.SiLUIt is the Sigmoid Linear Unit activation function in PyTorch, also known as Swish.

It has self-gating properties, is smoother than ReLU, and performs better on some tasks.

Function Definition

torch.nn.SiLU(inplace=False)

Mathematical Principle

SiLU(x) = x * sigmoid(x)

Usage Examples

Example 1: Basic Usage

Example

import torch
import torch.nn as nn

silu = nn.SiLU()

x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
output = silu(x)

print("Input:", x.tolist())
print("Output:", output.tolist())

Example 2: Comparison with ReLU

Example

import torch
import torch.nn as nn
import numpy as np

x = np.linspace(-3, 3, 7)
x_tensor = torch.tensor(x)

print("x       SiLU      ReLU")
print("-" * 30)
for xi in x_tensor:
    print(f"{xi.item():5.1f} {nn.SiLU()(xi.unsqueeze(0)).item():9.4f} {nn.ReLU()(xi.unsqueeze(0)).item():9.4f}")

Example 3: Use in MobileNet

Example

import torch
import torch.nn as nn

# MobileNetV3 uses SiLU
model = nn.Sequential(
    nn.Conv2d(3, 32, 3, stride=2, padding=1),
    nn.BatchNorm2d(32),
    nn.SiLU(),
    nn.Conv2d(32, 64, 3, padding=1),
    nn.BatchNorm2d(64),
    nn.SiLU()
)

x = torch.randn(1, 3, 224, 224)
output = model(x)

print("Input:", x.shape, "-> Output:", output.shape)

Use Cases

  • MobileNet: MobileNetV3
  • EfficientNet
  • Smooth activation: requires gating properties

Note: SiLU is more computationally complex than ReLU, but its performance is usually better.

PyTorch torch.nn 参考手册PyTorch torch.nn Reference Manual

Other Extensions