本文档适用于 containerd version >= v2.2

1. 绿色安装containerd

1) 下载安装文件

https://github.com/containerd/containerd/releases/download/v2.1.3/containerd-static-2.1.3-linux-amd64.tar.gz
* containerd  containerd-shim-runc-v2  containerd-stress  ctr

https://github.com/opencontainers/runc/releases/download/v1.4.0/runc.amd64

https://github.com/containernetworking/plugins/releases/download/v1.7.1/cni-plugins-linux-amd64-v1.7.1.tgz
* bandwidth  dhcp   firewall     host-local  loopback  portmap  sbr     tap     vlan
* bridge     dummy  host-device  ipvlan      macvlan   ptp      static  tuning  vrf

# 添加环境变量：Environment="PATH=/usr/local/containerd/bin:$PATH"
https://raw.githubusercontent.com/containerd/containerd/main/containerd.service

https://github.com/containerd/nerdctl/releases/download/v2.2.2/nerdctl-2.2.2-linux-amd64.tar.gz

alias nerdctl='/usr/local/containerd/bin/nerdctl --hosts-dir /usr/local/containerd/etc/certs.d --host /run/containerd/containerd.sock --data-root /usr/local/containerd/lib/nerdctl --cni-path /usr/local/containerd/bin --cni-netconfpath /usr/local/containerd/etc/cni/net.d'

2) containerd/config.toml

config.toml配置主要是给 CRI（Container Runtime Interface）接口用的。nerdctl和ctr使用的是 containerd 的原生 gRPC API，所以无法通过config.toml配置来设置参数，只能通过命令行参数来设置。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

# 注意 shim 的 socketRoot 目录（/run/containerd/s）是硬编码的（出于安全考虑），无法通过配置改变它。
version = 3
root = '{{.containerd_root}}/root'
state = '{{.containerd_root}}/state'
# 使用k8s时，需要禁用这个插件： io.containerd.monitor.container.v1.restart
disabled_plugins = ["io.containerd.differ.v1.erofs",
               "io.containerd.snapshotter.v1.blockfile", "io.containerd.snapshotter.v1.btrfs",
               "io.containerd.snapshotter.v1.devmapper", "io.containerd.snapshotter.v1.erofs",
               "io.containerd.snapshotter.v1.zfs", "io.containerd.snapshotter.v1.native",
               "io.containerd.tracing.processor.v1.otlp", "io.containerd.internal.v1.tracing",
               "io.containerd.nri.v1.nri"]

# ctr plugins ls 命令可以查看所有的插件
[plugins]

  # cri配置
  [plugins.'io.containerd.cri.v1.images'.registry]
    config_path = "{{.containerd_root}}/etc/certs.d"
  [plugins.'io.containerd.cri.v1.runtime']
    enable_cdi = false
    cdi_spec_dirs = ['{{.containerd_root}}/cdi']
    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      runtime_path = "{{.containerd_root}}/bin/containerd-shim-runc-v2"
      cni_conf_dir = "{{.containerd_root}}/etc/cni/custom-net.d"  # 覆盖全局配置，默认值为""，表示使用全局 CNI 配置目录
      [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runc.options]
        BinaryName = "{{.containerd_root}}/bin/runc"
        SystemdCgroup = true
        # containerd就是通过github.com\containerd\containerd\api\types\runc\options这个api来传Root参数的，如果没有传，就使用config.toml中Root配置的值。
        Root = "{{.containerd_root}}/state/runc"  # 如果该配置为""，就使用路径"/run/containerd/runc"
        CriuImagePath = ''
        CriuWorkPath = ''
    [plugins.'io.containerd.cri.v1.runtime'.cni]
      # containerd v2.1开始，bin_dir 被 bin_dirs 取代
      bin_dirs = ['{{.containerd_root}}/bin']
      conf_dir = "{{.containerd_root}}/etc/cni/net.d"  # 全局 CNI 配置目录
  [plugins.'io.containerd.grpc.v1.cri']
    disable_tcp_service = true
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    stream_idle_timeout = '4h0m0s'
    enable_tls_streaming = false
    [plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
      tls_cert_file = ''
      tls_key_file = ''

  # 通用配置
  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '{{.containerd_root}}/opt/image-verifier/bin'
  [plugins.'io.containerd.internal.v1.opt']
    path = '{{.containerd_root}}/opt'
  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = "{{.containerd_root}}/lib/io.containerd.snapshotter.v1.overlayfs"
  # 本地传输插件，主要负责容器镜像在本地文件系统上的导入（import）和导出（export）操作，还可以处理镜像的pull和push操作。
  [plugins.'io.containerd.transfer.v1.local']
    config_path = '{{.containerd_root}}/etc/certs.d'
    max_concurrent_downloads = 3
    concurrent_layer_fetch_buffer = 0
    max_concurrent_uploaded_layers = 3
    check_platform_supported = false

# 流处理器配置
[stream_processors]
  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
    accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '{{.containerd_root}}/etc/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG={{.containerd_root}}/etc/ocicrypt/ocicrypt_keyprovider.conf']

  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
    accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '{{.containerd_root}}/etc/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG={{.containerd_root}}/etc/ocicrypt/ocicrypt_keyprovider.conf']

2. containerd如何配置镜像加速

1) 给docker.io这个registry设置镜像

假设ctr的--hosts-dir参数设置为 /usr/local/containerd/etc/certs.d

$ tree /usr/local/containerd/etc/certs.d/
/usr/local/containerd/etc/certs.d/
└── docker.io
    └── hosts.toml

$ cat /usr/local/containerd/etc/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://055b251cd5000fb90fc3c01b214f2380.mirror.swr.myhuaweicloud.com"]
  capabilities = ["pull", "resolve"]

2) Setup Default Mirror for All Registries

1
2
3
4
5
6
7
8

$ tree /etc/containerd/certs.d
/etc/containerd/certs.d
└── _default
    └── hosts.toml

$ cat /etc/containerd/certs.d/_default/hosts.toml
[host."https://registry.example.com"]
  capabilities = ["pull", "resolve"]

3) 如何创建一个registry服务：

# 0. 部署registry
docker run -d -p 127.0.0.1:5000:5000 --restart always --name registry registry:2

# 1. 查看远程仓库有哪些镜像
curl http://localhost:5000/v2/_catalog

# 2. 查看镜像的标签列表
curl http://localhost:5000/v2/<name>/tags/list

3. 在linux系统中如何搭建windows和mac os环境

虽然linux桌面一言难尽，但是这不影响linux作为自己的主操作系统，因为linux中运行windows或者mac的虚拟机性能损耗极低，但是windows或者mac中运行linux虚拟机性能损耗极大。

确认BIOS中已启用虚拟化扩展（Intel VT-x 或 AMD SVM）

linux上检测是否支持kvm: sudo apt install cpu-checker; sudo kvm-ok

https://github.com/dockur/windows

https://github.com/dockur/macos

nerdctl run -d --name my-windows --privileged \
--env VERSION=2022 --env LANGUAGE=Chinese --env REGION=zh-CN \
--env KEYBOARD=zh-CN --env RAM_SIZE=8G --env CPU_CORES=4 \
--env USERNAME=docker --env PASSWORD=admin \
--mount type=bind,src=`pwd`/storage,dst=/storage \
--mount type=bind,src=`pwd`/shared,dst=/shared \
--cap-add CAP_NET_ADMIN --device /dev/kvm --device /dev/net/tun \
--ip 10.4.0.2 \
docker.io/dockurr/windows:latest

执行nerdctl exec my-windows ip a命令获取eth0网卡的ip，之后使用http://{eth0网卡的ip}:8006链接访问windows桌面

在容器内执行capsh --print命令的作用：检测--cap-add CAP_NET_ADMIN参数是否有生效

即使容器拥有 CAP_NET_ADMIN能力，容器内运行的进程也必须以 root 用户（UID 0）运行才能使用该能力

主机上的安全模块（如 AppArmor​ 或 SELinux）可能覆盖或限制了容器能力。

• AppArmor：检查容器的 AppArmor 配置文件。默认的 docker-default或类似配置文件可能明确拒绝了某些网络管理操作。
• SELinux：检查 SELinux 上下文和布尔值设置。

4. ctr 和 nerdctl 的区别

nerdctl的镜像构建依赖buildkit，可以从buildkit源码的examples/systemd/system目录下获取到buildkit.service和buildkit.socket。

ctr不支持镜像构建功能，但是可以通过--with-ns参数来对接“cni插件维护的网络命名空间”

ip netns add cni-containerd
export CNI_PATH=/usr/local/containerd/bin
export NETCONFPATH=/usr/local/containerd/etc/cni/net.d
# CNI_NETNSDIR环境变量的默认是：/var/run/netns
# 这里的 "ctr-net" 要和cni配置文件中的 "name" 对应起来
# ${NETCONFPATH}/99-ctr-net.conflist文件的内容：
# {
#   "cniVersion": "1.0.0",
#   "name": "ctr-net",
#   "plugins": [
#     {
#       "type": "loopback"
#     },
#     {
#       "type": "bridge",
#       "bridge": "cni0",
#       "isGateway": true,
#       "ipMasq": true,
#       "ipam": {
#         "type": "host-local",
#         "ranges": [
#           [
#             {
#               "subnet": "10.22.0.0/16",
#               "rangeStart": "10.22.0.100",
#               "rangeEnd": "10.22.0.200",
#               "gateway": "10.22.0.1"
#             }
#           ]
#         ],
#         "routes": [
#           {"dst": "0.0.0.0/0", "gw": "10.22.0.1"}
#         ],
#         "dataDir": "/var/lib/cni/mybridge"
#       }
#     },
#     {
#       "type": "tuning"
#     },
#     {
#       "type": "portmap",
#       "capabilities": {
#         "portMappings": true
#       },
#       "snat": true
#     }
#   ],
#   "dns": {
#     "nameservers": ["8.8.8.8", "8.8.4.4"]
#   }
# }
cnitool add ctr-net /var/run/netns/cni-containerd

ctr container create --runc-binary /usr/local/containerd/bin/runc --with-ns "network:/var/run/netns/cni-containerd" image_addr   container_name

从下面的代码得知，当ctr container create命令的runtime使用“io.containerd.runc.v2”，--runtime-config-path参数就会跳过，设置了也不会生效。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# cmd\ctr\commands\run\run_unix.go
func NewContainer(ctx context.Context, client *containerd.Client, cliContext *cli.Context) (containerd.Container, error) {
	//...
	var (
		opts  []oci.SpecOpts
		cOpts []containerd.NewContainerOpts
		spec  containerd.NewContainerOpts
	)
	//...
		opts = append(opts, oci.WithDefaultSpecForPlatform(platform), oci.WithDefaultUnixDevices)
	//...
	runtimeOpts, err := commands.RuntimeOptions(cliContext)
	//...
	spec = containerd.WithSpec(&s, opts...)
	cOpts = append(cOpts, spec)
	return client.NewContainer(ctx, id, cOpts...)
}

# cmd\ctr\commands\commands_unix.go:
func RuntimeOptions(cliContext *cli.Context) (interface{}, error) {
	// validate first
	if (cliContext.String("runc-binary") != "" || cliContext.Bool("runc-systemd-cgroup")) &&
		cliContext.String("runtime") != "io.containerd.runc.v2" {
		return nil, errors.New("specifying runc-binary and runc-systemd-cgroup is only supported for \"io.containerd.runc.v2\" runtime")
	}

	if cliContext.String("runtime") == "io.containerd.runc.v2" {
		return getRuncOptions(cliContext)
	}

	if configPath := cliContext.String("runtime-config-path"); configPath != "" {
		return &runtimeoptions.Options{
			ConfigPath: configPath,
		}, nil
	}

	return nil, nil
}

附录A 插件

所有插件都记录在 github.com/containerd/containerd/plugins 目录下

containerd运行时不再加载动态库插件，所以config.toml文件中名为plugin_dir的根配置已过时。

A.1 io.containerd.grpc.v1.cri 和 io.containerd.cri.v1.runtime 的区别

Support for the following properties of [plugins.\"io.containerd.grpc.v1.cri\".registry] is deprecated and will be removed in a future release.

• The CRIRegistryMirrors (mirrors) property. Users should migrate to use config_path.
• The CRIRegistryAuths (auths) property. Users should migrate to use ImagePullSecrets.
• The CRIRegistryConfigs (configs) property. Users should migrate to use config_path.

A.2 默认开启的插件

A.3 NRI is now enabled by default

NRI (Node Resource Interface) is a framework for plugging domain or vendor-specific logic into OCI-compatible container runtimes. It allows users to make changes to containers, perform extra actions, and improve the management of resources. NRI plugins are considered to be part of the container runtime, and access to NRI is controlled by restricting access to the systemwide NRI socket. See the “NRI” document for more details.

A.4 CDI is now enabled by default

CDI (Container Device Interface) provides a standard mechanism for device vendors to describe what is required to provide access to a specific resource such as a GPU beyond a simple device name. CDI is now part of the Kubernetes Device Plugin framework. See the Kubernetes Enhancement Proposal 4009.

A.5 Image verifier plugins

The transfer service now supports plugins that can verify that images are allowed to be pulled. Plugins like this can implement policy, such as enforcing that container images are signed, or that images must have particular names. Plugins are independent programs that communicate via command-line arguments and standard I/O. See more details in the image verifier plugin documentation.

A.6 criu

在 containerd 中，CRIU（Checkpoint/Restore In Userspace） 是一个用于实现容器或进程的检查点（Checkpoint）与恢复（Restore）功能的工具。

仓库地址： https://github.com/checkpoint-restore/criu.git

containerd 2.0之后，[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.*.options].CriuPath配置被移除，改为从PATH环境变量中找criu二进制可执行文件。

附录B 常见问题

B.1 io.containerd.content.v1.content 和 io.containerd.snapshotter.v1.overlayfs 和 io.containerd.runtime.v2.task 这三个目录的区别

content: 镜像层数据 (Blobs)， 只读

snapshotter: 可写层 (upperdir)，联合挂载视图

task: 运行时状态、IO 管道、配置文件、

删除时机：

• content: 镜像删除后 GC
• snapshotter: 容器删除后 GC
• task: 容器进程退出后 GC

state目录和 root目录下都包含 io.containerd.runtime.v2.task，这是为了将“运行时状态数据”与“运行时根文件系统”进行分离，这是 containerd 的标准设计

B.2 containerd 和 containerd-shim-runc-v2 和 runc

1. 当客户端调用containerd 来创建一个容器时，containerd接收到请求后，并不会直接去操作容器，而是创建一个叫做 containerd-shim-runc-v2的进程（父进程为systemd(1)），让这个进程去操作容器；

2. 之后，/usr/bin/containerd-shim-runc-v2会运行runc这个二进制文件去create、start容器，runc启动完容器后本身会直接退出，containerd-shim-runc-v2则会成为容器进程的父进程，负责收集容器进程的状态，上报给containerd。

上一篇 « 下一篇 »