There are two ways to capture packet for a given container
way1: capture in the given container
go to that container, use tcpdump(install it first if not available) in it.
required:
container has tcpdump
container no tcpdump but can install tcpdump on it from local or internet(change the container)
way2: capture in another container which shares the same net with the given container
In some case, you can’t change the given container or it’s complex to install tcpdump on the given container, so create a template container which has tcpdump installed and shared the same network with given container.
# create an image called tcpdump with tcpdump installed # then create a container with this image shares network with another container # rm means container will be destroy when exit. # 99c1bd6b342c is container ID
$ docker run --rm -it --net=container:99c1bd6b342c tcpdump # this will create unnamed container from image tcpdump and shared network with the container that you want to capture packet.
how to deploy docker env on lots of machine?
Use Docker Machine, docker machine provides a client to deploy docker env on lots machine(windows, mac, linux).
$ docker run --device=/dev/sdb:/dev/xvda -it ubuntu /bin/bash # /dev/sdb is host device # /dev/xvda is device in container
# default permission: rwm(read, write, create) # with permission $ docker run --device=/dev/sdb:/dev/xvda:r -it ubuntu /bin/bash
how to limit IO for a container
cgroups to accomplish this
Refer to docker io throttle, only supports DirectIO due to blkio cgroup limitation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
# these two policies can work together, not it only supports DirectIO, not buffer io --blkio-weight=0 Block IO weight (relative weight) accepts a weight value between 10 and 1000, default weight for each device --blkio-weight-device="" Block IO weight (relative device weight, format: DEVICE_NAME:WEIGHT), override default weight, example: --blkio-weight-device "/dev/sda:100"
--device-read-bps=[] Limit read rate (bytes per second) from a device --device-read-iops=[] Limit read rate (IO count per second) from a device --device-write-bps=[] Limit write rate (bytes per second) to a device --device-write-iops=[] Limit write rate (IO count per second) to a device
$ docker run -it --rm --device-write-bps /dev/sda:1mb ubuntu /bin/bash # /dev/sda is host device, it limits the rate of this container if it accesses /dev/sda
$ docker run -it --rm --device-write-iops /dev/sda:10 ubuntu /bin/bash
# blkio-weight is relative weight, that means if only one processA(docker process or other) with weight 100 access /dev/sda, it uses the 100% IO bandwidth, # if another processB access /dev/sda at same time who belongs to another blkio group with weight 200, 1/3 bandwidth for processA, 2/3 IO bandwith for processB
# test inside each container with directIO (container)# time dd if=/dev/zero of=test.out bs=1M count=1024 oflag=direct
--cpu-shares , -c Set this flag to a value greater or less than the default of 1024 to increase or reduce the container’s weight, and give it access to a greater or lesser proportion of the host machine’s CPU cycles. This is only enforced when CPU cycles are constrained. When plenty of CPU cycles are available, all containers use as much CPU as they need. In that way, this is a soft limit. It prioritizes container CPU resources for the available CPU cycles.
# !!!It does not guarantee or reserve any specific CPU access.!!!
# let's say there are four CPUS on host, if all containers are running CPU intensive workload # first container takes one CPU # second container takes one CPU # third container takes two CPU
# but if third container quits or sleep, the other two both take two CPUS !!!
$ docker run -it --rm --cpu-shares 1024 ubuntu /bin/bash $ docker run -it --rm --cpu-shares 1024 ubuntu /bin/bash $ docker run -it --rm --cpu-shares 2048 ubuntu /bin/bash
--cpu-period & --cpu-quota CPU quota (cpu_qota) is a feature of Linux Control Groups (cgroup). CPU quota control `how much CPU time a container can use`, `cpu_quota is the number of microseconds of CPU time` a container can use `per cpu_period`. For example configuring:
cpu_quota to 50,000 cpu_period to 100,000
The container will be allocated 50,000 microseconds per 100,000 microsecond period. `A bit like (see below) the use of 0.5 CPUs`. Quota can be greater than the period. For example:
cpu_quota to 200,000 cpu_period to 100,000
Now the container can use `200,000 microseconds of CPU time every 100,000 microseconds`. To use the CPU time there will either need to be multiple processes in the container, or a multi-threaded process. `This configuration is a bit like (see below) having 2 CPUs`.
cpu_quota allows setting an `upper bound on the amount of CPU time a container gets`. Linux enforces the limit even if CPU time is available. Quotas can hinder utilization while `providing a predictable upper bounds on CPU time.`
--cpuset-cpus Limit the specific CPUs or cores a container can use. A comma-separated list or hyphen-separated range of CPUs a container can use, if you have more than one CPU. The first CPU is numbered 0. A valid value might be 0-3 (to use the first, second, third, and fourth CPU) or 1,3 (to use the second and fourth CPU).
--cpus=<value> Specify how much of the available CPU resources a container can use. For instance, if the host machine has two CPUs and you set --cpus="1.5", the container is guaranteed at most one and a half of the CPUs. This is the equivalent of setting --cpu-period="100000" and --cpu-quota="150000". shortway for cpu-period and cpu-quota
# In order to set cpu_quota correctly, you need to know how many cpu can be used by container, then set cpu_quota and cpu_period correctly. $ docker run -it --rm --cpuset-cpus 0-1 --cpus=1.5 ubuntu /bin/bash # OR $ docker run -it --rm --cpuset-cpus 0-1 --cpu-period=100000 --cpu-quota=150000 ubuntu /bin/bash
# only alow cpu0 and cpu1 to run this container $ docker run -it --rm --cpuset-cpus 0-1 ubuntu /bin/bash
expose Nvidia GPU to a container
1 2 3 4
# must install nvidia runtime first on host $ apt-get install nvidia-container-runtime # run from image ubuntu with nvidia-smi command as entrypoint $ docker run -it --rm --gpus device=GPU-3a23c669-1f69-c64e-cf85-44e9b07e7a2a ubuntu nvidia-smi
limit memory used by a container
cgroups to accomplish this
1 2 3 4 5 6 7 8 9 10
# options followed by a suffix of b, k, m, g, to indicate bytes, kilobytes, megabytes, or gigabytes -m or --memory= The maximum amount of memory the container can use. If you set this option, the minimum allowed value is 6m (6 megabyte).
# !!!it's just cgroup limitation, it does not reserved such memory for a container!!! $ docker run -it --rm -m 100M ubuntu /bin/bash $ docker run -it --rm -m 10G ubuntu /bin/bash
# update docker memory or cpu when it's running $docker update --memory 123289600 --memory-swap 123289600 ubuntu $docker update --cpus 1 ubuntu
check stats for container
1 2 3 4
# it will show container memory limit and how much it's used now $ docker stats $container_id CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O 5ed 12.44% 104.8 MB / 104.9 MB 99.92% 4.861 kB / 648 B 9.138 GB / 10.16 GB
run container automatically when dockerd start
1 2
# run your container with --restart=always $ docker run --restart=always -it ubuntu /bin/bash
expose a port from container
iptable rule to accomplish this
1 2 3 4
# in this way you when you access host port, it redirect to container port # host port: 6000 # container port: 22 $ docker run --restart=always -p 6000:22 -it ubuntu /bin/bash
start a container with given ip and hostname
1 2 3 4 5
# check subnet of bridge that container uses $ docker network ls $ docker network inspect bridge # set with hostname and ip $ docker run --restart=always -p 6000:22 --hostname test --ip 172.16.0.2 -it ubuntu /bin/bash
1. Modify /lib/systemd/system/docker.service to tell docker to use our own directory instead of default /var/lib/docker. In this example, I am using /p/var/lib/docker Apply below patch.
$ diff -uP -N /lib/systemd/system/docker.service.orig /lib/systemd/system/docker.service --- /lib/systemd/system/docker.service.orig 2018-12-05 21:24:20.544852391 -0800 +++ /lib/systemd/system/docker.service 2018-12-05 21:25:57.909455275 -0800 @@ -10,7 +10,7 @@ # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker -ExecStart=/usr/bin/dockerd -H unix:// +ExecStart=/usr/bin/dockerd -g /p/var/lib/docker -H unix:// ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2
2. Stop docker service $ systemctl stop docker 3. Do daemon-reload as we changed docker.service file $ systemctl daemon-reload 4. rsync existing docker data to our new location $ rsync -aqxP /var/lib/docker/ /p/var/lib/docker/ 5. Start docker service $ systemctl start docker
inside docker permission denied to run some command
1 2
# run docker with --privileged=true and must run with /usr/sbin/init $ docker run --restart=always -p 6000:22 --hostname test --ip 172.16.0.2 --privileged=true -it ubuntu /usr/sbin/init
As there is no namespace for memory, cpu, hence when you run free, cat /proc/cpuinfo, it shows information about the host!!!, it’s not one that container can use, what you see is not true, it’s not what container can use, it’s also true, it’s true for host.
but df show mount information while container has its own mnt namespace, so it sees devices of its own not host.
# check disk size that can be used by container, please check https://cyun.tech/docker-persist-data # all limits you have with containers using something like the overlay2 filesystem are inherited from the parent filesystem. # Since docker does everything under /var/lib/docker, your available disk space on that filesystem are the same as the limits you'll see inside of a container.
# check cpu and memory can be used for this container, run docker inspec $ docker inspect centos | grep Cpuset 89: "CpusetCpus": "0-1", 90: "CpusetMems": "",
By default, docker sees the same disk size(while RW layer sits) as host and can use it as well, but in some case we want to limit storage size used by container, --storage-opt size=xxx can do this, but limit to some storage driver devicemapper, btrfs, overlay2, windowsfilter and zfs For the devicemapper, btrfs, windowsfilter and zfs drivers, user cannot pass a size less than the Default BaseFS Size. For the overlay2 storage driver, the size option is only available if the backing fs is xfs and mounted with the pquota mount option. Under these conditions, user can pass any size less than the backing fs size.
XFS supports disk quotasby user, by group, and by project. Project disk quotas allow you to limit the amount of disk space on individual directory hierarchies. You can configure both hard and soft limits on the number of disk blocks (or disk space), and the number of inodes, which limit the number of files a user can create. Quotas do not apply to the root user.
You must first enable quotas for users, groups, and/or projects by using a mount option when mounting for the XFS file system. After enabling quotas, use the xfs_quota command to set limits to view quota information.
# enable disk project quotas on /var $ cat /etc/fstab ... /dev/mapper/centos_dev-var /var xfs rw,pquota 0 0 ...
# report the overall quota state information: $ xfs_quota -x -c state User quota state on /var (/dev/mapper/centos_dev-var) Accounting: OFF Enforcement: OFF Inode: #20328 (5 blocks, 5 extents) Group quota state on /var (/dev/mapper/centos_dev-var) Accounting: OFF Enforcement: OFF Inode: #227342 (1 blocks, 1 extents) Project quota state on /var (/dev/mapper/centos_dev-var) Accounting: ON Enforcement: ON Inode: #227342 (1 blocks, 1 extents) Blocks grace time: [7 days] Inodes grace time: [7 days] Realtime Blocks grace time: [7 days]
# show quota $ xfs_quota -x -c 'report -h' /var Project quota on /var (/dev/mapper/centos_dev-var) Blocks Project ID Used Soft Hard Warn/Grace ---------- --------------------------------- #0 2.2G 0 0 00 [------]
$ docker run -it --storage-opt size=10G fedora /bin/bash
$ xfs_quota -x -c 'report -h' /var Project quota on /var (/dev/mapper/centos_dev-var) Blocks Project ID Used Soft Hard Warn/Grace ---------- --------------------------------- #0 2.2G 0 0 00 [------] #2 8K 10G 10G 00 [------] #3 8K 10G 10G 00 [------]
# what quota does # initialize project with ID: 100 $ mkdir -p /data/volumes/xfs32m/5m $ xfs_quota -x -c 'project -s -p /data/volumes/xfs32m/5m 100' /data/volumes/xfs32m
# set a 5M quota on project, id=100 $ xfs_quota -x -c 'limit -p bsoft=5m bhard=5m 100' /data/volumes/xfs32m
# Pid we see from host, inside container, it's Pid 1!!! $docker inspect --format {{.State.Pid}} $container_id $docker inspect --format {{.State.Pid}} $container_name 16755
enter docker namespace without docker exec
The first process of container, its parent is containerd-shim-runc-v2, but if you run docker exec -it $docker bash, its parent is containerd-shim-runc-v2 as well.
/proc/pid/ns/mnt the mount namespace /proc/pid/ns/uts the UTS namespace /proc/pid/ns/ipc the IPC namespace /proc/pid/ns/net the network namespace /proc/pid/ns/pid the PID namespace /proc/pid/ns/user the user namespace /proc/pid/root the root directory
# mount, uts, ipc, net, pid, user namespace $nsenter -t 16755 --mount --net --uts --ipc --pid --user --root /bin/bash # same as $dockerexec -it $container_name /bin/bash
dockerd setting
keep container running when docker service restarting
# build-dev is container's name $docker inspect --format="{{.Id}}" build-dev 6b667579c2a963767bb97b5ed35e4d56ca9a428da8b9fd067fac14b25712048a
# stop docker service, otherwise, edit config.v2.json will be lost as docker will rewrite this file if edited outside!!! $service stop docker $vim /var/lib/docker/containers/6b667579c2a963767bb97b5ed35e4d56ca9a428da8b9fd067fac14b25712048a/config.v2.json $service start docker
show disk usage of container
1 2 3 4 5
# NOTE: this list virtual size and real size # virtual size = real size + image size # real size is only RW layer size
$ docker ps --size
updating conf of existing container
how to set a pre-existing docker container’s restart policy
1 2
# docker update has very limit options(for mem, cpu, restart only) $ docker update --restart=always jason-dev
