Create docker image

There are several ways to create an image, you can create it by

Persisting Data

Overview

Sometimes you want to share data between container or you want to persist data even docker is deleted, there there two ways for it.

• bind mount
• volume

Docker CLI

Before going into docker cli, let’s see how docker client, daemon and registry works, here is an overview of these parts and how they work with each other.

Image and Container Concept

The major difference between a container(runtime) and an image(static) is the top writable layer. All writes to the container that add new or modify existing data are stored in this writable layer. When the container is deleted, the writable layer is also deleted. The underlying image layers remain unchanged.

Because each container has its own writable container layer, and all changes are stored in this container layer, multiple containers can share access to the same underlying image and yet have their own data state.

Storage driver

Overview

Docker supports many storage drivers like below, but if you work on ubuntu
use AUFS if linux kernel version is less than 4.0 otherwise use overlay(overlay2) as it has better performance.

AUFS

Introduction

Aufs is a stackable unification filesystem such as Union fs which unifies several directories and provides a merged single directory.

• overlay2 is the preferred storage driver, for all currently supported Linux distributions, and requires no extra configuration.
• aufs was the preferred storage driver for Docker 18.06 and older, when running on Ubuntu 14.04 on kernel 3.13 which had no support for overlay2

AUFS is not merged into linux main branch, only ubuntu/debian support it!!!

Chroot

Introduction

Every process/command in Linux/Unix like systems has a current working directory called root directory. chroot changes the root directory for current running process as well as its children.

it creates a virtualized environment in a Unix(linux) operating system, separating it from the main operating system’s directory structure. This process essentially generates a confined space with its own root directory, to run software programs. This virtual environment runs separately from the main operating system's root directory.

Any software program run in this environment can only access files within its own directory tree. It cannot access files outside of that directory tree. This confined virtual environment is often called a "chroot jail".

• debug application
  • show dynamic library used by an application
  • create and use a static library
  • create and use a dynamic library
  • show symbols in dynamics and static library
  • check process thread
  • do everything in memory
  • debug dynamic library info when runs a program
  • useful commands in binutils
  • process affinity
  • IRQ affinity setting
  • trace process by strace
  • show shared memory on the system
  • only show mem/cpu usage of a particular pid
  • show processes who opened a file or dir
  • force a running process to generate a core dump
  • signal
  • show environment variables of a running process.
  • check process name when knows pid
  • memory leak/ overflow tool
  • shutdown() vs close()
  • RESET TCP from application
  • install debuginfo of kernel or application on centos
  • show stack for give process(thread)
  • pkg-config
  • ld finding library
  • get hardware meta
  • lspci output
  • coredump without symbol table
  • network related
    • enable forwarding in kernel
    • IP address and Route
    • Deep into route table
    • prevent route missing after reboot
    • show arp entry
    • save tcpdump captured packet to file
    • packet shows as tcp segment of a reassembled pdu in wireshark
    • get two sides of established STREAM socket
    • show all sockets
    • check/set mtu or MAC
    • socket bind to no-local address
    • source port selection
    • create vlan interface and check its real dev
    • check tcp fastopen cookie
    • How to know if a network interface is tap, tun, bridge or physical
    • bind irq(s) of ethx to specific cpu
    • check rss of ethx
    • get bus info for a network device like eth0
    • create binding interface through sysfs
    • icmp ping is ok but tcp connect fails

debug application

Here are list of frequent commands that are used in daily life and some tips that may be useful to debug application

Performance knowledge

Memory Usage Metric

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

Show process memory usage by top
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

    1 root      20   0   38116   6136   3984 S   0.0  0.0   0:05.71 systemd


    VIRT(VSS): The total amount of virtual memory used by the task.
               It includes all code, data and shared libraries plus pages that have been swapped out(it not real physical memory current used)

    RES(RSS):  The non-swapped physical memory a task has used, CODE+DATA

    SHR:       The amount of shared memory used by a task.
               It simply reflects memory that could be potentially shared with other processes.

    %MEM:      Memory usage (RES)

Show process memory usage by smem (metric used by smem and ps)

    Swap: Swap size used by each process

    VSS(virtual set size)
         VSS (reported as VSZ from ps)is the total accessible address space of a process
         (all allocated virtual addresses like malloc, stack, map(shared library))
         This size also includes memory that may not be resident in RAM like mallocs that have been allocated but not written to.
         VSS is of very little use for determining real memory usage of a process.

    RSS(resident set size)
        RSS is the total memory actually held in RAM for a process. RSS can be misleading,
        because it reports the total all of the shared libraries that the process uses,
        even though a shared library is only loaded into memory once regardless of how many processes use it.
        RSS is not an accurate representation of the memory usage for a single process.

    PSS(Proportional set size)
        PSS differs from RSS in that it reports the proportional size of its shared libraries,
        i.e.if three processes all use a shared library that has 30 pages,
        that library will only contribute 10 pages to the PSS that is reported for each of the three processes.
        PSS is a very useful number because when the PSS for all processes in the system are summed together,
        that is a good representation for the total memory usage in the system.
        When a process is killed, the shared libraries that contributed to its PSS will be proportionally distributed to
        the PSS totals for the remaining processes still using that library.
        In this way PSS can be slightly misleading, because when a process is killed, PSS does not accurately represent the memory returned to the overall system.

    USS(unique set size)
        USS is the total private memory for a process, i.e. that memory that is completely unique to that process.
        USS is an extremely useful number because it indicates the true incremental cost of running a particular process.
        When a process is killed, the USS is the total memory that is actually returned to the system.
        USS is the best number to watch when initially suspicious of memory leaks in a process.

    For example, there are two processes share a library which takes 2M physical memory

                VSS     RSS     PSS     USS
    process A   20M     18M     17M     16M
    process B   20M     19M     18M     17M
    (RSS=USS+shared_library_memory, PSS=USS+shared_library_memory/shared_process_count)

load average

The load average is the average system load on a Linux server for a defined period of time. In other words, it is the CPU demand of a server that includes sum of the running and the waiting threads. on linux, it not only tracks running tasks, but also tasks in uninterruptible sleep (usually waiting for IO)

Measuring the load average is critical to understanding how your servers are performing; if overloaded, you need to kill or optimize the processes consuming high amounts of resources, or provide more resources to balance the workload.

For simple, let's assume a server with a single processor, if the load is less than 1, that means on average, every process that needed the CPU could use it immediately without being blocked. Conversely, if the load is greater than 1, that means on average, there were processes ready to run, but could not due to CPUs being unavailable.

For a single processor, ideal load average is 1.00, and anything above that is an action call to troubleshoot? Well, although it’s a safe bet, a more proactive approach is leaving some extra headroom to manage unexpected loads, many people tend to aim for a load number of about 0.7 to cater for the spikes

overloaded or not depends on how may cpus(not core) you have

You probably have a system with multiple CPUs. The load average numbers work a bit differently on such a system. For example, if you have a load average of 2 on a single-CPU system, this means your system was overloaded by 100 percent — the entire period of time, one process was using the CPU while one other process was waiting. On a system with two CPUs, this would be complete usage — two different processes were using two different CPUs the entire time. On a system with four CPUs, this would be half usage — two processes were using two CPUs, while two CPUs were sitting idle.

How to use Hexo tag and Next tag

hexo search setting