| |
Performance Diagnosis of 60,000 milliseconds for Linux
When you log on to solve a performance problem to a Linux server: In the first minute you should check what?
In Netflix, we have a huge cloud EC2 Linux, as well as a large number of performance analysis and diagnostic tools to monitor their performance. Including Atlas for cloud monitoring, as well as demand for case study of Vector. While these tools can help us solve most problems, but sometimes we still need to log in to a server instance, and run some standard Linux performance tools.
In this article, Netflix Performance Engineering team will explain to you pre 60 seconds to do a performance analysis of the best in the command line, using the standard Linux tools You should be able to get.
Before sixty seconds: Overview
By running the following ten commands, you will be able to within sixty seconds in a rough understanding of the process of running the system and resource usage. By viewing these error messages and resources saturation output of a command (they are easy to understand), you can be the next resource optimization. Saturation refers to a load beyond the limits of its resources can handle. Once saturation occurs, it will usually be exposed on the length of the request queue or waiting time.
uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP, ETCP 1
top
Some commands need to pre-install sysstat package. These commands are displayed information can help you implement USE method (for locating performance bottlenecks method), such as inspection of various resources (such as CPU, memory, disk, etc.) usage, saturation, and error messages. Also in the process of locating the problem, you can use these commands to exclude the possibility of some cause problems, help you narrow the scope of the inspection, the next step is to check the specified direction.
The following sections will execute these commands on a production environment as an example, a brief introduction of these commands. To learn more about using these tools, see their man pages.
1. uptime
$ Uptime
23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02
This is a method used to quickly view the system load average, which indicates the number of tasks to run (process) in the system. In the Linux system, these numbers contain the process needs to run in the CPU and is waiting for I / O (usually a disk I / O) process. It is merely a rough display of system load, it can look a little. You also need other tools to learn more about the specific situation.
These three figures show that for a minute, the total load within five minutes and fifteen minutes the system according to the average index compression ratio results obtained. We can see how the load on the system is changing with time. For example, you check a problem, and then saw one minute far less than the value corresponding to the value of 15 minutes, it may indicate the problem has passed, you can not be observed in time.
In the above example, the system load increases over time, because of the recent one-minute load value exceeds 30, while the average load for 15 minutes only 19. Such a significant gap contains a lot of meaning, such as CPU load. To further confirm the case, will have to run vmstat or mpstat command, these two commands, refer back to the third and fourth chapters.
2. dmesg | tail
$ Dmesg | tail [1880957.563150] perl invoked oom-killer: gfp_mask = 0x280da, order = 0, oom_score_adj = 0
[...]
[1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child
[1880957.563408] Killed process 18694 (perl) total-vm: 1972392kB, anon-rss: 1953348kB, file-rss: 0kB
[2320864.954447] TCP:. Possible SYN flooding on port 7001. Dropping request Check SNMP counters.
This command displays the most recent 10 system messages, if they exist. Finding errors can cause performance problems. The above example contains oom-killer, as well as TCP discards a request.
Do not miss this step! dmesg command is always worth a try.
3. vmstat 1
$ Vmstat 1procs --------- memory ---------- --- swap-- ----- io ---- -system-- ------ cpu -----
r b swpd free buff cache si so bi bo in cs us sy id wa st
34 0,020,088,979,273,708 5,918,280,005,610,961,300
32 0,020,088,992,073,708 five hundred and ninety-one thousand eight hundred and sixty 0,005,921,328,442,829,811 0 0
0,020,089,011,273,708 five hundred and ninety-one thousand eight hundred and sixty 32 0 0 0,095,012,154,991,000
32 0,020,088,956,873,712 five hundred ninety-one thousand eight hundred fifty-six 0,004,811,900,245,999,000 0
32 0,020,089,020,873,712 591.86 thousand 0,000,158,984,840,981,100
^ C
vmstat (8) is referred to as virtual memory statistics, which is a commonly used tool (created decades ago to BSD). A statistical summary of key print server in each row.
vmstat command specifies a parameter 1 run, to print summary statistics every second. Those columns of the first row (this version of vmstat) output, the average power is explicit since, instead of before the second value. Now, we skip the first row, unless you want to learn and remember each column.
Check these columns:
r: CPU is running and waiting for the number of running processes. Which provides a better signal than the average load to determine whether the CPU saturation, because it does not include I / O. He explained: "r" is greater than the number of CPU would represent saturated.
free: in kb units explicitly free memory. If the number of digits a lot, that you have enough free memory. "Free -m" command, the following is the seventh order, which may be a better description of the state of free memory.
si, so: Swap-ins and swap-outs. If they are not zero, the underrepresentation of your memory.
us, sy, id, wa, st: these are the average of all of the CPU CPU decomposition time. They are user time (user), the system time (the kernel) (system), idle (idle), waiting for I / O (wait), and the elapsed time (stolen) (by other visitors, or use Xen, visitors own independent domain driven).
CPU CPU decomposition time to confirm whether the busy state will increase the system time via the user time. Waiting for I / O has been the same time indicates a disk bottleneck; this is the CPU idle, because the task is blocked waiting for the pending disk I / O on. You can wait for the I / O is another form as the CPU is idle, which gives a clue why the CPU idle.
For I / O processing, the system time is very important. A more than 20% of the average system time can be worth further discussion: Perhaps the kernel when processing I / O efficiency is too low.
In the above example, CPU time is spent almost entirely user-level, indicates that the application taking up too much CPU time. The average CPU utilization rate is also 90%. This is not necessarily a problem; check the "r" saturation column.
4. mpstat -P ALL 1
$ Mpstat -P ALL 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
07:38:49 PM CPU% usr% nice% sys% iowait% irq% soft% steal% guest% gnice% idle
07:38:50 PM all 98.47 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.78
07:38:50 PM 0 96.04 0.00 2.97 0.00 0.00 0.00 0.00 0.00 0.00 0.99
07:38:50 PM 1 97.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
07:38:50 PM 2 98.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00
07:38:50 PM 3 96.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.03
[...]
This command prints each CPU CPU decomposition time, which can be used for a disproportionate use of checks. A separate CPU represents the very busy running a single-threaded application.
5. pidstat 1
$ Pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
07:41:02 PM UID PID% usr% system% guest% CPU CPU Command
07:41:03 PM 0 9 0.00 0.94 0.00 0.94 1 rcuos / 0
07:41:03 PM 0 4214 5.66 5.66 0.00 11.32 15 mesos-slave
07:41:03 PM 0 4354 0.94 0.94 0.00 1.89 8 java
07:41:03 PM 0 6521 1596.23 1.89 0.00 1598.11 27 java
07:41:03 PM 0 6564 1571.70 7.55 0.00 1579.25 28 java
07:41:03 PM 60004 60154 0.94 4.72 0.00 5.66 9 pidstat
07:41:03 PM UID PID% usr% system% guest% CPU CPU Command
07:41:04 PM 0 4214 6.00 2.00 0.00 8.00 15 mesos-slave
07:41:04 PM 0 6521 1590.00 1.00 0.00 1591.00 27 java
07:41:04 PM 0 6564 1573.00 10.00 0.00 1583.00 28 java
07:41:04 PM 108 6718 1.00 0.00 0.00 1.00 0 snmp-pass
07:41:04 PM 60004 60154 1.00 4.00 0.00 5.00 9 pidstat
^ C
pidstat command a bit like a statistical summary of top command for each process, but cyclic print summary statistics instead of a rolling top scraper. Which can be used for real-time viewing, but also you can see things (copy and paste) into your research records.
The above example shows two Java processes are consuming CPU. % CPU This column is the total of all CPU; 1591% indicates that the Java process consumes nearly 16 CPU.
6. iostat -xz 1
$ Iostat -xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
avg-cpu:% user% nice% system% iowait% steal% idle
73.96 22.21 0.00 3.73 0.03 0.06
Device: rrqm / s wrqm / s r / s w / s rkB / s wkB / s avgrq-sz avgqu-sz await r_await w_await svctm% util
xvda 0.00 0.23 0.21 0.18 4.52 2.08 34.37 0.00 9.98 13.80 5.42 2.44 0.09
xvdb 0.01 0.00 1.02 8.94 127.97 598.53 145.79 0.00 0.43 1.78 0.28 0.25 0.25
xvdc 0.01 0.00 1.02 8.86 127.79 595.94 146.50 0.00 0.45 1.82 0.30 0.27 0.26
dm-0 0.00 0.00 0.69 2.32 10.47 31.69 28.01 0.01 3.23 0.71 3.98 0.13 0.04
dm-1 0.00 0.00 0.00 0.94 0.01 3.78 8.00 0.33 345.84 0.04 346.81 0.01 0.00
dm-2 0.00 0.00 0.09 0.07 1.35 0.36 22.50 0.00 2.55 0.23 5.62 1.78 0.03
[...]
^ C
This is used to view the block device (disk) situation a great tool, either for work or load performance is. View columns:
r / s, w / s, rkB / s, wkB / s: These represent the number of times the device per second read and write times, read kb number, and writes the number of kb. These are used to describe the work load. Performance issues may simply be due to an excessive load is applied.
await: milliseconds of I / O average time consuming. This is the actual time consumed by the application, because it includes the queuing time and processing time. Greater than the expected average time may mean a saturated, device or equipment out of the question.
avgqu-sz: the average number of requests sent by the device. 1 illustrates the value is greater than the saturated (although the device can handle requests in parallel, in particular by a plurality of disks of virtual devices.)
% Util: equipment utilization. This value is a percentage showing the device in the second work in a busy state. If the value is greater than 60%, usually it indicates poor performance (it can be seen from await in), though it depends on the device itself. Usually means a value close to 100% saturated.
If the storage device is facing a lot of back-end disk logical disk devices, you may just mean that 100 percent utilization is currently processing some I / O occupied, however, the back-end disk may be far from saturated, and may be able to handle more much work.
Remember that disk I / O performance is not necessarily a poor procedural issues. Many techniques are usually asynchronous I / O, so that the application is not blocked and subject to delay (for example, read-ahead, and write buffer).
7. free -m
$ Free -m
total used free shared buffers cached
Mem: 245998 24545 221453 83 59 541
- / + Buffers / cache: 23944 222053
Swap: 0 0 0
The right of the two explicit:
buffers: buffer cache block devices for I / O's.
cached: page cache for file system.
We just do not want to check the size of close to zero, which may result in higher disk I / O (use iostat confirm), and worse performance. The example above looks pretty good, and each column has a lot of M size.
Compared to the first line, - / + memory usage buffers / cache will provide more accurate. Linux will not have access to temporary memory used as a cache, once the application when needed immediately re-assigned to it. It is used as part of the cache memory in fact, it can be considered free memory. To explain this, some even built a special website: linuxatemyram.
If you install ZFS on Linux, it will become even more confused, because its own ZFS file system cache is not the free -m. Sometimes the system has not found much free memory is available, but in fact they are nothing but stay in the memory of ZFS cache.
8. sar -n DEV 1
$ Sar -n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
12:16:48 AM IFACE rxpck / s txpck / s rxkB / s txkB / s rxcmp / s txcmp / s rxmcst / s% ifutil
12:16:49 AM eth0 18763.00 5032.00 20686.42 478.30 0.00 0.00 0.00 0.00
12:16:49 AM lo 14.00 14.00 1.36 1.36 0.00 0.00 0.00 0.00
12:16:49 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
12:16:49 AM IFACE rxpck / s txpck / s rxkB / s txkB / s rxcmp / s txcmp / s rxmcst / s% ifutil
12:16:50 AM eth0 19763.00 5101.00 21999.10 482.56 0.00 0.00 0.00 0.00
12:16:50 AM lo 20.00 20.00 3.25 3.25 0.00 0.00 0.00 0.00
12:16:50 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
^ C
This tool can be used to check the network interface throughput: rxkB / s and txkB / s, as well as whether the limit. The above example, eth0 received traffic reaches 22Mbytes / s, that is 176Mbits / sec (limit is a 1Gbit / sec)
We also offer a version% ifutil as equipment utilization (maximum receive and transmit) indicator. We can also use this tool Brendan nicstat measured value. As nicstat, sar displays this value is difficult to accurately achieved, even in this case, it did not live in normal operation (0.00).
9. sar -n TCP, ETCP 1
$ Sar -n TCP, ETCP 1Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015 _x86_64_ (32 CPU)
12:17:19 AM active / s passive / s iseg / s oseg / s
12:17:20 AM 1.00 0.00 10233.00 18846.00
12:17:19 AM atmptf / s estres / s retrans / s isegerr / s orsts / s
12:17:20 AM 0.00 0.00 0.00 0.00 0.00
12:17:20 AM active / s passive / s iseg / s oseg / s
12:17:21 AM 1.00 0.00 8359.00 6039.00
12:17:20 AM atmptf / s estres / s retrans / s isegerr / s orsts / s
12:17:21 AM 0.00 0.00 0.00 0.00 0.00
^ C
This is a summary view of some of the key indicators of TCP. These include:
active / s: Local initiate TCP connections per second (for example, by connect ()).
passive / s: remote TCP connections per second launched (for example, by accept ()).
retrans / s: TCP retransmission times per second.
The number of active and passive connections tend to describe a rough measure of the server load is very useful: the number of new connections accepted (passive), downlink connections (active). Can be understood as active connections are external, and internal connections are passive, although strictly speaking not entirely correct (for example, a localhost to localhost connections).
Retransmission is the emergence of a network and server issues a sign. It may be due to an unreliable network (for example, the public network) cause, and perhaps also may be due to server overload and packet loss. The above example shows that only every second a new TCP connection.
10. top
$ Toptop - 00:15:40 up 21:56, 1 user, load average: 31.09, 29.87, 29.92
Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie
% Cpu (s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 25190241 + total, 24921688 used, 22698073 + free, 60448 buffers
KiB Swap: 0 total, 0 used, 0 free 554208 cached Mem.
PID USER PR NI VIRT RES SHR S% CPU% MEM TIME + COMMAND
20248 root 20 0 0.227t 0.012t 18748 S 3090 5.2 29812: 58 java
4213 root 20 0 2722544 64640 44232 S 23.5 0.0 233: 35.37 mesos-slave
66128 titancl + 20 0 24344 2332 1172 R 1.0 0.0 0: 00.07 top
5235 root 20 0 38.227g 547004 49996 S 0.7 0.2 2: 02.74 java
4299 root 20 0 20.015g 2.682g 16836 S 0.3 1.1 33: 14.42 java
1 root 20 0 33620 2920 1496 S 0.0 0.0 0: 03.82 init
2 root 20 0 0 0 0 S 0.0 0.0 0: 00.02 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0: 05.35 ksoftirqd / 0
5 root 0 -20 0 0 0 S 0.0 0.0 0: 00.00 kworker / 0: 0H
6 root 20 0 0 0 0 S 0.0 0.0 0: 06.94 kworker / u256: 0
8 root 20 0 0 0 0 S 0.0 0.0 2: 38.05 rcu_sched
top command contains a lot before we have checked the index. Results can easily execute it to see compared to the previous command output is very different, indicating that the load is variable.
One drawback top is difficult to see trends in data over time change. vmstat and pidstat provide scrolling output more clearly. If you are not fast enough to suspend output (Ctrl-S to pause, Ctrl-Q to continue), some clues intermittent problem may have been lost and the screen is cleared.
Subsequent analysis
There are more commands and methods can be used for further analysis. View Brendan Velocity 2015 conference in Linux performance tool tutorial, which contains more than 40 commands, covering the observability benchmarking, tuning, static tuning, analysis, tracking and so on. |
|
|
|