| |
Today we received an alert message, which reads as follows:
------------------------------------
Alarm content: Free disk space is less than 15% on volume / var
------------------------------------
Alarm Level: PROBLEM
------------------------------------
Monitoring Project: Free disk space on / var (percentage): 10%
------------------------------------
Alarm Time: 2015.10.07-09: 56: 24
This alert email message has been very clear, is the space / var directory insufficient, we take a look at how it was.
First, to view the directory df -h when space remaining 9%, indicating that this is continually shrinking space.
# Df -h
Filesystem Size Used Avail Use% Mounted on
/ Dev / sda3 7.8G 908M 6.5G 13% /
/ Dev / sda6 7.8G 6.7G 746M 91% / var
/ Dev / sda5 7.8G 2.0G 5.5G 27% / usr
/ Dev / sda1 122M 12M 104M 10% / boot
tmpfs 48G 36K 48G 1% / dev / shm
/ Dev / shm 48G 36K 48G 1% / tmp
/ Dev / sda7 497G 391G 81G 83% / home
Then we found a large number of files in / var / spool / clientmqueue, consumed most of the space in here.
Casually took a look at what is in the end. I found a script to check the listener log.
# More dfs32Ct1KE012443
Start: 20140402205501
checking listener listener ... OK
checking listener listener_1525 ... OK
checking listener listener_1528 ... OK
checking listener listener_1523 ... OK
checking listener listener_1522 ... OK
End: 20140402205516
Further verification, to come up with new 5 files, view file contents as well.
clientmqueue] # ls -lrt | tail -5
-rw-rw ---- 1 Oracle smmsp 228 Oct 7 10:02 dft97221Lc010415
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:03 qft97231cW026036
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:03 dft97231cW026036
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:04 qft97241rm007778
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:04 dft97241rm007778
clientmqueue] # more dft97241rm007778
Start: 20151007100401
checking listener listener ... OK
checking listener listener_1525 ... OK
checking listener listener_1528 ... OK
checking listener listener_1523 ... OK
checking listener listener_1522 ... OK
End: 20151007100416
Description Basic can explain because checking listener script to produce a large number of log files.
Because this log file for we do not have much use, may be considered deleted, of course, will be reported directly to delete or wrong, you can slowly batch delete ls | xargs -n 10 rm
Remove the space immediately after the release. Released nearly 6G files.
# Df -h
Filesystem Size Used Avail Use% Mounted on
/ Dev / sda3 7.8G 908M 6.5G 13% /
/ Dev / sda6 7.8G 1.1G 6.4G 15% / var
/ Dev / sda5 7.8G 2.0G 5.5G 27% / usr
/ Dev / sda1 122M 12M 104M 10% / boot
tmpfs 48G 36K 48G 1% / dev / shm
/ Dev / shm 48G 36K 48G 1% / tmp
/ Dev / sda7 497G 391G 81G 83% / home
Now the problem is resolved, the problem is how we look at the matter, for the job crontab If you have set the output content that is sent in the form of mail to the corresponding user cron job, if this time will not start in sendmail these log files are generated under this path.
First crawled the latest contents of the file. You can see the high frequency of the generated file, a file almost every minute.
clientmqueue] # ll
total 64
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:08 dft97281ag005351
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:09 dft97292uQ011260
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:12 dft972C1Xg025752
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:13 dft972D11d025507
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:14 dft972E1IS008404
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:15 dft972F1Oi023669
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:16 dft972G1Xr006590
-rw-rw ---- 1 oracle smmsp 228 Oct 7 10:17 dft972H1I8022068
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:08 qft97281ag005351
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:09 qft97292uQ011260
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:12 qft972C1Xg025752
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:13 qft972D11d025507
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:14 qft972E1IS008404
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:15 qft972F1Oi023669
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:16 qft972G1Xr006590
-rw-rw ---- 1 oracle smmsp 919 Oct 7 10:17 qft972H1I8022068
See crontab -l can see that the frequency of checks script execution is still very high.
. 2-9,12-29,31-59 * * * * $ HOME / .xxxxprofile; $ HOME / dbadmin / scripts / lsnr_check.sh
For the listener checks, in fact, you do not need such frequent monitoring may be appropriate to slow down some of the frequency, according to the general machine settings or one hour 2 times inspection. For example, this setting:
. 9,39 * * * * $ HOME / .xxxxprofile; bash $ HOME / dbadmin / scripts / lsnr_check.sh
Clear the log files, log files are generated frequency is also reduced, but the problem still no cure index.
Check the logs for these, can be used as a background task, each time checks are not required to generate a large number of logs, one way is to directly shield the log, such as setting the following form.
. 9,39 * * * * $ HOME / .xxxxprofile; bash $ HOME / dbadmin / scripts / lsnr_check.sh> / dev / null 2> & 1
Such a solution to this problem came to an end, we can see a very small change after years and years of accumulation becomes an obvious problem, monitor settings too frequently but may have potential problems. |
|
|
|