Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Linux \ Linux processes in memory and memory cgroup statistics     - Installation and Configuration OpenVPN server and client on Ubuntu 15.04 (Server)

- Spark and Hadoop comparison (Server)

- Linux landing problem (Linux)

- Configuring a Linux operating system security management services (Linux)

- Linux foundation tutorial: how to modify the host name on CentOS or RHEL 7 (Linux)

- To compiler and install MariaDB-10.0.20 under CentOS 6.6 (Database)

- DM9000 bare Driver Design (Programming)

- PL / SQL -> UTL_FILE use presentation package (Database)

- Oracle11g Trigger Debugging Record Error: PLS-00201: identifier SYS.DBMS_SYSTEM 'must be declared (Database)

- Why learn Java EE (Programming)

- IBM Data Studio to create objects using ---- double quotes / sensitive issues and the table / column renaming (Database)

- How to use the on-screen keyboard in Linux (Linux)

- A simple shell script for monitoring in Linux (Linux)

- CentOS-based Kickstart automated installation practice (Linux)

- Increase Linux system security --chattr (Linux)

- C ++ Supplements - Smart Pointers (Programming)

- Ubuntu 14.10 / 14.04 / 12.04 virtual users to install the printing software Boomaga (Linux)

- MySQL simple operation notes under Linux (Database)

- Fedora 23 How to install LAMP server (Server)

- To share some very useful Vim command (Linux)

 
         
  Linux processes in memory and memory cgroup statistics
     
  Add Date : 2017-01-08      
         
         
         
  In the Linux kernel, the process memory usage and the memory usage statistics Cgroup have some of the same and different places.

Process memory statistics

In general, the process uses memory business mainly in the following situations:

(1) anonymous user space map pages (Anonymous pages in User Mode address spaces), like calling malloc allocation of memory, and the use of MAP_ANONYMOUS mmap; when the system memory is not enough, this part of the kernel can be swapped out of memory;

(2) user space file mapping page (Mapped pages in User Mode address spaces), contains the map file and map tmpfs; former such as mmap specified file, the latter such as IPC shared memory; when the system is not enough memory, the kernel can reclaim these pages, but you may need to synchronize data files before recovery;

(3) file cache (page in page cache of disk file); occurs in the program through the normal read / write to read and write files when the system is not enough memory, the kernel can recycle these pages, but may need to synchronize data files prior to recovery ;

(4) buffer pages, belong to the page cache; such as reading block device file.

Wherein (1) and (2) the process is counted as RSS, (3) and (4) belongs to the page cache.

Several files and process memory-related statistics:

/ Proc / [pid] / stat
(23) vsize% lu
        Virtual memory size in bytes.
(24) rss% ld
        Resident Set Size: number of pages the process has
        in real memory. This is just the pages which count
        toward text, data, or stack space. This does not
        include pages which have not been demand-loaded in,
        or which are swapped out.
RSS calculation:

Corresponding to the top of RSS column, do_task_stat

#define get_mm_rss (mm) \
    (Get_mm_counter (mm, file_rss) + get_mm_counter (mm, anon_rss))
That is RSS = file_rss + anon_rss

/ Proc / [pid] / statm
Provides information about memory usage, measured in pages.
The columns are:

  size (1) total program size
             (Same as VmSize in / proc / [pid] / status)
  resident (2) resident set size
             (Same as VmRSS in / proc / [pid] / status)
  share (3) shared pages (i.e., backed by a file)
  text (4) text (code)
  lib (5) library (unused in Linux 2.6)
  data (6) data + stack
  dt (7) dirty pages (unused in Linux 2.6)
See function proc_pid_statm

int task_statm (struct mm_struct * mm, int * shared, int * text,
           int * data, int * resident)
{
    * Shared = get_mm_counter (mm, file_rss);
    * Text = (PAGE_ALIGN (mm-> end_code) - (mm-> start_code & PAGE_MASK))
                                >> PAGE_SHIFT;
    * Data = mm-> total_vm - mm-> shared_vm;
    * Resident = * shared + get_mm_counter (mm, anon_rss);
    return mm-> total_vm;
}
top of SHR = file_rss. In fact, the process uses shared memory, but also to count file_rss because shared memory based tmpfs.

anon_rss and file_rss calculation

static int __do_fault (struct mm_struct * mm, struct vm_area_struct * vma,
        unsigned long address, pmd_t * pmd,
        pgoff_t pgoff, unsigned int flags, pte_t orig_pte)
{
    if (flags & FAULT_FLAG_WRITE) {
        if (! (vma-> vm_flags & VM_SHARED)) {
            anon = 1; /// anon page
...
        if (anon) {
            inc_mm_counter (mm, anon_rss);
            page_add_new_anon_rmap (page, vma, address);
        } Else {
            inc_mm_counter (mm, file_rss);
            page_add_file_rmap (page);
cgroup memory statistics

stat file

memory.stat file includes following statistics

# Per-memory cgroup local status
cache - # of bytes of page cache memory.
rss - # of bytes of anonymous and swap cache memory (includes
        transparent hugepages).
rss_huge - # of bytes of anonymous transparent hugepages.
mapped_file - # of bytes of mapped file (includes tmpfs / shmem)
pgpgin - # of charging events to the memory cgroup The charging.
        event happens each time a page is accounted as either mapped
        anon page (RSS) or cache page (Page Cache) to the cgroup.
pgpgout - # of uncharging events to the memory cgroup The uncharging.
        event happens each time a page is unaccounted from the cgroup.
swap - # of bytes of swap usage
dirty - # of bytes that are waiting to get written back to the disk.
writeback - # of bytes of file / anon cache that are queued for syncing to
        disk.
inactive_anon - # of bytes of anonymous and swap cache memory on inactive
        LRU list.
active_anon - # of bytes of anonymous and swap cache memory on active
        LRU list.
inactive_file - # of bytes of file-backed memory on inactive LRU list.
active_file - # of bytes of file-backed memory on active LRU list.
unevictable - # of bytes of memory that can not be reclaimed (mlocked etc).
Related code

static void
mem_cgroup_get_local_stat (struct mem_cgroup * mem, struct mcs_total_stat * s)
{
    s64 val;

    / * Per cpu stat * /
    val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_CACHE);
    s-> stat [MCS_CACHE] + = val * PAGE_SIZE;
    val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_RSS);
    s-> stat [MCS_RSS] + = val * PAGE_SIZE;
    val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_FILE_MAPPED);
    s-> stat [MCS_FILE_MAPPED] + = val * PAGE_SIZE;
    val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_PGPGIN_COUNT);
    s-> stat [MCS_PGPGIN] + = val;
    val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_PGPGOUT_COUNT);
    s-> stat [MCS_PGPGOUT] + = val;
    if (do_swap_account) {
        val = mem_cgroup_read_stat (& mem-> stat, MEM_CGROUP_STAT_SWAPOUT);
        s-> stat [MCS_SWAP] + = val * PAGE_SIZE;
    }

    / * Per zone stat * /
    val = mem_cgroup_get_local_zonestat (mem, LRU_INACTIVE_ANON);
    s-> stat [MCS_INACTIVE_ANON] + = val * PAGE_SIZE;
    val = mem_cgroup_get_local_zonestat (mem, LRU_ACTIVE_ANON);
    s-> stat [MCS_ACTIVE_ANON] + = val * PAGE_SIZE;
    val = mem_cgroup_get_local_zonestat (mem, LRU_INACTIVE_FILE);
    s-> stat [MCS_INACTIVE_FILE] + = val * PAGE_SIZE;
    val = mem_cgroup_get_local_zonestat (mem, LRU_ACTIVE_FILE);
    s-> stat [MCS_ACTIVE_FILE] + = val * PAGE_SIZE;
    val = mem_cgroup_get_local_zonestat (mem, LRU_UNEVICTABLE);
    s-> stat [MCS_UNEVICTABLE] + = val * PAGE_SIZE;
}

data structure

struct mem_cgroup {
...
    / *
     * Statistics. This must be placed at the end of memcg.
     * /
    struct mem_cgroup_stat stat; /// Statistics
};

/ * Memory cgroup statistics
 * Statistics for memory cgroup.
 * /
enum mem_cgroup_stat_index {
    / *
     * For MEM_CONTAINER_TYPE_ALL, usage = pagecache + rss.
     * /
    MEM_CGROUP_STAT_CACHE, / * # of pages charged as cache * /
    MEM_CGROUP_STAT_RSS, / * # of pages charged as anon rss * /
    MEM_CGROUP_STAT_FILE_MAPPED, / * # of pages charged as file rss * /
    MEM_CGROUP_STAT_PGPGIN_COUNT, / * # of pages paged in * /
    MEM_CGROUP_STAT_PGPGOUT_COUNT, / * # of pages paged out * /
    MEM_CGROUP_STAT_EVENTS, / * sum of pagein + pageout for internal use * /
    MEM_CGROUP_STAT_SWAPOUT, / * # of pages, swapped out * /

    MEM_CGROUP_STAT_NSTATS,
};

struct mem_cgroup_stat_cpu {
    s64 count [MEM_CGROUP_STAT_NSTATS];
} ____cacheline_aligned_in_smp;

struct mem_cgroup_stat {
    struct mem_cgroup_stat_cpu cpustat [0];
};
rss and cache

cache - # of bytes of page cache memory rss -. # of bytes of anonymous and swap cache memory (includes transparent hugepages).

static void mem_cgroup_charge_statistics (struct mem_cgroup * mem,
                     struct page_cgroup * pc,
                     long size)
{
...
    cpustat = & stat-> cpustat [cpu];
    if (PageCgroupCache (pc))
        __mem_cgroup_stat_add_safe (cpustat,
            MEM_CGROUP_STAT_CACHE, numpages);
    else
        __mem_cgroup_stat_add_safe (cpustat, MEM_CGROUP_STAT_RSS,
            numpages);

static void __mem_cgroup_commit_charge (struct mem_cgroup * mem,
                       struct page_cgroup * pc,
                       enum charge_type ctype,
                       int page_size)
{

    switch (ctype) {
    case MEM_CGROUP_CHARGE_TYPE_CACHE:
    case MEM_CGROUP_CHARGE_TYPE_SHMEM: // file cache + shm
        SetPageCgroupCache (pc);
        SetPageCgroupUsed (pc);
        break;
    case MEM_CGROUP_CHARGE_TYPE_MAPPED:
        ClearPageCgroupCache (pc);
        SetPageCgroupUsed (pc);
        break;
    default:
        break;
    }
    /// Updated statistics
    mem_cgroup_charge_statistics (mem, pc, page_size);


int mem_cgroup_cache_charge (struct page * page, struct mm_struct * mm,
                gfp_t gfp_mask)
{
...
    if (page_is_file_cache (page))
        return mem_cgroup_charge_common (page, mm, gfp_mask,
                MEM_CGROUP_CHARGE_TYPE_CACHE, NULL); /// file cache

    / * Shmem * /
    if (PageSwapCache (page)) {
        ret = mem_cgroup_try_charge_swapin (mm, page, gfp_mask, & mem);
        if (! ret)
            __mem_cgroup_commit_charge_swapin (page, mem,
                    MEM_CGROUP_CHARGE_TYPE_SHMEM);
    } Else
        ret = mem_cgroup_charge_common (page, mm, gfp_mask, /// shm memory
                    MEM_CGROUP_CHARGE_TYPE_SHMEM, mem);
You can see, cache contains shared memory and file cache

mapped_file

mapped_file - # of bytes of mapped file (includes tmpfs / shmem)

void mem_cgroup_update_file_mapped (struct page * page, int val)
{
... __mem_cgroup_stat_add_safe (Cpustat, MEM_CGROUP_STAT_FILE_MAPPED, val);
__do_fault -> page_add_file_rmap -> mem_cgroup_update_file_mapped.

inactive_anon

inactive_anon - # of bytes of anonymous and swap cache memory on inactive LRU list.

static int shmem_getpage_gfp (struct inode * inode, pgoff_t index,
    struct page ** pagep, enum sgp_type sgp, gfp_t gfp, int * fault_type)
{
...
        lru_cache_add_anon (page);

/ **
 * Lru_cache_add: add a page to the page lists
 * @page: The page to add
 * /
static inline void lru_cache_add_anon (struct page * page)
{
    __lru_cache_add (page, LRU_INACTIVE_ANON);
}
From here you can see, the shared memory will be increased INACTIVE_ANON.

inactive_file

inactive_file -. # of bytes of file-backed memory on inactive LRU list files use page cache (does not include shared memory)

static inline void lru_cache_add_file (struct page * page)
{
    __lru_cache_add (page, LRU_INACTIVE_FILE);
}
add_to_page_cache_lru -> lru_cache_add_file.

The sample program

#include < stdio.h >
#include < string.h >
#include < stdlib.h >
#include < errno.h >
#include < unistd.h >
#include < sys / stat.h >
#include < sys / types.h >
#include < sys / ipc.h >
#include < sys / shm.h >
#define BUF_SIZE 4000000000
#define MYKEY 26

int main (int argc, char ** argv) {
    int shmid;
    char * shmptr;

    if ((shmid = shmget (MYKEY, BUF_SIZE, IPC_CREAT)) == - 1) {
        fprintf (stderr, "Create Share Memory Error0m ~ Z% s \ n \ a", strerror (errno));
        exit (1);
    }

    if ((shmptr = shmat (shmid, 0,0)) == (void *) - 1) {
        printf ( "shmat error \ n!");
        exit (1);
    }

    memset (shmptr, '\ 0', 1000000000);

    printf ( "sleep ... \ n");
    while (1)
        sleep (1);

    exit (0);
}
Before and after the execution of the program, cgroup memory.stat values:

Before execution:

# Cat memory.stat
cache 1121185792
rss 23678976
rss_huge 0
mapped_file 14118912
inactive_anon 1002643456
active_anon 23687168
inactive_file 46252032
active_file 72282112
After performing:

# Cat memory.stat
cache 2121187328
rss 23760896
rss_huge 0
mapped_file 1014124544
inactive_anon 2002608128
active_anon 23736320
inactive_file 46247936
active_file 72286208

#ipcs -m
0x0000001a 229380 root 0 4000000000 1
Cgroup can see, the shared memory computing cache, mapped_file, inactive_anon in.

summary

The difference between (1) and cgroup rss rss process of
RSS all the physical memory process used for the process (file_rss + anon_rss), namely Anonymous pages + Mapped apges (containing the shared memory). cgroup RSS is (anonymous and swap cache memory), does not include shared memory. Neither contains file cache.

(2) cgroup cache contains the file cache and shared memory.
     
         
         
         
  More:      
 
- NFS-based services and service utilization Corosync DRBD high availability cluster configuration, respectively (Server)
- To install OwnCloud 7.0.4 under Ubuntu (Linux)
- Linux Systemd-- To start / stop / restart services in RHEL / CentOS 7 (Linux)
- C ++ How to determine the types of constants (Programming)
- SecureCRT use the configuration detailed tutorial (Linux)
- JavaScript basic types and type conversion (Programming)
- Ubuntu 14.04 Boot Repair (Linux)
- Oracle database online redo logs are several methods of recovery of deleted (Database)
- Git Rebase Tutorial: Using Git Rebase turn back the clock (Linux)
- Fedora 21 setting boot script (Linux)
- Install Unity 8 preview version of the desktop in Ubuntu (Linux)
- Android Sets the system screen brightness (Programming)
- Ubuntu 15.10 How to install TeamViewer 11 (Linux)
- Oracle study notes view (Database)
- Log in CentOS 6.5 Multi-user setting VNC (Server)
- Elaborate .NET Multithreading: Using Task (Programming)
- Java inheritance initialization problem (Programming)
- MySQL error: ERROR 1175: You are using safe update mode solution (Database)
- Linux Command - ps: a snapshot of the current process (Linux)
- Let Markdown code syntax highlighting and support Django1.6 (Linux)
     
           
     
  CopyRight 2002-2022 newfreesoft.com, All Rights Reserved.