Home IT Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Varnish configuration language VCL and its built-in variables     - Linux System Tutorial: Ubuntu on the desktop is disabled by default keyring to unlock tips (Linux)

- extundelete: the Linux-based open source data recovery tools (Linux)

- On the Web application attack techniques Common (Linux)

- ASM Disk Space Check (Database)

- MySQL restart process can not be taken lightly (Database)

- C ++ Const breaking rules (Programming)

- Java Cookie Comments (Programming)

- Linux can modify the maximum number of open files (Linux)

- Puppet Detailed centralized configuration management system (Server)

- How to install Git client in Ubuntu (Linux)

- Tsunami-udp accelerated network transmission (Linux)

- There is sort of a directed acyclic graph topology (Programming)

- Custom Android UI template Comments (Programming)

- Network security system (Network)

- Ubuntu system safe and caution sudo su command (Linux)

- Oracle Client Easy Connection error ORA-12154, TNS-03505 (Database)

- Vi syntax highlighting settings (Linux)

- Linux boot process (Linux)

- shell script error dirname: invalid option - b (Database)

- FreeBSD install Gnome 3 Desktop (Linux)

 
         
  Varnish configuration language VCL and its built-in variables
     
  Add Date : 2018-11-21      
         
       
         
  First, the Varnish configuration language VCL
All configurations of Varnish are configured via VCL (varnish configure language). It is a simple, domain-specific programming language that supports limited arithmetic and logic operations, allows string matching using regular expressions, allows users to use set custom variables, and supports if-statements , There are built-in functions and variables. Caching policies written in VCL are usually saved to a .vcl file, which needs to be compiled in binary format before being invoked by varnish. In fact, the entire cache strategy is composed of several specific subroutines, such as vcl_recv, vcl_fetch, etc., which are executed at different locations (or times), respectively, and without var customizing the subroutine for a location The implementation of the default definition of the code, the code is default.vcl in the comments of the code.
VCL strategy in the opening, the management process will be converted to C code, and then by the gcc compiler will be compiled into binary code C program. After the compilation is complete, the management is responsible for connecting it to the varnish instance, the child process. Just because the compilation is done outside the child process, it avoids the risk of loading the wrong format VCL. Therefore, the cost of modifying varnish is very small, it can also keep several versions of the old version is still cited configuration, but also to make the new configuration to take effect immediately. Older versions of the configuration are usually discarded when varnish restarts, and can be done using the vcl.discard command from varnishadm if manual cleanup is required.
 
Second, VCL syntax
The VCL is designed with reference to the C and Perl languages, so it is very easy to understand for people with C or Perl programming experience. The basic syntax is as follows:
(1) //, # or / * comment * / for comments
(2) sub $ name definition function
(3) does not support the cycle, with built-in variables
(4) use the return statement (return) to return control to varnish, no return value
(5) domain-specific
(6) Operators: = (assignment), == (equivalent comparison), ~ (pattern matching),! (Negate), && (logical AND), |
 
VCL functions do not accept parameters and do not return a value, therefore, it is not a true sense of the function, which also limits the data transfer within the VCL can only be hidden within the HTTP header. The VCL return statement is used to return control from the VCL state engine to Varnish instead of the default function, which is why the VCL only terminates the statement without a return value. At the same time, for each "domain", you can define one or more termination statements to tell Varnish what to do next, such as query cache or do not query the cache and so on.
 
Third, Varnish's built-in function
(1), vcl_recv function
Used to receive user requests, when the user request is successfully received after the call. Through some kind of action to decide how to handle the request.
This function has the following actions:
Pass: that the request is no longer local cache to find, and enter the pass mode, and will handle the request control to vcl_pass function
Pipe: the request is no longer local cache to find, and enter the pipe model, and the request control to vcl_pipe function. In this mode, the client does not perform any checking or operation on the client. Instead, the client and the back-end server directly establish the pipe and transmit the data directly in the pipe. In this case, the subsequent data in the keep-alive connection is also piped and does not appear in any log.
Error code [reason]: Returns the error code to the client and discards the request. Code indicates an error code, such as 404, 405, and so on. Reason Indicates an error message.
Lookup: The object that looks for the request in the cache. And based on the results of the search to control the request to vcl_hit or vcl_miss function.
 
(2), pipe function
This function is called when the pipe mode is entered. It is used to pass the client request directly to the backend server, and return the unchanged content directly to the client without changing the contents of the request and return. Until the connection is closed.
The function has the following actions:
Error code reason: Returns the error code and discards the request
Pipe:
 
(3), vcl_pass function
This function is called when the vcl_recv function executes the pass action to enter the pass mode. It is used to forward the client request directly to the backend server, which is not cached when the backend server responds to the client. Because the request is forwarded directly to the back-end server, the response data for that connection is up-to-date.
The function has the following actions:
Error code reason: Returns the error code and discards the request
Pass: that from the back-end server to obtain data, and request control to the vcl_fetch function
Restart: Restarts the entire vcl, and the request is re-examined. That is, the request rechecks the vcl_recv function. After the restart action is executed, the restart count is calculated and an error message is returned when the max_restarts maximum restart count is exceeded
 
 
(4), vcl_hit function
This function calls the function automatically if the request data is found in the cache after the lookup command is executed.
The function has the following actions:
Deliver: indicates that the data found in the cache is returned to the client and the control is passed to the vcl_deliver function
Pass: Enter the pass mode and give control to vcl_fetch. In this mode, although data is found in the cache, the data in the cache is not used, but is retrieved from the back-end server.
Error code reason: Returns the error code and discards the request
Restart: Restarts the entire vcl, and the request rechecks the vcl_recv function. After the restart action is executed, the restart count is calculated and an error message is returned when the max_restart maximum counter is exceeded.
 
(5), vcl_miss function
The function in the implementation of the lookup command, if the cache does not find the request data, then automatically call the function
The function has the following actions:
Fetch: that request data from the back-end server, and called control to vcl_fetch
Pass :: Enter the pass mode and give control to vcl_pass.
Error code reason: Indicates that the error code and information are returned and the request is discarded.
Restart: Restarts the entire vcl, and the request rechecks the vcl_recv function. After the restart action is executed, the restart count is calculated and an error message is returned when the max_restart maximum counter is exceeded.
 
 
(6), vcl_fetch function
When you want to get data from the back-end server or update the cache when the function is called, and according to a certain action to determine whether the data is cached, or directly back to the client.
The function has the following actions:
Deliver: Indicates that the resource or data obtained from the backend server is cached. And the control to the vcl_deliver function.
Hit_for_pass: Indicates that the data or resources fetched from the backend server are not cached and the control is passed to the vcl_deliver function. And subsequent requests to the object are passed directly to the vcl_pass function.
      Error code reason: Indicates that the error code and information are returned and the request is discarded.
Restart: Restarts the entire vcl, and the request rechecks the vcl_recv function. After the restart action is executed, the restart count is calculated and an error message is returned when the max_restart maximum counter is exceeded.
 
 
(7), vcl_deliver function
This function is called by returning the requested data to the client.
The function has the following actions:
Deliver: The request data is returned directly to the client
Error code reason
Restart
 
(8), vcl_timeout function
This function is called before the content of the cache expires. The function performs the following actions:
Discard: Clears the cache
Fetch: that from the back-end server to obtain resources to update
 
(9), vcl_discard function
This function is called after the cached content has expired, and has the following action:
Keep: Indicates that the expired cache object remains in the cache.
Discard: Clears from the cache

Fourth, Varnish work flow chart

Varnish processing request flow can be roughly divided into the following steps:
(1), first when the request arrived, to accept vcl_recv function of the inspection, in this function, you can perform pass, pipe, lookup, error operation.
(2) If the vcl_recv function in the implementation of the lookup operation will be in the cache to see whether the cache object. If there is, that cache hit, then the vcl_hit function to accept the check. In this function can be directly returned to the client cache objects, namely the implementation of deliver operation. You can also use the local cache object, from the back-end server to obtain the request object, the implementation of pass operation. If the cache object is not present locally, it indicates that the cache is missing and will be checked by the vcl_miss function. This function gets the latest resources from the back-end server, which will perform a pass or fetch operation.
(3) If vcl_recv function in the implementation of pass operation, said no longer local cache lookup, directly from the back-end server to obtain, will accept vcl_pass function check.
(4) If the vcl_recv function in the implementation of pipe operation, said no longer local cache to find, and the client and back-end server to establish a pipeline, follow-up client keep-alive connection request data transmission will be carried out in the pipeline, and The request data accepts the check of the vcl_pipe function. Until the data transfer is complete close the pipe connection.
(5) When you need to obtain resources or data from the back-end server, you need to call the vcl_fetch function, and according to a certain condition to determine whether the data or resources are cached locally, the implementation of deliver operation; or directly to the resources returned to the client, Do not cache the local, that is, the implementation of hit_for_pass operation
Finally, the data returned from the local cache, or from the back-end origin server, is returned to the client by calling the vcl_deliver function.

Fifth, Vcl's built-in variables
Vcl has a number of built-in variables, which are used in different stages.
When varnish accepts user requests, you can use built-in variables
Req.backend Specifies the back-end host
    Server.ip Specifies the back-end server ip
    Client.ip that client ip
    Req.request: Specifies the request method, such as GET, HEAD, POST and so on
    Req.url Specifies the url of the request
    Req.proto Indicates the version of the http protocol used by the client to initiate the request
    Req.http. < Header>: Indicates the http header information in the client request packet
    Req.restarts: Indicates the number of requests to restart; the default maximum value is 4
 
When varnish sends a request to the backend server, the built-in variables that can be used are:
    Beresp.request: Specifies the type of request, such as GET, HEAD
    Beresp.url Specifies the url of the request
    Beresp.proto Indicates the version of the http protocol used by the client to initiate the request
    Beresp.http.header represents the http header information in the client request message
    Beresp.ttl: Indicates the lifetime of the cache, in seconds
 
When the back-end server returns content, you can use built-in variables:
Obj.status: response status code that returns the content, such as 200,302 and so on
Obj.cacheable: Indicates whether the contents of the return cache, if http return to the status code 200,203,301,302,404,410, and there is a non-survival of 0, you can cache.
Obj.valid: whether valid http response;
Obj.response: Indicates the response status information of the returned content
Obj.ttl: Returns the cache lifetime of the content, in seconds
 
When varnish on the client response, you can use the built-in variables:
Resp.status: status code returned to the client
Resp.proto: return to the client's http protocol version
Resp.http. < Header>: Returns the header of the response message to the client
Resp.response: Return to the client's http status information
 
 
6, the management of Varnish cache content
1, to improve the cache hit rate
Effective ways to improve the cache hit rate are:
A, increase the cache object's time to live (TTL)
B, increase memory
 
2, clean up the cache
Sometimes a cache's expiration time has not yet expired, the cache object has been updated. Therefore, for this situation, for the cache administrator, you need to manually clean-up has expired or old cache object. For Varnish, it provides three ways to accomplish this: HTTP pruning (HTTP purging), disable a class of cache objects and forced cache miss (forced cache misses)
A, HTTP pruning
Http pugre function is through the Varnish telnet management port to send purge instructions to clear the cache object is not needed. The command format is:
/ Usr / local / varnish / bin / varnishadm -T 192.168.0.102:2000 purge.url < REGEXP>
REGEXP: is based on regular expression matching url
For example: # / usr / local / varnish / bin / varnishadm -T 192.168.0.102:2000 purge.url /a/varnish1.html
To delete all cached objects, you can use the following command
      For example: # / usr / local / varnish / bin / varnishadm -T 192.168.0.102:2000 purge.url ^. * $
To view a list of recently deleted cached objects, you can use the following command
      For example: # / usr / local / varnish / bin / varnishadm -T 192.168.0.102:2000 purge.list
 
In addition to deleting the cache object from the command line, you can do this by logging in to the telnet administration interface. Such as:
[Root @ varnish-server ~] # telnet 192.168.0.102 2000
Trying 192.168.0.103 ...
Connected to localhost.localdomain (192.168.0.103).
Escape character is '^]'.
200 154
-----------------------------
Varnish HTTP accelerator CLI.
-----------------------------
Type 'help' for command list.
Type 'quit' to close CLI session.
 
Purge.url /a/mz/2010/0421/11.html # Here is the removal of this page cache
200 0
 
Purge.url ^ / zm / a / d. * $ # Here is the clear / zm / a / d / directory all the letters d beginning with the cache page
200 0
 

B, mandatory cache miss
In the vcl_recv use return (pass) to force the request to the upstream server to obtain the content, but it will not be able to cache it. Use purge to remove the old cache object, but if the upstream server is down and can not get a new version of the content, this content will no longer respond to the client. Using req.has_always_miss = ture, Varnish searches for the appropriate content in the cache but always responds to "miss", so vcl_miss will be responsible for starting vcl_fetch to fetch new content from the upstream server and overwrite the old content with the new content cache . At this point, if the upstream server is down or unresponsive, the old content remains intact and can continue to serve clients that do not use req.has_always_miss = true until they expire or are removed by other means.
 
C, disable a class of cache objects (banning)
The ban () operation is only available in Varnish 3, the purge () operation in Varnish 2 is replaced with the ban () operation in Varnish 3, and the Varnish 3 uses the purge operation, but gives it new functionality . Ban () This operation was also supported by some people objection, therefore, for the time being to say this.
 
Seven, Varnish back-end storage type
Varnish supports many different types of backend storage, which can be specified with the -s option when varnishd starts. Types of backend storage include:
(1) file: use a specific file to store all of the cache data, and through the operating system mmap () system call to the entire cache file mapped to the memory area (if conditions permit);
(2) malloc: use malloc () library calls in the varnish start to the operating system for the specified size of the memory space to store the cache object;
(3) persistent (experimental): the same function with file, but can be persistent storage of data (that is, restart the varnish data will not be cleared); still in the testing period;
 
Varnish can not track whether a cache object is stored in the cache file, and thus there is no way to know whether the cache file on the disk is available, so the file storage method in the varnish stop or restart will clear the data. However, the persistent method is still in the testing phase. For example, if the cache object can not be processed beyond the cache space, it is only applicable to scenarios with huge buffer space.
 
Choose the appropriate storage method will help to enhance the system, from an empirical point of view, it is recommended that the memory space enough to store all the cache object using malloc method, on the contrary, file storage will have a better performance. However, it is important to note that varnishd actually uses more space than the -s option specifies, and generally requires more than 1K of storage per cache object, which means that for 1 million cache object scene, the use of the cache space will exceed the specified size of 1G or so. In addition, in order to save the data structure, etc., varnish itself will account for not a small memory space.
 
When specifying the cache type to be used for varnishd, the acceptable format for the -s option is as follows:
      Malloc [, size] or
    File [, path [, size [, granularity]]] or
      Persistent, path, size {experimental}
      
The granularity in file is used to set the buffer space allocation unit, the default unit is bytes, all other sizes will be rounded.
 
 
8, Varnish back-end state detection
Varnish can detect the health status of the back-end host and automatically remove it from the list of available back-end hosts when it determines that the back-end host fails, and automatically make it available once it becomes available again. To avoid miscarriage of justice, when Varnish detects a change in the health of the back-end host (such as when a back-end host suddenly becomes unavailable during a probe), it usually takes several consecutive probes to be marked as new Converted state.
 
Varnish detects that the health of the back-end server is set by the .probe, and the results can be retrieved from the req.backend.healthy variable, by the Backend_health in varnishlog or by debug.health in varnishadm.
 
Varnish detects configuration of back-end server health instances:
Backend web1 {
      .host = "www.xsl.com";
      .probe = {
              .url = "/.healthtest.html";
              .interval = 1s;
              .window = 5;
              .threshold = 2;
      }}
}}
.probe detection instructions commonly used are:
(1) .url: the URL of the requested host when detecting the health status of the backend host, defaults to "/";
(2) .request: the detailed format of the requested content when detecting the health status of the backend host, it will replace the detection mode specified by .url; for example:
      .request =
              "GET /.healthtest.html HTTP / 1.1"
              "Host: www.magedu.com"
              "Connection: close";
(3) .window: set to determine the back-end host health state based on the number of recent detections to the default is 8;
(4). Threshold: .window specified in the number of times, at least how many times it is successful to determine the back-end host is healthy to run; default is 3;
(5). Initial: Varnish at the start of the back-end host at least how many times the successful detection, the default with. Threshold;
(6). Expected_response: expect back-end host response status code, the default is 200;
(7). Interval: probe the request to send the cycle, the default is 5 seconds;
(8) .timeout: the expiry time of each probe request, the default is 2 seconds;
Thus, for the example above, this URL is http://www.xsl.com/.healthtest.html, which is used to probe the backend host at www.xsl.com every 1 second. In the last 5 probe requests At least 2 times is successful (response code is 200) to determine the back-end host to normal operation.
 
Nine, Varnish definition of multiple back-end host
Varnish can use the director directive to one or more approximate back-end host is defined as a logical group, and can specify the corresponding scheduling (also known as the selection method) to rotate the request to send to these hosts. Different directors can use the same back-end host, and a director can also use the "anonymous" back-end host (defined directly in the director). Each director must have its private name and must be invoked in the VCL after the definition. Any location in the VCL that can specify the back-end host can be replaced by a call to a defined director as needed.
Defines configuration instances for multiple back-end hosts
.backend web1 {# Display the definition of a back-end server
      .host = "backend1.xsl.com";
      .port = "80";
}}
Director webserver random {# Define a back-end server logical group whose scheduling is random (ie, random)
      .retries = 5;
      {
              .backend = web1;
              .weight = 2;
}}
{
      .backend = {# Define an anonymous backend host
              .host = "backend2.xsl.com";
              .port = '80';
      }}
      .weight = 3;
}}
}}
For the example above, web1 defines the back-end host for the display. If there are multiple, you need to use the director directive to define. In the above example, the director directive defines a logical group called webserver. Which need to specify the webserver group back-end server address and its ip and other parameters. In the webserver group, two back-end servers are defined, web1 is the back-end host defined for display, and "backend2.xsl.com" is the anonymous host.
When you use a director to define a logical group, because there are multiple hosts, you need to specify the scheduling method to transfer the request to the appropriate back-end server. There are two kinds of director in the transfer mode:
      Round-robin: Rotation, that is, the request is forwarded in turn to the next back-end host. The scheduling method has no other parameters, only need to know the back-end host.
      Random: Random scheduling, that is, from the back-end host in the random selection of a server to respond to requests. In the random mode, you need to specify the weight value for each backend host. Varnish 2.1.0, random selection method addition of two different forms of client and hash. The client type director uses client.identity as the pick factor, which means that the same request for client.identity will be sent to the same backend host. Client.identity defaults to cliet.ip, but it can also be modified in VCL to the desired identifier. Similarly, hash type directors use hash data as the pick factor, which means that requests for the same URL will be sent to the same back-end host, which is commonly used in multi-level cache scenarios. However, both client and hash, when it is inclined to use the back-end host is unavailable, will re-select the new back-end host.
Under the director level, you can also use the .retries parameter to set the number of attempts to find a healthy backend host.
     
         
       
         
  More:      
 
- Python basis: os module on the file / directory using methods commonly used functions (Programming)
- Web database security tips (Linux)
- HBase table data processing tab (Database)
- Du and df show disk space usage inconsistent Causes and Treatment (Linux)
- Oracle large table to clean truncate .. reuse storage (Database)
- Android Application Development: Genymotion can not start solving (Linux)
- Linux check disk parameters trapping lack amendments (Linux)
- Linux shell string interception and stitching (Linux)
- Linux file system management partition, format, mount - label mount (Linux)
- Laravel 4 Expansion Pack (Server)
- Ubuntu 12.04 installation DHCP Server (Server)
- Git uses a basic tutorial (Linux)
- Puppet Detailed centralized configuration management system (Server)
- How to configure security services under Linux (Linux)
- shellinabox: one uses AJAX Web-based terminal emulator (Linux)
- tcpdump Linux system security tools (Linux)
- Linux system security knowledge (Linux)
- C # Future: Tracking null reference (Programming)
- Git / Github use notes (Linux)
- CentOS 6.5 installation VNCServer implement graphical access (Server)
     
           
     
  CopyRight 2002-2016 newfreesoft.com, All Rights Reserved.