Home PC Games Linux Windows Database Network Programming Server Mobile  
           
  Home \ Server \ Nginx Proxy timeout Troubleshooting     - Use Bosh deploy CloudFoundry problems encountered on OpenStack (Server)

- Docker in the development and practice of IFTTT (Server)

- First start with Kali Linux 2.0 (Linux)

- How to configure HTTP Strict Transport Security (HSTS) (Linux)

- Fedora 8 configured Heartbeat (Linux)

- Build your own Web server under Ubuntu Linux system (Server)

- Linux Command Tutorial: du command to view disk space (Linux)

- Use HugePages optimize memory performance (Database)

- Linux / Windows setup is complete port range (Linux)

- RHEL7 unattended automatic installation DHCP + TFTP + SYSLINUX + TFTP + Kickstart (Linux)

- Unetbootin make use U disk loading Linux system (Linux)

- Linux Basic Course: tar command description (Linux)

- C # asynchronous delegates (Programming)

- Boot automatically remove Linux operating history records (Linux)

- How to upgrade to Ubuntu 14.04 Linux Kernel 4.4.1 LTS (Linux)

- Build Eclipse + Maven + Scala-IDEA the Scala Web development environment (Server)

- Syncthing: Private Security Synchronization tool to synchronize files between computers / folder (Linux)

- Simple Linux file system - df, du, ln (Linux)

- Oracle GoldenGate tutorial Second, configure and use (Database)

- Linux network security backdoor technology and advanced skill practice (Linux)

 
         
  Nginx Proxy timeout Troubleshooting
     
  Add Date : 2017-08-06      
         
         
         
  First, the environment

The current environment for nginx as front-end reverse proxy, upstream of the two tomcat.

Second, the reason

Due to recent project is the initial stage, more overtime on weekdays, time ran into the day without any problem, I got up early to pack equipment fun riding the subway toward home.

At this point, I'm happy listening to music sitting on the subway, the music suddenly stopped short, sounded the call ringtone. A bad feeling wells up, it appears to be a problem. Ever since I took the phone to see our boss's name flashed on the phone screen, a deep breath, picked up the phone. I heard the boss said we are now being given the client side so what what. Since the subway a lot of noise, and the signal is not very good, no small ask. Anyway, is a server-side problem, I will first answer down. At this point I have not got home, then said home look. Then the boss hung up the phone. I recall in my heart in the end to say what the problem is, but the memories for a long time did not hear the final say something.

Third, debugging

Turn on the computer at home, boarded the QQ, to see the group in the client side and then the problems reflected screenshot shows part of the request server resources appeared request timed out phenomenon. So I logged on to the server. First check whether each process is normal. After determining the normal access page under test it is normal. Because the client is called the interface file, the browser is no way to directly access the test, the problem can only be viewed by nginx logs.

Client side testing is https protocol, this time on the part of the log view https:

2015/08/06 19:15:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "

2015/08/06 19:16:11 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "

 2015/08/06 19:17:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "

2015/08/06 19:29:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "

Log format similar to the above content of these classes, see these, I think the problem may be some of the nginx proxy timeout, so he changed the time out of time setting.

The test again or have this problem, I carefully thought for a moment, suddenly found here testing is https, but I just like to modify http timeout, then modified again, was secretly excited think that this is OK now. I did not expect after modifying the client side of the test is still the same problem. At this point a bit depressed.

Since then I have real-time monitoring of the backlog, although again found an error, but there are some differences:

2015/08/06 19:47:31 [error] 17708 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: "

2015/08/06 19:50:11 [error] 12300 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: " www.xxx.com:443 "

2015/08/06 19:55:04 [error] 132648 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: " www.xxx.com:443 "

You can see the log inside the error code and error message changed. According to this prompt, it should be sent upstream reset request to nginx, which is why?

Online search a bit and found that most of the time out to say are still issues, said that the individual client's head get too big, but obviously here is the POST method. That was in accordance with the time out to find the problem to see whether the client side so want to consult when setting timeout, the client side is set to the 10s of the timeout. So it should not be a problem. Under 10s should be sufficient case processing and the server responds.

At this time also did not think the problem in any place, can only look at the multi-monitor write the log, so the http access log and error log and https access and error logs all the way to the tail of the real-time monitoring. At these logs to monitor them, suddenly I found a few strange phenomenon:

3.1, http error logs have connection time out phenomenon

2015/08/06 19:29:44 [error] 17708 # 0: * 1380926 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www. xxxx.com, request: "GET /xxx/xxx/xxx.png HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/image/xxx.png", host: "www. xxx.com ", referrer:" http://www.xxx.com/xxx/xxx/xxx.do?user_id=57&from=singlemessage&isappinstalled=1 "

2015/08/06 19:29:44 [error] 17708 # 0: * 1380930 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www. xxx.com, request: "GET /xxx/xxx/xxx.png HTTP / 1.1", upstream: "http://xx.xxx.xxx.xxx:8082/xxx/xxx/xxx.png", host: " www.xxx.com ", referrer:" http://www.xxx.com/xxx/xxx/xxx.do?user_id=57&from=singlemessage&isappinstalled=1 "

3.2, https access and error logs will be two requests at the same time point

# Error log

2015/08/06 21:58:59 [error] 22498 # 0: * 527 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx.xxx:8082/xxx/xxx/xxx.do", host : "www.xxx.com:443"

# Access log

xxx.xxx.xxx.xxx - - [06 / Aug / 2015: 21: 58: 59 +0800] "POST /xxx/xxx/xxx.do HTTP / 1.1" 200 1100 "-" "xxx / 1.1.0 ( iPhone Simulator; iOS 8.1; Scale / 2.00) "" - "

The above can be seen at the same time there have been two requests, and a success, a failure, and had a lot of access log 499 response code request. The 499 response code is said / * 499, client has closed connection * /. That active client closes the connection, or submit twice interval rapid post nginx This problem also occurs.

1, the client take the initiative to close the connection, because the long closes the connection after a timeout to set the time. This is back on the 10s will timeout and issues frequent occurrence of the phenomenon of time out.

2, submits a POST request too fast, nginx request will be considered to be unsafe, they take the initiative to refuse the connection. This may be a client side data uninterrupted test result, in this case, you can configure the following parameters for nginx configuration file does not take the initiative to close.

proxy_ignore_client_abort on;

However, this configuration is unsafe. At this time in order to solve the problem, he was deeply moved by the heart to the test configuration.

Sadly, this configuration parameter was found still not solve the problem. Heart drop happen again. But no, after all, the log can be found in all kinds of issues.

Excluded in determining the second point of the question, the return to the first point (that is, 3.1 on the issue), why the client side https protocol test, the same problem occurs in the http agreement. Here's shady way. So the focus from the start here.

Interface files test client is placed in an application directory, although not directly access interface file, but you can access the web application directory.

By accessing web directory and combine logs found where the problem is:

1, nginx proxy using the default polling, so every time might dispatch to a different back-end server. And at the moment when access to refresh the page, one of which would have some Caton, see the backlog will appear suddenly found that every time a card error.

2, while the error occurs, the normal log also request a successful appears. See page appeared in a refresh. This explains the phenomenon with a point in time before the emergence of the two requests.

3, this time in the back to see the error log is very easy to see the problem, and found that all belong to the error upstream in the same tomcat. On the note is a problem with this tomcat.

down in nginx out in this tomcat, client side test everything OK. To solve this problem. As to why there is this problem is the development of direct day moving too tomcat code. Problems can only work during the day to find the problem with the code.

IV Summary

After the troubleshooting, to sum up:

1, problems to calmly deal with, from the most basic start troubleshooting.

2, good use of the log of each application error tracking.

3, or to do it for each application clear in mind, for all kinds of error as much as possible to know about what it is. Where the investigation.
     
         
         
         
  More:      
 
- Linux install the Java Runtime Environment and the output Hello World under (CentOS) (Linux)
- Commentary Apache + Tomcat + JK implement Tomcat clustering and load (Server)
- Let Linux boot methods to enter characters interface and set FrameBuffer resolution methods (Linux)
- Debian 7.7 Installation and Configuration (Linux)
- Linux iptables port mapping settings (Server)
- How to use the ftp tool to transfer files between Windows and Linux (Linux)
- Ubuntu users Steam controller does not work solutions (Linux)
- Quick paging ROW_NUMBER conducted (Database)
- CentOS 7 open ports (Linux)
- Ubuntu 14.04 + xRDP + Xfce implement Windows Remote Desktop Connection (Linux)
- Caffe install under Ubuntu 14.04 64bit (Linux)
- Linux Bash share tips for getting started (Linux)
- Java Foundation - Getting Start (Programming)
- MariaDB 10.0.X, the dynamic column support JSON format to obtain data (Database)
- CentOS 6.5 using Virtualenv under development environment to build Python3 (Linux)
- Single list summarizes the basic operation (Programming)
- sed and awk in shell usage and some examples (Linux)
- Linux 0.12 kernel and modern kernels difference in memory management (Linux)
- Oracle user password problem (Database)
- Linux serial debugging tools xgcom install (Linux)
     
           
     
  CopyRight 2002-2020 newfreesoft.com, All Rights Reserved.