| |
First, the environment
The current environment for nginx as front-end reverse proxy, upstream of the two tomcat.
Second, the reason
Due to recent project is the initial stage, more overtime on weekdays, time ran into the day without any problem, I got up early to pack equipment fun riding the subway toward home.
At this point, I'm happy listening to music sitting on the subway, the music suddenly stopped short, sounded the call ringtone. A bad feeling wells up, it appears to be a problem. Ever since I took the phone to see our boss's name flashed on the phone screen, a deep breath, picked up the phone. I heard the boss said we are now being given the client side so what what. Since the subway a lot of noise, and the signal is not very good, no small ask. Anyway, is a server-side problem, I will first answer down. At this point I have not got home, then said home look. Then the boss hung up the phone. I recall in my heart in the end to say what the problem is, but the memories for a long time did not hear the final say something.
Third, debugging
Turn on the computer at home, boarded the QQ, to see the group in the client side and then the problems reflected screenshot shows part of the request server resources appeared request timed out phenomenon. So I logged on to the server. First check whether each process is normal. After determining the normal access page under test it is normal. Because the client is called the interface file, the browser is no way to directly access the test, the problem can only be viewed by nginx logs.
Client side testing is https protocol, this time on the part of the log view https:
2015/08/06 19:15:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "
2015/08/06 19:16:11 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "
2015/08/06 19:17:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "
2015/08/06 19:29:29 [error] 17709 # 0: * 1380648 upstream timed out (110: Connection timed out) while sending request to upstream, client: xxx, server: www.xxxx.com, request: " POST /xxx/pub/xxx.do HTTP / 1.1 ", upstream:" http: // xxxx: 8082 / xxx.do ", host:" www.xxxx.com:443 "
Log format similar to the above content of these classes, see these, I think the problem may be some of the nginx proxy timeout, so he changed the time out of time setting.
The test again or have this problem, I carefully thought for a moment, suddenly found here testing is https, but I just like to modify http timeout, then modified again, was secretly excited think that this is OK now. I did not expect after modifying the client side of the test is still the same problem. At this point a bit depressed.
Since then I have real-time monitoring of the backlog, although again found an error, but there are some differences:
2015/08/06 19:47:31 [error] 17708 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: "
2015/08/06 19:50:11 [error] 12300 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: " www.xxx.com:443 "
2015/08/06 19:55:04 [error] 132648 # 0: * 1381368 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/home/xxx.do", host: " www.xxx.com:443 "
You can see the log inside the error code and error message changed. According to this prompt, it should be sent upstream reset request to nginx, which is why?
Online search a bit and found that most of the time out to say are still issues, said that the individual client's head get too big, but obviously here is the POST method. That was in accordance with the time out to find the problem to see whether the client side so want to consult when setting timeout, the client side is set to the 10s of the timeout. So it should not be a problem. Under 10s should be sufficient case processing and the server responds.
At this time also did not think the problem in any place, can only look at the multi-monitor write the log, so the http access log and error log and https access and error logs all the way to the tail of the real-time monitoring. At these logs to monitor them, suddenly I found a few strange phenomenon:
3.1, http error logs have connection time out phenomenon
2015/08/06 19:29:44 [error] 17708 # 0: * 1380926 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www. xxxx.com, request: "GET /xxx/xxx/xxx.png HTTP / 1.1", upstream: "http://xxx.xxx.xxx:8082/xxx/image/xxx.png", host: "www. xxx.com ", referrer:" http://www.xxx.com/xxx/xxx/xxx.do?user_id=57&from=singlemessage&isappinstalled=1 "
2015/08/06 19:29:44 [error] 17708 # 0: * 1380930 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www. xxx.com, request: "GET /xxx/xxx/xxx.png HTTP / 1.1", upstream: "http://xx.xxx.xxx.xxx:8082/xxx/xxx/xxx.png", host: " www.xxx.com ", referrer:" http://www.xxx.com/xxx/xxx/xxx.do?user_id=57&from=singlemessage&isappinstalled=1 "
3.2, https access and error logs will be two requests at the same time point
# Error log
2015/08/06 21:58:59 [error] 22498 # 0: * 527 recv () failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: www.xxx.com, request: "POST /xxx/xxx/xxx.do HTTP / 1.1", upstream: "http://xxx.xxx.xxx.xxx:8082/xxx/xxx/xxx.do", host : "www.xxx.com:443"
# Access log
xxx.xxx.xxx.xxx - - [06 / Aug / 2015: 21: 58: 59 +0800] "POST /xxx/xxx/xxx.do HTTP / 1.1" 200 1100 "-" "xxx / 1.1.0 ( iPhone Simulator; iOS 8.1; Scale / 2.00) "" - "
The above can be seen at the same time there have been two requests, and a success, a failure, and had a lot of access log 499 response code request. The 499 response code is said / * 499, client has closed connection * /. That active client closes the connection, or submit twice interval rapid post nginx This problem also occurs.
1, the client take the initiative to close the connection, because the long closes the connection after a timeout to set the time. This is back on the 10s will timeout and issues frequent occurrence of the phenomenon of time out.
2, submits a POST request too fast, nginx request will be considered to be unsafe, they take the initiative to refuse the connection. This may be a client side data uninterrupted test result, in this case, you can configure the following parameters for nginx configuration file does not take the initiative to close.
proxy_ignore_client_abort on;
However, this configuration is unsafe. At this time in order to solve the problem, he was deeply moved by the heart to the test configuration.
Sadly, this configuration parameter was found still not solve the problem. Heart drop happen again. But no, after all, the log can be found in all kinds of issues.
Excluded in determining the second point of the question, the return to the first point (that is, 3.1 on the issue), why the client side https protocol test, the same problem occurs in the http agreement. Here's shady way. So the focus from the start here.
Interface files test client is placed in an application directory, although not directly access interface file, but you can access the web application directory.
By accessing web directory and combine logs found where the problem is:
1, nginx proxy using the default polling, so every time might dispatch to a different back-end server. And at the moment when access to refresh the page, one of which would have some Caton, see the backlog will appear suddenly found that every time a card error.
2, while the error occurs, the normal log also request a successful appears. See page appeared in a refresh. This explains the phenomenon with a point in time before the emergence of the two requests.
3, this time in the back to see the error log is very easy to see the problem, and found that all belong to the error upstream in the same tomcat. On the note is a problem with this tomcat.
down in nginx out in this tomcat, client side test everything OK. To solve this problem. As to why there is this problem is the development of direct day moving too tomcat code. Problems can only work during the day to find the problem with the code.
IV Summary
After the troubleshooting, to sum up:
1, problems to calmly deal with, from the most basic start troubleshooting.
2, good use of the log of each application error tracking.
3, or to do it for each application clear in mind, for all kinds of error as much as possible to know about what it is. Where the investigation. |
|
|
|