TCPIP FIN SENT PREMATURELY
 
*
Welcome, Guest. Please login or register. July 30, 2010, 12:06:18 PM


Login with username, password and session length


Pages: [1]   Go Down
  Print  
Author Topic: TCPIP FIN SENT PREMATURELY  (Read 1087 times)
0 Members and 1 Guest are viewing this topic.
Michael
Administrator
Hero Member
*****
Posts: 676


« Reply #6 on: March 12, 2010, 04:10:54 PM »

Justin,

any update on your situation?

Michael
Logged
John R Peck
Administrator
Senior Member
*****
Posts: 129


« Reply #5 on: February 19, 2010, 05:17:22 AM »

Well I was thinking "no" options for time-outs and buffers - in particular "tcp_finwait2" sounds appropriate - I have 1200 set, although no explanation in the manual as to what that means  Roll Eyes

Check your adapter settings for queue and pool sizes (to the max allowed) and for the interface use "tcp_nodelay=1"- those settings must be done before configuring any Ethernet Aggregation or EtherChannel.  

Check the interface MTU is correct and matches on both ends to avoid windowing delay problems as well as fragmentation.

From “smit tuning”, the following command can be built from the Tuning Network Option Parameters menu, to set options for next boot – you may find some of these are set already as a default, this is what I run with these days.  In particular I've tried various settings for tcp_timewait, tcp_keepidle and tcp__keepinit when faced with mail download hangs and cut offs over wide area connections.

no -r -o fasttimo='50' -o nbc_limit='65536' \
-o nbc_pseg_limit='20480' -o tcp_timewait='1' \
-o clean_partial_conns='1' -o rfc1323='1' \
-o tcp_keepidle='1200' -o tcp_keepinit='60' \
-o tcp_keepintvl='20' \
-o bcastping='0' -o directed_broadcast='0' \
-o ipignoreredirects='1' -o ipqmaxlen='150' \
-o ipsendredirects='0' -o ipsrcrouteforward='0' \
-o ipsrcrouterecv='0' -o ipsrcroutesend='0' \
-o tcp_pmtu_discover='0' -o udp_pmtu_discover='0'

In addition, for best tuning with TCP/IP over a fibre fabric, the following options apply:
no -r -o -o tcp_recvspace='393216 \
-o tcp_sendspace='393216 -o sb_max='1310720' \
-o udp_recvspace='655360' -o udp_sendspace='65536'

Whereas for best performance over a basic Ethernet network, in preference to a disk fabric, the following options would instead apply:
no -r -o -o tcp_recvspace='65536' \
-o tcp_sendspace='65536' -o sb_max='262144' \
-o udp_recvspace='65536' -o udp_sendspace='32768'

For web servers, the following lines would also be needed:
no -r -o -o sack='1' -o delayack='3' delayackports='{80}'

With AIX AIX 5.3 ML6, there was a fault with the default value for “ipfragttl” – the manuals say it should be “60”, but “2” is the apparently wrong default.
no -r -o -o ipfragttl='60'    # To fix the fault.

Logged
jlatus
New Member
*
Posts: 3


« Reply #4 on: February 18, 2010, 03:52:40 PM »

Thank you and have a nice vacation! I appreciate all these tips and they will be very helpful in the investigation......Justin
Logged
Michael
Administrator
Hero Member
*****
Posts: 676


« Reply #3 on: February 18, 2010, 03:22:09 PM »

I have no access to source code - so I shall make an assumption - that the socket close sends the FIN. I would expect it at that layer of abstraction - not in the application.

It has been a long long while since I have actually done any socket() programming, but applications may use socket_options (or something like that foo() name - as I said, it has been a while.)

Why would changing one server affect both - perhaps at the remote site - something unusual.

You can see if it is multi-threaded versus multi-process by checking with ps -emo THREAD - you will see the separate thread ID's if it is multi-threaded (e.g. 40 TID, odd numbers, and one PID.

Look into trace to get even lower than IPTRACE (which only looks at traffic, whereas TRACE takes a system call - and lower view.

Take a look at netpmon -v for an example of the trace command you could do (capturing it yourself, rather than the default netpmon processing).

p.s. - On vacation, so offline til March. Happy Hunting!
Logged
jlatus
New Member
*
Posts: 3


« Reply #2 on: February 17, 2010, 07:07:05 PM »

The application is CA Message Manager email archiving software. No open source unfortunately. It can start between 10 and 50 simultaneous SMTP conversations with the remote mail processor so I guess in that aspect it is multi-threaded. Each conversation starts a socket.

I don't know why the socket is closing before the QUIT that is the million dollar question at this point.

One big question I have - Is it TCPIP that is adding the FIN flag or is it the application? Or could it be either? Can the TCPIP stack decide it needs to FIN the connection independent of the application? Or can only the application cause this to happen? When I ran this by IBM they said they thought that the app was closing the socket resulting in the FIN, not AIX or AIX TCPIP.

One other issue I failed to mention. We are not 100% certain but this may have started when we moved the AIX LPAR from power5 hardware to the new power6. Not certain however. And why would moving to the new hardware cause this?

One other piece of information also.  We have two AIX LPARS running this software. One day we were running with 40 SMTP threads on server 1 and 13 threads on server 2. No errors were occurring. I increased the threads on server 2 from 13 to 40 (no change on server 1) and errors started on BOTH servers. Both servers point to the same remote mail processor. So the SMTP traffic from both servers goes to the same place.  So why would increasing sessions on one server cause BOTH servers to experience errors?

I am using IPTRACE...thanks

Logged
Michael
Administrator
Hero Member
*****
Posts: 676


« Reply #1 on: February 17, 2010, 06:28:24 PM »

I am wondering if this is a multi-process application, or multi-threaded application. And is it open-source, self-developed, or an IBM product.
My gut feeling is that the application is not waiting. Further, I am guessing that the FIN attribute gets sent when the socket gets closed - and my assumption/question - why is the socket closing before the "QUIT" message gets sent.

Further, are you using tcpdump or iptrace to capture the sessions including data? If not,, what and how are you capturing the message queues?

(And when I get back from my vacation I may have more questions for you. Please be patient!)
Logged
jlatus
New Member
*
Posts: 3


« on: February 16, 2010, 07:33:37 PM »

Aix 6.1 on a power6 8204 machine. An application on AIX transfers email for archiving to a remote IP address. The app does this by starting SMTP conversations to send the email being archived to the remote mail server. The number of simultaneous sessions that run is tailorable from 10 up to 50. Normally the SMTP session ends with the last packet sent with <CRLF> then "." the <CRLF>, the remote responds with "mail queued for delivery" we respond with "QUIT."  We have found if we set the number of simultaneous sessions too high, the last email packet is sent with TCPIP FLAGS PUSH ACK and FIN, which initiates a shutdown of the connection prematurely (before QUIT can be sent). This is intermittent. Obviously increasing the number of simultaneus sessions is increasing the network activity. Q; Why would this cause TCPIP to sometimes send the FIN prematurely? Is AIX seeing an error or delay that causes it to do this? Are there any "no" options to address this?.......Thanks
Logged
Pages: [1]   Go Up
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.10 | SMF © 2006-2009, Simple Machines LLC

Valid XHTML 1.0! Valid CSS! Dilber MC Theme by HarzeM
Page created in 0.682 seconds with 18 queries.



eXTReMe Tracker

Terms of Use and Privacy and Security Policies
Copyright 2001-2010 Michael Felt, John R Peck and ROOTVG.NET