Well I was thinking "no" options for time-outs and buffers - in particular "tcp_finwait2" sounds appropriate - I have 1200 set, although no explanation in the manual as to what that means  

Check your adapter settings for queue and pool sizes (to the max allowed) and for the interface use "tcp_nodelay=1"- those settings must be done before configuring any Ethernet Aggregation or EtherChannel.  

Check the interface MTU is correct and matches on both ends to avoid windowing delay problems as well as fragmentation.

From “smit tuning”, the following command can be built from the Tuning Network Option Parameters menu, to set options for next boot – you may find some of these are set already as a default, this is what I run with these days.  In particular I've tried various settings for tcp_timewait, tcp_keepidle and tcp__keepinit when faced with mail download hangs and cut offs over wide area connections.

no -r -o fasttimo='50' -o nbc_limit='65536' \
-o nbc_pseg_limit='20480' -o tcp_timewait='1' \
-o clean_partial_conns='1' -o rfc1323='1' \
-o tcp_keepidle='1200' -o tcp_keepinit='60' \
-o tcp_keepintvl='20' \
-o bcastping='0' -o directed_broadcast='0' \
-o ipignoreredirects='1' -o ipqmaxlen='150' \
-o ipsendredirects='0' -o ipsrcrouteforward='0' \
-o ipsrcrouterecv='0' -o ipsrcroutesend='0' \
-o tcp_pmtu_discover='0' -o udp_pmtu_discover='0'

In addition, for best tuning with TCP/IP over a fibre fabric, the following options apply:
no -r -o -o tcp_recvspace='393216 \
-o tcp_sendspace='393216 -o sb_max='1310720' \
-o udp_recvspace='655360' -o udp_sendspace='65536'

Whereas for best performance over a basic Ethernet network, in preference to a disk fabric, the following options would instead apply:
no -r -o -o tcp_recvspace='65536' \
-o tcp_sendspace='65536' -o sb_max='262144' \
-o udp_recvspace='65536' -o udp_sendspace='32768'

For web servers, the following lines would also be needed:
no -r -o -o sack='1' -o delayack='3' delayackports='{80}'

With AIX AIX 5.3 ML6, there was a fault with the default value for “ipfragttl” – the manuals say it should be “60”, but “2” is the apparently wrong default.
no -r -o -o ipfragttl='60'    # To fix the fault.

Thank you and have a nice vacation! I appreciate all these tips and they will be very helpful in the investigation......Justin

I have no access to source code - so I shall make an assumption - that the socket close sends the FIN. I would expect it at that layer of abstraction - not in the application.

It has been a long long while since I have actually done any socket() programming, but applications may use socket_options (or something like that foo() name - as I said, it has been a while.)

Why would changing one server affect both - perhaps at the remote site - something unusual.

You can see if it is multi-threaded versus multi-process by checking with ps -emo THREAD - you will see the separate thread ID's if it is multi-threaded (e.g. 40 TID, odd numbers, and one PID.

Look into trace to get even lower than IPTRACE (which only looks at traffic, whereas TRACE takes a system call - and lower view.

Take a look at netpmon -v for an example of the trace command you could do (capturing it yourself, rather than the default netpmon processing).

p.s. - On vacation, so offline til March. Happy Hunting!

The application is CA Message Manager email archiving software. No open source unfortunately. It can start between 10 and 50 simultaneous SMTP conversations with the remote mail processor so I guess in that aspect it is multi-threaded. Each conversation starts a socket.

I don't know why the socket is closing before the QUIT that is the million dollar question at this point.

One big question I have - Is it TCPIP that is adding the FIN flag or is it the application? Or could it be either? Can the TCPIP stack decide it needs to FIN the connection independent of the application? Or can only the application cause this to happen? When I ran this by IBM they said they thought that the app was closing the socket resulting in the FIN, not AIX or AIX TCPIP.

One other issue I failed to mention. We are not 100% certain but this may have started when we moved the AIX LPAR from power5 hardware to the new power6. Not certain however. And why would moving to the new hardware cause this?

One other piece of information also.  We have two AIX LPARS running this software. One day we were running with 40 SMTP threads on server 1 and 13 threads on server 2. No errors were occurring. I increased the threads on server 2 from 13 to 40 (no change on server 1) and errors started on BOTH servers. Both servers point to the same remote mail processor. So the SMTP traffic from both servers goes to the same place.  So why would increasing sessions on one server cause BOTH servers to experience errors?

I am using IPTRACE...thanks

I am wondering if this is a multi-process application, or multi-threaded application. And is it open-source, self-developed, or an IBM product.
My gut feeling is that the application is not waiting. Further, I am guessing that the FIN attribute gets sent when the socket gets closed - and my assumption/question - why is the socket closing before the "QUIT" message gets sent.

Further, are you using tcpdump or iptrace to capture the sessions including data? If not,, what and how are you capturing the message queues?

(And when I get back from my vacation I may have more questions for you. Please be patient!)

Can you unmount it?

Output of mount - grep for your mountpoint. Are there two filesystems mounted over it, or has something mounted over one of it's parent directories - masking the real mountpoint?

Otherwise, change the automount attribute to false, and restart.

Aix 6.1 on a power6 8204 machine. An application on AIX transfers email for archiving to a remote IP address. The app does this by starting SMTP conversations to send the email being archived to the remote mail server. The number of simultaneous sessions that run is tailorable from 10 up to 50. Normally the SMTP session ends with the last packet sent with <CRLF> then "." the <CRLF>, the remote responds with "mail queued for delivery" we respond with "QUIT."  We have found if we set the number of simultaneous sessions too high, the last email packet is sent with TCPIP FLAGS PUSH ACK and FIN, which initiates a shutdown of the connection prematurely (before QUIT can be sent). This is intermittent. Obviously increasing the number of simultaneus sessions is increasing the network activity. Q; Why would this cause TCPIP to sometimes send the FIN prematurely? Is AIX seeing an error or delay that causes it to do this? Are there any "no" options to address this?.......Thanks

Hi there

we have a situation here : on one of our Partition that rebooted, the FS disapeared  :
if we try to mount it, it says it s already mounted, but we see nothing !!
could it be a corruption of the inode ? or do you have some other ideas ?

thanks 

I would not recommend splitting the disks. All the time that you are busy the disk you moved will be complaining about a PMISSING volume. Instead, hopefully it is installed, I would use the mkcd command to create CDROM iso images of your rootvg and burn the iso images on a PC and use these to install on a second server, making changes as needed later.

Otherwise, I would break the mirror, and use alt_disk to clone a copy and move the cloned copy for an install on a second system (or add a new disk and clone to that. This is probably the ideal solution).

I'm having the following issue:

I have 2 old RS/6000 Power3 servers with 1GB of RAM and 4x19GB disks.

1 Server is still in production and within this company there isn't any information on this machine, so i must be carefull with it.

Now they need an exact copy of that machine into the spare RS/6000.
The server is running AIX5.1 ML04 .
There isn't any working NIM server around but the copy has to be made asap.

I've had the idea because rootvg is a mirror on 2 physical disks, to poweroff the production server.
Move 1 of the rootvg disks into the spare one, and boot with a Diagnostics CD to set the bootrecord and stuff, and  boot it, change IP adress and mirror it to another internal disk. Put the original rootvg disk back at the production machine again and I should be up & running with both machines within 1 day?

Can someone let me know if it has worked for you in the past?