on vacation. I'll look into it later.

Note the highlighted lines: it would appear you are using the 192.168.160.46 address on both nodes at the same time. This is going to give problems. I was blind before I guess.

Well, let's think a bit about switch failure as well.

You have dual NIC's, dual switches, etc.. If you look atthe way everything is cabled is there - perhaps - an unintended SPOF (single point of failure).

I know of a case where some clients could communicate, but others could not because there was a switch interconnect failure. HACMP got the service interfaces into the same switch - but since there was no connectivity to the other switch some clients got lost.

The cluster saw itself as correct - but not all the clients agreed. So, my next global view would be to inspect the infrastructure for side-effects. HACMP cannot see outside of the node/cluster.

Hi Michael,
Thanks for your thoughts.

The application is an Oracle database and it is used by clients on different subnets so it uses the default gateway. In our current setup, the rman recovery database is on the other node in the cluster which is on the same network. When a switch failure ocurred and we got the fail over, end users didn't notice because the database was still available. When we ran an RMAN backup, it sometimes worked and sometimes didn't depending on if it got the good interface.

I was reading an earlier post http://rootvg.net/forums/index.php/topic,392.0.html which looked similar to my broblem  in that it looked like the same error (switch failure) caused both our problems. In one of the replies (by John) it suggested it was caused by a problem with having two NICs on the same network which is like what we have. I was hoping that somehow, the persistnet address was casuing the problem and by changing the distribution preference to collocation with persistent label would do somethig. It didn't.

Hopefully, we will get our purchasing department sorted and have the bill paid so I can give the problem to IBM support. Its just so frustrating to see it looks like it should work, its just that it doesn't. I am thinking ethrchannel will be the way to go, but that is going to require downtime which is always a problem when you want high availability.

Cheers,
Bernie.

Hopefully, the error is obvious. I changed the names and left the addresses as is since they are in the private address space.

One thing, slightly off topic, but what does the => in the routing table indicate? I could not find anything in documentation on it and guessed it was meant ot refer to another entry, but when the route was buggered, it was associated with the card connected to the bad switch.

root@primary:/ > /usr/es/sbin/cluster/utilities/cltopinfo
Cluster Name: ora_prd_cluster
Cluster Connection Authentication Mode: Standard
Cluster Message Authentication Mode: None
Cluster Message Encryption: None
Use Persistent Labels for Communication: No
There are 2 node(s) and 2 network(s) defined

NODE primary:
        Network net_diskhb_01
                primary_hdisk5_01  /dev/hdisk5
        Network net_ether_01
                production    192.168.160.46
                primary_if2        10.0.2.1
                primary_if1        10.0.1.1

NODE secondary:
        Network net_diskhb_01
                secondary_hdisk5_01   /dev/hdisk5
        Network net_ether_01
                production    192.168.160.46
                secondary_if2 10.0.2.2
                secondary_if1 10.0.1.2

Resource Group ora_prd_group
        Startup Policy   Online On Home Node Only
        Fallover Policy  Fallover To Next Priority Node In The List
        Fallback Policy  Never Fallback
        Participating Nodes      primary secondary
        Service IP Label                 production
root@primary:/ > netstat -ni
Name  Mtu   Network     Address            Ipkts Ierrs    Opkts Oerrs  Coll
en0   1500  link#2      0.11.25.bf.5a.d8    977875     0   302223     3     0
en0   1500  10.0.1      10.0.1.1            977875     0   302223     3     0
en1   1500  link#3      0.11.25.bf.5a.77   8338784     0  6419315     4     0
en1   1500  10.0.2      10.0.2.1           8338784     0  6419315     4     0
en1   1500  192.168.160 192.168.160.47     8338784     0  6419315     4     0
en1   1500  192.168.160 192.168.160.46     8338784     0  6419315     4     0
lo0   16896 link#1                          296158     0   296507     0     0
lo0   16896 127         127.0.0.1           296158     0   296507     0     0
lo0   16896 ::1                             296158     0   296507     0     0
root@primary:/ > netstat -nr
Routing tables
Destination      Gateway           Flags   Refs     Use  If   Exp  Groups

Route Tree for Protocol Family 2 (Internet):
default          192.168.160.234   UG       26   5862416 en1      -      -   
10.0.1.0         10.0.1.1          UHSb      0         0 en0      -      -   =>
10.0.1/24        10.0.1.1          U         2    301844 en0      -      -   
10.0.1.1         127.0.0.1         UGHS      2     53424 lo0      -      -   
10.0.1.255       10.0.1.1          UHSb      0         4 en0      -      -   
10.0.2.0         10.0.2.1          UHSb      0         0 en1      -      -   =>
10.0.2/24        10.0.2.1          U         2    307716 en1      -      -   
10.0.2.1         127.0.0.1         UGHS      0     50209 lo0      -      -   
10.0.2.255       10.0.2.1          UHSb      0         4 en1      -      -   
127/8            127.0.0.1         U         9    172890 lo0      -      -   
192.168.160.0    192.168.160.47    UHSb      0         0 en1      -      -   =>
192.168.160/24   192.168.160.47    U         1    241283 en1      -      -   
192.168.160.46   127.0.0.1         UGHS      0      2921 lo0      -      -   
192.168.160.47   127.0.0.1         UGHS      6     17152 lo0      -      -   
192.168.160.255  192.168.160.47    UHSb      0         0 en1      -      -   

Route Tree for Protocol Family 24 (Internet v6):
::1              ::1               UH        0         0 lo0      -      -   
root@primary:/ > exit

G'day all,
I have read through the posts here and have found some descriptons close to my own problem but they are not quite the same.
Firstly, I want to state at the outset that I know using Etherchannel would overcome the issue - but hacmp is supposed to do the work and high availability means high availability - we have it because getting down time agreed to is difficult.

Now I have got that off my chest, I will describe the problem. I have a 2 node hacmp cluster using IPAT. Its running 5.3 cluster software on an AIX 5.3 machine. Our persistent address is on the same subnet as our service address (192.168.160.0/24) and the boot addresses on the two cards are on different subnets to each other and to the service address (10.0.1.0/24, 10.0.2.0/24). When the switch failed last year, the service address and the gateway moved to the working switch. However, looking at the routing table, we still had a route using the faulty interface (en1). Consequently, every second ip connection on the subnet of the service address (192.168.160) seemed to be sent to the bad route. I did try the boot address subnet and it worked fine but can't guarantee how well I checked it - limited down time meant not much time to test different scenarios. Everything  that went thorugh the default gateway worked so that clients didn't notice any difference.

I did some research and found seemingly related posts on the internet that kind of suggested the sort of behaviour we were seeing could be fixed by using dead gateway detection - I tried both active_dgd on the route and passive_dgd, neither made a difference.

Reading thorugh the posts here, it seems the problem is our choice of addresses and networks. At the same time as all this, our accounting has somehow managed to not pay our HACMP maintenance so I can't run this by IBM.

Does anyone have a suggestion of what might be the cause and a potential fix to our problems?

Any insight or comments would be appreciated!
Cheers,
Bernie.