Artur
2010-09-09 13:23:26 UTC
Hello
I face a problem with NFS v. 3 service running as failover resource
group on a SUN cluster
behind firewall. Firewall’s policy is just being tightened – from now
on only NFS is supposed to
be opened. Unfortunately SUN cluster’s behavior (asymmetric IP
communication) cannot be
properly addressed by FW. Please have a look at details below.
NFS v. 3 client first of all has to make UDP query to portmapper (111)
running on the NFS server
to obtain mountd and nfsd RPC program ports. Further query made to
mountd port will provide
handle used during future connection to nfsd. AFAIB SUN’s nfsd is
always bound to specific
(well known) port (as it also provides NFS v.4 service which does not
require calls separate to
mountd), but mountd port is allocated dynamically. To deal with NFS v.
3 firewall actually has to
trace portmapper communication and dynamically open ports for mountd,
nfsd, lockd. Both
Checkpoint and Cisco PIX are capable of doing this. But they trace
sessions between particular
IPs!
Here NFS server IP is actually logical SC IP address failing over
between 2 boxes. SUN Solaris
plays tricks here. rpcbind binds with IP 0.0.0.0 what means that it
listens on every single IP of
the server – also on logical nfs group IP. And that is OK. What is not
good is that it uses 0.0.0.0
also when sending RPC response packets resulting with source UDP
address picked up by
kernel – the one assigned according routing table (IP from smallest
not-depreciated subinterface
in appropriate direction). In my scenario NFS clients makes UDP query
to NFS server logical IP
but gets response from IP assigned to cluster’s node! Have a look at
snoop below:
a.b.c.d -> x.y.z.121 PORTMAP C GETPORT prog=100005 (MOUNT) vers=1
proto=TCP
x.y.z.111 -> a.b.c.d PORTMAP R GETPORT port=32872
This works well with clients which do not care about IP layer as long
as RPC layer
request/response sequence numbers match.
However FW is completely tricked by this – it does not consider
responses as coming within
same sessions. It may pass the response back but still will not track
conversation and
dynamically open ports for mountd and nfsd. Further client’s queries
to these ports will be
blocked.
I see 2 potential ways forward here:
1. Use static RPC program ports for additional programs required by
NFS3 (mountd and
lockd) and forget about FW whole “statefull inspection”, just
open specific ports – AFAIK
impossible in Solaris.
2. Make rpcbind behave nicely and respond on same IP it was queried –
not sure how,
maybe also impossible.
Actually I consider different source IP on RCP UDP responses as a
bug…
Wonder if there is a fix for it.
Do you know if it is properly addressed by SUN/Oracle?
Maybe some patch? Or special binaries for SUN clusters
Regards
Artur Pioro
I face a problem with NFS v. 3 service running as failover resource
group on a SUN cluster
behind firewall. Firewall’s policy is just being tightened – from now
on only NFS is supposed to
be opened. Unfortunately SUN cluster’s behavior (asymmetric IP
communication) cannot be
properly addressed by FW. Please have a look at details below.
NFS v. 3 client first of all has to make UDP query to portmapper (111)
running on the NFS server
to obtain mountd and nfsd RPC program ports. Further query made to
mountd port will provide
handle used during future connection to nfsd. AFAIB SUN’s nfsd is
always bound to specific
(well known) port (as it also provides NFS v.4 service which does not
require calls separate to
mountd), but mountd port is allocated dynamically. To deal with NFS v.
3 firewall actually has to
trace portmapper communication and dynamically open ports for mountd,
nfsd, lockd. Both
Checkpoint and Cisco PIX are capable of doing this. But they trace
sessions between particular
IPs!
Here NFS server IP is actually logical SC IP address failing over
between 2 boxes. SUN Solaris
plays tricks here. rpcbind binds with IP 0.0.0.0 what means that it
listens on every single IP of
the server – also on logical nfs group IP. And that is OK. What is not
good is that it uses 0.0.0.0
also when sending RPC response packets resulting with source UDP
address picked up by
kernel – the one assigned according routing table (IP from smallest
not-depreciated subinterface
in appropriate direction). In my scenario NFS clients makes UDP query
to NFS server logical IP
but gets response from IP assigned to cluster’s node! Have a look at
snoop below:
a.b.c.d -> x.y.z.121 PORTMAP C GETPORT prog=100005 (MOUNT) vers=1
proto=TCP
x.y.z.111 -> a.b.c.d PORTMAP R GETPORT port=32872
This works well with clients which do not care about IP layer as long
as RPC layer
request/response sequence numbers match.
However FW is completely tricked by this – it does not consider
responses as coming within
same sessions. It may pass the response back but still will not track
conversation and
dynamically open ports for mountd and nfsd. Further client’s queries
to these ports will be
blocked.
I see 2 potential ways forward here:
1. Use static RPC program ports for additional programs required by
NFS3 (mountd and
lockd) and forget about FW whole “statefull inspection”, just
open specific ports – AFAIK
impossible in Solaris.
2. Make rpcbind behave nicely and respond on same IP it was queried –
not sure how,
maybe also impossible.
Actually I consider different source IP on RCP UDP responses as a
bug…
Wonder if there is a fix for it.
Do you know if it is properly addressed by SUN/Oracle?
Maybe some patch? Or special binaries for SUN clusters
Regards
Artur Pioro