RE: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

cburnett@us.ibm.com

Subject: RE: [nfsv4] Out of sync stateid scenario that looks difficult to resolve
Message-ID: <OFEBB385C1.C476A3F2-ON87256D9F.005A0C43@us.ibm.com>
From: Carl Burnett <cburnett@us.ibm.com>
Date: Fri, 12 Sep 2003 11:46:31 -0500

See below.

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to cburnett@us.ibm.com)

"Noveck, Dave" <Dave.Noveck@netapp.com>
Sent by: nfsv4-admin@ietf.org
09/12/2003 11:08 AM

        To:     Carl Burnett/Austin/IBM@IBMUS, "Trond Myklebust" 
<trond.myklebust@fys.uio.no>
        cc:     <nfsv4@ietf.org>, <nfsv4-admin@ietf.org>, <spencer.shepler@sun.com>
        Subject:        RE: [nfsv4] Out of sync stateid scenario that looks difficult to resolve

> Just as food for thought, I think removing open share reservations from 
> NFS V4 would go along ways towards allowing a significantly more 
scalable 
> protocol. Are share reservations really worth the scalability issues 
they 
> present? Think about it. Locking and delegations could still be 
supported, 
> but with a different bootstrapping mechanism.

I don't think we can go there.  While the minor versioining rules
do allow us to remove features, getting a consensus on removing them
tends to be very hard (i.e. virtually impossible).  Even if you conclude
that shaer reservations are not worth the problems, all that gets you 
is to a conclusion that they shouldn't have been put in in the first 
place.

[CARL: I know, I couldn't help mentioning it anyway - sorry]

I do believe that we can get rid of the problems that share reservation
present, while still retaining the feature.  The problem with the
current approach is that all clients, even those that have no use
for share reservations, have to pay the price.  The reason is that
open_owner's currently provide two different functions: they serve
to distinguish the different owners of share reservations (they 
perform a lock_owner-like function for share reservations), and 
they serve to identify a seqid space for the replay protection
function.

[CARL: I agree there is future hope, but it does not help the
 pains of implementing RFC 3530 as defined.]

If you don't use share reservations, you don't need the first of
these functions, since one open_owner per client would serve as far
as that funcion goes.  The problem is that you still have that 
nasty second function so you still need open_owners.

[CARL: I have often pondered if it would be better for a
 UNIX client to not use OPENS, and instead use the special
 stateids until byte range locks are required. That presents
 issues in the form of unexpected application errors if
 there happen to be open_deny modes in use somewhere. Also,
 delegations can't be exploited without OPENS, which I view
 as unfortunate.
]

If we are talking about v4.1 however, the session extensions 
proposal can rescue us here.  That puts all of the replay protection
is its own layer, and therefore you don't need open_owner's and
associated sequence id's for replay protection, so the only use 
for open_owner's would be for those clients who want and need share 
reservations.  This requires some further discussion but I'm pretty 
sure it could also be the basis for eliminating the annoying stateid 
sequence increment that causes problems with interrupted requests. 

[CARL: yes I agree. I believe the sessions stuff
 is really going to be necessary to build an NFS that can truly
 be deployed in the real world. You have to get to a model
 that allows reasonable call concurrency with a limited number of
 state anchors (owners) and light weight server management of
 the state. In the simplest case, you would like the anchors to
 reduce down to the session and probably the clientid.
]

-----Original Message-----
From: Carl Burnett [mailto:cburnett@us.ibm.com]
Sent: Friday, September 12, 2003 10:42 AM
To: Trond Myklebust
Cc: nfsv4@ietf.org; nfsv4-admin@ietf.org; spencer.shepler@sun.com
Subject: Re: [nfsv4] Out of sync stateid scenario that looks difficult
to resolve

Trond writes:

"Are you perhaps assuming that open_owner == posix file handle (as per
the recommendation in the RFC). That is a concept which doesn't appear
to scale on the server side. We have already hit problems with it
under testing against existing commercial NFSv4 servers which quickly
run out of resources...

For that reason, we'd prefer to use a limited number of open_owners
(say related to the number of allowed simultaneous RPC requests on the
wire) which we share among all the users."

I completely agree that a model where every file object has a unique 
open_owner is non-scalable. We too believe in an approach where a pool 
owners can be shared among many objects. We do however continue to find 
new challenges with this approach that keep increasing the complexity of 
the client implementation. Even with a pool model, IMHO, its going to take 

some pretty serious NFS server hardware to serve V4 at any level of scale.

Just as food for thought, I think removing open share reservations from 
NFS V4 would go along ways towards allowing a significantly more scalable 
protocol. Are share reservations really worth the scalability issues they 
present? Think about it. Locking and delegations could still be supported, 

but with a different bootstrapping mechanism.

Trond also writes:

"Now I am curious, though. Where in the RFC does it say explicitly that
OPEN will invalidate an existing stateid, and why should it be necessary?"

I think this comes from the property, that within the open stateid, there 
is a seqid that the server should increment each time it processes an 
operation that uses the stateid. Thus the stateid continuely moves forward 

in time. As Spencer hinted, this is an addtion to any info within the 
opaque portion of the id that the server is managing, which could change 
with each use of the stateid as well.

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to cburnett@us.ibm.com)

Trond Myklebust <trond.myklebust@fys.uio.no>
Sent by: nfsv4-admin@ietf.org
09/11/2003 07:49 PM

        To:     spencer.shepler@sun.com
        cc:     nfsv4@ietf.org
        Subject:        Re: [nfsv4] Out of sync stateid scenario that 
looks difficult to resolve

>>>>> " " == Spencer Shepler <spencer.shepler@sun.com> writes:

    >> So now I'm confused. Does this also mean that you cannot
    >> upgrade/downgrade the open stateid if there is already a lock
    >> stateid hanging off it?

     > For a posix locking scheme, that doesn't apply.  The first
     > close() for a file will release all file locks (if they are
     > left) and therefore the lock stateid is no longer needed.

Are you perhaps assuming that open_owner == posix file handle (as per
the recommendation in the RFC). That is a concept which doesn't appear
to scale on the server side. We have already hit problems with it
under testing against existing commercial NFSv4 servers which quickly
run out of resources...

For that reason, we'd prefer to use a limited number of open_owners
(say related to the number of allowed simultaneous RPC requests on the
wire) which we share among all the users.

Now I am curious, though. Where in the RFC does it say explicitly that
OPEN will invalidate an existing stateid, and why should it be necessary?

Cheers,
  Trond

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www1.ietf.org/mailman/listinfo/nfsv4