Return-Path: X-Original-To: nfsv4@core3.amsl.com Delivered-To: nfsv4@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9C3EA3A684D for ; Wed, 20 Oct 2010 09:10:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.922 X-Spam-Level: X-Spam-Status: No, score=-1.922 tagged_above=-999 required=5 tests=[AWL=0.324, BAYES_00=-2.599, HELO_EQ_DE=0.35, HTML_MESSAGE=0.001, NORMAL_HTTP_TO_IP=0.001, UNPARSEABLE_RELAY=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ucZR5h1waAAL for ; Wed, 20 Oct 2010 09:10:54 -0700 (PDT) Received: from smtp-out-3.desy.de (smtp-out-3.desy.de [131.169.56.86]) by core3.amsl.com (Postfix) with ESMTP id B0E623A68B6 for ; Wed, 20 Oct 2010 09:10:53 -0700 (PDT) Received: from smtp-map-3.desy.de (smtp-map-3.desy.de [131.169.56.68]) by smtp-out-3.desy.de (DESY_OUT_3) with ESMTP id 33DC91185 for ; Wed, 20 Oct 2010 18:12:26 +0200 (MEST) Received: from adserv72.win.desy.de (adserv72.win.desy.de [131.169.97.58]) by smtp-map-3.desy.de (DESY_MAP_3) with ESMTP id 2BADB117C for ; Wed, 20 Oct 2010 18:12:26 +0200 (MEST) Received: from smtp-intra-1.desy.de (lb-40-26.desy.de) by adserv72.win.desy.de (Clearswift SMTPRS 5.3.4) with ESMTP id ; Wed, 20 Oct 2010 18:12:25 +0200 Received: from adxf2.win.desy.de (adxf2.win.desy.de [131.169.69.230]) by smtp-intra-1.desy.de (DESY-INTRA-1) with ESMTP id BE18F3E901; Wed, 20 Oct 2010 18:12:25 +0200 (MEST) Received: from tibook.kofemann.org ([131.169.40.26]) by adxf2.win.desy.de over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Wed, 20 Oct 2010 18:12:24 +0200 Message-ID: <4CBF14F6.9080907@desy.de> Date: Wed, 20 Oct 2010 18:12:38 +0200 From: Tigran Mkrtchyan User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100914 SUSE/3.0.8 Thunderbird/3.0.8 MIME-Version: 1.0 To: nfsv4@ietf.org References: <4CBD4A05.9050105@desy.de> In-Reply-To: Content-Type: multipart/alternative; boundary="------------020603050707030804050407" X-OriginalArrivalTime: 20 Oct 2010 16:12:24.0671 (UTC) FILETIME=[952F46F0:01CB7071] Cc: Marc Eshel , f.org%IBMUS@us.ibm.com Subject: Re: [nfsv4] Current implementation status of NFSv4.1 X-BeenThere: nfsv4@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: NFSv4 Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Oct 2010 16:10:55 -0000 This is a multi-part message in MIME format. --------------020603050707030804050407 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Here is the relevant information: SETUP server: sw: dCache-1.9.10-1 on MDS and DS hw: 5x DELL R510 2x4 core 2.27 GHz, 10 Gb/s NIC (intel), 12 GB RAM, 15 TB RAID6 SATA client: sw: RHEL 5.3 + 2.6.36-rc3-pnfs-all-latest + nfs-utils-1.2.1, hw: 32x DELL M600 blades 2x4 cores 2.5 GHz, 16 GB RAM, 1 Gb/s NIC mount options:/* */*/-t nfs4 -o minorversion=1,rsize=32768,wsize=32768/* network: hw: Force 10 Gbit switch In total: 256 cores vs. 5 DS + 1 MDS We run 8248 typical analysis batch jobs processing 33 TB of data. It took 4 days with ~330 MB/s. No crashes. We written in 10 days ~13 TB data from Stanford University to Hamburg with FTP client on a single node. Each file was between 100-200 GB. No crashes. With 50 parallel jobs we get up to network limit. With mo jobs performance started to degrade. I suspect, that there are no enough io thread in our server ( only 5 per DS ). Our admins say that we hit the disk limits on a single server as we do not stripe the files. This have to be sill investigated. All clients committed a group suicide after attempt to write into a single file (this was triggered by operational mistake). As this is not supported by our server I suspect that IOERROR or LAYOUT_TRYLATER error was returned. Unfortunately this happened during my vacation and I was not able to look at it. We have declared test success full. In total we have seen quite a good performance and reasonable stability. We was able to show our users that NFSv4.1 is real and can be used even if it's still experimental. We look forward to do similar test with some of the vendor systems as well. Regards, Tigran. P.S: here is the link to original presentation, but it full of 'foreign' words. http://117.103.105.177/MaKaC/materialDisplay.py?contribId=217&sessionId=60&materialId=slides&confId=3 On 10/19/2010 05:44 PM, Marc Eshel wrote: > Hi Tigran, > We are all interested, can you post it to this email list. > Thanks, Marc. > > > > From: > "William A. (Andy) Adamson" > To: > Tigran Mkrtchyan > Cc: > nfsv4@ietf.org > Date: > 10/19/2010 06:16 AM > Subject: > Re: [nfsv4] Current implementation status of NFSv4.1 > Sent by: > nfsv4-bounces@ietf.org > > > > On Tue, Oct 19, 2010 at 3:34 AM, Tigran Mkrtchyan > wrote: > >> We did extensive pNFS test last couple of mounts which we present this >> > week > >> at Conference on Computing in High Energy and Nuclear Physics in Taipei. >> I can provide numbers and slides if you need. >> > > Hi Tigran > > Very interesting - please forward them to me. > > Thanks > > -->Andy > > >> Regards, >> Tigran >> >> On 10/18/2010 10:30 PM, Spencer Shepler wrote: >> >> I am curious if anyone is interested and able to do a >> presentation at the upcoming WG meeting on 4.1 implementation >> status? Given the nature of the interop testing (NDAs) I am >> not asking for an industry report but more of an experience >> so far and status for single implementations. Covering >> things like: functionality implemented, issues found, issues >> unresolved, areas of concern, lessons learned for future work. >> That type of thing. >> >> Anyone interested? >> >> Thanks, >> Spencer >> >> p.s. Obviously, this could include NFSv4.0 as well. :-) >> _______________________________________________ >> nfsv4 mailing list >> nfsv4@ietf.org >> https://www.ietf.org/mailman/listinfo/nfsv4 >> >> >> _______________________________________________ >> nfsv4 mailing list >> nfsv4@ietf.org >> https://www.ietf.org/mailman/listinfo/nfsv4 >> >> >> > _______________________________________________ > nfsv4 mailing list > nfsv4@ietf.org > https://www.ietf.org/mailman/listinfo/nfsv4 > > > > --------------020603050707030804050407 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Here is the relevant information:

SETUP
 
server:
   sw: dCache-1.9.10-1 on MDS and DS
   hw: 5x DELL R510 2x4 core 2.27 GHz, 10 Gb/s NIC (intel), 12 GB RAM,  15 TB RAID6 SATA

client:
  sw: RHEL 5.3 + 2.6.36-rc3-pnfs-all-latest + nfs-utils-1.2.1,
  hw: 32x DELL M600 blades  2x4 cores 2.5 GHz, 16 GB RAM, 1 Gb/s NIC
  mount options: -t nfs4 -o minorversion=1,rsize=32768,wsize=32768

network:
   hw: Force 10 Gbit switch

In total: 256 cores vs. 5 DS + 1 MDS

We run 8248  typical analysis batch jobs processing 33 TB of data.
It took 4 days with ~330 MB/s. No crashes.

We written in 10 days ~13 TB data from Stanford University to Hamburg with FTP
client on a single node. Each file was between 100-200 GB. No crashes.

With 50 parallel jobs we get up to network limit. With mo jobs performance
started to degrade. I suspect, that there are no enough io thread in our server
( only 5 per DS ). Our admins say that we hit the disk limits on a single server
as we do not stripe the files. This have to be sill investigated.

All clients committed a group suicide after attempt to write into a single file
(this was triggered by operational mistake). As this is not supported by our
server I suspect that IOERROR or LAYOUT_TRYLATER error was returned.
Unfortunately this happened during my vacation and I was not able to look at it.

We have declared test success full. In total we have seen quite a good performance
and reasonable stability.  We was able to show our users that NFSv4.1 is real and
can be used even if it's still experimental.

We look forward to do similar test with some of the vendor systems as well.

Regards,
   Tigran.

P.S: here is the link to original presentation, but it full of 'foreign' words.
http://117.103.105.177/MaKaC/materialDisplay.py?contribId=217&sessionId=60&materialId=slides&confId=3

On 10/19/2010 05:44 PM, Marc Eshel wrote: 
Hi Tigran,
We are all interested, can you post it to this email list.
Thanks, Marc.



From:
"William A. (Andy) Adamson" <androsadamson@gmail.com>
To:
Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
Cc:
nfsv4@ietf.org
Date:
10/19/2010 06:16 AM
Subject:
Re: [nfsv4] Current implementation status of NFSv4.1
Sent by:
nfsv4-bounces@ietf.org



On Tue, Oct 19, 2010 at 3:34 AM, Tigran Mkrtchyan
<tigran.mkrtchyan@desy.de> wrote:
  
We did extensive pNFS test last couple of mounts which we present this 
    
week
  
at Conference on Computing in High Energy and Nuclear Physics in Taipei.
I can provide numbers and slides if you need.
    

Hi Tigran

Very interesting - please forward them to me.

Thanks

-->Andy

  
Regards,
   Tigran

On 10/18/2010 10:30 PM, Spencer Shepler wrote:

I am curious if anyone is interested and able to do a
presentation at the upcoming WG meeting on 4.1 implementation
status?  Given the nature of the interop testing (NDAs) I am
not asking for an industry report but more of an experience
so far and status for single implementations.  Covering
things like: functionality implemented, issues found, issues
unresolved, areas of concern, lessons learned for future work.
That type of thing.

Anyone interested?

Thanks,
Spencer

p.s. Obviously, this could include NFSv4.0 as well. :-)
_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4


_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4


    
_______________________________________________
nfsv4 mailing list
nfsv4@ietf.org
https://www.ietf.org/mailman/listinfo/nfsv4

--------------020603050707030804050407--