Re: [hybi] Background info: Properties of sendfile()

Roberto Peon <fenix@google.com> Fri, 06 August 2010 16:33 UTC

Return-Path: <fenix@google.com>
X-Original-To: hybi@core3.amsl.com
Delivered-To: hybi@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E70F53A6AED for <hybi@core3.amsl.com>; Fri, 6 Aug 2010 09:33:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.85
X-Spam-Level:
X-Spam-Status: No, score=-101.85 tagged_above=-999 required=5 tests=[AWL=0.126, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3lZWfNs98vIW for <hybi@core3.amsl.com>; Fri, 6 Aug 2010 09:33:00 -0700 (PDT)
Received: from smtp-out.google.com (smtp-out.google.com [74.125.121.35]) by core3.amsl.com (Postfix) with ESMTP id 002E33A68EC for <hybi@ietf.org>; Fri, 6 Aug 2010 09:32:59 -0700 (PDT)
Received: from hpaq6.eem.corp.google.com (hpaq6.eem.corp.google.com [172.25.149.6]) by smtp-out.google.com with ESMTP id o76GXUXo015892 for <hybi@ietf.org>; Fri, 6 Aug 2010 09:33:30 -0700
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=google.com; s=beta; t=1281112410; bh=t+dH9lwBw+8ONJ+C2HnmE0wZVHc=; h=MIME-Version:In-Reply-To:References:Date:Message-ID:Subject:From: To:Cc:Content-Type; b=OVA4FQ+r616iuUjTFKqE8sWUhByEAYEbVTMFUbn+iwJt7xG9E7O/zN0jeNkIuwonf BPRFWRtYWMzi/+3zbILKg==
DomainKey-Signature: a=rsa-sha1; s=beta; d=google.com; c=nofws; q=dns; h=mime-version:in-reply-to:references:date:message-id:subject:from:to: cc:content-type:x-system-of-record; b=ikEotd03OCyoAEGt4Xf6uxmZAm7w4Mu2gultfrcNOe+doGmtYj4LAl+ZcrbEIc7Xe 6vqOdmcfX1/aM6+s1PFZg==
Received: from ywa6 (ywa6.prod.google.com [10.192.1.6]) by hpaq6.eem.corp.google.com with ESMTP id o76GXHX7008047 for <hybi@ietf.org>; Fri, 6 Aug 2010 09:33:29 -0700
Received: by ywa6 with SMTP id 6so4131505ywa.37 for <hybi@ietf.org>; Fri, 06 Aug 2010 09:33:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.151.92.4 with SMTP id u4mr13643633ybl.395.1281112408336; Fri, 06 Aug 2010 09:33:28 -0700 (PDT)
Received: by 10.150.59.4 with HTTP; Fri, 6 Aug 2010 09:33:28 -0700 (PDT)
In-Reply-To: <AANLkTimUkx8_Ssiuh4vJsgiQb2xmUzr5uAb-TgG=bjyh@mail.gmail.com>
References: <4C5B1695.6070704@gmx.de> <F8E2F702-9F74-4316-B3B2-D5A731409ABF@apple.com> <4C5B2029.90403@gmx.de> <AANLkTim1WeCRfcPxXUNQcVhb4+t_TtDQDv2bXaxOQ=bk@mail.gmail.com> <01098AD0-FBF4-4A61-B565-947C95722BAA@apple.com> <AANLkTi=qQSND5BvUP5+P=wJ7E8SG6NncGZH8U8+VYwZ0@mail.gmail.com> <20100806004907.GF27827@shareable.org> <C0FC87B7-C51C-4B36-BC16-DBDB0B00A20F@gbiv.com> <20100806012845.GI27827@shareable.org> <AANLkTimuvuj87qwQi_1Gjg47-gGrCsDCQ5TDd5zvOw1N@mail.gmail.com> <20100806062928.GF20057@1wt.eu> <AANLkTimUkx8_Ssiuh4vJsgiQb2xmUzr5uAb-TgG=bjyh@mail.gmail.com>
Date: Fri, 06 Aug 2010 09:33:28 -0700
Message-ID: <AANLkTikxM2fOn_rh6tLjBsmSF_6PqKqAvjGqb_+T_+GY@mail.gmail.com>
From: Roberto Peon <fenix@google.com>
To: Jack Moffitt <jack@collecta.com>
Content-Type: multipart/alternative; boundary="000e0cd240c6042af7048d2a3a83"
X-System-Of-Record: true
Cc: "Roy T. Fielding" <fielding@gbiv.com>, "hybi@ietf.org" <hybi@ietf.org>
Subject: Re: [hybi] Background info: Properties of sendfile()
X-BeenThere: hybi@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Server-Initiated HTTP <hybi.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/hybi>
List-Post: <mailto:hybi@ietf.org>
List-Help: <mailto:hybi-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/hybi>, <mailto:hybi-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 06 Aug 2010 16:33:03 -0000

sendfile() doesn't require sending the entire file.
At some frame size, the overhead due to interleaving sendfile() with write()
should be negligible.
I don't think that sendfile() is a good reason to require total message
length up-front.
I still like having the total-message-length whenever possible-- it does
allow for a quick decision about where to store or process the data.
-=R

On Fri, Aug 6, 2010 at 6:49 AM, Jack Moffitt <jack@collecta.com> wrote:

> > At least I've seen it. Haproxy happily forwards traffic at 10 Gbps
> > with only 25% of one CPU core on a 2.66 GHz Core2Duo using splice().
> > And some file sharing sites make a big use of it ! Without splice,
> > it hardly achieves 10 Gbps at 100% CPU.
>
> It's not clear how much of the benefit we'd lose with fixed size
> lengths (and therefore mandatory chunking). At this point, it seems
> that kind of data would be valuable.
>
> The tradeoff *seems* to be simple intermediaries versus
> sendfile()/splice() efficiency. If sendfile() is severely affected, it
> may make sense to make the intermediaries life slightly more
> complicated. If not, I think we keep their job simple.
>
> jack.
>