Re: [hybi] Is there a traffic jam?

Jamie Lokier <> Tue, 14 April 2009 17:44 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 469A43A6E35 for <>; Tue, 14 Apr 2009 10:44:01 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -3.849
X-Spam-Status: No, score=-3.849 tagged_above=-999 required=5 tests=[AWL=-1.250, BAYES_00=-2.599]
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id tL3iOjFp7d1a for <>; Tue, 14 Apr 2009 10:44:00 -0700 (PDT)
Received: from ( []) by (Postfix) with ESMTP id 42B4928C12E for <>; Tue, 14 Apr 2009 10:44:00 -0700 (PDT)
Received: from jamie by with local (Exim 4.63) (envelope-from <>) id 1Ltmh3-0000b0-Of; Tue, 14 Apr 2009 18:45:09 +0100
Date: Tue, 14 Apr 2009 18:45:09 +0100
From: Jamie Lokier <>
To: Mario Balibrera <>
Message-ID: <>
References: <> <> <> <> <> <> <> <> <> <>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <>
User-Agent: Mutt/1.5.13 (2006-08-11)
Subject: Re: [hybi] Is there a traffic jam?
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Server-Initiated HTTP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 14 Apr 2009 17:44:01 -0000

Mario Balibrera wrote:
> We don't have to create something incompatible with existing
> frameworks, but the very nature of CGI is in diametric opposition to
> the concept of a streaming protocol.

No.  It's different, but there are ways to bring them quite close
together for a lot of practical applications.  It's far from
diametrically opposed, you just need good glue in the middle to hide
the protocol differences.

> It's clear that people working in CGI-like frameworks will _have_ to
> change the way they program, because they will be writing
> applications that are fundamentally different.

I disagree.  Some applications must be structured differently to work
well.  But many don't need radical changes, especially when efficiency
at the server is not a great concern.

For example, an application with:

    - Data stored in a database and/or files.
    - A HTML templating system.
    - A template containing HTML, references to other templates,
      and snippets of code generating some HTML based on the stored data.
    - The output must appear as either a whole web page, or embedded
      in part of a larger one.

That comprises probably 99%+ of dynamically generated web pages at the
moment (which don't update themselves).  Nearly everything which isn't
an interactive form fits that pattern.

Now you, the (CGI-level :-) programmer, are asked to do the same, but
it must update changed parts of the page in the browser, immediately
when the stored data updates.  (I'd say a lot of collaborative
editing, chatting and data presentation falls into that category.
Many widgets could do.  Are we still at 99%?)

Despite the use of asynchronous server events on the wire, it's
possible to implement that in CGI with the right glue.

In fact it's possible in some cases without changing the CGI script.

I'm told Microsoft already has a version of ASP.NET which does it by
regenerating whole pages on the server side and comparing with
previous output to decide when to send an update, and what to send.  I
don't know if it's smart enough to figure out _when_ to recalculate
those pages.

There are cleverer ways to do it with less calculation and only when
the requisite data changes, while still structuring the code as if
it's classic CGI and templating + database/file queries.

I'm not saying all applications should be written that way, but when
it works and especially when it works efficiently - why not?  It's a
good way to write applications when it works.

> Note that different != harder. Anyone in the world can write a CGI script
> because anyone in the world can type "cgi script" into google and get a
> thousand code snippets to paste in. Writing actual network applications is a
> big secret only because no one's writing the tutorials. (We should start).

I expect the same snippets as soon as we have CGI scripts (+ Python,
Java frameworks etc.) which do clever two-way interactive things with

The only difference will be "this script requires hybi-CGI support in
the web server for the dynamic parts to work" or whatever the
framework is called.

If people would like more discussion on techniques for client update
which don't require long-running server-side applications at the CGI
level (only the web server itself must manage those connections on
behalf of the application), I could expand on it but I expect there's
plenty of people here who have implemented that sort of thing already,
and the web isn't short of comet tutorials.

-- Jamie