RE: DoS attack ?

[John C Klensin <[e-mail]>]

--On Thursday, 06 December, 2001 10:09 -0800 Nicolas Popp
<[e-mail]> wrote:

> You can also do what most search engines would.
> You return a (small) range of ranked results in the set of
> results and your last result is a referral back to you for the
> next range in the set...Then you try to detect automated
> crawlers that recursively follow the referrals and slow them
> down to a halt.
> 
> So, just from that standpoint, it could be useful for the
> protocol to support the notion of results set range (query) as
> well as referral (response).

Brief observations:

Yeech.  

Been there and done that.  Does not scale well to VLDBs.
Returning of resulted based on sorted ranking doesn't either,
unless you retrieve everything (or high-performance record
pointers to everything) on the server, sort things out there,
and then, for efficiency, return part of it and cache everything
else.  If you are trying to conserve bandwidth or improve user
presentation speed, this works (although it still doesn't scale
terribly well).  If you are trying to conserve server resources,
it is usually bad news.

<Incorporate several observations about race conditions and
server-based search state here>

How would you feel about getting back an randomly-chosen subset
of specified maximum size each time, sampled with replacement ?
:-)

Let's discuss next week.

    john