RE: DoS attack ?

[Nicolas Popp <[e-mail]>]

>(1)One practical path is to give in the protocol a way for the server to
>say "I'm sorry 

>(2) As soon as you do "paged results", you force the server to keep state.

No. You simply refetch (the referral query sends me back all the facets and
the new range). All large search engines are stateless and handle paged
results. Their caching strategies is typically based on query keyword
frequency (the cache is built as indexing time based on the query
distribution, more than keeping the "last requests" (they actually do a
little bit of that too). Since they are stateless, their response is
consistent with the query (at any time their return the set of results for
the query within the specified range, but that set is not immutable. The
fact that the index is slowing changing makes that strategy acceptable from
a user perspective).

Having said that I don't disagree with (1). I just think you need to do
both.

-Nico



-----Original Message-----
From: Patrik Fältström [mailto:[e-mail]]
Sent: Thursday, December 06, 2001 10:34 PM
To: Nicolas Popp; 'John C Klensin'; YangWoo Ko
Cc: [e-mail]
Subject: RE: DoS attack ?


--On 01-12-06 10.09 -0800 Nicolas Popp <[e-mail]> wrote:

> You can also do what most search engines would.
> You return a (small) range of ranked results in the set of results and
> your last result is a referral back to you for the next range in the
> set...Then you try to detect automated crawlers that recursively follow
> the referrals and slow them down to a halt.
> 
> So, just from that standpoint, it could be useful for the protocol to
> support the notion of results set range (query) as well as referral
> (response).

We have been through this when looking at other protocols....and I would
urge you to learn from earlier mistakes (and successes).

(1) One practical path is to give in the protocol a way for the server to
say "I'm sorry, but I will not do that operation you requested. Instead I
did the following". This generic response can be "you only got 10 records
even though the result set is larger".

(2) As soon as you do "paged results", you force the server to keep state.
Depending on whether the protocol is stateful or stateless, it is harder or
easier for the server to know when to remove the cached search. Further, as
soon as you start doing pages results, you end up getting problems with
sorting the result, handling of database changes between the two fetches
(i.e. can the server re-issue the query for the second fetch, or do the
server really have to cache the result set and return the second part at
the second fetch) and million of other problems.

So, my suggestion is "don't go there".

    paf