RE: DoS attack ?

[Patrik Fältström <[e-mail]>]

--On 01-12-06 10.09 -0800 Nicolas Popp <[e-mail]> wrote:

> You can also do what most search engines would.
> You return a (small) range of ranked results in the set of results and
> your last result is a referral back to you for the next range in the
> set...Then you try to detect automated crawlers that recursively follow
> the referrals and slow them down to a halt.
> 
> So, just from that standpoint, it could be useful for the protocol to
> support the notion of results set range (query) as well as referral
> (response).

We have been through this when looking at other protocols....and I would
urge you to learn from earlier mistakes (and successes).

(1) One practical path is to give in the protocol a way for the server to
say "I'm sorry, but I will not do that operation you requested. Instead I
did the following". This generic response can be "you only got 10 records
even though the result set is larger".

(2) As soon as you do "paged results", you force the server to keep state.
Depending on whether the protocol is stateful or stateless, it is harder or
easier for the server to know when to remove the cached search. Further, as
soon as you start doing pages results, you end up getting problems with
sorting the result, handling of database changes between the two fetches
(i.e. can the server re-issue the query for the second fetch, or do the
server really have to cache the result set and return the second part at
the second fetch) and million of other problems.

So, my suggestion is "don't go there".

    paf