[Din] distributed search engine

[Stan Srednyak <[e-mail]>]

hi DINRG,

It seems that there is a high demand for development of decentralized
Internet. While decentralization must address many issues, in particular,
social networks, it is of utmost importance to develop decentralized search
engines, as search engines are the gateway to the web. Search engine
industry has been monopolized by a few large companies. While there are
many negative impacts that this centralization has had ( i.e., manipulation
of the rank), the one that is particularly conspicuous is the secrecy of
the ranking algorithms. It is highly desirable to have open ranking
algorithms and allow the users to choose from a variety of algorithms.

Some time ago I started working on the design and implementation of a
decentralized search engine. The basic idea of my approach is that while
computational power needed to realize a distributed search engine is
immense, it is possible to design a communication protocol that would
orchestrate data collection, analysis, ranking, and serving search queries
to users and split the work load among participating nodes. The
participating nodes in my design are computers of ordinary internet users.
I have developed corresponding algorithms that are necessary to realize
search and ranking operations on a distributed network of user computers.
One of the challenges lies in achieving acceptable latency (<1 second) in
serving search transactions. According to my estimates, it is possible to
achieve latency comparable to the existing search engines. In addition, I
have shown that it is possible to guarantee the computation and delivery of
the true rank (the one that is actually being requested by the user) to the
end users ( of course, there is the obvious problem that in decentralized
architectures the nodes may try to manipulate the rank, and rank some pages
unjustifiably high or low. Nonetheless, it is possible to design a network
communication protocol in such a way that it is highly improbable that
malicious nodes can manipulate the rank, as long as their total fraction is
below a certain threshold).

 Some of the details of the project can be found at https://rorur.com. To
incentivize people to maintain "search nodes" ( analogously to Ethereum
nodes), I proposed an architecture that allows individuals and companies to
advertise on this network, quite analogously to what is done on the usual
search engines, with the difference that the revenue is distributed to the
node maintainers. There are some details on how to achieve this in a secure
fashion, and some of them can be found on the site linked above. I will be
rolling out the first stage of this project quite soon, and I would like to
know if there is any interest in this project. Of course , this has to be a
collaborative project. It is impossible to run it on individual hardware (
although it is possible to deploy it on a centralized data center). There
will be several stages in the deployment, in particular, several versions
of the communication protocol. I will detail on these stages in a
forthcoming publication. There are various roles you can participate in the
project, from maintaining a node, to software development, to algorithm
design. I will be very happy to hear from you. I think this project ties in
really well with the spirit of this group and more generally, with the
spirit of IETF. As explained in the white paper, this project, if
successful, can lead to the transformation of the web into a "knowledge
system". There is a large discussion that is necessary here, but to make it
brief, it may allow for creation of personalized search and personal
knowledge graphs. It can also be instrumental in creating more robust
Internet infrastructure. I will try to develop this project in close
collaboration with the IETF community, because the issues it addresses have
to do with fundamental aspects of the web, at the level of protocols and
data routing. If I am not mistaken in my calculations, distributed search
operations can be added on top of the standard protocol stack and thus
become part of everyday web operation.



best regards,
Stan Srednayk