[Din] Rorur : decentralized search engine

[Stan Srednyak <[e-mail]>]

Dear colleagues,

This is an update from rorur.com, the distributed search engine project.

We successfully implemented basic functionality of the search engine - data
crawl, indexing , ranking and query service and tested it on moderate size
clusters on AWS.

Now we would like to move to the next stage of decentralizing the web
search which would consist in full web crawl and data analysis. This is a
computation for which we do not have resources. We estimate ( see below)
that for a meaningful operation we would need ~100 8CPU machines with 2TB
SSD and Gb pipe. If you are interested, join us in this effort to construct
a decentralized open source search engine.

We have hardware partners who can provide the necessary servers distributed
worldwide. At the moment, they charge 300$/month per server , BW included.
Of course you can run your own node, as the network is decentralized and
free to join. You can run smaller machines, there is in fact no minimum
hardware requirement. You can run it from your laptop in the background.

There are also multiple algorithmic and programming issues on which our
team is working. They are mostly centered around distributed verifiable
computing with web data with the purpose of constructing distributed
versions of knowledge graphs, as the latter are known to be of central
importance for high quality search. If you have the qualifications to carry
out such research you are welcome to contact us.


Feasibility analysis:

Our code can be successfully run on an 8CPU machine with an SSD. With a Gb
connection we saw performance at 100 pages/second for crawl and analysis.
Given ~5*10^10 pages on the web we estimate that with 100 such machines the
crawl can be completed in ~2months. According to estimates from Common
Crawl, the total indexable data ~ few PB. We perform data cleaning which
includes removal of JS and non-English characters. This results in at least
10-fold reduction in size ( we typically see substantially more). This
brings the size of cleaned data ~ few hundred TB. The index size is
comparable to the data. Rank data is usually much smaller, for the ranks
that we tried. We estimate that 1-200TB total disk space should be enough
for a proof of principle demo.



best regards,
Stan Srednyak