Naslov
Deliverable 4.2: ALVIS Peers: Prototype for Distributed Retrieval

Autori
Luu, Toan ; Klemm, Fabius ; Podnar Žarko, Ivana ; Rajman, Martin ; Aberer, Karl

Izvornik
Alvis Deliverable 4.2

Vrsta, podvrsta
Ostale vrste radova, izvještaj

Godina
2006

Ključne riječi
peer-to-peer overlay networks; information retrieval

Sažetak
Web search over peer-to-peer (P2P) overlay networks shows promise to enable attractive search scenarios operating at a large scale. However, the design of effective techniques for P2P indexing and retrieval raises a number of technical challenges due to potentially unscalable bandwidth consumption, and the unavailability of global document collection statistics. In this report, we present \textsc{; ; Alvis peers}; ; , a full-text P2P retrieval engine designed to offer retrieval performance comparable to centralized solutions while scaling to a very large number of peers. To cope with problem of unscalable bandwidth consumption in the P2P network, the engine implements a novel retrieval model that indexes highly-discriminative keys (HDKs)---terms and term sets appearing in a limited number of collection documents. We have shown, both theoretically and experimentally, the proposed indexing and retrieval models are scalable in terms of generated network traffic while the retrieval performance is comparable to centralized solutions. Therefore, the novel indexing model and prototype system represent a unique contribution in the area of distributed and decentralized information retrieval. We present the architecture and detailed design of our fully-functional P2P retrieval engine. The architecture is layered: On top of the P2P layer is a distributed information retrieval layer that builds a distributed index and uses a ranking component to produce ranked answers to user queries. We present algorithms for computing the keys that are indexed by our search engine, and for maintaining a global inverted index in a completely decentralized fashion. Next, we extend the `standard' P2P layer to maintain global document collection statistics which are essential for our key-generation algorithms and ranking, and introduce special mechanisms for congestion control in the P2P overlay to efficiently handle the specific indexing load. The prototype \textsc{; ; Alvis peers}; ; can be used either as a stand-alone system when it indexes documents stored by peers forming the P2P network, or it can integrate inverted indexes produced by external engines supporting sophisticated methods for document pre-processing, e.g. the Alvis pipeline or Fedora.

Izvorni jezik
Engleski

Znanstvena područja
Računarstvo

Napomena
FP6 project "Alvis: Superpeer Semantic Search Engine"

POVEZANOST RADA