HSSA: A Novel Hybrid Shard Selection Algorithm for performance enhancement of distributed processing system

Distributed processing systems are widely used for query search operations , Where the large-size data is partitioned into different numbers of nodes for parallel processing and replication operations. In an exhaustive search approach, for a given query, all the data nodes or shards are searched to find relevant documents matching the user query. Using the sharding technique, we search selected shards to retrieve relevant data for the given query. The conventional shard selection algorithm has significant challenges: Shard ranking Shard cutoff estimation, high latency, less throughput, and high cost in processing extensive size data. Among them are CORI, ReDDe, RankS , and SHiRE are the most popular ones. The limitations of these algorithms are that the performance tends to decrease with the increasing data size, affecting search efficiency and effectiveness. To overcome these challenges, we propose a novel hybrid shard selection algorithm (HSS) to enhance search effectiveness and efficiency. The proposed HSS algorithm is designed and tested with medium and large-size datasets (Gov2, clueweb 9) considering precision, recall, and MAP performance metrics. Considering average throughput, the HSS algorithm performs 21%, 16%, and 12% better compared to CORI, Ranks , and SHiRE algorithms. Similarly, in terms of average latency, the HSS algorithm performs 14.2%, 9.4%, and 8.2% better compared to CORI, RankS , and SHiRE algorithms.