Exploiting Sharing Join Opportunities in Big Data Multiquery Optimization with Flink

Complexity 2020:1-25 (2020)

Abstract

Multiway join queries incur high-cost I/Os operations over large-scale data. Exploiting sharing join opportunities among multiple multiway joins could be beneficial to reduce query execution time and shuffled intermediate data. Although multiway join optimization has been carried out in MapReduce, different design principles are not considered. To bridge the gap of not considering the optimization of Big Data platforms, an end-to-end multiway join over Flink, which is called Join-MOTH system, is proposed to exploit sharing data granularity, sharing join granularity, and sharing implicit sorts within multiple join queries. For sharing data, our previous work, Multiquery Optimization using Tuple Size and Histogram system, has been introduced to consider the granularity of sharing data opportunities among multiple queries. For sharing sort, our previous work, Sort-Based Optimizer for Big Data Multiquery, has been introduced to consider the implicit sorts among join queries. For sharing join, additional modules have been tailored to the J-MOTH optimizer to optimize sharing work by exploiting shared pipelined multiway join among multiple multiway join queries. The experimental evaluation has demonstrated that the J-MOTH system outperforms the naive and the state-of-the-art techniques by 44% for query execution time using TPC-H queries. Also, the proposed J-MOTH system introduces maximal intermediate data size reduction by 30% in average over Hadoop-like infrastructures.

Download options

PhilArchive



    Upload a copy of this work     Papers currently archived: 72,891

External links

Setup an account with your affiliations in order to access resources via your University's proxy server

Through your library

Analytics

Added to PP
2020-12-22

Downloads
3 (#1,369,006)

6 months
1 (#386,040)

Historical graph of downloads
How can I increase my downloads?

Author's Profile

References found in this work

No references found.

Add more references

Citations of this work

No citations found.

Add more citations

Similar books and articles

Openness in the Social Sciences: Sharing Data.Joan E. Sieber - 1991 - Ethics and Behavior 1 (2):69 – 86.
What Should Be the Data Sharing Policy of Cognitive Science?Mark A. Pitt & Yun Tang - 2013 - Topics in Cognitive Science 5 (1):214-221.
Cracking the Code: Using Data to Combat the Opioid Crisis.Catherine Martinez - 2018 - Journal of Law, Medicine and Ethics 46 (2):454-471.
Data Sharing in the Context of Health-Related Citizen Science.Mary A. Majumder & Amy L. McGuire - 2020 - Journal of Law, Medicine and Ethics 48 (S1):167-177.