RT doctoral thesis
T1 Navigating diverse datasets in the face of uncertainty
A1 Álvarez Ayllón, Alejandro
A2 Ingeniería Informática
K1 Big Data
K1 Data analysis
K1 Equally-Distributed Dependencies
K1 Self-Organizing Maps
AB When exploring big volumes of data, one of the challenging aspects is their diversityof origin. Multiple files that have not yet been ingested into a database system maycontain information of interest to a researcher, who must curate, understand and sievetheir content before being able to extract knowledge.Performance is one of the greatest difficulties in exploring these datasets. On theone hand, examining non-indexed, unprocessed files can be inefficient. On the otherhand, any processing before its understanding introduces latency and potentially unnecessary work if the chosen schema matches poorly the data. We have surveyed thestate-of-the-art and, fortunately, there exist multiple proposal of solutions to handledata in-situ performantly.Another major difficulty is matching files from multiple origins since their schemaand layout may not be compatible or properly documented. Most surveyed solutionsoverlook this problem, especially for numeric, uncertain data, as is typical in fieldslike astronomy.The main objective of our research is to assist data scientists during the explorationof unprocessed, numerical, raw data distributed across multiple files based solely onits intrinsic distribution.In this thesis, we first introduce the concept of Equally-Distributed Dependencies,which provides the foundations to match this kind of dataset. We propose PresQ,a novel algorithm that finds quasi-cliques on hypergraphs based on their expectedstatistical properties. The probabilistic approach of PresQ can be successfully exploited to mine EDD between diverse datasets when the underlying populations canbe assumed to be the same.Finally, we propose a two-sample statistical test based on Self-Organizing Maps(SOM). This method can outperform, in terms of power, other classifier-based twosample tests, being in some cases comparable to kernel-based methods, with theadvantage of being interpretable.Both PresQ and the SOM-based statistical test can provide insights that driveserendipitous discoveries.
YR 2023
FD 2023
LK http://hdl.handle.net/10498/30338
UL http://hdl.handle.net/10498/30338
LA eng
DS Repositorio Institucional de la Universidad de Cádiz
RD 05-nov-2026