Analytical Processing Systems ‒ DIAS ‐ EPFL

The ongoing data explosion necessitates that the database software efficiently uses the available hardware and exploits data properties, to enable timely business intelligence. Additionally, data and hardware become increasingly heterogeneous: modern servers are adopting a variety of hardware accelerators to increase their energy efficiency while data scientists analyze a wide variety of heterogeneous datasets to gain insights. In this line of work, we are enabling query engines to adapt to the available hardware and data formats as well as automatically exploit domain-specific properties by generating specialized query engines on demand, achieving the performance of specialized engines without the extra development effort and time.

Analytical engines with Context-Rich Processing

As modern data pipelines continue to collect, produce, and store a variety of data formats, extracting and combining value from traditional and context-rich sources such as strings, text, video, audio, and logs becomes a manual process where such formats are unsuitable for RDBMS. To tap into the dark data, domain experts analyze and extract insights and integrate them into the data repositories. This process can involve out-of-DBMS, ad-hoc analysis, and processing, resulting in ETL, engineering effort, and suboptimal performance. While AI systems based on ML models can automate the analysis process, they often further generate context-rich answers. Using multiple sources of truth, for either training the models or in the form of knowledge bases, further exacerbates the problem of consolidating the data of interest.

We envision an analytical engine co-optimized with components that enable context-rich analysis. Firstly, as the data from different sources or resulting from model answering cannot be cleaned ahead of time, we propose using online data integration via model-assisted similarity operations. Secondly, we aim for a holistic pipeline cost- and rule-based optimization across relational and model-based operators. Thirdly, with increasingly heterogeneous hardware and equally heterogeneous workloads ranging from traditional relational analytics to generative model inference, we envision a system that just-in-time adapts to the complex analytical query requirements. To solve increasingly complex analytical problems, ML offers attractive solutions that must be combined with traditional analytical processing and benefit from decades of database community research to achieve scalability and performance effortless for the end user.

Query Approximation

Modern analytical engines rely on Approximate Query Processing (AQP) to provide faster response times than the hardware allows for exact query answering. However, existing AQP methods impose steep performance penalties as workload unpredictability increases. Specifically, offline AQP relies on predictable workloads to create samples that match the queries in a priori to query execution, providing reductions in query response times when queries match the expected workload. As soon as workload predictability diminishes, existing online AQP methods create query-specific samples with little reuse across queries, producing significantly smaller gains in response times. As a result, existing approaches cannot fully exploit the benefits of sampling under increased unpredictability.

At DIAS we analyze sample creation and propose a framework for building, expanding, and merging samples to adapt to the changes in workload predicates. Our framework speeds up online sampling processing as a function of sample reuse, ranging from practically zero to full online sampling time and from 2.5x to 19.3x in a simulated exploratory workload.

Data Cleaning

Data cleaning has become an indispensable part of data analysis due to the increasing amount of dirty data. Data scientists spend most of their time preparing dirty data before it can be used for data analysis. At the same time, the existing tools that attempt to automate the data cleaning procedure typically focus on a specific use case and operation and are unaware of the analysis that users perform. Thus, specialized tools exhibit long running times or fail to process large datasets. In this project, we focus on approaches that address the coverage and performance issues of data cleaning, while also integrating data cleaning tasks seamlessly into the data analysis process.

Elastic & Distributed Query Engines

We build transactional and analytical engines that leverage native cloud functionality, such as elasticity and distribution. We provide fine-grained elasticity through cross-cutting system designs, spanning throughout the whole software virtualization stack, whereas we build our distributed query processing systems on top of Spark and other parallel frameworks.

Modern Storage

Storage hardware has improved dramatically in the past decade. It is now possible to have storage bandwidths in the hundreds of GB/s on a single server, approaching memory bandwidth. Conventional analytical engines rely on in-memory caching to avoid disk accesses and provide timely responses by keeping the most frequently accessed data in memory. However, high bandwidth storage performance is sufficiently close to memory bandwidth so that storing the input data on HBS can be as fast as full in-memory processing for many workloads. In this line of work we explore how high performance analytical systems must be redesigned for the high bandwidth era.

Query Accelerators

Traditionally, query engines are optimized for CPUs, but nowadays modern servers are becoming increasingly heterogeneous and equipped with multiple hardware accelerators, like GPUs. In this line of work, we investigate how different accelerators can be used by the query engine to increase its performance as well as provide isolation between queries. We design new hardware-conscious algorithms, study how existing ones perform across different micro-architectures and investigate multi-device query execution. Lastly, we provide engine designs that generalize device-specific approaches to achieve efficient heterogeneous-device execution through just-in-time code generation.

Publications

Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America

R. Bütler Sauvain; L. Larrieu; L. F. Lunde; M. Maxence; B. Nordén et al.

Swiss Federal Institute for Forest, Snow and Landscape Research WSL, Switzerland, 2024.

Analytical engines with Context-Rich Processing

Query Approximation

Data Cleaning

Elastic & Distributed Query Engines

Modern Storage

Query Accelerators

Publications

Field Guide to Northern Tree-related Microhabitats: Descriptions and size limits for their inventory in boreal and hemiboreal forests of Europe and North America

Data Champions Lunch Talks – Green Bytes: Data-Driven Approaches to EPFL Sustainability

Comparison of Three Viral Nucleic Acid Preamplification Pipelines for Sewage Viral Metagenomics

How to Support Students to Develop Skills that Promote Sustainability

How to Support Students Giving Each Other Constructive Feedback, Especially When It Is Difficult to Hear

How teachers can use the 3T PLAY trident framework to design an activity that develops transversal skills

The conceptual foundations of innate immunity: Taking stock 30 years later

Radio-Activities: Architecture and Broadcasting in Cold War Berlin

No Last One

All That is Solid

Characterization of the gut-bone marrow axis through bile acid signaling

Studies of crystal collimation for heavy ion operation at the LHC

Engineering novel protein interactions with therapeutic potential using deep learning-guided surface design

Querying the Digital Archive of Science: Distant Reading, Semantic Modelling and Representation of Knowledge

Hybrid organic/metal-oxide shells on semiconductor nanocrystals via colloidal atomic layer deposition

Electrical and Optical Manifestations of Flat Band Physics in Van der Waals Materials

Plasmonically enhanced molecular junctions for investigation of atomic-scale fluctuations in self-assembled monolayers

Exact Obstacle Avoidance for Robots in Complex and Dynamic Environments Using Local Modulation

S-acylation and lipid exchange at the ER-Golgi membrane contact sites regulate pathogen entry in human cells

Serial Dependence in Human Visual Perception and Decision-Making

Data-Driven Methods for Controller Design in Atomic Force Microscopy

Quantum-mechanical effects in photoluminescence from thin crystalline gold films

Quantifying the effects of rainfall temporal variability on landscape evolution processes

Towards a metabolic theory of catchments: scaling of water and carbon fluxes with size

Biohybrid Superorganisms—On the Design of a Robotic System for Thermal Interactions With Honeybee Colonies

Dataset to accompany publication “Quantum-mechanical effects in photoluminescence from thin crystalline gold films”

Impact of CO2-rich seawater injection on the flow properties of basalts

Architecture of a decentralised decision support system for futuristic beehives

TSLAM: a tag-based object-centered monocular navigation system for augmented manual woodworking.

Water weakening and the compressive brittle strength of carbonates: Influence of fracture toughness and static friction

Integration of the Pilatus3 detector for soft X-ray diagnostics on TCV

Siloxide tripodal ligands as a scaffold for stabilizing lanthanides in the +4 oxidation state

High-Permittivity Polysiloxanes for Bright, Stretchable Electroluminescent Devices

An extension of the stochastic sewing lemma and applications to fractional stochastic calculus

A bridge between trust and control: computational workflows meet automated battery cycling

Capsizing due to friction-induced twist in the failure of stopper knots

A CTCF-dependent mechanism underlies the Hox timer relation to a segmented body plan

Gradient High-Q Dielectric Metasurfaces for Broadband Sensing and Control of Vibrational Light-Matter Coupling

SOMOphilic alkyne vs radical-polar crossover approaches: The full story of the azido-alkynylation of alkenes

Dielectric elastomer actuator-based valveless pump as Fontan failure assist device: introduction and preliminary study

Tree diversity reduces variability in sapling survival under drought

Probing structural and dynamic properties of MAPbCl3 hybrid perovskite using Mn2+ EPR

Benchmarking machine-readable vectors of chemical reactions on computed activation barriers

Quantum radio astronomy: Data encodings and quantum image processing

Comprehensive Memory Safety Validation: An Alternative Approach to Memory Safety

A Method of Moments Estimator for Interacting Particle Systems and their Mean Field Limit

A setup for hard x-ray time-resolved resonant inelastic x-ray scattering at SwissFEL

An aircraft assembly process formalism and verification method based on semantic modeling and MBSE

Stellar Metallicities and Gradients in the Isolated, Quenched Low-mass Galaxy Tucana

Twisted pair transmission line coil – a flexible, self-decoupled and robust element for 7 T MRI

3D Printing of Double Network Granular Elastomers with Locally Varying Mechanical Properties

Dopant-additive synergism enhances perovskite solar modules

Proliferation-driven mechanical compression induces signalling centre formation during mammalian organ development

Spin-Reorientation-Driven Linear Magnetoelectric Effect in Topological Antiferromagnet Cu3TeO6

Is There a Special Role for Ovarian Hormones in the Pathogenesis of Lobular Carcinoma?

Stick-slip-to-stick transition of liquid oscillations in a U-shaped tube

Opportunities and challenges in design and optimization of protein function

Impact of beam-coupling impedance on the Schottky spectrum of bunched beam

Gaussian universality of perceptrons with random labels

Diffusion of brain metabolites highlights altered brain microstructure in type C hepatic encephalopathy: a 9.4 T preliminary study

Teaching about non-deterministic physics: an almost forgotten fundamental contribution of Marie Curie

The angiogenic growth of cities

Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms

Euclid: Improving the efficiency of weak lensing shear bias calibration

Randomized flexible GMRES with deflated restarting

An interior penalty coupling strategy for isogeometric non-conformal Kirchhoff-Love shell patches

Stress tolerance of lightweight glass-free PV modules for vehicle integration

On the Arithmetic and Geometric Fusion of Beliefs for Distributed Inference

Alpha1-antitrypsin improves survival in murine abdominal sepsis model by decreasing inflammation and sequestration of free heme

Quantitative prediction of crystallization in laser powder bed fusion of a Zr-based bulk metallic glass with high oxygen content

A cut-cell method for the numerical simulation of 3D multiphase flows with strong interfacial effects

Speciation of Lanthanide Metal Ion Dopants in Microcrystalline All-Inorganic Halide Perovskite CsPbCl3

Can Gas Consumption Data Improve the Performance of Electricity Theft Detection?