What Is a Data Lakehouse?
A data lakehouse is a modern platform that combines the functions of a data lake and data warehouse. Data lakehouses have the flexibility of a data lake for storing unstructured data while providing the data management tools and features of a data warehouse, coupling them strategically in a larger data storage/management system. Integrating these two data solutions allows users to leverage the best of both worlds.
In this article:
How Does a Data Lakehouse Work?
A data lakehouse can handle several main aspects of organizing data to support use cases traditionally associated with a data lake or warehouse. One issue it addresses is the need to combine easy data ingestion capabilities (for raw data) while enabling high-performance business intelligence (BI) using extract, transform and load (ETL) processes to organize data.
A traditional data lake organizes data with an Optimized Row Columnar (ORC) or Parquet file system. More modern open source tools support atomicity, consistency, isolation, and durability (ACID) database transactions for big data projects.
Data science and machine learning (ML) tools traditionally support the Parquet and ORC file formats. However, these tools have also adopted data frames to make optimizing data for new models easier.
While a standard data warehouse can fine-tune the structure of SQL data for many common queries, a data lakehouse optimizes the layout of a traditional data warehouse format like ORC or Parquet files on the fly.
Data Lakehouse Benefits and Challenges
While data lakehouses are gaining popularity, many organizations are reluctant to adopt them. Businesses should evaluate the advantages and hurdles of implementing a data lakehouse to determine the right strategy.
Benefits
Proponents of data lakehouses emphasize several advantages, including their ability to serve as a central repository for an organization’s entire data. Data lakehouse architectures help mitigate standard data lakes’ governance and administration challenges. Another advantage is that it decouples storage and compute to enable higher scalability and flexibility.
A large organization has increasingly complex business requirements, making it highly dependent on data to support ML and other advanced use cases such as decision intelligence (DI). A data lakehouse offers combined capabilities to enable highly intricate ML and data science projects.
The attributes of this architecture that help achieve business objectives include the ability to enforce complex schemas and support ACID transactions (a major benefit of a data warehouse).
Challenges
However, implementing a data lakehouse can be challenging. Organizations should avoid treating the lakehouse as a complete, uniform solution for everything. It contains many complex components that require separate management.
Another challenge of adopting a data lakehouse approach is migrating all existing data warehouses to the company’s data lake before experiencing the benefits. Migration is often time-consuming and expensive and carries risks associated with outages and latency.
It also requires designing the lakehouse infrastructure for scalability and cost-effectiveness. A poorly planned data lakehouse won’t support round-the-clock analytics for business-critical use cases. Some data lakehouse vendors require customers to use specific tools, which might not be compatible with all existing tools connected to the organization’s data lake.
Data Lakehouse Architectural Options
AWS and Databricks are the two main proponents of the data lakehouse concept, so they define a lakehouse’s architecture. Data lakehouse systems usually include the following five layers.
Ingestion Layer
The first layer pulls data from various sources and delivers it to the next layer. It combines data streaming and batch processing, using various protocols to connect to internal and external resources (i.e., an RDMS, NoSQL database, CRM application, etc.). Components used during the ingestion stage might include Amazon’s Data Migration Service (DMS) to import data and Apache Kafka to stream data.
Storage Layer
The second layer stores the data in various cost-effective platforms like Amazon S3. The client has tools to read objects from the data store, allowing many APIs and other components to access and use the data. A data lakehouse is most useful for cloud repo services that separate storage and compute, although it can also work on-premises.
Metadata Layer
This layer is the main component that differentiates data lakehouses from other storage architectures. It is a centralized catalog providing metadata about each object in the data lake, allowing users to implement various management features (i.e., ACID transactions, caching, versioning, and zero-copy cloning).
The metadata layer enables schema architectures like star or snowflake schemas. It allows organizations to manage schemas and provides data auditing and governance functionality. Schema management includes evolution and enforcement features, allowing users to control data quality by rejecting writes that don’t meet the schema.
A unified management interface makes auditing and access control easier.
API Layer
This layer hosts several APIs that allow end-users to process data quickly and perform advanced analytics tasks. For instance, a metadata API helps identify the objects required for a given application. Some ML libraries can read formats like Parquet, enabling direct queries of the metadata layer. Other APIs help developers optimize the data structure and transformation.
Consumption Layer
The data consumption layer includes tools and applications that support analytics tasks like data visualization, ML jobs, queries, and business intelligence (BI) dashboards.
Is the Data Lakehouse Solution Right for Your Business?
Organizations should consider the following capabilities when evaluating if they should implement a data lakehouse:
- Advanced and BI analytics—the data lakehouse should be able to analyze optimally structured data for standard business intelligence tools like Tableau and Power BI. It should also allow data scientists to use datasets for AI and machine learning tasks.
- Less data redundancy—the lakehouse should allow the organization to keep all data in a single place rather than replicating data sets across different data lakes and warehouses.
- Data observability—a data lakehouse helps reduce the data moving into different systems. Storing everything in one place makes data governance and maintenance easier.
- Simplified data management—organizations can leverage a unified architecture instead of a multi-tiered architecture for managing data.
- Enhanced data security—a data lakehouse enables access controls and other data security mechanisms to protect workflows and data.
- Flexible analytics—business requirements evolve, so the solution must support changing data storage and consumption patterns.
- Reduced storage costs—capturing and storing data in raw formats helps cut costs. The data lakehouse will scale to accommodate any data storage needs.
Depending on the organization’s specific needs, there may be more considerations, but the above list is a good starting point for assessing whether a data lakehouse is an appropriate solution.
Related content: Read our guide to data warehouse vs data lake and S3 data lake.
Data Storage and Management with Cloudian
Regardless of which solution you choose you are likely to have data that is less frequently, if ever, accessed, consuming valuable space. Cloudian allows you to store this less used but no less valuable data at a reduced price on appliances that are scalable and integrate with existing NAS and cloud services.