A data lakehouse is a modern platform that combines the functions of a data lake and data warehouse. Data lakehouses have the flexibility of a data lake for storing unstructured data while providing the data management tools and features of a data warehouse, coupling them strategically in a larger data storage/management system. Integrating these two data solutions allows users to leverage the best of both worlds.
In this article:
A data lakehouse can handle several main aspects of organizing data to support use cases traditionally associated with a data lake or warehouse. One issue it addresses is the need to combine easy data ingestion capabilities (for raw data) while enabling high-performance business intelligence (BI) using extract, transform and load (ETL) processes to organize data.
A traditional data lake organizes data with an Optimized Row Columnar (ORC) or Parquet file system. More modern open source tools support atomicity, consistency, isolation, and durability (ACID) database transactions for big data projects.
Data science and machine learning (ML) tools traditionally support the Parquet and ORC file formats. However, these tools have also adopted data frames to make optimizing data for new models easier.
While a standard data warehouse can fine-tune the structure of SQL data for many common queries, a data lakehouse optimizes the layout of a traditional data warehouse format like ORC or Parquet files on the fly.
While data lakehouses are gaining popularity, many organizations are reluctant to adopt them. Businesses should evaluate the advantages and hurdles of implementing a data lakehouse to determine the right strategy.
Proponents of data lakehouses emphasize several advantages, including their ability to serve as a central repository for an organization’s entire data. Data lakehouse architectures help mitigate standard data lakes’ governance and administration challenges. Another advantage is that it decouples storage and compute to enable higher scalability and flexibility.
A large organization has increasingly complex business requirements, making it highly dependent on data to support ML and other advanced use cases such as decision intelligence (DI). A data lakehouse offers combined capabilities to enable highly intricate ML and data science projects.
The attributes of this architecture that help achieve business objectives include the ability to enforce complex schemas and support ACID transactions (a major benefit of a data warehouse).
However, implementing a data lakehouse can be challenging. Organizations should avoid treating the lakehouse as a complete, uniform solution for everything. It contains many complex components that require separate management.
Another challenge of adopting a data lakehouse approach is migrating all existing data warehouses to the company’s data lake before experiencing the benefits. Migration is often time-consuming and expensive and carries risks associated with outages and latency.
It also requires designing the lakehouse infrastructure for scalability and cost-effectiveness. A poorly planned data lakehouse won’t support round-the-clock analytics for business-critical use cases. Some data lakehouse vendors require customers to use specific tools, which might not be compatible with all existing tools connected to the organization’s data lake.
AWS and Databricks are the two main proponents of the data lakehouse concept, so they define a lakehouse’s architecture. Data lakehouse systems usually include the following five layers.
The first layer pulls data from various sources and delivers it to the next layer. It combines data streaming and batch processing, using various protocols to connect to internal and external resources (i.e., an RDMS, NoSQL database, CRM application, etc.). Components used during the ingestion stage might include Amazon’s Data Migration Service (DMS) to import data and Apache Kafka to stream data.
The second layer stores the data in various cost-effective platforms like Amazon S3. The client has tools to read objects from the data store, allowing many APIs and other components to access and use the data. A data lakehouse is most useful for cloud repo services that separate storage and compute, although it can also work on-premises.
This layer is the main component that differentiates data lakehouses from other storage architectures. It is a centralized catalog providing metadata about each object in the data lake, allowing users to implement various management features (i.e., ACID transactions, caching, versioning, and zero-copy cloning).
The metadata layer enables schema architectures like star or snowflake schemas. It allows organizations to manage schemas and provides data auditing and governance functionality. Schema management includes evolution and enforcement features, allowing users to control data quality by rejecting writes that don’t meet the schema.
A unified management interface makes auditing and access control easier.
This layer hosts several APIs that allow end-users to process data quickly and perform advanced analytics tasks. For instance, a metadata API helps identify the objects required for a given application. Some ML libraries can read formats like Parquet, enabling direct queries of the metadata layer. Other APIs help developers optimize the data structure and transformation.
The data consumption layer includes tools and applications that support analytics tasks like data visualization, ML jobs, queries, and business intelligence (BI) dashboards.
Organizations should consider the following capabilities when evaluating if they should implement a data lakehouse:
Depending on the organization’s specific needs, there may be more considerations, but the above list is a good starting point for assessing whether a data lakehouse is an appropriate solution.
Related content: Read our guide to data warehouse vs data lake and S3 data lake.
Regardless of which solution you choose you are likely to have data that is less frequently, if ever, accessed, consuming valuable space. Cloudian allows you to store this less used but no less valuable data at a reduced price on appliances that are scalable and integrate with existing NAS and cloud services.
