So you’re building an app for your enterprise. While your primary purpose for building that app may take countless forms, that purpose is at least partly driven by the desire to collect and store “big data” that can be translated into business intelligence (BI). Why? BI allows you to translate your data into actionable information that helps you drive better business decisions.
Enterprises have long relied on BI to help them move their businesses forward. Years ago, translating BI into actionable information required the help of data experts. Today, technology supports BI that can be made accessible to people at all levels of an enterprise.
All that BI data needs to live somewhere. The data storage solution you choose for enterprise app development positions your business to access, secure, and use data in different ways. That’s why it’s helpful to understand the basic options, how they’re different, and which use cases are suitable for each.
Basic Definitions of the Basic Options
Data warehouses and data lakes are both widely used for storing data. But what is a data warehouse, and what is a data lake? Let’s establish the basics:
- Data lake definition: Primarily used to store raw, unprocessed data flowing directly from an organization’s various operational and transactional systems and other sources of internal or external data. Picture an actual lake fed by a variety of sources (e.g., rivers, streams, rainwater): Water/data flows in and is simply stored as-is.
- Data warehouse definition: Primarily used to store refined, processed data. This means that the data being stored has already been cleansed and categorized. Picture a typical warehouse: The product/data is processed and packaged before being stored in its assigned area.
Getting Clear on Other Concepts: Data Oceans, Reservoirs, and Marts
You may also have heard of data oceans, data reservoirs, and data marts. These concepts are related to data lakes and warehouses and are most easily understood in comparison. Basically:
- Data ocean vs. Data lake: A data lake is often used for data specific to a certain part of the business. A data ocean, on the other hand, comprises unprocessed data from the entire scope of the business.
- Data reservoir vs. Data lake: Some businesses use the concept of a data reservoir to distinguish between a totally unrefined data lake and a data reservoir that’s been partly filtered, secured, and made ready for consumption. Imagine a lake that’s been drained and filtered into a reservoir, creating a source of potable drinking water.
- Data warehouse vs. Data mart: A data mart is a subset of the data warehouse tailored to the needs of a specific team or line of business. Think of it as a storage room within your warehouse used to store only data within a specific scope.
To keep things simple, we’ll keep our discussion focused on the question of data lake vs. data warehouse.
When weighing the question of data lake vs. data warehouse, it’s helpful to understand the key differences in data lake and data warehouse architecture. These differences drive how they can be used, when, and by whom, as well as the security required for each.
While data lakes are schemaless, data warehouses use very precise schema. In this way, data lakes are similar to non-relational NoSQL databases, while data warehouses are similar to relational SQL databases.
NoSQL databases can store data in many ways, allowing for great flexibility in how data is input and stored. Accordingly, NoSQL databases tend to work well with data lakes. SQL databases require you to use predefined schema to structure your data before you’re able to work with it. SQL databases integrate well with data warehouses.
Again, data lakes contain raw, unrefined data while data warehouses contain cleansed, processed data.
Data lakes tend to be difficult to navigate for people unused to working with unprocessed data. They’re more commonly used by highly skilled data scientists. Data scientists can draw a variety of data from the lakes, using specialized tools to process and translate it into actionable BI.
Business analysts tend to favor data warehouses. Since the stored data has been organized according to predefined schema, the data in warehouses can be easily navigated, understood, and interpreted by a range of users.
Due to the data-cleansing process, data warehouse environments have more transparent information security and more well-established data protection controls. Data lakes present more information security challenges, given the large amounts of raw data flowing in from different sources. As a result, strict data protection controls must be applied for data lakes. These controls are especially important in regulated environments with compliance requirements (e.g., healthcare, financial services).
Data lakes’ schemaless structure offers more flexibility than a traditional data warehouse. Data lakes don’t have mandatory data cleansing efforts.
Data Lake Advantages and Anti-patterns
In general, data lakes are good for analyzing data from different, diverse sources from which initial data cleansing can be problematic. In particular, data lake advantages include:
- Easy integration with the Internet of Things (IoT), since data such as IoT device logs and telemetry can be collected and analyzed
- Good integration with machine learning (ML), given the schemaless structure and ability to store large amounts of data
- Flexibility provided by the schemaless structure that:
- Supports analysis of data from social networks and mobile devices
- Supports large heterogeneous, multiregional, and microservices environments
That said, data lakes’ flexibility can lead to misuse, causing anti-patterns that create more problems than they solve. For example, “data swamps” are data lakes containing low-quality, unrefined data. “Data graveyards” are data lakes containing data that’s collected in large quantities but never used.
Data Warehouse Advantages and Anti-patterns
Data warehouses offer advantages such as:
- Good ability to analyze relational data coming from online transaction processing (OLTP) systems and business apps (e.g., ERP, CRM, and HRM systems)
- Good integration with uniform data sources (especially relational sources), making it a fit for small- to medium-sized businesses
Data warehouse anti-patterns include:
- Data silos, in which information security controls lead to limited access such that useful data isn’t accessible to people who may have benefited from it, impeding productivity and collaboration
- Higher probability of distortion of BI analysis results due to premature or incorrect data cleansing — since data quality is often subjective, with different analysts having different tolerances for what constitutes quality
Given these anti-patterns, some data scientists prefer data lakes. With lakes, they can work with raw data, deliberately applying techniques to make sense of the data while ensuring its integrity.
All major cloud providers — including AWS, Google Cloud Platforms, and Azure — offer managed data lake solutions. Major cloud providers tend to offer data lakes rather than data warehouses, given data lakes integrate better with organizations’ systems and are better optimized for cloud environments. That said, cloud data warehouse options include AWS Redshift, Google BigQuery, Azure SQL Data Warehouse, Oracle Autonomous Data Warehouse, and Snowflake Data Warehouse.
These managed solutions offer some of cloud computing’s well-known benefits, including cost-effectiveness, simplified operations, enterprise-level security and compliance, and integrations with other managed cloud services (e.g., cloud storage, cloud ML services).
Which Solution Is Right for Your Enterprise App?
With all that said, which BI data storage solution is right for your enterprise’s app development efforts — a data lake or a data warehouse? After all, both provide good data storage solutions for suitable use cases. The answer may be one or both, as businesses can use both solutions simultaneously.
Generally speaking, data warehouse use cases tend to be common for small- to medium-sized businesses, while data lake use cases are more common for larger enterprises. That said, the answer to the question often depends on your data sources. For example:
- If you use an SQL database or ERP, CRM, and HRM systems, data warehouses will fit well into your enterprise environment.
- If your data comes from disparate data sources (e.g., NoSQL, IoT logs and telemetry, social data, mobile, and web analytics), data lakes are likely a good choice.
Making the right choice can be central to making sure that your enterprise app delivers optimal value to your business. After all, the data you capture is only truly valuable if you can translate it into actionable BI.
Still confused about choosing the right data storage solution for your enterprise app? Let us know!
Dmitriy Stepanenko joined the Distillery team in 2016. His experience is diverse; he has worked as a database administrator, information security analyst, and software developer. With extensive knowledge of both software development and IT operations, Dmitry now leads Distillery's DevOps department. He holds a master’s degree in information security, and his passions include continuous delivery, cloud computing, and the DevOps culture.