---
title: "A data and AI platform for the mid-sized company: layers, key decisions and roadmap"
description: "The layers that make up a data and AI platform, how to choose between data lake, data warehouse and lakehouse, cloud or open source, what MLOps involves, and a phased roadmap to help a mid-sized company turn data into value."
url: https://bidatia.com/en/ideas/plataforma-de-datos-e-ia-capas-decisiones-y-hoja-de-ruta
lang: en
date: 2026-04-14
updated: 2026-11-11
author: "Santos Vicente"
keywords: ["data platform", "lakehouse", "data lake", "data warehouse", "mlops", "etl elt", "self-service analytics", "data culture"]
publisher: Bidatia
---

# A data and AI platform for the mid-sized company: layers, key decisions and roadmap

> The layers that make up a data and AI platform, how to choose between data lake, data warehouse and lakehouse, cloud or open source, what MLOps involves, and a phased roadmap to help a mid-sized company turn data into value.

## Executive summary

- A data platform manages the full data lifecycle: ingestion, transformation, analysis, visualisation and use in AI models.
- It is organised into three layers: ingestion and storage, processing and governance, and consumption and analytics.
- The lakehouse architecture combines the flexibility of the data lake with the management of the data warehouse; ELT suits semi-structured data better.
- The choice between cloud and open source depends on budget, team and long-term goals.
- AI requires MLOps (models, registry, deployment and monitoring), and agents must access data securely and under control.
- It is built in phases: discovery, pilot, scaling, and AI and automation, with data culture as a key piece.

Many mid-sized companies generate more data every day than they are able to use: the ERP, the CRM, the website, each department's spreadsheets, application logs. The value lies not in accumulating it, but in processing it, organising it and putting it within reach of the people who make decisions. That requires a **data platform**: not just another repository, but an ecosystem that manages the full data lifecycle, from ingestion to analysis and use in artificial intelligence models.

This article sets out, for whoever leads data in the organisation, the building blocks of that platform, the architectural decisions to be made and the order in which it makes sense to build it.

## What a data platform delivers

For a mid-sized company, the platform is not a technological luxury; it is the precondition for four things:

- **Evidence-based decisions**: moving from intuition to reliable indicators that steer strategy.
- **Operational efficiency**: automating processes and optimising resources with up-to-date information.
- **Innovation**: designing products and services based on customer and market behaviour.
- **Business autonomy**: letting teams explore their own data without always depending on IT.

## The three layers of the platform

A complete data platform is organised into three connected layers:

| Layer | Purpose | Main components |
|---|---|---|
| Ingestion and storage | Collect, centralise and retain data from all sources with integrity and availability | Sources (operational databases, APIs, logs, social media, real-time streams), data lake, data warehouse, ETL/ELT pipelines |
| Processing and governance | Turn data into useful, reliable information | Processing engine (cleansing, enrichment, aggregation), data catalogue, security and compliance policies |
| Consumption and analytics | Put the value in the hands of end users | Dashboards and reports, self-service analytics, AI platform |

The order matters: if the foundation fails, nothing built on top of it will be reliable.

## The foundation: data lake, data warehouse or lakehouse

The first architectural decision is where and how data is stored.

| | Data lake | Data warehouse |
|---|---|---|
| What it holds | Large volumes of raw data, structured or not | Structured, modelled data |
| Schema | No upfront schema required | Schema defined in advance |
| What it is for | Flexibility, future analysis, data science | Recurring business reporting and analysis |
| Cloud examples | Azure Data Lake Storage, Google Cloud Storage, Amazon S3 | Azure Synapse Analytics, Google BigQuery, Amazon Redshift |
| Open-source options | Hadoop HDFS, Apache Iceberg, Delta Lake | Apache Druid, ClickHouse |

The current trend is the **lakehouse** architecture, which combines the flexibility of the data lake with the management and analytical performance of the data warehouse. In practice, it lets you store data once, in open formats, and serve it both to reports and to AI models.

### ETL or ELT

There are two approaches to moving data between systems:

- **ETL (extract, transform, load)**: data is transformed before it reaches its destination. Common with structured data and stable rules.
- **ELT (extract, load, transform)**: data is loaded into the data lake first and transformed there. A better fit for semi-structured data and lakehouse architectures, because it keeps the original data and allows transformations to be rerun.

## The consumption layer: self-service analytics

This is the visible face of the platform, the part that turns technical complexity into answers for the business. **Self-service** lets users run their own queries and analysis without going through IT, which brings agility (faster answers), autonomy (curiosity and experimentation) and efficiency (IT can focus on higher-value work).

A good BI tool should offer clear visualisation of complex data, interactivity (filters and drill-down) and the ability to connect to several sources at once. Among the most widely used are Microsoft Power BI, tightly integrated with the Microsoft ecosystem; Google Looker Studio, simple and closely tied to Google Analytics and BigQuery; Amazon QuickSight, with built-in AI capabilities; and Apache Superset, open source and highly customisable.

Self-service works when there are shared, validated data models underneath. Without them, each department builds its own version of the indicators and the numbers stop adding up.

## Cloud or open source

The choice depends on budget, in-house technical resources and long-term goals.

| | Cloud platform | Open source |
|---|---|---|
| Advantages | Automatic scaling, maintenance handled by the provider, preconfigured security, integrated services | No licence costs, full control of the technology, high customisation |
| Drawbacks | Usage-based variable costs, risk of vendor lock-in | Requires a specialised technical team, slower implementation, full responsibility for security |

Among the major providers, Microsoft Azure stands out for its integration with the Microsoft ecosystem (with Microsoft Fabric as its bid to unify its data services), Google Cloud for its AI capabilities and the performance of BigQuery on large-scale queries, and AWS for the breadth and maturity of its service catalogue. For many mid-sized companies, the deciding factor is the ecosystem they already use and the skills of the team that will run the platform.

## The artificial intelligence platform

A modern data platform is not complete without AI capabilities, and deploying models means managing their lifecycle (known as **MLOps**). It has four basic building blocks:

1. **Model development**: environments to train and evaluate classification, regression, clustering or generative models.
2. **Model registry**: a central place to store trained models and their versions.
3. **Deployment**: exposing models as services (APIs) that other applications can consume.
4. **Monitoring**: tracking performance in production and detecting drift.

The major providers offer complete environments for this, such as Azure Machine Learning, Vertex AI and Amazon SageMaker. The loop closes with feedback: data from the data warehouse feeds training, models are deployed and monitored, and what is learnt in production improves the next version.

With this foundation in place, use cases open up such as personalising the customer experience, real-time anomaly detection, intelligent process automation and predictive analytics for decision-making.

### AI agents on company data

A further step is deploying **AI agents** that let employees interact with data conversationally: an assistant that answers questions about sales using the data warehouse, or an agent that summarises daily reports or technical documentation. These agents need an architecture that gives them **secure, controlled** access to data, respecting the same permissions and classification that apply to people.

## Data culture: the piece you cannot buy

Technology alone does not create value; the people who use it do. Two practices help build a data culture:

- **A knowledge repository**: an internal wiki documenting indicator definitions, data lineage and user guides.
- **Communities of practice**: cross-functional groups where business and data teams share analysis, solve problems and learn from each other.

## A phased roadmap

The platform is built incrementally and with a focus on value. A phased approach reduces risk and makes adoption easier:

| Phase | Goal | Activities |
|---|---|---|
| 1. Preparation and discovery | Lay the groundwork | Define high-impact indicators and use cases, inventory internal and external sources, assess cloud versus open source |
| 2. Minimum viable product | Show value early with a pilot | Set up the storage foundation, build the pilot's load pipelines, develop the pilot dashboard and validate it with users |
| 3. Scaling and integration | Reach more business areas | Integrate new sources (ERP, CRM), implement governance and the data catalogue, train business users |
| 4. AI and automation | Automate processes with models | Develop models (sales forecasting, segmentation), establish MLOps, deploy AI agents for knowledge management |

How long each phase takes depends on the starting point, the number of sources and the resources available; what matters is that each one ends with something the business actually uses.

## Where to start

A data and AI platform is not a project that is delivered and closed, but an ongoing journey. The most effective approach for a mid-sized company is to choose one or two use cases with clear impact, build a pilot that solves them end to end and grow on what already works, building in governance and data culture from the outset rather than at the end. At Bidatia we support organisations along that path, from architecture design to getting the first use cases up and running.

## Frequently asked questions

### What is the difference between a data lake, a data warehouse and a lakehouse?
A data lake stores large volumes of raw data with no upfront schema; a data warehouse stores structured, modelled data for reporting. A lakehouse combines both: it stores data once, in open formats, and offers the management and analytical performance of a data warehouse.

### Is a cloud platform or an open-source one better?
It depends on budget, team and goals. The cloud provides scaling and managed maintenance in exchange for variable costs and some vendor dependency; open source avoids licences and gives full control, but requires a specialised team and full responsibility for security.

### What is MLOps and why is it needed?
It is the lifecycle management of AI models: development, versioning, deployment and monitoring. Without MLOps, models stay as lab experiments or degrade in production without anyone noticing.

### Where should a mid-sized company start?
With one or two high-impact use cases and an inventory of its data sources. That is enough to build a pilot that demonstrates value; from there, the platform is extended to more areas and AI and automation are added.