THE CHALLENGE Every morning, the same question: how did the batch go?
The problem: errors detected late, reports with yesterday's data and credentials scattered across the team.
THE SOLUTION
AlmIA
AlmIA monitors data loads on Microsoft Fabric, translates every run into a business status, diagnoses errors and lets you relaunch without opening Fabric. Everything goes through a gateway installed in your network, which keeps your credentials and your data.
Warning: 1 h 45 min, 135% above its historical 90th percentile of 44 min.
AlmIA answers four questions
How did today's batch go?
Daily dashboard with indicators, a 24-hour timeline and the status of every pipeline.
What failed and what does it affect?
Diagnosis of each error, a proposed fix and the downstream impact.
Can I relaunch it now?
Run, abort and schedule pipelines from the portal, with their parameters.
Who sees and operates each load?
Read or write permissions by project or by pipeline.
HOW IT WORKS
From the gateway to the portal, every day and with no manual registration
- 1Connect
The gateway is installed in your network and is the only component that talks to Fabric.
- 2Discover
It syncs workspaces, pipelines and activities every day and on demand.
- 3Organise
It assigns each pipeline to its layer (bronze, silver or gold) and to its projects.
- 4Diagnose
It classifies every run, proposes the cause of the error and calculates its downstream impact.
- 5Act
It launches, aborts and schedules pipelines through the gateway's Airflow.
No open ports. The gateway always initiates the connection to the portal and works behind your firewall.
THE STATUSES
Statuses the business understands
Every Fabric run is translated into a business status. If several apply, the most severe one wins.
KO
It failed or was cancelled.
Warning
It finished, but with a warning.
In progress
Running or waiting for its scheduled time.
OK
It finished with no issues.
THE WARNINGS
A warning alerts you before bad data gets through
Retries or handled failure
It finished, but after retrying or after going through an error branch of the pipeline.
Slower than usual
It took longer than the 90th percentile of its own run history.
Zero rows written
It wrote no data in a pipeline that normally does.
SLA missed
It finished after the time committed to the business for that data.
THE MONITOR
The daily dashboard tells you how the batch went
Indicators
Total runs and how many are OK, with a warning, failed, in progress or pending.
24-hour timeline
Active pipelines per hour and markers for errors, warnings, runs in progress and scheduled runs.
Detail by workspace
Pipelines sorted by layer, scheduled time, latest activities and a link to Fabric.
Filters
Date, technology, workspace, layer and status, with automatic refresh.
INCIDENTS
Every error comes with its probable cause and its impact
Diagnosis
Probable reason for the error or warning and a proposed fix.
Downstream impact
Which pipelines and layers are affected, based on the dependencies.
Follow-up
The team marks each incident as open, under review or fixed, with a comment.
Runs
The full list of all pipelines, filterable by workspace and status.
RUNS AND SCHEDULES
Launch, cancel and schedule pipelines without opening Fabric
Run now
Launch a pipeline from its card. If it has parameters, the portal asks for them in a form and the run shows as in progress until it finishes in Fabric.
Schedules
Recurring runs with a cron expression or on a specific date, with their parameters. The gateway triggers them when they are due.
History and volumes
Per pipeline: duration of each run, rows read and written, activity detail and how they evolve over time.
Abort and recover: a running execution can be cancelled from the history and, if the portal or the gateway restarts, the gateway reconciles its status with Airflow.
CATALOGUE AND METADATA
A live inventory of Fabric, with no manual registration
Automatic discovery
Lists the workspaces the service identity can see, so they can be assigned to the client.
Synced catalogue
Workspaces, pipelines, activities and parameters, every day and on demand.
Lineage and volumes
Source and target tables, rows read and written, Delta and Spark statistics.
Fabric capacity
The gateway checks every 5 minutes whether it is active; the portal shows it on the Home page.
PROJECTS
Organise loads by business project, with their layers
Its own structure, history and monitor
Each project groups workspaces and pipelines, even from different technologies, with its own dependency canvas.
Permissions by project
The administrator decides who sees each project, with read or write access, and even which pipelines within it.
ADMINISTRATION
Three roles: each one sees and does only what is theirs
Superadmin
Registers clients, rotates their gateway secret and assigns them their workspaces. Does not operate client data.
Admin
Manages their client's projects, schedules, tags and users, and accesses their gateway's panel.
User
Views and, with write permission, runs the pipelines of the projects assigned to them.
Identity managed with Keycloak: user registration and blocking, password change and reset, and sessions with expiry and automatic renewal.
SECURITY
Your data and your credentials stay in-house
One gateway per client
With its own Airflow, its own database and its own credentials. It shares nothing with other clients.
Credentials in your network
Fabric keys are configured in the gateway panel and never reach the central portal.
Outbound connection
The gateway opens the connection with its own secret. No inbound ports need to be opened.
Only facts, never data
The portal receives a heartbeat and a summary of runs, never rows of data or parameters.
Permissions denied by default: a new user sees no project until their administrator assigns it, and the portal discards data from unassigned workspaces.
ROADMAP
After Fabric come Databricks and Snowflake
Databricks and Snowflake
The monitor already separates the view by technology; what is missing is each platform's connector in the gateway.
AI-powered diagnosis
Today the diagnosis uses rules on the error message; it is ready for a language model.
Deeper Fabric integration
View the schedules created in Fabric and link to each run in the Monitoring hub.
Built for: Data and operations teams working with Microsoft Fabric, platform leads, data governance and IT.
FAQ
Frequently asked questions
Straight answers to what we're asked most.
Which platforms does AlmIA work with?
Today, Microsoft Fabric. Databricks and Snowflake are on the roadmap: the monitor already separates the view by technology and what is missing is each platform's connector in the gateway.
Does AlmIA copy my data outside my network?
No. The gateway is installed in your network, keeps the Fabric credentials and always opens the connection towards the portal. The portal only receives a heartbeat and a summary of runs, never rows of data or parameters.
Do I need to open ports in the firewall?
No. The connection is always outbound, from the gateway to the portal, with its own secret.
What is a warning in AlmIA?
A run that finished but deserves attention: after retries, slower than its historical 90th percentile, with no rows written when it normally writes them, or outside the SLA committed to the business.
How are errors diagnosed?
Today with rules on the error message, which propose the probable reason and the fix and calculate the downstream impact. The diagnosis is ready to incorporate a language model.
Want to see AlmIA with your data?
We'll show you with a case like yours.