Skip to content
senn-techsenn-tech
← All references03 · IT

SIEM & security monitoring with real-time correlation

Reference client: Logistics and trading group in Tyrol, Austria · ~100 employees · 4 sites

Customer data in copy and images has been neutralised.

~30integrated systems
~60monitors & correlation rules
~7Mevents per day
30 scorrelation interval
01

Starting point

Firewall, email gateway, Microsoft 365, endpoint protection, phone system, cameras, servers, switches: every system diligently generates logs. But no one ever read them: scattered across a dozen consoles, each with its own login, its own interface, and its own alert behavior.

Anomalies were noticed when something was broken. Not when it was starting to break. And a group of around 100 employees is simply too small for a dedicated security team.

Abb. 01Activity center: live logs from every source in one place, filterable by source, severity and host.
Live log view of the activity center

Abb. 01Activity center: live logs from every source in one place, filterable by source, severity and host.

02

The hard part

Collecting logs is the easy part. The hard part is deciding what may interrupt a human being, because a system that reports too much gets dismissed within two weeks and is then worse than none at all: it simulates security where none is happening any more.

An anomaly is not yet a finding. An example from our own operations: for several nights the DNS server reported “disk full”. The measurement was correct, because maintenance of the query database briefly needs twice the space and releases it afterwards. As an alert it was worthless. Three of those a night would have been enough to leave the fourth unread as well.

The second kind points at something that no longer exists in that form. A 98 percent collapse in mail volume looks like an outage; in fact it was a decommissioned mailbox whose journal nobody had switched off.

The most uncomfortable class, though, is the third: a monitor can be green because it sees nothing. And that is not the same as there being nothing to see. The whole difference sits between “no alert” and “no detection”, and from the outside the two are indistinguishable. That is exactly why the pipeline monitors itself.

Abb. 02Threats: correlation across all sources. The timeline shows when it is quiet and when a pattern forms, instead of every raw event on its own.
Threat timeline with severity breakdown

Abb. 02Threats: correlation across all sources. The timeline shows when it is quiet and when a pattern forms, instead of every raw event on its own.

03

Solution

A central SIEM was set up on the company's own infrastructure: a lean log pipeline accepts syslog, GELF and API data from around 30 systems: from Microsoft 365 and endpoint security through Proxmox clusters, switches and UPS units to the phone system.

On top of it sits a custom-built monitoring layer: around 60 monitors and real-time correlation rules check every 30 seconds for patterns such as brute-force attempts, impossible sign-in locations or expiring certificates. Scoring, deduplication and prioritization are rule-based. Only what requires action gets delivered; a local language model is on call for deep analysis of individual incidents.

Relevant alerts go out immediately as push and email to the people responsible; everything lower-priority is collected into a digest twice a day.

VectorVictoriaLogsPostgreSQLRule EnginePush + Mail
Abb. 03Dashboard: only what needs attention on top, all signals as status cards below. Quiet reliably means quiet here.
SIEM dashboard with signal cards and a needs-attention area

Abb. 03Dashboard: only what needs attention on top, all signals as status cards below. Quiet reliably means quiet here.

04

How it is built

The design is a funnel with four cleanly separated stages: collect, store, evaluate, deliver. Each can be replaced on its own. We swapped the log store during live operation without touching a single rule.

Collection uses a lean collector that accepts and normalises syslog, GELF and API sources. Evaluation then does not re-query the log store for every rule but works on a rolling time window, and that is the reason the interval stays stable at 30 seconds no matter how much is arriving.

Every monitor is its own small service with exactly one job, one owner and one alert text. Not a monolith with sixty switches. A monitor that reports nonsense gets switched off without touching the other 59.

Delivery runs on two tracks: anything that requires action goes out immediately as push and email to the owner, everything else is collected into a digest twice a day. That separation is not a convenience. It is the actual countermeasure against alert fatigue.

Abb. 04M365 security: every cloud sign-in is checked, locations and apps are listed (users pseudonymised).
M365 security view with sign-in monitoring

Abb. 04M365 security: every cloud sign-in is checked, locations and apps are listed (users pseudonymised).

Abb. 05Endpoints: EDR detections from every server and workstation feed the same picture via syslog (host names replaced).
Endpoint detection view with EDR events

Abb. 05Endpoints: EDR detections from every server and workstation feed the same picture via syslog (host names replaced).

FirewallMail gatewayMicrosoft 365Endpoint protectionPhone systemCamerasServersSwitchesCollectnormaliseCorrelationrules + AIOne live viewAlerts: mail + push
Every system already wrote logs. What was missing was the single place that reads them.
05

Day-to-day operation

The first monitor watches the pipeline itself. If ingestion stalls, events are not lost loudly but quietly, and the picture then looks calm while it is in fact blind. A stalled ingestion journal is therefore an alert with a priority of its own.

Suppressions exist, but they are narrow and justified: sign-ins from our own network are scored differently from those outside it. A blanket night-time mute we rejected deliberately. Attacks do not observe office hours, and the hours when nobody is looking are exactly the interesting ones.

Every false positive is fixed at the source. Not by raising the threshold. Raising thresholds is the route by which a security system goes quiet without anyone noticing.

Abb. 06Infrastructure: 7 nodes, over 90 VMs and per-pair replication, monitored live.
Proxmox monitoring with storage replication

Abb. 06Infrastructure: 7 nodes, over 90 VMs and per-pair replication, monitored live.

Abb. 07Perimeter: nearly 36,000 automatically banned attacker IPs, broken down by scenario and origin.
CrowdSec view with ban list and attack scenarios

Abb. 07Perimeter: nearly 36,000 automatically banned attacker IPs, broken down by scenario and origin.

06

Outcome

Around seven million events per day now flow through the pipeline: read by machines and not by people. Seconds rather than chance separate an event from an alert.

Security incidents, failed backups or storage filling up are noticed before they become a problem, without additional staff and without licence costs for a cloud SIEM.

Abb. 08Visual analytics: attack origins, security levels and a 30-day integrity heatmap.
Analytics with world map and integrity heatmap

Abb. 08Visual analytics: attack origins, security levels and a 30-day integrity heatmap.

07

What we learned

A monitor for terminal server load showed plausible but incorrect absolute values for weeks. The cause was a mix-up in definition: the metric in use was a single process's share and not the load of the system, and in the raw data the two are named almost identically. The rule since: pin down what a number means first. Then display it.

A detector for stalled storage replication was blind to exactly the fault we had built it for. It checked a counter that keeps advancing even during the failure. That lesson now sits in the build instructions for every new monitor: before it goes live, the fault case must have been induced artificially once and the alert actually seen. A monitor that has never fired is unproven.

Both cases have the same shape. It was not the system that failed but our assumption about what was being measured. Which is why the first question about a new alert is not “is this bad?” but: what exactly does this number count, and what would actually happen if the fault occurred?

Similar problem?

Tell us what you're planning, a short call clarifies whether it pays off.