Home / Blog / General / What is SIEM? (Part 3): How does SIEM work?

October 13, 2018

In parts 1 and 2 of this 3-part series, we answered the question ‘What is SIEM?’, plus also covered the detection, response and recovery process following a cyber attack.

» What is SIEM? (Part 1): Cyber Security 101
» What is SIEM? (Part 2): Detection, response & recovery
» What is SIEM? (Part 3): How does SIEM work?

In part 3, we take a detailed look at how SIEM platforms analyse log data to uncover the suspicious activity of a cyber attack – so let us take a look at how SIEM works.

SIEM platforms thrive on data.

The SIEM collects and processes data. At the core of the data is the log. The log, also known as a log file, is a data file of the operation of a machine or application. Log file types include FlatFile, SysLog, Microsoft event logs, File Integrity Monitoring (FIM) data, UDLA (Universal Database Log Adaptor), SDEE (Security Device Event Exchange), Checkpoint data, and more.

Logs provide data on a device’s activity, health, how a user interacts with the device, as well as application performance and activity.

What is SIEM? How does SIEM work?

SIEM data collection sources

The SIEM collects log data from a wide variety of sources, data-related to systems, networks and network devices and applications. The log data that is collected could be related to security, unauthorised activities, misuse of company resources, user activity, file integrity and many other common computing activities.

A log provider is a device or application that creates and provides log files. Common devices and application that provide log data are:

Microsoft windows; UNIX or Linux; and AS400 servers; Firewalls; Switches and Routers; Intrusion Prevention Systems, or Intrusion Detection Systems; and application servers, such as web, database, and email servers.

A log source is a type of log, or a unique log originating from a specific host.

For example, a Windows server would include three log sources by default. One for each event log type:

– Application
– System
– Security

These devices and applications in your network are generating thousands of raw logs at every moment. The logs are generated in a wide variety of formats and they represent varying degrees of importance in maintaining and protecting your network.

Log messages

Logs typically take the form of log messages. Logs are generated from many different types of systems, therefore there are many different types of log messages. This log data isn’t always easily deciphered when viewed in its raw state!

What happens to the log data when it is received by a SIEM?

So, what happens to the log message and the data within?

Typically, with the leading ‘Next-Gen’ SIEM platforms, the log message is subject to several processes.

The log message is “time normalised” to ensure timestamps all reflect the same time zone. Then meta data extraction and tagging is performed, actively parsing and breaking down the log message into core components.

Then every identified log message is assigned a classification and a common event to help define the type of activity described in the message. Threat and risk contextualisation may also be performed, to evaluate each log and provide a risk-based priority value.

SIEM collection data sources

Meta data extraction and tagging

Meta data extraction and tagging actively parses and breaks down the log message into core components, identifying what kind of data the log message contains, the meaning behind the message, and the relative importance of the action described in the message.

Essentially, a SIEM platform takes the raw log message and parses it, making the data easy to use – the data parsed from the log message is call meta data.

Meta data is stored in databases using common meta data fields, so it can quickly be referenced for more efficient searching, allowing for in-depth reporting and analysis.

Categories of meta data

By parsing data into common meta data fields, searching for that data in the mass of log messages is made easy using consistent terminology, or a consistent language.

Meta data can then be further categorised…

Contextual meta data are fields that are directly parsed from log messages. They are usually text-based in nature, and are descriptive in a relation to the message. Examples include subject, domain origin and object.

Quantitative meta data are fields that are directly passed from log messages and can be used for numeric comparisons. Examples included, impacted host bytes received, rate and size.

In addition to parsing meta data directly from the log message, a SIEM can derive some meta data from the logs. Derived meta data are fields that use parsed information in log messages to provide additional context about the message. Examples include: Entity origin; Known host impacted; and Direction.

SIEM - Uniform Data Classification (UDC)

Uniform Data Classification

The leading SIEM platforms will assigns every identified log message a Classification and a Common event.

Classifications define the broad range of activity. There are three main types of classification (with sub-classifications beneath):

– Audit
– Operations
– Security

Common events are not intended to be highly specific but provide a more descriptive context for the activity described in the log message.

Threat and risk contextualisation

By converting many different types of log messages from many types of systems into a common language (or common meta data fields), a SIEM platform can compare like-to-like, and then correlate data that otherwise might not have been relatable (and therefore useful).

Events (security events)

An Event is a log having more operational, security or compliance relevance and typically include logs that are classified as Errors, Failures, or Attacks.

Events are actionable log items only, meaning they are more significant than other logs. Actionable logs include items such as login failures, hacking attempts and the granting of elevated permissions. Or system-related items, such as service or software failures and reboot messages.

Events typically make up 5% or less of the total logs collected. Once events have been identified, they are used as part of real-time monitoring, compiling reports, performing investigations and defining alarms.

Alarms and reports

A capable SIEM also aids in the creation and issuance of alarms and reports, often including a real-time alarming system to notify users and systems when certain criteria are met. In addition, the alarming system should include smart response actions for responding to certain scenarios observed by the SIEM platform. Reports then allow users to generate detailed or summary reports based on log data collected by the platform.

SIEM is complex – and everyone knows it

As we’ve seen, SIEM platforms can seem complex. The capabilities and intelligence built into a SIEM is impressive – but this does mean a skills investment and complexity… for the users, for support teams and for the organisation.

Any while businesses rely more and more on IT teams to deliver core business projects, day-to-day IT operations AND maintain security – with limited resources and budgets – it is no wonder that many organisations have realised it is not viable to build their own fully staffed and resourced 24/7 Security Operations Centre (SOC), to secure their critical business information.

Comtact's UK Security operation Centre (SOC)

Missed parts 1 and 2?

» What is SIEM? (Part 1): Cyber Security 101
» What is SIEM? (Part 2): Detection, response & recovery
» What is SIEM? (Part 3): How does SIEM work?


Outsourced SOC (and SIEM)

Managing the complexities of a SIEM platform, keeping pace with the latest security threats, as well as managing people, processes and associated technologies is a tall order. As well as factoring in the time and cost to build, train and retain your own 24/7 Security Operations Centre (SOC).

Whether fully outsourced Security, or working in partnership with internal teams, an outsourced Security Operations Centre will help you to quickly scale your security, keep pace with ever-changing threats – and ultimately ensure effective security outcomes, at a lower cost of doing it yourself.


Related articles:


About Comtact Ltd.

Comtact Ltd. is a government-approved Cyber Security and IT Managed Service Provider, supporting clients 24/7 from our ISO27001-accredited UK Security Operations Centre (SOC).

Located at the heart of a high security, controlled-access Tier 3 data centre, Comtact’s state-of-the-art UK Cyber Defence Centre (SOC) targets, hunts & disrupts hacker behaviour, as part of a multi-layered security defence, to help secure some of the UK’s leading organisations.