Rsyslog using severities, templating and datadog integration

Basic setup

So I will assume you installed syslog and have some default config running for it with syslog_facility set to log_local0. With that lets assume you are using rsyslog and you have config for it in

/etc/rsyslog.conf

which could have all config in this file or separated config files that are included with

$IncludeConfig /etc/rsyslog.d/*.conf

So we will use that and add drupal.conf in rsyslog.d directory whit following content

local0.panic;local0.alert;local0.crit;local0.err;local0.warning;  -/var/log/drupal-alert.log

This code will make that specific drupal severities are logged, those that need alerting, from panic to warning, if you want to log all, just write local0.* and all will be sent to log.  You can also have several log lines/files where you log stuff, like one for warnings and one for just errors or whatever you like and each can be used with any other tool and 3rd party integration

Severities

Drupal outputs severities from emergency to debug

$severity = RfcLogLevel::EMERGENCY;
$severity = RfcLogLevel::ALERT;
$severity = RfcLogLevel::CRITICAL;
$severity = RfcLogLevel::ERROR;
$severity = RfcLogLevel::WARNING;
$severity = RfcLogLevel::NOTICE;
$severity = RfcLogLevel::INFO;
$severity = RfcLogLevel::DEBUG;
$levels = RfcLogLevel::getLevels();

with method

$logger->log($level, $message);

which will send whole drupal message and the severity(level) to local0 facility. That is one of the reasons why there is no !severity token in default message format for syslog

!base_url|!timestamp|!type|!ip|!request_uri|!referer|!uid|!link|!message

although you can add it with !severity wherever you want. 
Some people do as they don't get it outputted in their logs, but this is due to rsyslog template that is used for outputting logs, and changing the template you can include severity, without adding it to drupal message.

Rsyslog Templating

On both Ubuntu the default syslog format is set to RSYSLOG_TraditionalFileFormat with its low-precision timestamps and unstructured message field.
If you wanted more detail or structure you could use one of the other built-in formats like RSYSLOG_SyslogProtocol23Format or create your own.
So for above example you would put

local0.panic;local0.alert;local0.crit;local0.err;local0.warning;local0.notice  -/var/log/drupal-alert.log; RSYSLOG_SyslogProtocol23Format

code like this, adding template name after log file declaration. Also when doing this, you need to restart service for changes to take effect.

sudo service rsyslog restart

after that you will have completely different log output in your logs.

Last thing we can do is our custom templates, which we define in the same conf file like

$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated::fulltime%,%HOSTNAME%,%syslogtag%,%msg%\n"

local0.panic;local0.alert;local0.crit;local0.err;local0.warning;local0.notice  -/var/log/drupal-alert.log; precise

and now we have our output look like whatever we want it to look. It can even be an SQL that would be used to insert data into DB.

$template dbFormat,"insert into SystemEvents (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql

What we want to do is make a modern format template, used in many places and that is JSON. So we will try it like this

template(name="outfmt" type="list" option.jsonf="on") {
         property(outname="@timestamp" name="timereported" dateFormat="rfc3339" format="jsonf")
         property(outname="host" name="hostname" format="jsonf")
         property(outname="severity" name="syslogseverity-text" format="jsonf")
         property(outname="message" name="msg" format="jsonf")
 }

and then add this template to 

local0.panic;local0.alert;local0.crit;local0.err;local0.warning;local0.notice  -/var/log/drupal-alert.log; outfmt

or just use local0.* /var/log/drupal-alert.log; outfmt to cover all severites.

So now we should have completely different output which should correspond to JSON format, ohh and yes this is newer templating process, which can be seen here
https://www.rsyslog.com/doc/v8-stable/configuration/templates.html
The one with the $template variable can also be used, but it is considered legacy. New templating with template() statement has few types available, above is list type, string type which looks like legacy can also be used, there are also plugin and subtree types, "list" will be mostly used and which properties can be set for that type you can see in before mentioned link.

Also to get full list of rsyslog properties then can be used can be find here https://www.rsyslog.com/doc/master/configuration/properties.html

Datadog

Lets use this logs to send it to datadog, after you installed service to your machine, you should go for integration configuration, which should be /etc/datadog-agent/conf.d/ here you can make /etc/datadog-agent/conf.d/drupal.d directory and add to it conf.yaml file. Fill it in with these lines so it is added to datadog config,

init_config:

instances:

## Log section
logs:

-   path: /var/log/drupal-alert.log
    service: drupal
    source: my_site
    sourcecategory: drupal
    type: file

After creating this new config, you should restart datadog agent with

sudo service datadog-agent restart

To see how I setup datadog through UI, check more in this post:

https://www.drupaldump.com/setting-datadog-alert-drupal-logs

Extras

1.If you are using some older versions of rsyslog, you maybe won't be able to use newer options like auto formatting of JSON (option.jsonf="on"), so you will need to make it by yourself. So do it like this

template(name="dr_json" type="list" option.json="on") {
  constant(value="{")
  constant(value="\"message\":\"") property(name="msg" regex.expression="\\|(.*)" regex.type="ERE" )
  constant(value="\",\"host\":\"") property(name="hostname")
  constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
  constant(value="\",\"type\":\"") property(name="msg" regex.expression="[^|]*" regex.type="ERE" )
  constant(value="\"}\n")
}

Also I used regex to extract type of logs to attribute and message to be without it.

2. This table might come in handy to know which severities you can get from rsyslog,  the keywords error, warn and panic are deprecated and should not be used anymore.

3. One more extra tip is if you want to have an alias of your host added to log, edit your

/etc/rsyslog.conf  and add this variable, so you dont have output like host: localhost but host: hex02

$LocalHostName hex02