Setting up datadog to alert on drupal logs

So before we made an setup for rsyslog and some advanced log output that can be read by datadog (or others).

In this post I will show you how to setup monitor, alerting and drupal syslog module. So let's first start with drupal syslog, I usually set this setup for syslog format

!type|!message|!ip|!request_uri|!referer|!uid

so there is no fluff in message I save to logs.  After all the setup we made in previous post we will receive proper logs in datadog and we can do some monitoring, so I first made an error monitor. Below is export you can import

{
	"id": 18192899,
	"name": "Hex Drupal Errors",
	"type": "log alert",
	"query": "logs(\"status:(error OR err) source:hex-test\").index(\"main\").rollup(\"count\").last(\"15m\") > 5",
	"message": "@slack-hex-alert <@marko.b> <@mmarko>",
	"tags": [],
	"options": {
		"queryConfig": {
			"timeRange": {
				"from": 1588519508813,
				"to": 1588605908813,
				"live": true
			},
			"queryString": "status:(error OR err) hex-test",
			"queryIsFailed": false,
			"logset": {
				"name": "main",
				"type": "logs",
				"dailyLimit": null,
				"rateLimited": null,
				"scopeId": "1301",
				"query": "",
				"retention": 15,
				"readDataAccess": true,
				"id": "1301",
				"dailyQuotaDisabled": true
			},
			"track": "logs",
			"indexes": [
				{
					"name": "main",
					"type": "logs",
					"dailyLimit": null,
					"rateLimited": null,
					"scopeId": "1301",
					"query": "",
					"retention": 15,
					"readDataAccess": true,
					"id": "1301",
					"dailyQuotaDisabled": true
				}
			]
		},
		"aggregation": {
			"metric": "count",
			"type": "count",
			"groupBy": ""
		},
		"thresholds": {
			"comparison": ">",
			"period": {
				"value": "last_15m",
				"seconds": 900
			},
			"critical": "5",
			"timeAggregator": "avg",
			"warning": 1
		},
		"enable_logs_sample": true,
		"notify_audit": false,
		"locked": false,
		"timeout_h": 0,
		"include_tags": true,
		"new_host_delay": 300,
		"notify_no_data": false,
		"renotify_interval": 0,
		"no_data_timeframe": 2
	}
}

it is pretty basic, we check for status:error in logs and send email when there is some threshold met, I used low one of 5 errors in 5 minutes, with warnings on 1 in 5 minutes as I want to know all of the errors of happening and act on them, your setup and needs might be different.

Finally if you want to be alerted in some other way than email, check possible integrations here
https://docs.datadoghq.com/monitors/notifications/?tab=is_alert#integrations
and how to set them up. I used slack and won't explain details here.

And one more thing, to test all of your setup it is best to make some test errors or warnings, so I used drush and sent some like

 drush php:eval '\Drupal::logger("test_error")->error("This is an test error");'