Icinga2 PagerDuty Configuration

| Comments

After some stumbling around the web finding a few nuggets of information on how to configure PagerDuty event reporting with Icinga2. I have what I belive to be a fully function solution.

The Icinga Integration Guide is a good basis to start off, with a few gotchas for Icinga2.

  1. The format for the notification and command objects needed for Icinga2
  2. The Icinga2 env vars need prefixed by “ICINGA_” as the pagerduty_icinga.pl script is looking for them
1
2
3
4
while ((my $k, my $v) = each %ENV) {
  next unless $k =~ /^ICINGA_(.*)$/;
  $event{$1} = $v;
}

Below is a manifest example that uses the Icinga2 Puppet Module with my commited objects to configure PagerDuty Integration. Note: You’ll need to download the pagerduty_icinga.pl script locally for this manifest to work correctly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
# Configure PagerDuty Alerting Service
#
# Template Examples:
# http://monitoring-portal.org/wbb/index.php?page=Thread&postID=204321
# https://lists.icinga.org/pipermail/icinga-users/2014-May/008201.html
class icinga2::arin::pagerduty (
  $pagerduty_service_apikey = undef,
) {

  include stdlib

  # Install Perl dependencies
  $pagerduty_dependencies_packages = [ 'perl-libwww-perl', 'perl-Crypt-SSLeay' ]
  ensure_packages ( $pagerduty_dependencies_packages )

  # Install PagerDuty Alerting Script
  file { 'pagerduty_icinga.pl':
    ensure  => file,
    path    => '/etc/icinga2/scripts/pagerduty_icinga.pl',
    owner   => 'root',
    group   => 'root',
    mode    => '0755',
    content => template('icinga2/pagerduty_icinga.pl.erb'),
  }

  # Create PagerDuty Icinga User
  icinga2::object::user { 'pagerduty':
    display_name => 'PagerDuty Notification User',
    pager        => $pagerduty_service_apikey,
    target_dir   => '/etc/icinga2/objects/users',
    states       => [ 'OK', 'Critical' ],
  }

  ## Configure Cron for Icinga User
  cron { 'icinga pagerduty':
    ensure   => present,
    command  => '/usr/local/bin/pagerduty_icinga.pl flush',
    user     => 'icinga',
    minute   => '*',
    hour     => '*',
    monthday => '*',
    month    => '*',
    weekday  => '*',
  }

  ## Configure Icinga2 PagerDuty Notification Command for Service
  icinga2::object::notificationcommand { 'notify-service-by-pagerduty':
    command            => ['"/icinga2/scripts/pagerduty_icinga.pl"', '"enqueue"', '"-f"', '"pd_nagios_object=service"', '"--verbose"'],
    cmd_path           => 'SysconfDir',
    template_to_import => 'plugin-notification-command',
    env                         => {
      '"ICINGA_CONTACTPAGER"'     => '"$user.pager$"',
      '"ICINGA_NOTIFICATIONTYPE"' => '"$notification.type$"',
      '"ICINGA_SERVICEDESC"'      => '"$service.name$"',
      '"ICINGA_HOSTNAME"'         => '"$host.name$"',
      '"ICINGA_HOSTALIAS"'        => '"$host.display_name$"',
      '"ICINGA_SERVICESTATE"'     => '"$service.state$"',
      '"ICINGA_SERVICEOUTPUT"'    => '"$service.output$"',
    },
  }

  ## Configure Icinga2 PagerDuty Notification Command for Hosts
  icinga2::object::notificationcommand { 'notify-host-by-pagerduty':
    command            => ['"/icinga2/scripts/pagerduty_icinga.pl"', '"enqueue"', '"-f"', '"pd_nagios_object=host"', '"--verbose"'],
    cmd_path           => 'SysconfDir',
    template_to_import => 'plugin-notification-command',
    env                         => {
      '"ICINGA_CONTACTPAGER"'     => '"$user.pager$"',
      '"ICINGA_NOTIFICATIONTYPE"' => '"$notification.type$"',
      '"ICINGA_HOSTNAME"'         => '"$host.name$"',
      '"ICINGA_HOSTALIAS"'        => '"$host.display_name$"',
      '"ICINGA_HOSTSTATE"'        => '"$host.state$"',
      '"ICINGA_HOSTOUTPUT"'       => '"$host.output$"',
    },
  }

  ## Configure Apply Notification to Hosts
  icinga2::object::apply_notification_to_host { 'pagerduty-host':
    assign_where => 'host.vars.enable_pagerduty == "true"',
    command      => 'notify-host-by-pagerduty',
    users        => [ 'pagerduty' ],
    states       => [ 'Up', 'Down' ],
    types        => [ 'Problem', 'Acknowledgement', 'Recovery', 'Custom' ],
    period       => '24x7',
  }

  ## Configure Apply Notification to Services
  icinga2::object::apply_notification_to_service { 'pagerduty-service':
    assign_where => 'service.vars.enable_pagerduty == "true"',
    command      => 'notify-service-by-pagerduty',
    users        => [ 'pagerduty' ],
    states       => [ 'OK', 'Warning', 'Critical', 'Unknown' ],
    types        => [ 'Problem', 'Acknowledgement', 'Recovery', 'Custom' ],
    period       => '24x7',
  }
}

For those who want to configure this manually, below are examples of the objects needed.

Icinga2 Apply Objects

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apply Notification "pagerduty-host" to Host {
    assign where host.vars.enable_pagerduty == "true"
  command = "notify-host-by-pagerduty"
  users = [ "pagerduty" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom ]
  states = [ Up, Down ]
}

apply Notification "pagerduty-service" to Service {
    assign where service.vars.enable_pagerduty == "true"
  command = "notify-service-by-pagerduty"
  users = [ "pagerduty" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom ]
  states = [ OK, Warning, Critical, Unknown ]
}

Icinga2 Notification Command Objects

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
object NotificationCommand "notify-host-by-pagerduty" {

  import "plugin-notification-command"

  command = [ SysconfDir + "/icinga2/scripts/pagerduty_icinga.pl", "enqueue", "-f", "pd_nagios_object=host", "--verbose" ]

  env = {
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_CONTACTPAGER" = "$user.pager$"
    "ICINGA_HOSTOUTPUT" = "$host.output$"
    "ICINGA_HOSTSTATE" = "$host.state$"
  }
}

object NotificationCommand "notify-service-by-pagerduty" {

  import "plugin-notification-command"

  command = [ SysconfDir + "/icinga2/scripts/pagerduty_icinga.pl", "enqueue", "-f", "pd_nagios_object=service", "--verbose" ]

  env = {
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_CONTACTPAGER" = "$user.pager$"
    "ICINGA_SERVICEDESC" = "$service.name$"
    "ICINGA_SERVICESTATE" = "$service.state$"
    "ICINGA_SERVICEOUTPUT" = "$service.output$"
  }
}

Icinga2 User Object

1
2
3
4
5
object User "pagerduty" {
  display_name = "PagerDuty Notification User"
  pager = "YOURAPIKEYGOESHERE"
  states = [  OK,  Critical, ]
}

Comments