検証環境

CentOS 7.3 1611
Vagrant

リファレンス

https://access.redhat.com/documentation/ja-JP/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/ch-abrt.html

インストールと起動

sudo yum install abrt-cli
sudo systemctl enable abrtd.service
sudo systemctl start abrtd

UNIT FILE                                     STATE   
abrt-ccpp.service                             enabled 
abrt-oops.service                             enabled 
abrt-pstoreoops.service                       disabled
abrt-vmcore.service                           enabled 
abrt-xorg.service                             enabled 
abrtd.service                                 enabled

検証なので puppet や chef は使わずに手作業でペペっと

abrt-pstoreoops.service は Vagrant で使えるのかどうか分からない
- カーネルパニックのログを pstore インタフェース( platform persistent storage ) であれこれする仕組み
- https://github.com/abrt/abrt/wiki/pstore-oops

プロセスを SIGSEGV で kill する

22.2. ABRT のインストールとそのサービスの起動の手順をなぞって試す。sleep プロセスを SIGSEGV で kill して動作を確かめるとよいらしい

$ sleep 100 &
[1] 2954
$ kill -SEGV 2954
$ fg
-bash: fg: job has terminated
[1]+  Segmentation fault      (core dumped) sleep 100

/var/log/messages にログが残る

Dec 15 10:53:21 localhost abrt-hook-ccpp: Process 2954 (sleep) of user 1000 killed by SIGSEGV - dumping core

abrt-cli list で abrtd がキャッチしたクラッシュを一覧できる

$ sudo abrt-cli list
id 3cd5767798e6794fa2593aa1ee29f52c91e41818
reason:         sleep killed by SIGSEGV
time:           Thu Dec 15 10:53:21 2016
cmdline:        sleep 100
package:        coreutils-8.22-18.el7
uid:            1000 (vagrant)
count:          1
Directory:      /var/spool/abrt/ccpp-2016-12-15-10:53:21-2954

The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges

さらに /var/spool/abrt/ にプロセス実行時のあれこれのデータが残る

$ sudo ls -hal /var/spool/abrt/ccpp-2016-12-15-10:53:21-2954
total 312K
drwxr-x---. 2 root abrt 4.0K Dec 15 11:08 .
drwxr-x--x. 4 root abrt   94 Dec 15 11:03 ..
-rw-r-----. 1 root abrt    6 Dec 15 10:53 abrt_version
-rw-r-----. 1 root abrt    4 Dec 15 10:53 analyzer
-rw-r-----. 1 root abrt    6 Dec 15 10:53 architecture
-rw-r-----. 1 root abrt  190 Dec 15 10:53 cgroup
-rw-r-----. 1 root abrt    9 Dec 15 10:53 cmdline
-rw-r-----. 1 root abrt    9 Dec 15 10:53 component
-rw-r-----. 1 root abrt 1.3K Dec 15 10:53 core_backtrace
-rw-r-----. 1 root abrt 376K Dec 15 10:53 coredump
-rw-r-----. 1 root abrt    1 Dec 15 10:55 count
-rw-r-----. 1 root abrt  298 Dec 15 10:53 dso_list
-rw-r-----. 1 root abrt 2.3K Dec 15 10:53 environ
-rw-r-----. 1 root abrt    0 Dec 15 10:53 event_log
-rw-r-----. 1 root abrt   14 Dec 15 10:53 executable
-rw-r-----. 1 root abrt    4 Dec 15 10:53 global_pid
-rw-r-----. 1 root abrt   21 Dec 15 10:53 hostname
-rw-r-----. 1 root abrt   25 Dec 15 10:53 kernel
-rw-r-----. 1 root abrt   10 Dec 15 10:55 last_occurrence
-rw-r-----. 1 root abrt 1.3K Dec 15 10:53 limits
-rw-r-----. 1 root abrt  135 Dec 15 10:53 machineid
-rw-r-----. 1 root abrt 1.6K Dec 15 10:53 maps
-rw-r-----. 1 root abrt  138 Dec 15 10:53 open_fds
-rw-r-----. 1 root abrt  393 Dec 15 10:53 os_info
-rw-r-----. 1 root abrt   37 Dec 15 10:53 os_release
-rw-r-----. 1 root abrt   21 Dec 15 10:53 package
-rw-r-----. 1 root abrt    4 Dec 15 10:53 pid
-rw-r-----. 1 root abrt    6 Dec 15 10:53 pkg_arch
-rw-r-----. 1 root abrt    1 Dec 15 10:53 pkg_epoch
-rw-r-----. 1 root abrt   19 Dec 15 10:53 pkg_fingerprint
-rw-r-----. 1 root abrt    9 Dec 15 10:53 pkg_name
-rw-r-----. 1 root abrt    6 Dec 15 10:53 pkg_release
-rw-r-----. 1 root abrt    6 Dec 15 10:53 pkg_vendor
-rw-r-----. 1 root abrt    4 Dec 15 10:53 pkg_version
-rw-r-----. 1 root abrt 1.2K Dec 15 10:53 proc_pid_status
-rw-r-----. 1 root abrt   13 Dec 15 10:53 pwd
-rw-r-----. 1 root abrt   23 Dec 15 10:53 reason
-rw-r-----. 1 root abrt    4 Dec 15 10:53 runlevel
-rw-r-----. 1 root abrt   10 Dec 15 10:53 time
-rw-r-----. 1 root abrt    4 Dec 15 10:53 type
-rw-r-----. 1 root abrt    4 Dec 15 10:53 uid
-rw-r-----. 1 root abrt    8 Dec 15 10:53 username
-rw-r-----. 1 root abrt   40 Dec 15 10:53 uuid
-rw-r-----. 1 root abrt  155 Dec 15 10:53 var_log_messages

各ファイルの詳細は長いので gist にまとめた https://gist.github.com/hiboma/5b7caa8e0a4a9e3254aeda9d86b86a72

自作のバイナリでのテスト

こんな C のコードを書いてテストをした

static int i = 0;

int main() {
    char buffer[16];

    for(;;) 
        buffer[i++] = 'a';
}

実行するとインデックスでの参照がスタック領域を超えページフォルトを起こし、SIGSEGV で kill される

/var/log/messages にログが残るが、先ほどとは内容が違う

Dec 15 10:57:09 localhost kernel: a.out[3057]: segfault at 7fff75726000 ip 0000000000400502 sp 00007fff75724710 error 6 in a.out[400000+1000]
Dec 15 10:57:09 localhost abrt-hook-ccpp: Process 3057 (a.out) of user 1000 killed by SIGSEGV - dumping core
Dec 15 10:57:09 localhost abrt-hook-ccpp: Failed to create core_backtrace: dwfl_getthread_frames failed: No DWARF information found
Dec 15 10:57:09 localhost abrt-server: Executable '/home/vagrant/a.out' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Dec 15 10:57:09 localhost abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2016-12-15-10:57:09-3057' exited with 1
Dec 15 10:57:09 localhost abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2016-12-15-10:57:09-3057'

RPM として提供されていないバイナリの場合は ProcessUnpackaged を yes にしておかないと abrtd の採取対象とならないようだ

$ sudo perl -i'' -nlpe 's/ProcessUnpackaged = no/ProcessUnpackaged = yes/' /etc/abrt/abrt-action-save-package-data.conf

再度同じ手順を踏むと abrtd がキャッチしてくれた (設定変更後の abrtd の再起動はいらないみたい)

$ sudo abrt-cli list
id 856651691a20719d3e5387eda67265af645f64da
reason:         a.out killed by SIGSEGV
time:           Thu Dec 15 11:02:27 2016
cmdline:        aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
uid:            1000 (vagrant)
count:          2
Directory:      /var/spool/abrt/ccpp-2016-12-15-11:02:27-3114

The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges