systemd System and Service Manager
What is this?
systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts.
systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services,
offers on-demand starting of daemons, keeps track of processes using Linux control groups,
supports snapshotting and restoring of the system state, maintains mount and automount points
and implements an elaborate transactional dependency-based service control logic.
The ($) and (#) signs before all commands, just indicates how to enter the commands.
so don't actually type them. (#) means you have to be root, ($) means normal user
- 1 systemctl usage
- 2 journalctl usage
- 3 systemd timers
- 4 analyzing and performance
- 5 debugging
- 6 tips and tricks
- 7 systemd for Administrators
- 8 documentation for developers
let's start with some basics. To access the logs of the journal use the journalctl tool.
To have a first look at the logs, just type in:
If you run this as root you will see all logs generated on the system, from system components the same way
as for logged in users. The output you will get looks like a pixel-perfect copy of the traditional /var/log/messages format,
but actually has a couple of improvements over it:
- Lines of error priority (and higher) will be highlighted red.
- Lines of notice/warning priority will be highlighted bold.
- The timestamps are converted into your local time-zone.
- The output is auto-paged with your pager of choice (defaults to less).
This will show all available data, including rotated logs.
Browsing logs this way is already pretty nice.
But requiring to be root sucks of course, even administrators tend to do most of their work as unprivileged users these days.
By default, Journal users can only watch their own logs, unless they are root or in the adm group.
To make watching system logs more fun, you could add yourselve to adm:
# usermod -a -G adm yourusername
After logging out and back in as yourusername you have access to the full journal of the system and all users:
If invoked without parameters journalctl will show you the current log database.
Sometimes one needs to watch logs as they grow, where one previously used tail -f /var/log/messages:
$ journalctl -f
Yes, this does exactly what you expect it to do: it will show you the last ten logs lines,
and then wait for changes and show them as they take place.
When invoking journalctl without parameters you'll see the whole set of logs, beginning with the oldest message stored.
That of course, can be a lot of data. Much more useful is just viewing the logs of the current boot:
$ journalctl -b
This will show you only the logs of the current boot, with all the gimmicks mentioned.
But sometimes even this is way too much data to process.
So let's just listing all the real issues to care about: all messages of priority levels ERRORS and worse,
from the current boot:
$ journalctl -b -p err
But, if you reboot only seldom the -b makes little sense, filtering based on time is much more useful:
$ journalctl --since=yesterday
And there you go, all log messages from the day before at 00:00 in the morning until right now. Awesome!
Of course, we can combine this with -p err or a similar match. But suppose, we are looking for something that happened on the
15th of October, or was it the 16th?
$ journalctl --since=2012-10-15 --until="2011-10-16 23:59:59"
And there we go, we found what we were looking for. But i noticed that some CGI script in Apache was acting up earlier today, let's see what Apache logged at that time:
$ journalctl -u httpd --since=00:00 --until=9:30
There we found it. But... , wasn't there an issue with that disk /dev/sdc? Let's figure out what was going on there:
$ journalctl /dev/sdc
Ouch ! a disk error! Hmm, maybe quickly replace the disk before we lose data.
Wait... didn't I see that the vpnc binary was nagging? Let's check for that:
$ journalctl /usr/sbin/vpnc
I don't get this, this seems to be some weird interaction with dhclient, let's see both outputs, interleaved:
$ journalctl /usr/sbin/vpnc /usr/sbin/dhclient
As you can see here with the given examples, Journalctl is a pretty advanced tool, than can track down pretty much anything.
But we're not done yet. Journalctl has some more to offer, which will be showed in the section Advanced Usage.
Internally systemd stores each log entry with a set of implicit meta data.
This meta data looks a lot like an environment block, but actually is a bit more powerful.
This implicit meta data is collected for each and every log message, without user intervention.
The data will be there, and wait to be used by you. Let's see how this looks:
$ journalctl -o verbose -n $ Fri, 2013-11-01 19:22:34 CET [s=ac9e9c423355411d87bf0ba1a9b424e8;i=4301;b=5335e9cf5d954633bb99aefc0ec38c25;m=882ee28d2;t=4ccc0f98326e6;x=f21e8b1b0994d7ee] PRIORITY=6 SYSLOG_FACILITY=3 _MACHINE_ID=a91663387a90b89f185d4e860000001a _HOSTNAME=epsilon _TRANSPORT=syslog SYSLOG_IDENTIFIER=avahi-daemon _COMM=avahi-daemon _EXE=/usr/sbin/avahi-daemon _SYSTEMD_CGROUP=/system/avahi-daemon.service _SYSTEMD_UNIT=avahi-daemon.service _SELINUX_CONTEXT=system_u:system_r:avahi_t:s0 _UID=70 _GID=70 _CMDLINE=avahi-daemon: registering [epsilon.local] MESSAGE=Joining mDNS multicast group on interface wlan0.IPv4 with address 172.31.0.53. _BOOT_ID=5335e9cf5d954633bb99aefc0ec38c25 _PID=27937 SYSLOG_PID=27937 _SOURCE_REALTIME_TIMESTAMP=1351029098747042
(I cut out a lot here, I don't want to make this story overly long. Without the -n parameter it shows you
the last 10 log entries, but I cut out all but the last.)
With the -o verbose switch we enabled verbose output. Instead of showing a pixel-perfect copy of classic
/var/log/messages that only includes a minimimal subset of what is available,
we now see all the details the journal has about each entry, but it's highly interesting: there is user credential information.
Now, as it turns out the journal database is indexed by all of these fields, out-of-the-box! Let's try this out:
$ journalctl _UID=70
And there you go, this will show all log messages logged from Linux user ID 70.
As it turns out you can easily combine these matches:
$ journalctl _UID=70 _UID=71
Specifying two matches for the same field will result in a logical OR combination of the matches.
All entries matching either will be shown, i.e. all messages from either UID 70 or 71
If you specify two matches for different field names, they will be combined with a logical AND.
All entries matching both will be shown now, meaning that all messages from processes named avahi-daemon and host bwg-inc.
$ journalctl _HOSTNAME=bwg-inc _COMM=avahi-daemon
But of course, that's not fancy enough for us. We must go deeper:
$ journalctl _HOSTNAME=bwg-inc _UID=70 + _HOSTNAME=epsilon _COMM=avahi-daemon
The + is an explicit OR you can use in addition to the implied OR when you match the same field twice.
The line above means: show me everything from host bwg-inc with UID 70, or of host epsilon with a process name of avahi-daemon.
And now it becomes Magic
Who can remember all those values a field can take in the journal, I mean, who has that kind of photographic memory?
Well, the journal has:
$ journalctl -F _SYSTEMD_UNIT
This will show us all values the field _SYSTEMD_UNIT takes in the database, or in other words:
the names of all systemd services which ever logged into the journal. This makes it super-easy to build nice matches.