Easy and Efficient System Alerts
Anybody setting up a system that operates a cash making – or extra to the purpose, doubtlessly dropping – enterprise wants to just accept that it could possibly go mistaken. Line-by-line whereas writing the code, the programmer should logically account for what happens in each department. That’s the place choices are made. Choices must be recorded.
Is there an assumption {that a} result’s constructive? What if it’s unfavourable? This occurred prior to now a number of years in a lot banking code when rates of interest went beneath zero which had been assumed unattainable. Code crashed or produced deceptive outcomes.
The ethical of the story is that each doable consequence ought to be explicitly acted upon, although this may be tedious. As a minimum, there ought to be a log. What’s a log?
The time period comes from shipping in the 15th century the place to estimate pace an precise log of wooden can be thrown over the aspect hooked up to a rope. It was knotted at common intervals (that is the place the time period “knots” for pace at sea comes from). It was vital to notice the instances at which these knots handed behind the ship. The notes have been themselves termed “logs”.
Such a log would, at a minimal, report the time. That is true of in the present day’s pc system logs. They sometimes include a time, a standing and a message:
2023-02-27T12:01:16.217 CRITICAL markets Could not create container: not sufficient reminiscence
Earlier in pc system historical past, staff would take turns watching log-files being printed out dwell. The function was generally colloquially termed the “watchdog”.
Lately there are too many methods and related logs to bodily watch, however logs stay very important. When there’s a system failure they’re looked for indications of the trigger.
Nevertheless, logs will also be watched programmatically and used for alerting. The topic of this weblog is organising a easy system for doing that.
There are numerous log alerting methods available on the market. One of the best recognized might be Datadog. There’s additionally Logtail, Papertrail, Splunk, Logstash and others. These are nicely put collectively merchandise with a number of nice options, equivalent to wonderful UIs, subtle dwell looking by way of internet interfaces and generally question languages and alerting. They require varied ranges of set up they usually have prices, both by way of volume-based tiered methods or month-to-month funds. For a bootstrapped enterprise, this may be problematic, as an illustration when a surge of logs – indicating a doable essential drawback that must be solved – pushes quantity on to a different tier. Ought to the “log ransom” be paid?
As a substitute, I recalled from earlier instances certainly the only log watcher: Swatchdog. It’s slightly venerable software program. Its file historical past from its supply obtain reveals dates in 2015, nevertheless it was written a lot earlier – the 90s or probably 80s by Todd Atkins.
To put in it, do
(It was referred to as “Swatch” nonetheless a watch producer objected!)
It’s troublesome to search out documentation on-line. As a substitute – it’s proper there within the implementation! You merely use the Linux pod2text command:
pod2text `which swatchdog`|much less
Alternatively, generate an HTML file with
pod2html `which swatchdog` > swatchdog.html
Its language is Perl, which has fallen out of favour with the rise of Python. Python is way simpler to jot down – and browse. Additional, the bash shell language encompasses most of Perl’s performance now. Nevertheless, within the days swatchdog was written, Perl was the default language for system directors. It had a big energetic and enthusiastic consumer and developer base. They produced many packages that have been distributed by way of CPAN. CPAN has “…has 213,554 Perl modules in 44,051 distributions, written by 14,345 authors”. In actual fact some giant methods have been written in Perl.
Perl has important options that make it nicely fitted to some duties, notably within the system administration house. It’s well-known for its succinctness (although that is one explanation for the problem to learn its code). It has a really expressive regular expression language. These two facets make it work nicely for automated log-watching.
swatchdog runs a course of for every file it watches. In its easiest kind it makes use of an everyday expression that every log-line matches towards and a command that runs within the case of a constructive match.
For our system ProfitView, most code is in Python by which we use the logging package deal. Due to this fact we use quite simple Perl regular expressions that match these kind of logs. In follow, in manufacturing, we search for cases of the time period CRITICAL. In growth and pre-prod we use the system selectively in additional intensive methods. We’re possible to make use of it in manufacturing extra extensively because the system evolves.
Such “crucial” strains shall be logged in circumstances by which our programmers decide {that a} code path being hit signifies a critical system drawback that should rectified or a minimum of examined within the brief time period. For instance:
logging.crucial(f"Could not create {useful resource}: not sufficient reminiscence")
would possibly create the log-line cited above.
We wanted an operationally easy system that had no different dependence on enterprise crucial features. Additional, since we wished it for use throughout our property, it wanted to be lightweight by way of system sources. swatchdog itself actually suits this invoice – it’s a small Perl program. We wished to maintain to this mannequin for its integration into our personal operation. Due to this fact we adopted a “minimal vital” rule in choices on implementation.
Python Wrappers
swatchdog is designed to be run from the command line, managed by intensive choices. Our utilization for the medium time period a minimum of shall be easy. We’re unlikely to must be alerted to troublesome to search out identifiers. Due to this fact we received’t want to contemplate many of those choices.
Nevertheless, we do have to have our swatchdog utilization automated. We use systemd. Due to this fact we wanted to introduce wrapper scripting for each the beginning and cease of the set up.
We initially used bash scripts for this goal, nonetheless, principally because of the want for a number of ranges of special-character escaping, this proved to be complicated and thus error-prone. A easy Python script solved the issue. A single script was used with an choice to decide on to both begin it or shut it down.
[Unit]
Description=Swatchdog Service
After=community.goal
[Service]
Kind=forking
ExecEnvironmentFile=/and many others/swatchdog/surroundings
ExecStart=/usr/bin/env ${PYTHON_EXECUTABLE} /and many others/swatchdog/swatchdog.py
ExecStop=/usr/bin/env ${PYTHON_EXECUTABLE} /and many others/swatchdog/swatchdog.py -t
[Install]
WantedBy=multi-user.goal
The script itself can also be easy as could be seen here.
Configuration
Tailoring our swatchdog implementation to your wants is – sure, easy. It depends on two elements: a controlling JSON file, specifying which log-files to look at and which “watchfor” information to make use of for them.
There may be additionally an surroundings file housed in /and many others/swatchdog/env. It’s formatted as required by systemd’s EnvironmentFile mechanism, however really utilized by this implementation in three other ways! This coupling shouldn’t be ultimate, however expedient. The aim of the file is to retailer native or (to some degree) secret knowledge. In our fast case it shops our Slack API key and monitoring channel title and the trail to the Python set up we use.
It’s value clarifying how this file’s utilization is applied. It’s a easy “tag=worth” checklist – nothing extra. Nevertheless systemd Unit information have some guidelines that barely complicate the file’s use – see beneath.
Constrained by systemd’s Atmosphere system to make use of an EnvironmentFile file, we nonetheless wished to have just one place for this sort of native knowledge. Because the file’s format is genuinely about so simple as it’s doable to be, this labored – although not fairly trivially.
The info from this file is required in two different locations: the primary is within the watchfor information the place the Python interpreter is used to ship alerts to Slack. The second is within the script that does the Slack message ship – it wants the Slack API key. So, Python must learn the file and so does Perl! This can be a little complicating, however in the long run, not excessively so – and it really works.
systemd
There’s limitless criticism of systemd within the Linux neighborhood and I count on it is going to be correctly outmoded within the subsequent few years. I haven’t wanted to study a lot about it till now. It’s a curiously un-robust system. Specifically in Exec strains, it seems that the primary “phrase” is particular (the “command”) and these can’t be substituted utilizing Atmosphere mechanisms – however subsequent phrases in these strains could be. Utilizing /usr/bin/env is allowed, nonetheless, for the command.
ExecStart=/usr/bin/env ${PYTHON_EXECUTABLE} /and many others/swatchdog/swatchdog.py
swatchdog “watchfor” information
The “watchfor” swatchdog configuration information are snippets of Perl which can be pulled into the complete script for the aim of matching. They permit using perlcode strains. These are strains of arbitrary perl which can be executed “round” the matching line(s). They are often configured to run at varied ranges. We simply want strains that learn the env file and extract the textual content of the Python interpreter path. That is really a small perl script in itself – I’m mildly stunned to see that the Python model is shorter – one line (an enormous a part of perl’s enchantment was at all times its terseness). It’s messy to have the entire script sitting in every watchfor .cfg file, so we place this in a perl “module” .pm file and import it with “use”.
package deal Env;
use strict;
use warnings;
sub parameterName {
my ($file, $parameter) = @_;
my $worth; open my $fh, '<', $file or die "Couldn't open file '$file': $!";
whereas (my $line = <$fh>) {
chomp $line;
if ($line =~ /^$parameter=(.*)$/) {
$worth = $1;
final; # we discovered what we have been in search of, no have to proceed studying the file
}
}
shut $fh;
return $worth;
}
1;
and
perlcode use lib '/and many others/swatchdog/watchfor';
perlcode use Env;
perlcode my $python_executable = Env::parameterName('/and many others/swatchdog/env', 'PYTHON_EXECUTABLE');
watchfor /RegularExpressionToFind/
exec $python_executable /and many others/swatchdog/swatchdog_slack.py test_server 'Message to ship' "$0"
Howdy My AI Good friend
There’s a bit story with this script. I spent about half a irritating day endlessly looking Stackoverflow and many others tweaking the script and retrying to get it working – to no avail.
This sort of factor was at all times the issue with Perl. It’s a prime purpose why Python was such a revolution (the opposite was that it was object-oriented, which was a brand new idea). Again then, “Perl Monks” would get respect by with the ability to internalise its arcana (a bit like “KDB/Q Gods” extra just lately). My private expertise was that, in distinction to Perl, Python simply did what you requested it to, first time.
So I believed, who can I ask to assist me? Perl is so outdated that what’s on the web is dispersed and in confusingly threaded bug-reports. Who do I do know who can nonetheless recall all these things?
So I requested ChatGPT
write a perl script that searches a file each line of which is of the shape
<tag>=<worth>for a tag PYTHON_EXECUTABLE and writes the worth of it right into a variable $python_executable that may then be utilized in the remainder of the script
I had a totally functioning outcome 30 seconds later…
I pasted into the watchfor file (every line prefixed with perlcode) – it immediatedly labored flawlessly. That actually gave me pause. I requested a pair extra questions to assist put the code in a module and name it.
It’s an incredible instance of using ChatGPT as a sensible coding instrument – to assist with facets of labor the place you’ll be able to acknowledge your lack of experience.
The Slack message script
As talked about above, the Python to learn the env file is a one-liner:
with open(sys.argv[Slack.ENVS]) as envs:
env = dict(v.strip().break up('=') for v in envs.readlines())
Operation
swatchdog use is initiated within the traditional manner with
sudo systemctl allow swatchdog
which provides a symlink on the applicable location within the /and many others/systemd/system hierarchy. Due to this fact, on startup, the Python script is executed selecting the run_watchfors() operate (since there’s no choices chosen).
The script works by way of the JSON file, lastly accumulating the right log file and “watchfor” file to run. The watchfor information are assumed to be in /and many others/swatchdog/watchfor, nonetheless this may be modified within the argument handling of swatchdog.py.
{
"logfiles": [
{
"directory": "/home/directory/path",
"files": [
{
"name": "first_logfile.log",
"watchfor": "first_logfile_config.cfg"
},
...
]
},
...
]
}
It then executes swatchdog as a script within the regular manner, which is by use of subprocess.run(). Thus, for every log file monitored there may be one swatchdog executable occasion. Inside the swatchdog Perl script there may be additionally some course of administration leading to two processes for every. Due to this fact in a system utilizing an association like ours that’s of affordable complexity there shall be many swatchdog initiated processes.
Regular Perform
swatchdog detects the addition of textual content to the file it screens. It calls a script to get the Python executable to execute as described above. It then seems to be for strains in its related common expression “watchfor” beginning watchfor. It then applies the Perl Regular Expression textual content on that line to the textual content added to the watched log file. If there’s a match, it executes the instructions on the following strains
perlcode Use ....
watchfor / [CRITICAL/
exec $PYTHON_EXECUTABLE /etc/swatchdog/swatchdog_slack.py server_name 'server [A CRITICAL error has occourred]' "$0"
On this instance it seems to be for the string “CRITICAL” occurring after an area and left square-bracket character ([). The common expression is positioned between two terminating characters, on this case slashes (/) – although different characters might be used. The left square-bracket is important to Perl common expressions, subsequently it should be escaped with a number one backslash.
Within the case that there’s a match, the “exec” directive specifies that an exterior command is for use. Alternatively, inside swatchdog instructions – like “mail” to ship emails – might be used. Within the present case, the Python file to ship Slack messages known as as described beneath.
The info is shipped in 4 arguments as required within the Python file. The second-to-last of those makes use of a Perl variable $0 which comprises the entire textual content that was added to the log file and detected by swatchdog. In case you want to ship an alternate location of the EnvironmentFile that may be handed because the final parameter. It defaults to /and many others/swatchdog/env.
Shutdown
It’s doable to place collectively an environment friendly and stylish course of administration system that can work for a swatchdog set up like this one, in all probability utilizing swatchdog’s --pid-file choice which writes the method id to a file. Nevertheless, that was not thought vital in follow. As a substitute, for shutdown, the Python script known as once more, this time with the -t choice. The Linux comfort script pgrep is then used to assemble all of the process-ids, that are then merely terminated with os.kill() and the SIGKILL sign. That is crude however easy and efficient. Due to the selection to make use of swatchdog as a wholly separate system to different purposeful processes, there may be minimal threat of issues with this “brute pressure” method.
Notification
We choose notifications by way of Slack. For this, we initially used a easy REST interface. Nevertheless, as a consequence of a later problem with particular character escaping, we determined to make use of the official Slack API within the hope of avoiding these issues. In actual fact, this didn’t assist with the particular character escaping. However the official API is cleaner so we continued utilizing it. It’s extra applicable to make use of that API for Slack notifications and we haven’t wanted to think about generalising to different notification methods at this juncture.
To execute these notifications we launched one other simple Python script. It merely codecs the notification textual content minimally and passes it to the Slack API.
As famous, a greater course of administration scheme might be utilised. There’s a small threat that one other system whose course of names are ‘swatch’ could also be put in. If this occurred it’s possible they’d be killed throughout a shutdown of the Swatchdog service and this would possibly trigger issues. At this stage this isn’t thought of a threat.
Slack’s API sends messages inside JSON wrappers. To take action it should first escape these parts of the messages which can be particular characters in JSON. This escaping shouldn’t be reversed when output in Slack, leading to textual content that isn’t as simply readable as it could usually be. Specifically, we use JSON internally in order that error messages usually comprise JSON – many characters of which should naturally be escaped. This drawback has not but been solved. It’s thought of an annoyance, however non-critical.
The usage of swatchdog solves a big drawback for us – notification of system warnings – in a easy manner. The system is free and open supply. It makes use of few sources.
It’s easy sufficient to be simply understood from scratch in lower than an hour, in order that further log-files could be added by competent non-experts.
The system could be prolonged with out important threat by a junior developer.
Total, it’s a good answer for a minimum of as much as mid-sized software program installations.
The code is available with the MIT license. Please submit Points and Pull Requests.