The main reason for networking is communication. While networking, crucial messages have to be passed between network devices so as to keep track of events as they occur. As a system administrator or a Developer Operations (DevOps) personnel, keeping track of activities ongoing over a network is very vital, and is very useful for solving problems when ever they surface.
The method of logging most times, is considered as time consuming or stressful. In the end, the effort is usually worth it. However, with syslog, all of that stress is reduced, as you could get to automate the logging process.All you just have to do is to go over the logs whenever a problem comes up and tackle the problems as the logs indicate.
Syslog is a known standard for message logging. Most times, the system that does the logging and the software that gets to generate them tend to interfere during processes. But syslog helps separate the software generating the logs from the system that stores the logs, thereby making the process of logging less complicated and stressful.
In other words, syslog is an open system, designed to help monitor network devices or systems and send events to a logging server. It ensures that messages are distinguished based on the priority of the messages and the sort of network device that is sending the message.
Apart from helping with the generating and storing of logs, it can also be used for security auditing as well as general analysis and debugging of system messages.
The syslog standard is available for use across different network devices such as routers, switches, load balancers, intrusion protection systems etc. by using the User Datagram Protocol of port 514 to communicate messages to the logging servers.
A syslog message follows either the legacy-syslog or BSD-syslog protocol and takes the following format:
- PRI message section
- HEADER message section
- MESSAGE section
A syslog message cannot ever go past 1024 bytes.
PRI message section
PRI is also known as the Priority Value part of the syslog message, and recall earlier that I talked about syslog sending logs messages according to the level of priority and also the type of network device or facility, here is where all that information is displayed. This part represents the facility and severity section of the syslog message.
The priority value is obtained by calculating the product of the facility number (the part of the system sending the message) by 8 and then adding the numerical value of the severity (this is the level of importance of the message according to the system.
Priority value = (Facility number * 8) + Severity
HEADER message section
While the PRI part was more about the system, the header part is more about the information that comes with the syslog event.
It contains the message timestamp, the hostname or the IP address of the system. The format of the timestamp field is:
MM dd hh:mm:ss
Where:
MM is the month in which the syslog was sent as an abbreviation. This means the month come in the form of Jan, Feb, Mar, Apr etc.
dd is the day of the month in which the message was sent. When the day is not double digits, the value is represented by a space and the number instead of a 0 and the number. This means “ 7” is used to depict 7 instead of “07”.
hh is the hour of the day when the message was sent, using the 24 hour time format. With values between 00 and 23, with 00 and 23 inclusive.
mm is the minute of the hour when the message was sent. With values between 00 and 59, with 59 inclusive.
ss is the second of the minute when the message was sent. With values between 00 and 59, with 59 inclusive.
An example of the above is:
Mar 8 22:30:15
MESSAGE section
This most times is where all of the needed information lies. It contains the name of the program, the process that led to the generation of the message and the text of the message itself.
The message part is usually in the format: program[pid]: message_text.
Example:
The following is a sample syslog message: <133>Feb 25 14:09:07 webserver syslogd: restart. The message corresponds to the following format: <priority>timestamp hostname application: message.
In the end, after generating the message, parsing it is a different ball game. You can parse the syslog using a programming language such as python, using regular expressions, using xml parser and you can also parse using json. A log parser like syslog-ng works perfectly with Python. It allows you write your own parser in python, allowing for much more control over the parsing potentials.
Python is very popular for scraping data, so you can easily find modules for scrapping the needed data from the syslog which makes it easier to process messages, query databases etc. If you intend using syslog-ng, you can get the OSE configuration file and include it in the file.
However, you should ensure that the PYTHON_PATH environment variable includes the path to the Python file and then you export the PYTHON_PATH environment variable.
For example:
export PYTHONPATH=/opt/syslog-ng/etc
The Python object is initiated only once, when syslog-ng OSE is started or reloaded. That means it keeps the state of internal variables while syslog-ng OSE is running. Python parsers consist of two parts. The first is a syslog-ng OSE parser object that you use in your syslog-ng OSE configuration, for example, in the log path.
This parser references a Python class, which is the second part of the Python parsers. The Python class processes the log messages it receives, and can do virtually anything that you can code in Python.
parser <name_of_the_python_parser>{ python( class("<name_of_the_python_class_executed_by_the_parser>") ); }; python { import re class MyParser(object): def init(self, options): '''Optional. This method is executed when syslog-ng is started or reloaded.''' return True def deinit(self): '''Optional. This method is executed when syslog-ng is stopped or reloaded.''' return True def parse(self, msg): '''Required. This method receives and processes the log message.''' return True }; When you finally get to parse your syslog file, you can then get to act on those issues that have been causing problems.
Most times, you would find the paths to the directories where the problem lies, so you can easily navigate directories using the “cd” command.
With syslog, you are able to save more time and improve efficiency.
