What Are Log Files and Why Are They Useful
Log files are files that contain historical records of events and other data from an application. Some common examples of events that you may find in a log file:
- Login attempts or failures
- Traffic on a network
- Things (website URLs, files, etc.) that have been accessed
- Password changes
- Application errors (used in debugging)
- and many, many more
By making a historical record of events that have happened, log files are extremely important pieces of evidence when investigating:
- What has happened?
- When has it happened?
- Where has it happened?
- Who did it? Were they successful?
- What is the result of this action?
What Does a Log File Look Like?
Log files come in all shapes and sizes. However, a useful log will contain at least some of the following attributes:
- A timestamp of the event (I.e. Date & Time)
- The name of the service that is generating the logfile (I.e. SSH is a remote device management protocol that allows a user to login into a system remotely)
- The actual event the service logs (i.e., in the event of a failed authentication, what credentials were tried, and by whom? (IP address)).
Common Locations of Log Files
Windows features an in-built application that allows us to access historical records of events that happen. The Event Viewer is illustrated in the picture below:
Operating system log files on this version of Linux (and frequently software-specific log files like apache2) are kept in the /var/log directory. To view a list of all the log files on the system, type ls in the /var/log directory:
dessy@virtualmachine:/var/log$ ls -lah total 724K drwxrwxr-x 9 root syslog 4.0K Nov 14 10:59 . drwxr-xr-x 13 root root 4.0K Oct 26 2020 .. drwxr--r-x 3 root root 4.0K Nov 14 10:56 amazon drwxr-xr-x 2 root root 4.0K Oct 26 2020 apt -rw-r----- 1 syslog adm 11K Nov 14 11:03 auth.log -rw-rw---- 1 root utmp 0 Oct 26 2020 btmp -rw-r--r-- 1 root root 7.3K Nov 14 10:59 cloud-init-output.log -rw-r--r-- 1 syslog adm 251K Nov 14 10:59 cloud-init.log drwxr-xr-x 2 root root 4.0K Oct 7 2020 dist-upgrade -rw-r--r-- 1 root adm 36K Nov 14 10:59 dmesg -rw-r--r-- 1 root adm 36K Nov 14 10:56 dmesg.0 -rw-r--r-- 1 root root 12K Oct 26 2020 dpkg.log
Looking Through Log Files
Hundreds or even thousands of entries and numerous events can be found in log files very fast. Distinguishing useful information from meaningless data is challenging when analyzing log files. Software programs called Security Information and Event Management (SIEM), which include tools like Splunk, are focused on collecting logs for analysis.
Fortunately for us, the majority of operating systems already include a set of tools that let us browse log files. We’ll be using Linux’s grep command in this space.
Grep is a command used to look for specific text within a file. It examines the entire file for any text that fits our input after accepting a provided input (a text or value).
To use grep, we must first determine the location of the log file that we want to search for. By default, grep will use your current working directory.
Now that we know where our log files are, we can begin to proceed with learning how to use
grep. To use grep, we need to do three things:
- Call the command.
- Specify any options that we wish to use (this will later be explained), but for now, we can ignore this.
- Specify the location of the file we wish to search through (
grepwill first assume the file is in your current directory unless you tell it otherwise by providing the path to the file i.e. /path/to/our/logfile.log).
For example, in the terminal below, we are using
grep to look through the log file for an IP address. The log file is located in our current working directory, so we do not need to provide a path to the log file – just the name of the log file.
dessy@virtualmachine:~ grep "192.168.1.30" webaccess.log 192.168.1.30 - - [14/Nov/2022:00:53:07 +0000] "GET / HTTP/1.1" 200 13742 192.168.1.30 - - [14/Nov/2022:00:53:43 +0000] "HEAD
In the terminal above, we can see two entries in this log file (webaccess.log) for the IP address “192.168.1.30”. Here are some ideas for things you may want to use grep to search a log file for:
- A name of a computer.
- A name of a file.
- A name of a user account.
- An IP address.
- A certain timestamp or date.
As previously mentioned, we can provide some options to
grep to enable us to have more control over the results of grep. The table below contains some of the common options that you may wish to use with
|-i||Perform a case insensitive search. For example, “helloworld” and “HELLOWORLD” will return the same results||
|-E||Searches using regex (regular expressions). For example, we can search for lines that contain either “thm” or “tryhackme”||
|-r||Search recursively. For example, search all of the files in a directory for this value.||
Further options available in grep can be searched within grep‘s manual page via
~ Source: Tryhackme.