AoC 2022 Day 2: Log Analysis in Cyber Security

Posted on by

What Are Log Files and Why Are They Useful

Log files are files that contain historical records of events and other data from an application. Some common examples of events that you may find in a log file:

  • Login attempts or failures
  • Traffic on a network
  • Things (website URLs, files, etc.) that have been accessed
  • Password changes
  • Application errors (used in debugging)
  • and many, many more

By making a historical record of events that have happened, log files are extremely important pieces of evidence when investigating:

  • What has happened?
  • When has it happened?
  • Where has it happened?
  • Who did it? Were they successful?
  • What is the result of this action?

What Does a Log File Look Like?

 

Log files come in all shapes and sizes. However, a useful log will contain at least some of the following attributes:

  1. A timestamp of the event (I.e. Date & Time)
  2. The name of the service that is generating the logfile (I.e. SSH is a remote device management protocol that allows a user to login into a system remotely)
  3. The actual event the service logs (i.e., in the event of a failed authentication, what credentials were tried, and by whom? (IP address)).

Common Locations of Log Files

Windows

Windows features an in-built application that allows us to access historical records of events that happen. The Event Viewer is illustrated in the picture below:

a picture of the event viewer on Windows

Linux

Operating system log files on this version of Linux (and frequently software-specific log files like apache2) are kept in the /var/log directory. To view a list of all the log files on the system, type ls in the /var/log directory:

Listing log files within the /var/log directory
dessy@virtualmachine:/var/log$ ls -lah
total 724K
drwxrwxr-x   9 root      syslog          4.0K Nov 14 10:59 .
drwxr-xr-x  13 root      root            4.0K Oct 26  2020 ..
drwxr--r-x   3 root      root            4.0K Nov 14 10:56 amazon
drwxr-xr-x   2 root      root            4.0K Oct 26  2020 apt
-rw-r-----   1 syslog    adm              11K Nov 14 11:03 auth.log
-rw-rw----   1 root      utmp               0 Oct 26  2020 btmp
-rw-r--r--   1 root      root            7.3K Nov 14 10:59 cloud-init-output.log
-rw-r--r--   1 syslog    adm             251K Nov 14 10:59 cloud-init.log
drwxr-xr-x   2 root      root            4.0K Oct  7  2020 dist-upgrade
-rw-r--r--   1 root      adm              36K Nov 14 10:59 dmesg
-rw-r--r--   1 root      adm              36K Nov 14 10:56 dmesg.0
-rw-r--r--   1 root      root             12K Oct 26  2020 dpkg.log

Looking Through Log Files

Hundreds or even thousands of entries and numerous events can be found in log files very fast. Distinguishing useful information from meaningless data is challenging when analyzing log files. Software programs called Security Information and Event Management (SIEM), which include tools like Splunk, are focused on collecting logs for analysis.

Fortunately for us, the majority of operating systems already include a set of tools that let us browse log files. We’ll be using Linux’s grep command in this space.

Grep 101

Grep is a command used to look for specific text within a file. It examines the entire file for any text that fits our input after accepting a provided input (a text or value).

To use grep, we must first determine the location of the log file that we want to search for. By default, grep will use your current working directory.

Now that we know where our log files are, we can begin to proceed with learning how to use grep. To use grep, we need to do three things:

  • Call the command.
  • Specify any options that we wish to use (this will later be explained), but for now, we can ignore this.
  • Specify the location of the file we wish to search through (grep will first assume the file is in your current directory unless you tell it otherwise by providing the path to the file i.e. /path/to/our/logfile.log).

 

For example, in the terminal below, we are using grep to look through the log file for an IP address. The log file is located in our current working directory, so we do not need to provide a path to the log file – just the name of the log file.

Using grep to look in a log file for activity from an IP address
dessy@virtualmachine:~ grep "192.168.1.30" webaccess.log
192.168.1.30 - - [14/Nov/2022:00:53:07 +0000] "GET / HTTP/1.1" 200 13742
192.168.1.30 - - [14/Nov/2022:00:53:43 +0000] "HEAD

 

In the terminal above, we can see two entries in this log file (webaccess.log) for the IP address “192.168.1.30”. Here are some ideas for things you may want to use grep to search a log file for:

  • A name of a computer.
  • A name of a file.
  • A name of a user account.
  • An IP address.
  • A certain timestamp or date.

As previously mentioned, we can provide some options to grep to enable us to have more control over the results of grep. The table below contains some of the common options that you may wish to use with grep.

Option Description Example
-i Perform a case insensitive search. For example, “helloworld” and “HELLOWORLD” will return the same results grep -i "helloworld" log.txt and grep -i "HELLOWORLD" log.txt will return the same matches.
-E Searches using regex (regular expressions). For example, we can search for lines that contain either “thm” or “tryhackme” grep -E "thm|tryhackme" log.txt
-r Search recursively. For example, search all of the files in a directory for this value. grep -r "helloworld" mydirectory

Further options available in grep can be searched within grep‘s manual page via man grep

~ Source: Tryhackme.

Challenge Solution