System Clock Skewed? Read this Post, Especially if You Don’t Have Time

“Time keeps on slippin’ slippin’ slippin’, into the future”
-“Fly Like An Eagle”, The Steve Miller Band

CDNs (content delivery networks) were a great invention. Put pieces of content at the edge of the network to speed up distribution and take the load off of your web and application servers. Now your servers aren’t bogged down sending all the gifs, stylesheets, and other static content so you can let the app servers focus on the dynamic bits instead of serving your logo for the millionth time.

Your web server’s system time can keep on slippin’ far into the future if Network Time Protocol (NTP) is not properly configured. Having accurate system time is critical for application logic, scheduled jobs, and of course logging. With logging, if the system time is off, log forensics and log correlation of security events across systems becomes a nightmare, if not impossible. Many software components on your servers rely on accurate system time, including the Signal Sciences agent. The agent uses the local system clock to generate timestamps when logging detections and other events. As a result, ensuring accurate system time on all your web servers is a must for robust security — and this is especially true for virtual machine based deployments.

In addition to NTP not being configured, there has been research published on attacks against the NTP protocol. Some of which result in shifting time on NTP clients. This research is also referenced in this Ars Technica article. Another threat consideration is a malicious insider, who could modify system time in attempts to hide events or manipulate time sensitive transactions.

“Watching” Time

How can we mitigate the risk of skewed timestamps, missing data, and possible attacks on system time? First, NTP is a beautiful thing. Make sure you have a consistent NTP implementation across all your servers. Second, periodically verify the system time on your servers is not skewed beyond an acceptable threshold. Fortunately for Signal Sciences customers, you already have visibility into your server time! Since time is such a crucial part of security, our agent provides visibility on your server clock skew. This is just one of the many metrics reported on by the agent, and it can be found via the UI by navigating to the agent’s graph page. On that page, scroll down towards the bottom to find the Agent Clock Skew graph.

The screen shot above is from a VM, where the clock skew is obviously drastically off until it was corrected.

The screen shot above is from an agent running in a popular PaaS environment. A one second clock skew is relatively negligible, but depending on your circumstances perhaps it could be an issue.

If you’re not a Signal Sciences customer, don’t worry, keep reading! In the next section I’ve published two scripts, one is a proof-of-concept Bash shell script that can be used to help monitor system time across many servers.

Monitoring Clock Skew

Ok, visibility on clock skew — great! Pretty graph — great! But what if you are like many of our customers and have hundreds of agents deployed. That’s a lot of looking at pretty graphs to find hosts with clock skew — not great! Good news, Signal Sciences makes all data available via it’s API — great!!! This means you now have the ability to automate monitoring of clock skew across your web server farm.

Recommended reading: The Ultimate Guide to Web Application and API Protection (WAAP)

With the API you can integrate with any monitoring system you may have, however, I’ve provided an example of checking clock skew with just two scripts. The first script, SigSciApiPy, handles authenticating and pulling agent metrics from the API. The second script is shown below, clock_skew.py, loops through the agent data looking for the value of host.clock_skew. Host.clock_skew is the median number of seconds that the particular agent’s clock is skewed compared to the time on our servers, over the past 5 minutes. So for example, if we have “host.clock_skew”: 30, that means that we saw a median skew of 30 seconds over the past 5 mins. The clock_skew.py script will print a warning or alert message to the console based on defined thresholds.

#!/usr/bin/env python # # clock_skew.py # This script reads json data from /tmp/sigsci-agents.json # and prints a message if an agent's clock skew is greater # or equal to the defined thresholds. # # SigSciApiPy (https://github.com/signalsciences/SigSciApiPy) is # an easy way to pull agent data from the Signal Sciences API and # save it to /tmp/sigsci-agents.json # # Example usage with SigSciApi.py: # ./SigSci.py --agents > /tmp/sigsci-agents.json; ./clock_skew.py # # Thresholds WARN = 3 ALERT = 5 import json agents = open('/tmp/sigsci-agents.json', 'r') data = json.load(agents) skew = 0 count = 0 agents.close() for agent in data['data']: skew = int(agent['host.clock_skew']) if skew >= ALERT: print("ALERT: Clock skew for is ".format(agent['agent.name'], skew)) count  = 1 elif skew >= WARN: print("WARNING: Clock skew on is ".format(agent['agent.name'], skew)) count  = 1 if count == 0: print("Everything is running on time!")

Download or view the script on Github.

An example run of these scripts together looks like this:

./SigSci.py --agents > /tmp/sigsci-agents.json; ./clock_skew.py
WARNING: Clock skew on host1 is 4
ALERT: Clock skew for host2 is 5
WARNING: Clock skew on host3 is 4
WARNING: Clock skew on host4 is 3
ALERT: Clock skew for host5 is 5
ALERT: Clock skew for host6 is 6

Note, if you do use clock_skew.py, you’ll want to set the WARN and ALERT threshold variables to values that make sense to you.

If you are not a Signal Sciences customer, below is a script that can achieve a similar result in monitoring for clock skew. If shell scripts are not your thing, there are many other tools that may be a better fit in your environment. For example, monitoring systems like Nagios or Zabbix, or configuration management tools such as Ansible, Chef, or Puppet. I’m sure there are other, and even better, ways to monitor for clock skew. If you have suggestions please add them in the comments section.

#!/usr/bin/env bash # clock_skew.sh - Check remote hosts for clock skew (time drift), and # output a warning or alert message based on defined thresholds. # # Instructions: # Create a hosts.txt file that contains the list of hosts to be checked. # # hosts.txt should contain one host entry per line, and username and # port fields are optional. Format: # # <username>@<host or IP address>:<port> # # Note: this is a proof-of-concept script so prompting for a single # password to access all the hosts in your host.txt file may not be # feasible. Consider modifying the script to handle prompting for multiple # passwords, or use ssh keys. # # Set threshold variables below to your tolerance level. # # Thresholds WARN=10 ALERT=20 # prompt for ssh password. read -s -p "Enter password:" password echo export SSHPASS=$password count=0 # loop through hosts.txt file entries. for host in `cat hosts.txt`; do IFS=':' read -a host_config <<< "$" h=$ if [ ! -z $ ]; then p=$ else p=22 fi # get local and remote date/time. local_time=`date` remote_time=`sshpass -e ssh -p $ $ date` # convert the date/time to seconds from the Epoch. local_sec=`date --date="$"  %s` remote_sec=`date --date="$"  %s` # calculate the time skew. skew=`expr $ - $` # output messages if thresholds are met. if [ $skew -ge $ALERT ]; then echo "ALERT: Clock skew on $ is $." ((count  )) elif [ $skew -ge $WARN ]; then echo "WARNING: Clock skew on $ is $." ((count  )) fi done if [ $count -eq 0 ]; then echo "Everything is running on time!" fi

Download or view the script on Github.

Conclusion

I think we can all agree that ensuring you have accurate system time across your servers is critical for security. Also, having the ability to monitor for time skew is equally as important. Signal Sciences gives you this ability via its API, and with a few simple scripts you can monitor time skew across all your web servers. Even if you are not a Signal Sciences customer you should choose a monitoring method that makes sense for you, and start monitoring. For Signal Sciences customers, I hope the example scripts I’ve provided help you to jump start your clock skew monitoring efforts in a… timely manner.