Protecting our customers Our security research team has built and deployed a rule to protect…
Signal Sciences Agent Monitoring
In your efforts to defend your network and applications, deploying detection and protection security controls is a large part of the job. Once deployed, the focus switches to monitoring events, alerts, and other analysis. However, one important task that should not be left out is establishing health monitoring of security controls. Monitoring the health of your security controls helps ensure your defenses by enabling you to quickly identify and react to control failures. It would be tragic to suffer a security breach only to realize later that it was human error that disabled a security control, which let the attack go undetected.
For many organizations, the Signal Sciences agent is a key security control over production web applications for both detection and protection. As a result, monitoring and ensuring the health of the Signal Sciences agent deployments are important. With Signal Sciences, the UI, also known as the Console, provides an Agents page that displays agent status, and the API enables proactive automated monitoring via tools like Splunk, Datadog, and any homegrown solution. However, a recently released feature called Agent Alerts gives customers the ability to quickly turn on proactive agent health monitoring with alert notifications that integrate into existing channels and tools.
As you can see in the screenshot above, there are two monitoring options that can be easily enabled:
Low Traffic Alert
The first option will alert when the average requests per second (RPS) across all agents falls below 10. Obviously, if your site does not have an average RPS higher than 10, it may not be useful to enable this alert. However, if your site does have a high average RPS, this alert could indicate one of the following issues (though the possible causes are not limited to these three examples):
- Something has caused the agent process to unexpectedly stop. For example, the operations team may stop the agent service to perform unscheduled maintenance or troubleshooting.
- There may be a problem starting or deploying new agents, especially in dynamic environments where server instances are recycled frequently. For example, an erroneous update to a configuration script may have a typo that prevents the agent from starting.
- There could be an issue, external to the web servers, that is preventing traffic from reaching the web servers. For example, an unintended redirect was implemented on the load balancer.
No Agents Online Alert
The second alert will let you know when all agents for a site are effectively offline. This alert is suitable to enable for any site you are protecting with Signal Sciences. In fact, I would recommend all customers enable it. This alert is one worth immediate attention.
After you’ve enabled any of the two options, you’ll want to configure one of the integration options to receive the alerts. In the screenshot above, I’ve added a mailing list integration, so I will receive alerts via e-mail. For a full list of available integrations, see the documentation link on integrations. Also, for reference, here is the documentation link for setting up agent alerts.
Using the CLI to Manage Agent Alerts
Using the UI to manage Agent Alerts is great, but there’s an API too! Where there’s an API, there’s bound to be a CLI tool to make life even easier. One CLI tool for Signal Sciences API is pysigsci. This Python package provides the CLI tool and an API wrapper module that can be used to script your own automation. In this post, I’ll just focus on the CLI tool and its options for managing Agent Alerts. This tool will be especially helpful if you have Signal Sciences deployed to numerous sites within your organization. Below are example commands for various scenarios.
Enable both Agent Alerts for a site
$ pysigsci --enable agent-alerts --site mysite_shortname
Enable both Agent Alerts for all sites
$ pysigsci — enable agent-alerts-all-sites
Enable a specific Agent Alert for a site
If you only want to enable one of the alerts, you can specify which one using the –alert-tag-name option. The two possible agent tag names for Agent Alerts are requests_total and agent_scoreboards. The example command below uses agent_scoreboards, which will alert when the site’s online agent count is zero.
$ pysigsci --disable agent-alerts --site mysite_shortname–alert-tag-name agent_scoreboards
Enable a specific Agent Alert for all sites
$ pysigsci --enable agent-alerts-all-sites
--alert-tag-name requests_total
You can use the same commands for disabling the alerts, just use the --disable option instead of the --enable option.
Add an integration
The last step to configuring the alerts is to add an integration. This determines how the alert is delivered. For demonstration purposes, I’ll use the mailing list integration as the example. With the pysigisci tool, the –data option allows you to provide a JSON payload to create or update configuration.
$ pysigsci --add integration --data
'{"type":"mailingList",
"url":"ops@example.com",
"events":["agentAlert"]}'
Note, you can configure Agent Alerts with more than one integration. For example, you may want to receive Agent Alerts via Slack and send an Agent Alert to VictorOps. For more details on the integrations API endpoints, an example JSON payloads, see the API documentation.
Conclusion
Monitoring the health of security controls to minimize the risk of control failures is an important part of defending your network and applications. For Signal Sciences customers, the Agent Alerts feature provides instant health monitoring over agent activity. In addition, with the Signal Sciences API you can easily script automation to manage this feature or take advantage of the pysigsci CLI tool. To explore what else can be done with the Signal Sciences API, view the documentation here.
Related Posts
