Observability

ThreatIngestor comes with a few configurable options for observability:

  • Logging
  • Metrics
  • Notifications

By default, ThreatIngestor will print some debug logs to stderr, and will send some basic metrics to a local statsd server on the default port, if it finds one.

Logging

ThreatIngestor uses Loguru for logging, and will pass in any config you define in your config.yml in a logging: section.

Any options accepted by Loguru’s configure function can be defined in your config.yml. You can also use the supported environment variables to change logging options without modifying the config.

Metrics

ThreatIngestor uses statsd, through the python-statsd library, to track a few different types of metrics. If you’d like to track counts for each type of artifact, error rates, how long it takes your sources/operators to run, set up a statsd server and point it to a frontend like Graphite or a (paid) service like Datadog or Librato.

Any options accepted by StatsClient can be defined in your config.yml, in a statsd section.

For example:

statsd:
    prefix: 'threatingestor'

This will tell ThreatIngestor to prefix all its metrics with threatingestor, which is useful if you have more than one tool feeding into your statsd server.

Notifications

ThreatIngestor uses Notifiers for notifications, and will pass in any config you define in your config.yml in a notifiers: section.

Notifiers supports several providers, including email, common chat serices, push notifications, and more.

Here’s an example using HipChat:

notifiers:
    provider: hipchat
    defaults:
        team_server: https://myteamserver
        room: 'ROOMID'
        token: MYTOKEN
        id: ID
        message_format: text
        notify: false

For documentation on required parameters for each provider, take a look at the Notifiers Providers docs. Anything you define in the config will be passed into this NotificationHandler logging interface.