Dashboard
Getting Started

Configuring the Gremlin Agent

This documentation page shows you how to configure the Gremlin Agent. You can configure Gremlin using either environment variables or a configuration file, although we recommend using the configuration file in most cases. This guide assumes you've already installed and authenticated the Agent.

Using the configuration file

Gremlin's configuration file is located in a file named config.yaml. On Linux, this file lives at /etc/gremlin/config.yaml, and on Windows, its path is C:\ProgramData\Gremlin\Agent\config.yaml. By default, Gremlin ships with a commented-out version of this file. Use the comments or the following example to set up your configuration.

Note
Any changes to config.yaml require a restart of the gremlind service.

Accepted variables

Variable Description
identifier a string that uniquely identifies a Gremlin installation (defaults to the hosts IP address)
team_id the ID of your Gremlin team
tags key-value pairs that help you target this machine during experiments
team_secret the secret value (used by secret authentication only)
team_certificate the public portion of the certificate-pair (required for signature auth only)
team_private_key the private portion of the certificate-pair (required for signature auth only)
https_proxy the url of a proxy in between Gremlin and api.gremlin.com
ssl_cert_file the ssl certificate for your https proxy
push_metrics set to false if you would like to disable system metrics used for experiment charting (defaults to true)
collect_processes enables/disables Process Collection (as of Linux Agent 2.25.0 default is true)
iam_role role the agent uses to retrieve remote cloud config values

Example: config.yaml

YAML

## Gremlin Identifier; uniquely identifies this machine with Gremlin
## AWS ARNs are also supported (see below section on supported services and credentials)
## (can also set with GREMLIN_IDENTIFIER environment variable)
identifier: gremlin-01

## Gremlin Team Id; you can find this value at https://app.gremlin.com/settings/teams
## (can also be set with GREMLIN_TEAM_ID environment variable)
team_id: 11111111-1111-1111-1111-111111111111

## Gremlin Client Tags; Tag your machine with key-value pairs that help you target this machine during experiments
## (can also set with GREMLIN_CLIENT_TAGS environment variable)
tags:
  service: pet-store
  interface: http

## Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`
## (can also set with GREMLIN_TEAM_SECRET environment variable)
#team_secret: 11111111-1111-1111-1111-111111111111

## Gremlin Team Certificate, should not be set when using `team_secret`.
## Paste certificate content here or a paths to the file (prefixed with `file://`)
## AWS ARNs are also supported (see below section on supported services and credentials)
## (can also set with GREMLIN_TEAM_CERTIFICATE_OR_FILE environment variable)
team_certificate: |
  -----BEGIN CERTIFICATE-----
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  1111111111111111111111111111111111111111111111111111111111111111
  11111111111111111111111111111111111111111111111111==
  -----END CERTIFICATE-----

## Gremlin Team Certificate, should not be set when using `team_secret`.
## Paste certificate content here or a paths to the file (prefixed with `file://`)
## AWS ARNs are also supported (see below section on supported services and credentials)
## (can also set with GREMLIN_TEAM_PRIVATE_KEY_OR_FILE environment variable)
team_private_key: file:///var/lib/gremlin/key.pem

## HTTPS Proxy, set this when routing outbound Gremlin HTTPS traffic through a proxy
## (can also set with HTTPS_PROXY or https_proxy environment variables)
https_proxy: https://localhost:3128

## SSL CERT FILE, set this when using a https proxy with a self-signed certificate
## Paste certificate content here or a paths to the file (prefixed with `file://`)
## (can also set with SSL_CERT_FILE environment variable)
ssl_cert_file: file:///var/lib/gremlin/proxy_cert.pem

## Push Metrics, tell Gremlin whether to send system metrics to the control plane for charting the impact of experiments in
## real time. Metrics are only collected during active experiments, and only metrics relevant to the experiment are collected.
## defaults to `true`
## (can also set with PUSH_METRICS environment variable)
push_metrics: true

## Collect Process Data, data about running processes is sent to Gremlin for service discovery.
## defaults to `true` as of Gremlin Linux Agent 2.25.0
collect_processes: true


Restarting the Gremlin Agent

To restart the Gremlin Agent and apply your new changes, run the following commands (depending on your environment):

Linux
SHELL

sudo systemctl restart gremlind

Windows
POWERSHELL

Restart-Service -Name gremlind

Using environment variables

The daemon supports the following environment variables

Environment Variable Description
GREMLIN_TEAM_ID Your Team ID (required for authentication)
GREMLIN_TEAM_SECRET Your Team Secret (should only require secret or PEM certificates, not both)
GREMLIN_TEAM_PRIVATE_KEY_OR_FILE Your PEM-encoded private key or path to a file containing the private key (required for authentication). Paths should be prefixed with file:// . For example, on Unix: file:///var/lib/gremlin/key.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/key.pem
GREMLIN_TEAM_CERTIFICATE_OR_FILE The PEM-encoded public-key certificate or path/filename to the file containing your PEM-encoded public-key certificate (required for authentication). Paths should be prefixed with file://. For example, on Unix: file:///var/lib/gremlin/cert.pem , on Windows: file://C:/ProgramData/Gremlin/Agent/cert.pem
GREMLIN_IDENTIFIER Custom name for this client (default as the host's IP address)
GREMLIN_CLIENT_TAGS Comma-separated list of custom tags to assign to this client. For example: GREMLIN_CLIENT_TAGS="zone=us-east1,role=mysql,foo=bar"

The following are optional environment variables that may be required in Docker containerized deployments:

Environment Variable Description
GREMLIN_BYPASS_USERNS_REMAP Indicates to run the Gremlin Daemon on the same Docker namespace as the host. By default, Gremlin does not launch with a namespace defined. To set this variable, change the value to 1
GREMLIN_DOCKER_IMAGE Indicates the location and version of the Gremlin Docker image to use. This variable can be used to indicate an internal repository. For example: GREMLIN_DOCKER_IMAGE=registry.hub.docker.com/gremlin/gremlin:latest
net=host Indicates to run the Gremlin Daemon to use the host network. By default, Gremlin network experiments will not affect the host network for containerized environments.
GREMLIN_IAM_ROLE Indicates to run the Gremlin Daemon with the specified role when retrieving remote config values.
net=host Indicates to run the Gremlin Daemon to use the host network. By default, Gremlin network experiments will not affect the host network for containerized environments.
pid=host Indicates to run the Gremlin Daemon to affect host processes. By default, Gremlin process killer will not affect host processes for containerized environments.

In addition, the following standard Linux environment variables allow proxy configuration:

Environment Variable Description
https_proxy In the form http[s]://[username:password@]address:port
SSL_CERT_FILE Indicates location of the proxy certificates used to authenticate traffic with the proxy. For Example: SSL_CERT_FILE=/etc/ssl/certs/ca-example.crt

Configuration value priorities

If both the environment variables and the configuration file are used when configuring Gremlin, the environment variable values will take priority. For example if you have the GREMLIN_IDENTIFIER environment variable set to foo and in the config.yaml file the identifier field is set to bar, Gremlin will end up using foo as its identifier when registering with the API.

Remote config values

The Gremlin agent supports using AWS ARNs for the following config values in either ENVVARs or config file:

  • identifier
  • team_id
  • team_secret
  • team_certificate
  • team_private_key

When remote configuration values are used, Gremlin does not store the corresponding configuration values on the hard disk of the system where it runs. When Gremlin is configured to use Secret Authentication option (i.e. using a team_secret), the agent must store temporary session tokens in the .credentials file, which is only readable by the Gremlin system user.

The following services are supported for reading values:

  • ssm
  • secretsmanager

The agent requires some credentials to exist on the machine to be able to retrieve the remote values.The credentials preference follows the default AWS order.

Passing environment variables to Systemd or SysV

While configuring the daemon with config.yaml is recommended, process managers like systemd and sysvinit can pass environment variables directly to the daemon process from /etc/default/gremlind. Your installation comes with an example of this file at /etc/default/gremlind.example

SHELL

# Gremlin Identifier; uniquely identifies this machine with Gremlin
GREMLIN_IDENTIFIER=gremlin-01

# Gremlin Team ID; you can find this value at https://app.gremlin.com/settings/teams
GREMLIN_TEAM_ID=11111111-1111-1111-1111-111111111111

# `gremlin init` will run automatically when the gremlind service starts.
# Supply extra options to `gremlin init` via this variable
GREMLIN_INIT_OPTS="--tag service=pet-store --tag interface=http"

# Gremlin Team Secret, should not be set when using `team_certificate`+`team_private_key`
GREMLIN_TEAM_SECRET=11111111-1111-1111-1111-111111111111

You can set any of the environment variables listed in the previous section in the configuration file.

Tags

The Gremlin Agent automatically detects tags from the host system, such as operating system, zone, and local hostname. Gremlin also detects additional tags from certain cloud providers, including Amazon Web Services and Azure. You can also create custom tags and assign them to your Gremlin Agent(s) during installation. See Network Tags for instructions on adding custom tags to Gremlin.

Zone tags are required for the Zone Reliability Test.

Process Collection

In order to run dependency tests using Gremlin Reliability Management (RM), the Gremlin agent must be configured to collect process data by enabling Process Collection. When Process Collection is enabled, the Gremlin Agent collects process information from hosts, containers, and Kubernetes objects, and uses it to detect dependencies.

As of Linux Agent version 2.25.0, Gremlin automatically enables Process Collection.

To learn more about enabling or disabling process collection, see Process Collection.

No items found.
Previous
Previous
This is some text inside of a div block.
Compatibility
Installing the Gremlin Agent
Authenticating the Gremlin Agent
Configuring the Gremlin Agent
Managing the Gremlin Agent
User Management
Integrations
Health Checks
Notifications
Command Line Interface
Updating Gremlin
Quick Start Guide
Services and Dependencies
Detected Risks
Reliability Tests
Reliability Score
Targets
Experiments
Scenarios
GameDays
Overview
Deploying Failure Flags on AWS Lambda
Deploying Failure Flags on AWS ECS
Deploying Failure Flags on Kubernetes
Classes, methods, & attributes
API Keys
Examples
Container security
General
Linux
Windows
Chao
Helm
Glossary
Alfi
Additional Configuration for Helm
Amazon CloudWatch Health Check
AppDynamics Health Check
Application Level Fault Injection (ALFI)
Blackhole Experiment
CPU Experiment
Certificate Expiry
Custom Health Check
Custom Load Generator
DNS Experiment
Datadog Health Check
Disk Experiment
Dynatrace Health Check
Grafana Cloud Health Check
Grafana Cloud K6
IO Experiment
Install Gremlin on Kubernetes manually
Install Gremlin on OpenShift 4
Installing Gremlin on AWS - Configuring your VPC
Installing Gremlin on Kubernetes with Helm
Installing Gremlin on Windows
Installing Gremlin on a virtual machine
Installing the Failure Flags SDK
Jira
Latency Experiment
Memory Experiment
Network Tags
New Relic Health Check
Overview
Overview
Overview
Overview
Overview
Packet Loss Attack
PagerDuty Health Check
Preview: Gremlin in Kubernetes Restricted Networks
Private Network Integration Agent
Process Collection
Process Killer Experiment
Prometheus Health Check
Role Based Access Control
Running Failure Flags experiments
Scheduling Scenarios
Shared Scenarios
Shutdown Experiment
Slack
Teams
Time Travel Experiment
Troubleshooting Gremlin on OpenShift
User Authentication via SAML and Okta
Users
Webhooks
Integration Agent for Linux
Test Suites
Restricting Testing Times
Reports
Process Exhaustion Experiment
Enabling DNS collection