When setting up PromLTS, we ask users to insert remote_read and remote_write details into their prometheus.yml. Since Prometheus can be installed a lot of different ways depending on your setup, you might not directly encounter the prometheus.yml file when you first start. This is a quick guide to where you’ll find that configuration for different methods of installation.
Whenever the configuration file is edited for an existing Prometheus process, Prom needs to reload to pick up the changes as described in the Prometheus documentation. There are two methods: send a SIGHUP (or hangup) signal to the process, or send a POST request to Prometheus’s http://prom/-/reload endpoint. The reload endpoint isn’t available by default so for each method we’ve also included how to activate that endpoint. You can send the POST request with curl, for example (use your own Prometheus endpoint here):
curl -X POST http://localhost:9090/-/reload
To ensure your new prometheus.yml file was loaded correctly, you can check the configuration in the Prometheus UI at http://localhost:9090/config
The most straightforward way to install Prometheus is to download the latest version, unzip it, and run Prometheus right out of the box. When you follow this method, there will be a prometheus.yml file for you to use in the prometheus-<version> directory. You can simply paste the remote configuration options into there, and then run Prometheus using the config.file flag:
You can preview the default yaml file on github: https://github.com/prometheus/prometheus/blob/release-2.7/documentation/examples/prometheus.yml
If you are updating an already-running Prometheus, it can be triggered to reload prometheus.yml in 2 ways, as detailed in the Prometheus documentation:
1. Send a SIGHUP signal - this can be done from the terminal using the kill command. Get the process ID for prometheus using pgrep prometheus or ps auxw | grep prometheus and send the signal to PID:
kill -HUP 1234
2. Send a POST request to /-/reload endpoint - this requires Prometheus to be launched with the flag:
./prometheus --config.file=/path/to/file/prometheus.yml --web.enable-lifecycle
You can check if this flag is enabled through the Prom UI at http://localhost:9090/flags
When this launches, Prometheus starts running straight away, using the same default prometheus.yml file included in the downloaded package.
To override this default configuration you’ll need to mount a prometheus.yml file from your system onto the docker container. Copy the yaml from github as above, or download and unzip the package and take the prometheus.yml file from there. Then add the remote settings, and take note of the path to the file.
You can then mount the file from your machine into the docker container using the -v flag, either by using docker run as in the Prometheus installation guide, or by inserting it into your docker-compose file:
$ docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
Restarting Prometheus running in a container to load the new config can be achieved using the same two methods as above, with some changes:
1. Send a SIGHUP signal: use docker ps to get the name of your container, and then docker kill to send the signal to the container name or ID, e.g.
2. The POST to /-/reload method as above, but it requires the --web.enable_lifecycle flag to have been added to docker-compose.yml or the docker run command beforehand:
$ docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --web.enable-lifecycle
Kubernetes (or k8s for short) supports Prometheus metrics natively; Prometheus in return can automatically discover Kubernetes metrics if it’s running in the k8s cluster in its own container. You can find dozens of articles on how to deploy Prometheus to k8s on the web, for example this guide, but most of them provide all the files you’ll need so you might not spot where the prometheus.yml file is hidden.
When deploying to k8s one of the things you create is a ConfigMap which provides the configuration files for the applications within your deployment. Often this is created from a file called config-map.yaml (though the name could be anything, this is a good standard to stick to). If you look at the example config-map.yaml provided for the article above, you can see pretty quickly what we’re looking for:
The prometheus.yml file is embedded inside the config-map.yml file, in the “data” section, so that’s where you can add the remote_read/write details. Just make sure that the indentations are correct, since they’ll be more indented than in the standard config file.
If Prometheus is already running in Kubernetes, reloading the configuration can be interesting. First of all create the amended config-map.yml as above, then use the kubectl replace command to update it for the prometheus pod: $ kubectl replace -f config-map.yml
The new config-map.yml will be rolled out to the containers in the pod within a minute or two, after which you can trigger Prometheus to reload using the /-/reload endpoint provided you’ve passed the --enable.web_lifecyle flag.
There’s no specific way to send a SIGHUP signal to a k8s pod, so if you haven’t got that flag enabled, you’ll need to scale the pod down to 0 containers and back up to the desired number again. All the new containers will use the new configuration. If you’ve only been using local storage for Prometheus that could mean losing your data, but a standard Kubernetes deployment for Prometheus uses a separate storage volume.
For future reloads, you can update the prometheus-deployment.yml file to add --enable.web_lifecyle to the arguments passed to the container, e.g.:
Then you can roll out an update to the prometheus pod, which will restart all the containers with the flag and the new configuration.
Kubernetes can be quite complex even for standardised deployments like Prometheus, so a deployment management tool called Helm was developed to make deploying applications easier. Quick installation instructions can be found here if you don’t already have it set up.
Once Helm is installed, setting up Prometheus is as easy as helm install stable/prometheus but again that will only use a default configuration (which includes k8s service discovery, Alertmanager and more in this case). To add additional configuration settings, they need to be provided in a values.yaml file. The default values.yaml file can be found on github with the rest of the Prometheus ‘chart’ (the helm package format):
This file is massive! But if you scroll down – or more reasonably, use ctrl-F and search – you can find the prometheus.yml file in there, in a section titled serverFiles.
Copy the contents of values.yaml to a local file, update that with the remote_read/write details provided by PromLTS, and then deploy with helm:
helm install stable/prometheus --name my-release -f values.yaml
It’s also possible to pass the extra configuration as flags at launch, but it’s not pretty and it’s not as easy to replicate later. This is approximately how you would do it, just to show how unwieldy it is:
Updating the configuration for an existing Prometheus container which was installed via helm is achieved using the upgrade command:
helm upgrade -f values.yaml my-release stable/prometheus
Helm rolls out the minimum change possible, so all it will do is distribute the config to the existing Prometheus containers and reload them. --web.enable_lifecycle is active by default in helm version of Prom, so if you don’t see values coming into PromLTS, you should be able to send a POST to reload - or to receive a helpful error message.
© 2018 Metricfire Limited.
All Rights Reserved.
29-31 South William Street