Secure Prometheus https scrapper by TLS with mutual TLS authentication

This post was inspired and parly based on Julien Pivotto’s blog postlocal copy

Main difference from Julen’s approach that I’m going to use 2 way TLS authentication between Prometheus and node-exporter instead of one way password authentication in Julien’s post.

Generating TLS certificates.

We will need a self signed cert for https scrapper and exporters. Standard stuff, you have done it million times.

Create directories to store certificates

mkdir -p /usr/local/etc/prometheus/certs
chown -R prometheus:prometheus /usr/local/etc/prometheus
chmod og= /usr/local/etc/prometheus/certs
cd /usr/local/etc/prometheus/certs

Creating CA

  1. Create CA_openssl.cnf file for CA with following content 
     # cat CA_openssl.cnf
 subjectKeyIdentifier=hash
 basicConstraints=critical, CA:TRUE
 keyUsage = keyCertSign,digitalSignature,cRLSign
 nsCertType = sslCA
  2. Create a private key and self-signed certificate for CA. Cert will expire in 10 years, so do not forget to set a reminder.
    1. CSR and private key in same command 
      # openssl req -newkey rsa:2048 -keyout CA.key -out CA.csr -nodes
Generating a RSA private key
............................................................................+++++
............+++++
writing new private key to 'CA.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:NSW
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example CA    
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:monitoring.example.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

      # ls
monitoring:/usr/local/etc/prometheus/certs/test@[20:59] # ls
CA.csr				CA_openssl.cnf  CA.key
    2. Sign that CSR 
       # openssl x509 -req -days 3650 -in CA.csr -signkey CA.key -out CA.cert -extfile CA_openssl.cnf
 Signature ok
 subject=C = AU, ST = NSW, O = Example CA, CN = monitoring.example.com
 Getting Private key

       # ls CA*
 CA.cert		CA.csr		CA.key		CA_openssl.cnf

    3. Check results. Cert must have CA:TRUE and Digital Signature, Certificate Sign

       openssl x509 -in CA.cert -text -noout
 Certificate:
     Data:
         Version: 3 (0x2)
         Serial Number:
             39:35:3c:18:ed:70:53:c6:43:fa:69:20:f0:f0:b6:ba:78:a8:30:04
         Signature Algorithm: sha256WithRSAEncryption
         Issuer: C = AU, ST = NSW, O = Example CA, CN = monitoring.example.com
         Validity
             Not Before: Feb  7 10:05:06 2022 GMT
             Not After : Feb  5 10:05:06 2032 GMT
         Subject: C = AU, ST = NSW, O = Example CA, CN = monitoring.example.com
         Subject Public Key Info:
             Public Key Algorithm: rsaEncryption
                 RSA Public-Key: (2048 bit)
                 Modulus:
                     00:a2:66:5b:30:ad:61:10:b2:9f:16:b2:7a:ac:62:
                     === CUT ===
                     55:83
                 Exponent: 65537 (0x10001)
         X509v3 extensions:
             X509v3 Subject Key Identifier:
                 30:F9:C0:EB:90:66:3D:0F:8A:7F:01:7A:11:3E:2B:C2:22:87:7F:F6
             X509v3 Basic Constraints: critical
                 CA:TRUE
             X509v3 Key Usage:
                 Digital Signature, Certificate Sign, CRL Sign
             Netscape Cert Type:
                 SSL CA
     Signature Algorithm: sha256WithRSAEncryption
          61:f2:1f:90:cf:03:0c:c4:d6:f8:f6:81:7d:66:90:a7:df:12:
          === CUT ===
          73:a3:f2:3b
    4. Delete CSR, we no longer need it 
       # rm CA.csr

Generate certificate(s) for prometheus AKA scrapper

  1. Create config file. scrapper. 
       # cat monitoring.example.com.cnf
   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   prompt = no

   [req_distinguished_name ]
   CN = monitoring.example.com

   [v3_req]
   # authorityKeyIdentifier=keyid,issuer:always
   basicConstraints=critical,CA:false
   keyUsage = digitalSignature,keyEncipherment, dataEncipherment
   extendedKeyUsage = clientAuth
   nsCertType = client

    Replace values for CN with FDQN of your prometheuth host (or just fake domain, it does not mater what is there)

  2. Generate private key and CSR 
       monitoring:/usr/local/etc/prometheus/certs/test@[21:31] # openssl req -newkey rsa:2048 -keyout monitoring.example.com.key -out monitoring.example.com.csr -config monitoring.example.com.cnf -nodes
   Generating a RSA private key
   .......+++++
   .....................+++++
   writing new private key to 'monitoring.example.com.key'
   -----
  3. Signef CSR by CA key 
       # openssl x509 -req -days 3650 -CA CA.cert -CAkey CA.key -CAcreateserial -CAserial CA.serial -extensions v3_req -out monitoring.example.com.cert -extfile monitoring.example.com.cnf -in monitoring.example.com.csr
   Signature ok
   subject=CN = monitoring.example.com
   Getting CA Private Key
  4. Check that cert has following extensions.TLS Web Client Authentication is what impotent there. 
       # openssl x509 -text -noout -in monitoring.example.com.cert
   === CUT ===
   X509v3 extensions:
         X509v3 Basic Constraints: critical
             CA:FALSE
         X509v3 Key Usage:
             Digital Signature, Key Encipherment, Data Encipherment
         X509v3 Extended Key Usage:
             TLS Web Client Authentication
         Netscape Cert Type:
             SSL Client
   === CUT ===

Generate certificates for exporters

There are 2 approaches to exporters certificates

  1. Configure Prometheuth to do not verify exporters certificates. In this case you can use any certs on exportes you find fancy. You event can uses same self-signed cert for all exporters in your network.
  2. Use “proper” certs for exporters signed by CA. This way you will need individual cert for each exporter with matching FDQN so Prometheus will be able to verify that it is talking to right host. I did not find a way to use prevent prometheuth from checking that target and CN in certificate match.

It may be a good idea to generate exporters cert on other machine not where Prometheus server runs. But we are trying to protect server so if bad guys get access to exporters certs it is already too late.

Following pretty much same steps as for scrapper certificate to create exporters certificates.

  1. Create CNF files You will need one file for each exporter you have. Please put FDQN of exporter to CN and DNS.1 fields inside CNF file.

    !!! These FDQN have to match hostnames you put under target list in Prometheus config file prometheus.yaml. !!!

    Also notice that extendedKeyUsage is serverAuth for exporter’s certs.

      # cat host1.example.com.cnf
  [req]
  distinguished_name = req_distinguished_name
  req_extensions = v3_req
  prompt = no

  [req_distinguished_name ]
  CN = host1.example.com

  [v3_req]
  # authorityKeyIdentifier=keyid,issuer:always
  basicConstraints=critical,CA:false
  keyUsage = digitalSignature,keyEncipherment, dataEncipherment
  extendedKeyUsage = serverAuth
  nsCertType = client
  subjectAltName = @alt_names


  [alt_names]
  DNS.1 = host1.example.com
  2. Generate CSR and sign it by CA 
       # openssl req -newkey rsa:2048 -keyout host1.example.com.key -out host1.example.com.csr -config host1.example.com.cnf -nodes
   Generating a RSA private key
   ..............................................+++++
   ....+++++
   writing new private key to 'host1.example.com.key'
   -----
   # openssl x509 -req -days 3650 -CA CA.cert -CAkey CA.key -CAcreateserial -CAserial CA.serial -extensions v3_req -out host1.example.com.cert -extfile host1.example.com.cnf -in host1.example.com.csr
   Signature ok
   subject=CN = host1.example.com
   Getting CA Private Key
   # rm host1.example.com.csr
  3. Repeat for each host with exporter(s). Multiple exporters on same host but different ports may share certificate

Configuring scrapper.

Add following under scrape_config in prometheuth.yaml

scrape_configs:
  - job_name: 'node_exporter'
    scheme: https
    tls_config:
        cert_file: /usr/local/etc/prometheus/certs/monitoring.example.com.cert
        key_file: /usr/local/etc/prometheus/certs/monitoring.example.com.key
        ca_file: /usr/local/etc/prometheus/certs/CA.cert
 #       insecure_skip_verify: true
    static_configs:
      - targets:
          - host1.example.com
          - host2..example.com
          - host3..example.com

    relabel_configs:
      - source_labels: [__address__] # populate list of hosts to scan from targets list
        target_label: __param_target
      - source_labels: [__param_target] # Add label 'instance' to identify targets.
        target_label: instance
      - source_labels: [__param_target]
        target_label: __address__  # Add port number to targets automatically.
        replacement: $1:9010

Uncomment insecure_skip_verify: true if you use self-signed certificates for exporters. Otherwise FQDN names under - targets: must match CN/DNS.1 names in scrapper certificates. You probaly can use IP addresses there, but in this case certificates should have appropriate IP.1 entry under alt_names in cnf files. I did not tested it.

Do not forget to reload or restart prometheus to pick up config change.

Configure node_exporter

I’m using FreeBSD,but it should be pretty much same for linux.

  1. Create a user for node_exporter.

    TLS support in node_exporter is still experimental. So bugs are expected. One problem I found with TLS that node_exporter first drop root privileges and then attempts read certificates. By default node_exporter runs as nobody:nobody, so to allow it read certs I will need make certs readable for user nobody, which is very bad idea. So let’s create a dedicated unprivileged user to run node_exporter.

     # adduser -D  -w no -s /usr/sbin/nologin

 Username: node_exporter
 Full name: node_exporter
 Uid (Leave empty for default):
 Login group [node_exporter]:
 Login group is node_exporter. Invite node_exporter into other groups? []:
 Login class [default]:
 Shell (sh csh tcsh bash rbash nologin) [nologin]:
 Home directory [/home/node_exporter]:
 Home directory permissions (Leave empty for default):
 Use password-based authentication? [no]:
 Lock out the account after creation? [no]:
 Username   : node_exporter
 Password   : <disabled>
 Full Name  : node_exporter
 Uid        : 1007
 Class      :
 Groups     : node_exporter
 Home       : /home/node_exporter
 Home Mode  :
 Shell      : /usr/sbin/nologin
 Locked     : no
 OK? (yes/no): yes
 adduser: INFO: Successfully added (node_exporter) to the user database.
 Add another user? (yes/no): no
 Goodbye!
  2. Install node_exporter 
    # pkg install node_exporter
  3. Create directory to store certificate and config. 
     mkdir -p /usr/local/etc/node_exporter/certs
 chown -R node_exporter:node_exporter /usr/local/etc/node_exporter
 chmod og= /usr/local/etc/node_exporter/certs
  4. Copy certificates from whatever system you created them. 3 files are required
    • host specific private key host1.example.com.key
    • host specific certificate host1.example.com.cert
    • CA certificate CA.cert
     /usr/local/etc/node_exporter/certs# ls
 CA.cert			host1.example.com.cert	host1.example.com.key

  5. create web-config.yaml

     # cat /usr/local/etc/node_exporter/web-config.yaml
 tls_server_config:
   # Certificate and key files for server to use to authenticate to client.
   cert_file: /usr/local/etc/node_exporter/certs/host1.example.com.cert
   key_file: /usr/local/etc/node_exporter/certs/host1.example.com.key

   # Server policy for client authentication. Maps to ClientAuth Policies.
   # client_auth_type: "NoClientCert"
   client_auth_type: "RequireAndVerifyClientCert"

   # CA certificate for client certificate authentication to the server.
   client_ca_file: /usr/local/etc/node_exporter/certs/CA.cert

  6. Add parameters to /etc/rc.config

     # cat /etc/rc.config | grep node_exporter_
 node_exporter_enable="YES"
 node_exporter_user=node_exporter
 node_exporter_group=node_exporter
 node_exporter_args="--web.config=/usr/local/etc/node_exporter/web-config.yaml"
 node_exporter_listen_address="a.b.c.d:9010"
  7. Start node_exporter 
    # /usr/local/etc/rc.d/node_exporter start

It is done. your prometheuth metrics are more secure than before. Enjoy!!

Updated: