Open in app

Sign in

Write

Sign in

Install Grafana-Prometheus-Node_Exporter Using Ansible

Najib Radzuan
devops4me
Published in
8 min readMay 15, 2022

--

Introduction

You can accomplish so much with Ansible, and it’s so simple to get started with it. The more you use it, the more it becomes a part of you. Prometheus, Grafana, and Node-Exporter are all good options for a monitoring stack. Because you can use a small or large scale to practise a variety of standard modules.

I will guide you on how you can use the Ansible script to install Grafana and Prometheus in your Host/Main monitor server and install node_exporter in other servers that you would like to monitor.

Use-Case

Step 1: Set with static inventory file and Pinging the Target Nodes

  • As a best practice, I prefer to create a new folder for each project and create a config file in it.

Note: Remember that Ansible will process the below list and use the first file found, all others are ignored.

ANSIBLE_CONFIG (environment variable if set)
ansible.cfg (in the current directory)
~/.ansible.cfg (in the home directory)
/etc/ansible/ansible.cfg
  • So, make a directory named grafana-prometheus under the home directory and switch to it.
$ mkdir dynamic-inventory
$ cd dynamic-inventory

Create a file named inventory.txt

$ sudo vi inventory.txt
  • Paste the content below into the inventory.txt file.
[monitorserver]
db_server   ansible_host=<YOUR-DB-SERVER-IP>   ansible_user=ec2-user  ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
[nodeservers]
server1  ansible_host=<YOUR-WEB-SERVER-IP>  ansible_user=ec2-user  ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>
server2  ansible_host=<YOUR-WEB-SERVER-IP>  ansible_user=ec2-user  ansible_ssh_private_key_file=~/<YOUR-PEM-FILE>

Note: Don’t forget the change the IP addresses of the target nodes and the path of your pem key.

  • Create file named ansible.cfg under the the dynamic-inventory directory.
  • Paste the content below into ansible.cfg file.
[defaults]
host_key_checking = False
inventory=inventory.txt
interpreter_python=auto_silent
localhost_warning=false

Validate and check the inventory.

ansible-inventory --graph
  • Check the connectivity to the target nodes.
$ ansible all -m ping
  • So, that’s great! We have connected to the target nodes with a static inventory.

Step 2: Create a role to install node-exporter

To install a binary or a tarball with an IaC or configuration manager such as ansible, you must first master a basic case. A good example is the node exporter. First and foremost, the ansible-galaxy command aids in the construction of a role’s skeleton:

ansible-galaxy init roles/node-exporter

Then we can set some defaults variables in the default directory :

node_exporter_version: "1.1.2"
node_exporter_bin: /usr/local/bin/node_exporter
node_exporter_user: node-exporter
node_exporter_group: "{{ node_exporter_user }}"
node_exporter_dir_conf: /etc/node_exporter

And now the main file of the tasks directory :

- name: check if node exporter exist
  stat:
    path: "{{ node_exporter_bin }}"
  register: __check_node_exporter_present
- name: create node exporter user
  user:
    name: "{{ node_exporter_user }}"
    append: true
    shell: /usr/sbin/nologin
    system: true
    create_home: false
- name: create node exporter config dir
  file:
    path: "{{ node_exporter_dir_conf }}"
    state: directory
    owner: "{{ node_exporter_user }}"
    group: "{{ node_exporter_group }}"
- name: if node exporter exist get version
  shell: "cat /etc/systemd/system/node_exporter.service | grep Version | sed s/'.*Version '//g"
  when: __check_node_exporter_present.stat.exists == true
  changed_when: false
  register: __get_node_exporter_version
  
- name: download and unzip node exporter if not exist
  unarchive:
    src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
    dest: /tmp/
    remote_src: yes
    validate_certs: no
- name: move the binary to the final destination
  copy:
    src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
    dest: "{{ node_exporter_bin }}"
    owner: "{{ node_exporter_user }}"
    group: "{{ node_exporter_group }}"
    mode: 0755
    remote_src: yes
  when: __check_node_exporter_present.stat.exists == false or not __get_node_exporter_version.stdout == node_exporter_version
- name: clean
  file:
    path: /tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/
    state: absent
- name: install service
  template:
    src: node_exporter.service.j2
    dest: /etc/systemd/system/node_exporter.service
    owner: root
    group: root
    mode: 0755
  notify: reload_daemon_and_restart_node_exporter
- meta: flush_handlers
- name: service always started
  systemd:
    name: node_exporter
    state: started
    enabled: yes

We need to create node_exorter.service.j2 file in templates directory :

[Unit]
Description=Node Exporter Version {{ node_exporter_version }}
After=network-online.target
[Service]
User={{ node_exporter_user }}
Group={{ node_exporter_user }}
Type=simple
ExecStart={{ node_exporter_bin }}
[Install]
WantedBy=multi-user.target

Finally the handler file in the handler direct

- name: reload_daemon_and_restart_node_exporter
  systemd:
    name: node_exporter
    state: restarted
    daemon_reload: yes
    enabled: yes

Step 3: Create a role for Prometheus and its configuration

We can initialize a Prometheus role :

ansible-galaxy init roles/prometheus

We can set some defaults variables in the default directory :

prometheus_dir_configuration: "/etc/prometheus"
prometheus_retention_time: "365d"
prometheus_scrape_interval: "30s"
prometheus_node_exporter: true
prometheus_node_exporter_group: "all"
prometheus_env: "production"
prometheus_var_config:
  global:
    scrape_interval: "{{ prometheus_scrape_interval }}"
    evaluation_interval: 5s
    external_labels:
      env: '{{ prometheus_env }}'
  scrape_configs:
    - job_name: prometheus
      scrape_interval: 5m
      static_configs:
        - targets: ['{{ inventory_hostname }}:9090']

Since we define a part of the Prometheus configuration, the header exactly with prometheus_var_config.Now I create tasks in the main.yml file of the tasks directory :

- name: update and install prometheus
  apt:
    name: prometheus
    state: latest
    update_cache: yes
    cache_valid_time: 3600
- name: prometheus args
  template:
    src: prometheus.j2
    dest: /etc/default/prometheus
    mode: 0644
    owner: root
    group: root
  notify: restart_prometheus
- name: prometheus configuration file
  template:
    src: prometheus.yml.j2
    dest: "{{ prometheus_dir_configuration }}/prometheus.yml"
    mode: 0755
    owner: prometheus
    group: prometheus
  notify: reload_prometheus
- name: start prometheus
  systemd:
    name: prometheus
    state: started
    enabled: yes

Then, we create the prometheus.yaml.j2 file:

#jinja2: lstrip_blocks: "True"
{{ prometheus_var_config | to_nice_yaml(indent=2) }}
{% if prometheus_node_exporter_group %}
- job_name: node
  scrape_interval: 15s
  metrics_path: /metrics
  static_configs:
  - targets:
{% for server in groups[prometheus_node_exporter_group] %}
    - '{{ server }}:9100'
{% endfor %}
{% endif %}

And the prometheus.j2 file for the Prometheus CLI :

ARGS="--web.enable-lifecycle --storage.tsdb.retention.time={{ prometheus_retention_time }} --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

Finally handlers of this Prometheus role;We have two handlers :

  • Restart with the systemd service
  • Reload with a curl on the prometheus API.
- name: restart_prometheus
  systemd:
    name: prometheus
    state: restarted
    enabled: yes
    daemon_reload: yes
- name: reload_prometheus
  uri:
    url: http://localhost:9090/-/reload
    method: POST
    status_code: 200

Step 4: Create a role for Grafana

Now we can create a last role to install Grafana server package and start it. Just edit the main.yml file in the tasks directory :

- name: install gpg
  apt:
    name: gnupg,software-properties-common
    state: present
    update_cache: yes
    cache_valid_time: 3600
- name: add gpg hey
  apt_key:
    url: "https://packages.grafana.com/gpg.key"
    validate_certs: no
- name: add repository
  apt_repository:
    repo: "deb https://packages.grafana.com/oss/deb stable main"             
    state: present
    validate_certs: no
- name: install grafana
  apt:
    name: grafana
    state: latest
    update_cache: yes
    cache_valid_time: 3600
- name: start service grafana-server
  systemd:
    name: grafana-server
    state: started
    enabled: yes
- name: wait for service up
  uri:
    url: "http://127.0.0.1:3000"
    status_code: 200
  register: __result
  until: __result.status == 200
  retries: 120
  delay: 1
- name: change admin password for grafana gui
  shell : "grafana-cli admin reset-admin-password {{ grafana_admin_password }}"
  register: __command_admin
  changed_when: __command_admin.rc !=0

Don’t forget to set your Admin password for start and you can set it in default directory:

grafana_admin_password: "abc1234"

Step 5: Create Ansible Playbook

Create an Ansible playbook file like the below :

- name: install monitoring stack
  hosts: monitorserver
  become: yes
  roles:
  - prometheus
  - grafana
- name: install node-exporter
  hosts: nodeservers
  become: yes
  roles:
  - node-exporter

Step 6: Testing

You can run this playbook file via the below command:

ansible-playbook -i inventory.txt playbook.yml```devops4me@ % ansible-playbook -i inventory.txt playbook.yml
/usr/local/Cellar/ansible/5.7.1/libexec/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,PLAY [install monitoring stack] ****************************************************************************************************************************TASK [Gathering Facts] *************************************************************************************************************************************
ok: [18.136.207.64]TASK [prometheus : update and install prometheus] **********************************************************************************************************
ok: [18.136.207.64]TASK [prometheus : prometheus args] ************************************************************************************************************************
ok: [18.136.207.64]TASK [prometheus : prometheus configuration file] **********************************************************************************************************
ok: [18.136.207.64]TASK [prometheus : start prometheus] ***********************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : install gpg] *******************************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : add gpg hey] *******************************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : add repository] ****************************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : install grafana] ***************************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : start service grafana-server] **************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : wait for service up] ***********************************************************************************************************************
ok: [18.136.207.64]TASK [grafana : change admin password for grafana gui] *****************************************************************************************************
ok: [18.136.207.64]PLAY [install node-exporter] *******************************************************************************************************************************TASK [Gathering Facts] *************************************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]TASK [node-exporter : check if node exporter exist] ********************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]TASK [node-exporter : create node exporter user] ***********************************************************************************************************
[WARNING]: 'append' is set, but no 'groups' are specified. Use 'groups' for appending new groups.This will change to an error in Ansible 2.14.
ok: [54.251.28.26]
ok: [13.215.254.70]TASK [node-exporter : create node exporter config dir] *****************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]TASK [node-exporter : if node exporter exist get version] **************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]TASK [node-exporter : download and unzip node exporter if not exist] ***************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]TASK [node-exporter : move the binary to the final destination] ********************************************************************************************
skipping: [54.251.28.26]
skipping: [13.215.254.70]TASK [node-exporter : clean] *******************************************************************************************************************************
changed: [13.215.254.70]
changed: [54.251.28.26]TASK [node-exporter : install service] *********************************************************************************************************************
ok: [13.215.254.70]
ok: [54.251.28.26]TASK [node-exporter : meta] ********************************************************************************************************************************TASK [node-exporter : service always started] **************************************************************************************************************
ok: [54.251.28.26]
ok: [13.215.254.70]PLAY RECAP *************************************************************************************************************************************************
13.215.254.70              : ok=9    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0   
18.136.207.64              : ok=12   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
54.251.28.26               : ok=9    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Improvement

We can improve this solution by apply the following steps:

1. Security:

1.1 Encrypt The Credentials/Password/Private key

Ansible Vault encrypts variables and files so you can protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in playbooks or roles

2. Agility:

2.1 Continuous Integration and Continuous Delivery:

Integrate our Ansible script with CI/CD and apply it to other projects or needs.

2.2 Disable fact gathering :

When a playbook executes, each play runs a hidden task, called gathering facts, using the setup module. This gathers information about the remote node you're automating, and the details are available under the variable ansible_facts. But if you're not using these details in your playbook anywhere, then this is a waste of time. You can disable this operation by setting gather_facts: False in the play.

Conclusion

Finally, with all the above steps we have learned :

  1. How to create a Dynamic Ansible Inventory file.
  2. How to provision and installed Grafana & Prometheus using Ansible script
  3. How to automatically installed Node Exporter via Ansible

Additional:

You may find the full code for this post from here-> DevOps4Me Global Code

DevOps
Automation
Ansible
Grafana
Prometheus

--

--

Najib Radzuan
devops4me

Written by Najib Radzuan

108 Followers

DevOps | DevSecOps | Global DevOps Ambassador | CDF Ambassador | Digital Transformation [https://linktr.ee/devops4me]

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams