Monitor OPNsense internet connection

!README.md

Monitor OPNsense internet connection

OPNsense sometimes "loses" the WAN facing connection when that is "reinitialised" (modem reboot/WAN restart).

Monitoring the WAN facing connection and taking corrective action is the solution proposed by this script.

opnsense_ping_check.sh will ping IP addresses to determine if the WAN connection is still working, and try to restore the WAN if needed by bringing the interface down and up, or through interface reconfiguration and if needed through a system reboot (OPNsense).

So to install this script:

1. Copy it to your OPNsense instance.
2. Update values in the script (network interface name, IPs you want to ping).
3. Execute it once interactively (copies script, and adds action)

To enable monitoring, a cron job needs to be added in OPNsense that calls the action created by opnsense_ping_check.sh.

Using the OPNsense UI, the generated action can be enabled as a cron job. Cron jobs are added under System>Settings>Cron.
Add an entry:

1. Tick Enable;
2. Set minutes to */5 (or other delay - be reasonable);
3. Select "ping_check" as the command;
4. Set a description such as "Ping check and recover connection"
5. Click save.

Screenshot while adding action to the cron jobs:

.pre-commit-config.yaml

---
files: ^(.*\.(py|json|md|sh|yaml|cfg|txt))$
exclude: ^(\.[^/]*cache/.*)$
repos:
  - repo: https://github.com/verhovsky/pyupgrade-docs
    rev: v0.3.0
    hooks:
      - id: pyupgrade-docs
  - repo: https://github.com/executablebooks/mdformat
    # Do this before other tools "fixing" the line endings
    rev: 0.7.17
    hooks:
      - id: mdformat
        name: Format Markdown
        entry: mdformat  # Executable to run, with fixed options
        language: python
        types: [markdown]
        args: [--wrap, '75', --number]
        additional_dependencies:
          - mdformat-toc
          - mdformat-beautysh
          # -mdformat-shfmt
          # -mdformat-tables
          - mdformat-config
          - mdformat-black
          - mdformat-web
          - mdformat-gfm
  - repo: https://github.com/asottile/blacken-docs
    rev: 1.16.0
    hooks:
      - id: blacken-docs
        additional_dependencies: [black==22.6.0]
        stages: [manual]   # Manual because already done by mdformat-black
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      # - id: no-commit-to-branch
      #   args: [--branch, main]
      # - id: check-yaml
      #   args: [--unsafe]
      # - id: debug-statements
      - id: end-of-file-fixer
      - id: trailing-whitespace
        exclude: \.md$
      # - id: check-json
      - id: mixed-line-ending
      # - id: check-builtin-literals
      # - id: check-ast
      - id: check-merge-conflict
      - id: check-executables-have-shebangs
      - id: check-shebang-scripts-are-executable
      # - id: check-docstring-first
      - id: fix-byte-order-marker
      - id: check-case-conflict
      # - id: check-toml

#   - repo: https://github.com/adrienverge/yamllint.git
#     rev: v1.23.0
#     hooks:
#       - id: yamllint
#         args:
#           - --no-warnings
#           - -d
#           - '{extends: relaxed, rules: {line-length: {max: 90}}}'
#   - repo: https://github.com/asottile/pyupgrade
#     rev: v2.31.0
#     hooks:
#       - id: pyupgrade
#         args:
#           - --py37-plus
#   - repo: https://github.com/psf/black
#     rev: 22.1.0
#     hooks:
#       - id: black
#         args:
#           - --safe
#           - --quiet
#           - -l 79
#   - repo: https://github.com/Lucas-C/pre-commit-hooks-bandit
#     rev: v1.0.5
#     hooks:
#       - id: python-bandit-vulnerability-check
#         args: [--skip, 'B101,B311', --recursive, .]
#
#   - repo: https://github.com/fsouza/autoflake8
#     rev: v0.3.1
#     hooks:
#       - id: autoflake8
#         args:
#           - -i
#           - -r
#           - --expand-star-imports
#           - custom_components
#   - repo: https://github.com/PyCQA/flake8
#     rev: 4.0.1
#     hooks:
#       - id: flake8
#         additional_dependencies:
#           - pyproject-flake8==0.0.1a2
#           - flake8-bugbear==22.1.11
#           - flake8-comprehensions==3.8.0
#           - flake8_2020==1.6.1
#           - mccabe==0.6.1
#           - pycodestyle==2.8.0
#           - pyflakes==2.4.0
#   - repo: https://github.com/PyCQA/isort
#     rev: 5.10.1
#     hooks:
#       - id: isort

  - repo: https://github.com/codespell-project/codespell
    rev: v2.2.6
    hooks:
      - id: codespell
        args:
          # - --builtin=clear,rare,informal,usage,code,names,en-GB_to_en-US
          - --builtin=clear,rare,informal,usage,code,names
          - --ignore-words-list=wan
          - --skip="./.*"
          - --quiet-level=2

#   - repo: https://github.com/pre-commit/mirrors-mypy
#     rev: v0.931
#     hooks:
#       - id: mypy
#         args:
#           - --ignore-missing-imports
#           - --install-types
#           - --non-interactive
#           - --check-untyped-defs
#           - --show-error-codes
#           - --show-error-context
#         additional_dependencies:
#           - zigpy==0.43.0

opnsense_ping_check.sh

#!/bin/sh
# Script for OPNsense to monitor WAN facing connection
# - When failing:
#   1. down and up the interface, check again
#   2. Try interface reconfiguration, check again
#   3. When still failing, reboot OPNsense

# The second part of this script will also install
# an action on opnsense and copy the script to a system
# location.
#
# Therefore, to install this script:
#  a. Copy it to your OPNsense instance.
#  b. Execute it once interactively (copies script, and adds action)
#
# Using the OPNsense UI, this action can be enabled as a cron
# job.  Cron jobs are added under System>Settings>Cron.
# Add an entry that you:
#  1. Enable
#  2. Set minutes to "*/5"
#  3. Select "ping_check" as the command
#  4. Set a description such as "Ping check and recover connection"
#  5. Click save.
#
# Based on a script that was likely adapted from
#   http://blog.martinshouse.com/2014/06/pfsense-auto-reboot-if-internet.html
#

# Configuration parameters

# First IP to ping to check if connection is up
IP1=8.8.8.8  # Google DNS Server 1
# Second IP to ping to check if connection is up
IP2=8.8.4.4  # Google DNS Server 2
# Interface to down and up
INTERFACE=igb0
# Minimum uptime
MIN_UPTIME=120

# ENABLE_LOGGING=true

TARGET_ACTION=/usr/local/opnsense/service/conf/actions.d/actions_ping_check.conf
TARGET_LOCATION=/usr/local/sbin/ping_check.sh

# Logging function
log_message() {
  if [ "$ENABLE_LOGGING" = "true" ]; then
    echo "$(date +%Y-%m-%d.%H:%M:%S) - $1" >> /var/log/ping_check.log
  fi
}

# Function to ping both IP addresses
ping_ips() {
  count=$1
  ip1_count=$(ping -o -s 0 -c "$count" "$IP1" | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

  if [ "$ip1_count" -ne 0 ]; then
    return 1
  fi

  ip2_count=$(ping -o -s 0 -c "$count" "$IP2" | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')

  if [ "$ip2_count" -ne 0 ]; then
    return 1
  fi

  return 0
}

# Function to restart the interface
restart_interface() {
  ifconfig $INTERFACE down
  ifconfig $INTERFACE up
}

# Function to reconfigure the WAN interface
reconfigure_wan() {
  configctl interface reconfigure wan
}

# Function to reboot the system
reboot_system() {
  /usr/local/etc/rc.reboot
}

# Main script logic
main() {
  # Testing uptime to run script only xx seconds after boot
  curtime=$(date +%s)
  uptime=$(sysctl kern.boottime | awk -F'sec = ' '{print $2}' | awk -F',' '{print $1}')
  uptime=$((curtime - uptime))

  if [ $uptime -gt $MIN_UPTIME ]; then
    log_message "Testing Connection (System uptime: $uptime seconds)"

    ping_ips 10
    if ping_ips 10; then
      log_message "Restarting interface '$INTERFACE'"
      restart_interface
      ping_ips 10
    fi

    if ping_ips 10; then
      log_message "Reconfiguring WAN interface"
      reconfigure_wan
      ping_ips 10
    fi

    if ping_ips 10; then
      log_message "Rebooting system"
      reboot_system
    fi
  fi
}

# Optional, add opnsense action for this script, which can then be added
# as a cron job in the UI:
if [ ! -r "$TARGET_LOCATION" ] ; then
  cp "$0" "$TARGET_LOCATION"
  chmod +x "$TARGET_LOCATION"
fi

if [ ! -r "$TARGET_ACTION" ] ; then
  cat > "$TARGET_ACTION" <<EOACTION
[load]
command:$TARGET_LOCATION
parameters:
type:script
message:starting ping check
description:ping_check
EOACTION
  # Restart configd service to have action appear in the menus
  service configd restart
fi

# Run the main script logic
main

OPNsense_ping_cron.png

Hello, I have this script running perfectly. Thanks! One question. I sometimes have a ISP outage (ie. 5 hours long) which won't be fixed by a reboot. Will this be rebooting every 2 minutes until it comes back up? If I change the MIN_UPTIME to 3600 then it would only reboot once a hour correct?

The script would reboot at most every 5 minutes because that is the periodicity of the cron job which is higher than the 2 minutes.

Setting MIN_UPTIME to 3600 will limit the reboots to once every hour.

The script would reboot at most every 5 minutes because that is the periodicity of the cron job which is higher than the 2 minutes.

Setting MIN_UPTIME to 3600 will limit the reboots to once every hour.

Perfect! Thanks again.

Does this leave an entry in the logs, or will I get an e-mail if this triggers? Is there a good way to set that up? It'd be nice to know how often this is popping off. By the way, thank you for creating this much needed tool!

@ericbakeragilix There currently isn't anything in the script that logs something.
A logger -t TAG "MESSAGE could be added whenever the reboot is triggered.

A quick does does not reveal any "OPNsense method" to trigger an email notification from a script.
Further, as this script triggers when the network is absent, the mail would need to be added to a local mail queue or sent later from the script.
The script could be extended to append the date it requests a reboot to a file. Then - on each execution - it could check if that file exists. Ifs, try to notify about the reboot through a mail server with the contents of that file and delete it when the notification could be sent. That could be a simple https API call to some webhook that takes care of the rest.

Great script! As of note, one more step to do is to restart the configd service so the cron job will show on the GUI:

service configd restart

@edominicci Thank you for the feedback - I added this to the script.

Great script!

For use cases where the Up/Down restart of the NIC doesn't work I found this command "configctl interface reconfigure wan" to function in my system. It's the equivalent to the reload button in Interfaces --> Overview menu

@felixalberto26 Thank you for this feedback - it helps save a reboot. I updated the script.