bb-check-etl.sh

The ETL monitoring script, bb-check-etl.sh, monitors the health of the ETL service.

Overview

The script resides in the [RUNTIME_DIR]/scripts directory and runs as an extension script in the Big Brother framework. It reports to Big Brother based on the output of the etl_running script which checks whether the ETL service is running or not. The host on which the ETL service is configured to run is also displayed in Big Brother.

Status indicators

The ETL Monitoring script uses the Big Brother color codes for status indicators. The status codes can be green or red.

Red
The ETL service is down. This status requires immediate attention.
Green
The ETL service is running. Under normal operating conditions, the status is generally green. This condition requires no response.

Logs

The ETL service logs to two files in the [LOGGING_DIR]/debug directory:

etl.log
Information from ETL runs. This file may contain data transformation warnings and errors.
Note: Transformation errors do not constitute a failed ETL run. For example, if a corrupt row of data in one of the source tables causes transformation errors, this is treated as a "skipped record" and gets logged.
etl-out.log
This file contains logs from the ETL scheduler and logs from ETL runs. It is created by the babysit process on ETL startup.