In cloud-native environments, visibility is critical. This project introduced a “blackbox monitoring” solution capable of collecting logs and metrics from containerized services without modifying application code. The system uses lightweight agents written in Go, a backend powered by ElasticSearch, and a custom Grafana dashboard for real-time observability.
A major challenge was handling log spikes (>100k events/min) without data loss. I implemented local buffering, retry logic, and LZ4 compression. The system features YAML-based configuration and integrates seamlessly with Prometheus, making it easy to scale in DevOps pipelines
