Investigation performed on NYVM2528.edc.un.org: Server logs confirm a clean, graceful reboot occurred on March 31, 2026 at 16:29 with no evidence of crash, kernel panic, or malicious activity. Root cause could not be determined from OS-level logs alone as pre-shutdown logs were not preserved.
Investigated the April 5 SCOM downtime alerts for MediaNet by checking Docker container logs and health on both application nodes (mnet-p-app-01 and mnet-p-app-02), and found no evidence of any outage — all requests returned 200 OK throughout the entire incident window, with no container crashes or restarts, confirming the alerts were false positives on the SCOM side.
Diagnosed and resolved a MongoDB replica set majority write blockage caused by a downed node and a stale secondary, recovering the replica set via initial sync.
Identified 100% disk usage on the root filesystem by narrowing down space consumption with du, tracing it to /home/_osmprod/imposm/diffs/ where 852 orphaned temp files had accumulated. Cross-referencing imposm's state file confirmed these were all from failed runs since February 27, none of which were ever applied to the database.