Wazuh Indexer + Dashboard error after power failure / LVM corruption / rm -rf /var/lib/wazuh-indexer

Gejala umum:

• Wazuh Dashboard stuck / tidak bisa login
• OpenSearch Security not initialized
• Unauthorized di curl 127.0.0.1:9200
• Indexer tidak stabil setelah reboot
• Cluster health tidak normal
• Security index hilang 

---

🧠 ROOT CAUSE (ringkasan teknis)

Biasanya kombinasi:

1. Power failure / VM crash
2. LVM / disk state tidak clean
3. /var/lib/wazuh-indexer corrupt atau di-reset
4. Security index (.opendistro_security) hilang
5. securityadmin.sh belum bootstrap ulang

---

STEP 0 — CEK STATUS AWAL (WAJIB LOGGING)

systemctl status wazuh-manager --no-pager
systemctl status wazuh-indexer --no-pager
systemctl status wazuh-dashboard --no-pager

---

📌 LOG OUTPUT CONTOH (ERROR STATE)

wazuh-indexer.service - active (running)
but:
OpenSearch Security not initialized

---

STEP 1 — CEK INDEXER CONNECTIVITY

curl -k https://127.0.0.1:9200

❌ ERROR:

OpenSearch Security not initialized

atau

Unauthorized

---

STEP 2 — CEK SECURITY INDEX

curl -k https://127.0.0.1:9200/_cat/indices?v

❌ JIKA BERMASALAH:

(no output atau tidak ada .opendistro_security)

---

STEP 3 — CEK CLUSTER HEALTH

curl -k https://127.0.0.1:9200/_cluster/health?pretty

❌ ERROR:

OpenSearch Security not initialized

---

STEP 4 — STOP SEMUA SERVICE

systemctl stop wazuh-dashboard
systemctl stop wazuh-manager
systemctl stop wazuh-indexer

---

STEP 5 — CLEAN STATE INDEXER

⚠️ hanya lakukan jika crash / corruption / reset

rm -rf /var/lib/wazuh-indexer/nodes
rm -rf /var/lib/wazuh-indexer/indices
rm -rf /var/lib/wazuh-indexer/_state

❗ JANGAN hapus:

• /etc/wazuh-indexer
• certs

---

STEP 6 — START INDEXER

systemctl start wazuh-indexer

Tunggu 60–120 detik

---

STEP 7 — VALIDASI CLUSTER READY

curl -k https://127.0.0.1:9200

✔ HARUS BALIK JSON:

{
  "cluster_name": "wazuh-cluster"
}

---

STEP 8 — BOOTSTRAP SECURITY (INI INTI FIX)

export JAVA_HOME=/usr/share/wazuh-indexer/jdk
export OPENSEARCH_JAVA_HOME=/usr/share/wazuh-indexer/jdk

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh \
-cd /etc/wazuh-indexer/opensearch-security/ \
-icl -nhnv \
-h 127.0.0.1 -p 9200 \
-cert /etc/wazuh-indexer/certs/admin.pem \
-key /etc/wazuh-indexer/certs/admin-key.pem \
-cacert /etc/wazuh-indexer/certs/root-ca.pem

---

📌 LOG SUCCESS (HARUS MUNCUL)

Clusterstate: GREEN
.opendistro_security index does not exists → created
SUCC: internalusers updated
SUCC: roles updated
SUCC: actiongroups updated
Done with success

---

STEP 9 — RELOAD SYSTEM

systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
systemctl restart wazuh-manager

---

STEP 10 — VALIDASI FINAL

curl -k https://127.0.0.1:9200

✔ OUTPUT NORMAL:

{
  "cluster_name": "wazuh-cluster",
  "version": {...}
}

---

🧠 TROUBLESHOOT MATRIX (PENTING)

❌ “OpenSearch Security not initialized”

→ security index belum dibuat

---

❌ “Unauthorized”

→ security aktif tapi user belum login

---

❌ indexer tidak listen 9200

→ service crash / Java / disk issue

---

❌ cluster RED/YELLOW

→ shard issue / disk corruption

---

CRITICAL INSIGHT 

Kasus kamu terjadi karena:

power failure + LVM issue
→ indexer state corrupt
→ security index hilang
→ bootstrap ulang pakai securityadmin.sh

---

BEST PRACTICE

1. Jangan pernah:

rm -rf /var/lib/wazuh-indexer/*

---

2. Gunakan snapshot (recommended):

/var/lib/wazuh-indexer → backup harian

---

3. Proxmox:

• enable graceful shutdown
• delay stop 60–120 detik untuk Java service 

---

📌 SUMMARY SINGKAT

Kalau error ini muncul lagi:

“OpenSearch Security not initialized”

lakukan urutan:

stop service → clean state → start indexer → securityadmin.sh → restart stack

Subscribe to receive free email updates: