磁盘达到阈值导致ES无法删除数据

method [POST], host [http://xx.xxx.xxx.xxx:9200], URI [/security_event/_delete_by_query?slices=1&requests_per_second=-1&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&refresh=true&conflicts=proceed&wait_for_completion=true&timeout=1m], status line [HTTP/1.1 403 Forbidden]\n{\"took\":19,\"timed_out\":false,\"total\":8,\"deleted\":0,\"batches\":1,\"version_conflicts\":0,\"noops\":0,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1.0,\"throttled_until_millis\":0,\"failures\":[{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"ecb098ef-2e3a-4c7a-a282-4484cabb362f\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"30c1da71-25c5-4e1f-a58f-95f6f5abfc52\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"687e14ba-0bc8-466a-83b1-294a3f9b2422\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"e0f25c75-cce3-4c44-9691-d4c79ecb72e1\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"dbfabe19-3318-420c-a7f0-eb74eab25d43\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"222fcf5e-8b72-4bd8-b3c8-a8c615db2ae9\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"690f57a2-1306-4749-8dfc-a1f0c5d926e5\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"280ff8cc-f43a-40ee-9ae3-9af48960832b\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}]}

排查思路。之前有一个清理数据的接口。用来清理大屏的历史数据。运维人员根据往常一样执行接口。但是报错了反馈到我这里。

一开始看到报错有很多403。联想到之前这个环境做了安全检测。关闭了很多端口。之前es没有认证以为被扫到了，询问运维人员得知es在内部网络并没有开放出去。于是重新审视问题报错。看到

index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

索引是只读模式不允许删除，但是并没有人去主动操作es。深入查询后发现es会检测磁盘使用率达到85%(默认)会将索引模型调整。检查磁盘后确实使用率达到了95%

解决方案：

临时解除只读限制

curl -X PUT "localhost:9200/security_event/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '

永久配置磁盘水印

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } } '

后续交给运维人员清理磁盘数据。

Kafka 多监听/网络隔离转发

老大难问题