Compare commits

...

2 Commits

Author SHA1 Message Date
zerosaturation
cacc686e9a fix:notification/moderation service healthcheck 配置错误
两个服务的 healthcheck 配置都有历史遗留 bug,导致 docker ps
一直显示 unhealthy / health: starting:

1) notificationService
   - 现象: docker-compose healthcheck 路径写 /healthz,但 pkg/health
     只注册 /health (backend/pkg/health/health.go:29),wget 永远 404
   - 修复: compose 路径 /healthz -> /health,Dockerfile HEALTHCHECK
     同样改 /health (双保险,rebuild 后不会回退)

2) moderationService
   - 现象: main.go 没启 health server (只暴露 Dubbo Triple 协议),
     compose 原 healthcheck 端口 20011 永远 timeout,
     Dockerfile HEALTHCHECK 也指向 20011 同样错
   - 修复: compose 用 healthcheck: test: ["NONE"] 显式覆盖 image
     HEALTHCHECK,Dockerfile 改成 HEALTHCHECK NONE (image-level
     也禁掉,rebuild 后不会回退)

gateway depends_on moderationservice: service_started (不依赖 healthy),
不影响入口可用性。健康状态靠 gateway 间接覆盖。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-21 12:12:46 +08:00
zerosaturation
7bf4666cdb fix:restart 命令卡死问题
deploy.sh restart 命令原本 down && up --force-recreate + sleep 10,
在以下场景会卡死:
1) docker-compose up 中途异常退出,server-side 进程没回传 EOF,
   本地 ssh 客户端没设 ServerAliveInterval,死链检测不到,
   ssh 进程挂 25+ 分钟不退出 (2026-07-21 实测)
2) up --wait 等所有服务 healthy 才返回,但本 compose 里有
   oss-cors-init (restart: no, exited(0) 永远不 running) +
   moderationservice/notificationservice healthcheck 写错,
   永远卡住

修改:
- ssh_cmd / ssh_cmd_batch / scp_cmd 加 ServerAliveInterval=30 +
  ServerAliveCountMax=3,死链 90s 主动断
- restart 去掉前置 docker-compose down (up --force-recreate 自带 stop+create)
- restart 去掉末尾 sleep 10,改 polling gateway healthy (timeout 180s)

端到端实测 1:19 通过。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-21 12:12:27 +08:00
3 changed files with 42 additions and 14 deletions

View File

@ -220,6 +220,7 @@ HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=3 \
ENTRYPOINT ["/app/statisticservice"]
# ---- Runtime Stage: NotificationService (通知 + uniCloud 推送) ----
# health 端点路径是 /health(不是 /healthz),见 backend/pkg/health/health.go:29
FROM --platform=linux/amd64 alpine:3.19 AS notificationservice
RUN apk add --no-cache ca-certificates tzdata
@ -230,11 +231,14 @@ COPY --from=builder /tmp/notificationservice /app/notificationservice
EXPOSE 20010
HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:21010/healthz || exit 1
CMD wget --no-verbose --tries=1 --spider http://localhost:21010/health || exit 1
ENTRYPOINT ["/app/notificationservice"]
# ---- Runtime Stage: ModerationService (举报 / 反馈 / 自动隐藏) ----
# moderationService main.go 没启 health server(只暴露 Dubbo Triple 协议),
# image-level HEALTHCHECK NONE 防 rebuild 后回退到错误的 20011 HEALTHCHECK。
# 运行时 compose 用 healthcheck: test: ["NONE"] 显式覆盖,这里 NONE 是双保险。
FROM --platform=linux/amd64 alpine:3.19 AS moderationservice
RUN apk add --no-cache ca-certificates tzdata
@ -244,8 +248,7 @@ COPY --from=builder /tmp/moderationservice /app/moderationservice
EXPOSE 20011
HEALTHCHECK --interval=10s --timeout=5s --start-period=15s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:20011 || exit 1
HEALTHCHECK NONE
ENTRYPOINT ["/app/moderationservice"]

View File

@ -116,25 +116,30 @@ SSH_KEY_PATH="$HOME/.ssh/id_rsa" # SSH 密钥路径,默认使用 ~/.ssh/id_r
# ==================== SSH 别名 ====================
# 优先使用 SSH 密钥,如果失败则使用密码
# SSH 加 ServerAliveInterval/ServerAliveCountMax 防止长任务时死链(被 NAT/firewall 静默断开)
# 又不报 EOF,本地 ssh 进程一直挂着等 server-side stdout 关闭。
# 历史教训:2026-07-21 ./deploy.sh restart 起了 3 个容器后 docker-compose 异常退出,
# 但 stderr/stdout 没回传,本地 ssh 客户端挂着 25 分钟才被用户手动 kill。
# 30s 一次心跳,3 次 (90s) 没回应就判死链,SSH 进程会主动退出。
ssh_cmd() {
if [ -n "$SERVER_PASSWORD" ]; then
sshpass -p "$SERVER_PASSWORD" ssh -o StrictHostKeyChecking=no -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
sshpass -p "$SERVER_PASSWORD" ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
else
ssh -o StrictHostKeyChecking=no -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
fi
}
ssh_cmd_batch() {
if [ -n "$SERVER_PASSWORD" ]; then
sshpass -p "$SERVER_PASSWORD" ssh -o StrictHostKeyChecking=no -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
sshpass -p "$SERVER_PASSWORD" ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
else
ssh -o StrictHostKeyChecking=no -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -p "$SERVER_PORT" "$SERVER_USER@$SERVER_HOST" "$@"
fi
}
scp_cmd() {
if [ -n "$SERVER_PASSWORD" ]; then
sshpass -p "$SERVER_PASSWORD" scp -o StrictHostKeyChecking=no -P "$SERVER_PORT" "$@"
sshpass -p "$SERVER_PASSWORD" scp -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -P "$SERVER_PORT" "$@"
else
scp -o StrictHostKeyChecking=no -P "$SERVER_PORT" "$@"
scp -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -P "$SERVER_PORT" "$@"
fi
}
@ -702,9 +707,24 @@ main() {
exit 1
fi
# ⚠️ 不要 `down && up` 两步走:
# 1) `up -d --force-recreate` 本身就会 stop+create,先 down 浪费 5-10s
# 2) 先 down 会一次性把 postgres/redis 也停了,但这俩根本没 env_file,
# 无理由重建,白增 cascade 时间和数据卷 IO 风险
# ⚠️ 不要 `up --wait`:会等**所有**服务 healthcheck 变 healthy 才返回。
# 但本 compose 里:
# - `topfans-oss-cors-init` 是 `restart: "no"` 的一次性任务,跑完 exited(0),
# `--wait` 把它当 hang 永远等变 running → 卡死
# - `topfans-moderationservice` healthcheck URL 端口写错(20011 → 应 21011)
# - `topfans-notificationservice` healthcheck 路径写错(/healthz → 服务没暴露)
# 这俩历史遗留 bug 让 `--wait` 永远不退出。
# ⚠️ 不要 `sleep 10`:固定 sleep 不准,等不到 healthcheck 也白等。
# ✅ 改成 polling gateway healthy:gateway 是 Layer 6 末梢,depends_on 所有
# 业务服务 + redis,它的 healthcheck 通过 = 入口可对外服务。
# moderation/notification 是 internal healthcheck 不影响 gateway,放过即可。
# 给 180s timeout 上限防极端情况。
print_step "🔄 重启服务"
ssh_cmd "cd ${SERVER_PATH} && docker-compose -f docker-compose.prod.yml down && docker-compose -f docker-compose.prod.yml up -d --force-recreate"
sleep 10
ssh_cmd "cd ${SERVER_PATH} && docker-compose -f docker-compose.prod.yml up -d --force-recreate && timeout 180 bash -c 'while [ \"\$(docker inspect --format={{.State.Health.Status}} topfans-gateway 2>/dev/null)\" != \"healthy\" ]; do sleep 3; done && echo gateway-healthy'"
print_msg "$GREEN" "✅ 服务重启完成"
;;

View File

@ -523,7 +523,7 @@ services:
expose:
- "20010"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:21010/healthz || exit 1"]
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:21010/health || exit 1"]
<<: *healthcheck
deploy:
resources:
@ -536,6 +536,12 @@ services:
# 内容审核服务(举报 / 反馈 / 自动隐藏)
# 依赖 user / asset / notification Dubbo 服务,以及 Postgres + Redis
# ⚠️ healthcheck: test: ["NONE"]:moderationService/main.go 没启 health server
# (只暴露 Dubbo Triple 协议在 20011),加 healthcheck 会永远 unhealthy。
# Dockerfile.services 的 HEALTHCHECK 指向 20011 也是错的(指向 dubbo 端口),
# 必须用 test: NONE 在 compose 层显式覆盖 image 的 HEALTHCHECK。
# gateway depends_on moderationservice: service_started(不依赖 healthy),
# 所以不影响入口。健康状态靠 gateway 间接覆盖。
moderationservice:
image: topfans/moderationservice:latest
build:
@ -572,8 +578,7 @@ services:
expose:
- "20011"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:20011 || exit 1"]
<<: *healthcheck
test: ["NONE"]
deploy:
resources:
limits: