|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。, Y$ m" F9 _) v7 m0 }3 n5 d5 k
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。/ q# \- o4 v0 \8 f5 ^1 ^! w
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
: v* c* \, R$ `接着设置语言和本地化设置：
# h" y" Z1 S0 q& f6 W' Yexport LC_ALL=en_US.UTF-8) C: F7 \4 @& U0 {. s' Z, e3 y
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
$ ^9 y0 s; o2 H! k但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？7 S7 o1 x1 @& K, O5 |6 Q/ W' q
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
# r9 r; m; q. aRecipe: gitlab::database_migrations
+ h1 ^2 D5 c. B9 n; ? * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
/ A# U; J: o7 N' j * rails_migration[gitlab-rails] action run' X: l ^' C% r
* bash[migrate gitlab-rails database] action run* a5 |0 h! A& s- A: Q. ?
, Q6 B- v$ w% k/ D& n0 s- x$ Z
================================================================================: g( U9 `: E0 U0 P, ]
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
s% U3 N5 s7 p; D. d/ h ================================================================================
; K2 N& P6 _ H8 T# n7 E $ k3 N; G+ q5 ^8 E% _( `
Mixlib::ShellOut::ShellCommandFailed
; q5 O# C! h9 j V( [ ------------------------------------
9 L" y; v9 ~2 b- V Command execution failed. STDOUT/STDERR suppressed for sensitive resource$ t8 C) l/ ^9 _ y
! }, k* h7 `$ m3 C Resource Declaration:
# Q" O# C `) M ---------------------
: j( C/ t1 ]$ \* y suppressed sensitive resource output
4 Q* K8 T& Z8 p4 ?
1 w# t8 a+ e$ M# c2 K Compiled Resource:
$ ~5 G( q6 e$ }' ^. U2 R0 a$ I1 ~ ------------------
. S* t) z4 ^4 z# Z5 Q% X1 z+ v suppressed sensitive resource output- O! n- R/ g5 b4 m- | ~
+ f. V- Y( O, x" {# W5 b9 F4 s System Info:) q' Z8 E! Y9 D. I7 e# P
------------4 E/ m! i. A0 A' V2 }* e# j+ g4 {: W7 B
chef_version=15.17.4
* u/ ?8 ~% a; d$ g2 `) \( U platform=ubuntu3 P( C; v0 K+ q0 [7 n( e) d
platform_version=20.04
& ~" z1 X2 F4 V1 o2 y8 { ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]6 ^! l$ I$ q2 g( @5 y9 p0 e
program_name=/opt/gitlab/embedded/bin/chef-client# v5 P( z" l# E% m% a' g: d0 T
executable=/opt/gitlab/embedded/bin/chef-client' R1 u/ n/ b/ A- W
2 y7 G$ [! {7 E. l- I- u, X
) B2 y; C* y' h9 _/ M5 v
================================================================================- {& k9 Y: t4 |8 S( Y% l
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
% y- ?1 B0 s$ P* x" i& n" j" B# b ================================================================================
! C1 P0 `! I' w
% S# c( f* I* J- D Mixlib::ShellOut::ShellCommandFailed
" Z5 p" J# G( G+ [$ `. p ------------------------------------
* H: o9 G+ P2 J( t bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
# Y: X/ }& H* A) p/ ?. d2 b 8 h. K* ~# V* x% \$ f
Resource Declaration:
( v! i: R/ `) y* ]5 C ---------------------1 Q# A& w' s) ?# Q( T0 x3 ~
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb/ M8 X: [) E( i
3 b; a7 z4 K- x. e) I3 h! w 51: rails_migration "gitlab-rails" do
/ ^1 Y+ [ ^" G5 f' v 52: rake_task 'gitlab:db:configure' G% C3 k3 ~& A# J: d
53: logfile_prefix 'gitlab-rails-db-migrate'& k% @5 M9 r, R8 X
54: helper migration_helper
6 o5 G; { D* ?* E( F' K 55:
) i! U3 T% A3 H! O 56: environment env_variables: I; t/ V1 v2 e9 c5 f
57: dependent_services dependent_services5 Q9 G( Q" A$ X, d* ]: `: h; V
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately# ^# f \; X; u! v |& U
59: notifies :run, "ruby_block[check remote PG version]", :immediately9 H, U+ i0 B% ^' ?1 `
60: - A. X$ u% g: r( Q5 w6 y D1 \/ @
61: only_if { migration_helper.attributes_node['auto_migrate'] }
?5 s* H; ]) C1 `3 n5 Z 62: end
0 `+ v" w% ~1 M1 U+ _, j7 M 4 p; d. X p" [2 ^, p- w( [
Compiled Resource:
. w+ T6 H. g& @3 } ------------------4 w9 H% l! v5 @; B' E* S
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
% J/ ] _! b' e9 x: W; j 3 v- }9 }7 q0 Q) S; @! @
rails_migration("gitlab-rails") do
9 u0 L9 {5 C$ |9 b: B, V6 n6 K action [:run]0 J6 U. Q$ d! P/ D- m
default_guard_interpreter :default" |/ e7 K- }' T! K% O" r
declared_type :rails_migration
! m7 H7 w8 ^# R2 ?! d cookbook_name "gitlab"6 P0 z& }0 ?# G1 Z8 U: c
recipe_name "database_migrations"
) b& A4 G: B- e1 Q/ t9 Z rake_task "gitlab:db:configure"$ A, W5 h3 C" ?
logfile_prefix "gitlab-rails-db-migrate"
7 V& z) G* Z. G4 V5 p5 U& Y helper "*sensitive value suppressed*"( h+ a, S' P' n2 n6 j
environment "*sensitive value suppressed*"
" ?. p Z# y* x H2 K- i dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]6 l3 L4 y+ J% C
only_if { #code block }2 ]1 e I7 Y, p4 h
end8 _; x: h, S0 N7 c
- O* K! P/ _; A" q
System Info:
* `) t! t8 W) l+ d# L. e ------------
3 H0 g1 G9 b$ O% |3 N, Y& C chef_version=15.17.4
z8 M. @+ d2 U1 X: m" l2 c4 s, @ platform=ubuntu; _8 m* g3 z' }/ l3 U0 M& g
platform_version=20.04
$ o" H1 l7 g- G6 c& M& L ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]1 @2 a' o$ C# ?3 @" \
program_name=/opt/gitlab/embedded/bin/chef-client
" N/ b$ G7 G; [6 K7 J: ^! J' d executable=/opt/gitlab/embedded/bin/chef-client- W( W- I# _, n1 ]6 [: t
7 w1 V% p9 E' E/ H" F3 J+ FRunning handlers:
5 D1 B* ]) y( m p5 e. |3 A" {There was an error running gitlab-ctl reconfigure:
1 Y+ N0 L. m2 u: Srails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource% @# }) a a! W
Running handlers complete
' _+ w% x2 C6 K* B; MChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
. a, Z0 n! N6 F. ?* t9 d. _1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；* A) m- m/ A. }: D
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；) W0 E: S6 D2 v! K! m; A- a
3. 再次运行 gitlab-rake db:migrate，确认不会再报错； @3 ~2 M0 l* h
4. 运行 gitlab-ctl reconfigure;
" V% _. A$ |3 e5. 重启Gitlab：gitlab-ctl restart。
2 b4 Y- S& U6 \9 h; H( R需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
9 N' O1 e1 F$ ^; `6 T参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated! K5 q& X2 a( _8 s+ _
2. gitlab-ctl reconfigure failing due to migration issue
8 p8 F. g) ]: X# b O' {; @3 CAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
