|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。' R! _5 R( h I1 ~9 P
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
a# r- ?" |* O; M# [运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。+ L" c3 e) M# V/ l
接着设置语言和本地化设置：* ^+ i- B! ~% d
export LC_ALL=en_US.UTF-8/ e& j6 z3 k& p/ S9 U5 }
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
$ r3 l) [, g D' o7 k1 B5 R( m但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？& e/ R3 r1 x6 e s% ~2 Z% j+ k
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：; y; e1 q: M3 q( m F8 l8 @& B
Recipe: gitlab::database_migrations
: }8 U' x; d% k2 I2 O * ruby_block[check remote PG version] action nothing (skipped due to action :nothing) ^8 v0 u( n# O# k: C1 o
* rails_migration[gitlab-rails] action run+ w, j1 O5 W0 `! L
* bash[migrate gitlab-rails database] action run7 P+ D" _& P7 `- \- Q
* m$ }( f" P7 Y2 Q4 e% } ================================================================================
6 j7 A- {- l5 a8 Q3 Z$ \ Error executing action `run` on resource 'bash[migrate gitlab-rails database]': t7 B4 D2 T- I; ^7 ]6 ]) Z, f4 b
================================================================================$ S/ f$ D! k" \* m
: I+ ~2 e, d f6 M
Mixlib::ShellOut::ShellCommandFailed
4 O Z7 _0 {+ R- c8 Y; G; k5 M/ U" _ ------------------------------------
7 g* F% Z; s: z) P; L6 b) t! g Command execution failed. STDOUT/STDERR suppressed for sensitive resource8 g. N& {1 N8 ` U# G- k
! l+ u# t, o, w2 S Resource Declaration:
/ U9 X" V" A& V0 b ---------------------
7 ]# g$ S3 q$ L# |. Q2 @+ E suppressed sensitive resource output; G( Q" B6 O% [: q9 ~4 B
# K6 u# e6 K( |& N; V4 U Compiled Resource:
; E+ U6 P6 T6 v1 }" n3 H7 |. V ------------------+ E5 q$ k7 M- B% R+ t( \$ i
suppressed sensitive resource output
2 V( c3 Z7 h6 a& p
" o7 P* N& `: h0 o System Info:; k% t2 c& a) \9 v f' \% j9 @
------------
9 \- ^* u* A2 \8 R( v chef_version=15.17.4
2 `7 ]/ g! K( h platform=ubuntu
" P q: L$ l' y9 t' A) Z* |! w platform_version=20.04
5 Q/ L0 u/ \( E' D ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
1 n0 x7 }5 C' L1 [ program_name=/opt/gitlab/embedded/bin/chef-client
8 C% M5 H$ ~4 }$ F executable=/opt/gitlab/embedded/bin/chef-client: i O2 O0 I, H" k/ q
/ K( q) k" t! b: p* F
) w6 e8 M& p7 G( a/ o ================================================================================/ E- g8 Y1 e# [2 Z
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
" W0 Q2 e: s% a5 }* F( }3 Y ================================================================================
* W- j: N; y4 v ' w$ Z# \( v+ x9 s
Mixlib::ShellOut::ShellCommandFailed
' ~2 l' V& ?7 J# L ------------------------------------
% F1 ?7 j$ r4 [ bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
7 {$ v- G$ k' X) F 0 ?' F0 I/ ?" x2 O& h
Resource Declaration:3 t& f5 Z" b* r `4 a
---------------------
' Z) ~: ?8 h# W6 D: V # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb" n3 R4 F* U, F. G* O
1 H" ` O$ I3 ^, a7 z 51: rails_migration "gitlab-rails" do; T& u( P' _% g, |8 \/ o4 m
52: rake_task 'gitlab:db:configure'* p" O4 ~ A# `. n; ?$ \
53: logfile_prefix 'gitlab-rails-db-migrate'% M8 G8 q3 B3 |, b4 V0 V
54: helper migration_helper0 I* I! z2 B" V( \/ K$ p
55:
8 @, r: N9 F1 g9 ~ 56: environment env_variables
7 [6 H5 D W# L% K* Y 57: dependent_services dependent_services
9 P8 c# R$ }, i 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
9 T+ k+ ?" Z/ W) d% U G 59: notifies :run, "ruby_block[check remote PG version]", :immediately
2 Y w9 ~" k5 \9 i2 u 60:
3 c* J9 r% W- [9 o% N 61: only_if { migration_helper.attributes_node['auto_migrate'] }/ e( e' y3 Q7 g3 S9 E( P" W
62: end7 _% w! @3 O, L$ Q) I1 x
Q" p! ?8 }$ N y4 S
Compiled Resource:
& j& w, [# j- E& L( Z7 X$ ~2 b3 C0 V ------------------
~+ Q5 d6 { N- m! ~% j # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'2 O. o9 m4 X& e; P$ [7 C ]! O
$ [4 K, p6 B: p! f( T' p
rails_migration("gitlab-rails") do
9 Z+ j/ I# j ? action [:run]
! h( b. p& q4 K% w9 D default_guard_interpreter :default
+ A% v5 Y1 ^2 m. Q% H0 ~* D declared_type :rails_migration
6 u4 D1 b$ a% B3 a% I cookbook_name "gitlab"
2 \0 ]0 W4 l1 { K9 X X& a recipe_name "database_migrations"
( o; e; E0 l6 A9 F" Z5 ]% Y rake_task "gitlab:db:configure"
1 q2 _- v: w! }# |) h logfile_prefix "gitlab-rails-db-migrate"- a: D# m8 a3 K& C p& b: R8 S7 J
helper "*sensitive value suppressed*"
% I$ d3 Y7 V* d) m environment "*sensitive value suppressed*": W( y* f$ X: J- A
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
H% l. B- D4 ] only_if { #code block }1 e! u8 _* X2 D2 O
end
8 R! E7 C* p) g
8 U4 s2 b5 q' s; M2 Q System Info:
0 F: O2 x2 w# U1 }- w( n7 q ------------
. k& x' {* e$ Q/ t chef_version=15.17.4
0 n! n' \ o% y platform=ubuntu; N) }# z9 Q: E' X
platform_version=20.04. Q5 K6 S0 G8 t
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]7 B9 F( M, O6 H2 h
program_name=/opt/gitlab/embedded/bin/chef-client
0 |6 ]& X; ~6 E6 S7 g+ a executable=/opt/gitlab/embedded/bin/chef-client, V' ~) \7 S. \6 }
9 W2 O$ Q/ y, D* z: m! n d
Running handlers:/ ~! E0 @$ A: c( v1 c* ]
There was an error running gitlab-ctl reconfigure:
9 j/ ~$ t! e6 j6 b3 a6 S, a. {rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource N% A/ Z( R& V4 m7 h
Running handlers complete2 a. ?4 Z5 @, M. d& @. h/ C
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
" w& c- ?0 ^* J+ R3 s1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；8 l' a+ a: N# e' ] n
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；# k! Q, z e: t2 F/ r% g
3. 再次运行 gitlab-rake db:migrate，确认不会再报错；3 l" j- z5 | y# t
4. 运行 gitlab-ctl reconfigure;
/ X5 c) O) c. M# g5. 重启Gitlab：gitlab-ctl restart。# R5 i% ^1 u# s
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
" a/ x+ A$ e& S参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated" P; ]7 }) C" `
2. gitlab-ctl reconfigure failing due to migration issue
0 E$ e) T+ G# a4 [; x' c( a0 g% \+ RAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
