|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。8 I1 b, A% P& `) c
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
' K* q* ?8 U" Q- k& P/ z8 {8 I0 m运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。! }; K* v- X$ o+ M+ |1 g# n
接着设置语言和本地化设置：$ m* }$ h4 d/ F& d
export LC_ALL=en_US.UTF-8+ j. g0 f: @1 @* Q5 ?
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。, d/ k, _- F+ a1 ^( r9 h
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？! @5 I3 p L$ C
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：; ?; n$ l7 }3 k: R l y& _
Recipe: gitlab::database_migrations
3 a9 _1 W6 u: z, ~0 B5 U * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
) e# c, `1 R$ [, j+ ^' D: x * rails_migration[gitlab-rails] action run
2 W$ ^4 f- B9 O * bash[migrate gitlab-rails database] action run8 E3 D# h) A6 F1 D2 w+ v3 j. t
% W3 `( g( C% `6 w/ w ================================================================================
9 R. x' d# \) N Error executing action `run` on resource 'bash[migrate gitlab-rails database]'$ u2 s% E' A9 |
================================================================================0 L% N3 [. o n. E" J* Z
/ w. S5 `; ^4 s* x" ?3 v& R( d" w: ?
Mixlib::ShellOut::ShellCommandFailed
. {& h/ ~1 c" p7 g5 D ------------------------------------
4 [* \: t) g6 ?' F0 l1 G7 L& l Command execution failed. STDOUT/STDERR suppressed for sensitive resource
4 }+ R w+ B: o6 z" y ! P7 s5 C0 U; Z6 R
Resource Declaration:; l* ]! [* m* O1 Z& a& y
---------------------
4 {5 H2 h0 V9 c. d4 h R suppressed sensitive resource output. @, Q' ]$ b. u$ N1 U
: y8 E5 _" G% n; o
Compiled Resource:1 ]; y1 K' F. X% K( Y, o& x
------------------
/ |5 X7 l- D, t1 X% v suppressed sensitive resource output
1 {+ u/ n8 [( P# I( I2 [- G/ l
) u) [( C3 U5 {2 P0 I. t8 U System Info:
' f% ~9 g$ T$ U ------------
+ e% F' w( N+ F$ e% ~! @( M chef_version=15.17.4
, W/ h5 G# @' i G P platform=ubuntu7 ~/ R& J% z5 V& S- ?5 M- ~0 U7 _
platform_version=20.04) A) Q3 H* e! i; Q0 p$ \
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]# s$ e/ I9 t. o
program_name=/opt/gitlab/embedded/bin/chef-client
# l9 |/ E6 x2 h X; ^2 o# e! ^ executable=/opt/gitlab/embedded/bin/chef-client% j# K' z* I2 x' L- E5 g
H8 {- B+ @1 z9 f: J( D) Y . d" r5 J0 ~( n1 D
================================================================================5 T! \" J, b$ \) C- x7 I: g
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
* B8 m" h8 w3 i6 `1 u8 Q ================================================================================
' ?$ ]5 G* Z# O' `1 b* Q 8 [0 p+ H1 c3 C1 u/ ?& t
Mixlib::ShellOut::ShellCommandFailed$ ?5 T X0 a0 o5 P1 _( T
------------------------------------
# E ^% l; u3 ?2 m* J bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource2 T: ^4 i5 f6 } n
# O# @# R; B/ X. R: O) [- ]; U Resource Declaration:) Z. g) X5 S; d3 E$ P" @
---------------------
9 p6 P( ?! q) Z) A0 \ # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
# @; v2 D9 [) N8 m, E1 O
) C6 ~3 p; l; _' y2 t+ J 51: rails_migration "gitlab-rails" do
2 ]3 C5 g6 _- k9 I3 j" G 52: rake_task 'gitlab:db:configure'! s& `3 B; G+ `2 r' o/ y
53: logfile_prefix 'gitlab-rails-db-migrate'6 m: R' \9 s2 `2 X4 t
54: helper migration_helper
8 ^- x% w2 [8 m4 P! a ^ 55:
( F, K4 J/ }7 b0 g+ `+ h 56: environment env_variables0 R) u e6 q* L; a6 D" s
57: dependent_services dependent_services
/ m5 b. Q- z% u; A5 m 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
! b3 q7 U/ h Z2 ? 59: notifies :run, "ruby_block[check remote PG version]", :immediately9 M6 B T7 F* I3 Q8 R' l- Z% m+ `1 B
60:
$ i' v% c" U2 p! q$ [$ s6 m 61: only_if { migration_helper.attributes_node['auto_migrate'] }9 u+ X* y% Z7 ?$ |# x, j+ h- V+ h5 [
62: end
' v6 n* o6 \8 c6 e Y' j3 F$ K & X) N9 h) } |. g. A! k
Compiled Resource:* h2 Z$ ]1 E2 Q& E3 C# l" q) m
------------------
; A0 a" n& P; d # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
3 a- d, J$ ^3 v! D! \0 ] - j% S2 c2 t* |$ Y& s4 K' w
rails_migration("gitlab-rails") do
3 v; y) H( U1 S. x7 b8 Y% [0 Y8 p action [:run]
. C9 U5 H% _) W w8 }% F+ r default_guard_interpreter :default
5 X% v; j( ^5 f& F$ ~& Q4 x declared_type :rails_migration
! r* S. V. W+ F9 z cookbook_name "gitlab"- \$ p7 }8 |; ^9 e6 O; n
recipe_name "database_migrations"8 }9 O. l& E$ {& X, I! H# E
rake_task "gitlab:db:configure"# D/ G' d, ~' w# s5 _ A& ~" x
logfile_prefix "gitlab-rails-db-migrate"9 H7 |4 y6 x, Z" [4 t4 y! n
helper "*sensitive value suppressed*"0 n, Q3 l0 x* q
environment "*sensitive value suppressed*" ~1 d4 \7 p3 |/ f' u# E; ~5 A
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
+ e! Y: ]: D, O; |- Q: ]5 c only_if { #code block } I+ ]1 }& M, u; l* z
end
, E1 Y. Z+ p9 k3 U 6 P! N! R [$ O
System Info:) {& G8 D" c' o" @
------------
/ W) M/ t9 ^( x0 e) u chef_version=15.17.4! l/ D( o# y) J6 r- V) z' e) ?
platform=ubuntu2 u2 |' l5 b* ~: a
platform_version=20.04
6 ], K3 K. i+ e8 a2 g1 }& ] ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
! I+ s7 ~% @& W& `$ |( N, D program_name=/opt/gitlab/embedded/bin/chef-client9 {9 ?: \' w) x9 g
executable=/opt/gitlab/embedded/bin/chef-client
; M$ K% q! w" ~( E ! c! M' y; r- q* C/ k" a* `1 C
Running handlers:
9 Q1 h s0 h* Q( J( X" q$ vThere was an error running gitlab-ctl reconfigure:
7 U3 h0 { }; N# A' irails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource" y! c( `" \8 O, d) _
Running handlers complete
2 W% N% w( D, F8 iChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
6 |: U' |, W* w1 L7 {1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；. z* X8 \* U' M+ I5 h
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
% ?. m! C9 z/ |3. 再次运行 gitlab-rake db:migrate，确认不会再报错；% @& `, D" G) y! ^$ Y
4. 运行 gitlab-ctl reconfigure;- D) U. {# `0 Y
5. 重启Gitlab：gitlab-ctl restart。
) \/ Z8 C, ?* G8 _" z需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。& c% A! o7 S$ G+ _2 R
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
( e- H5 H. X/ S4 g2. gitlab-ctl reconfigure failing due to migration issue
4 B. P- |- K E* x: k$ ^5 jAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
