|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
* L/ V+ O: z8 \& F4 V6 o1 m登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
+ i, K! Q& D c运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
4 M) x# W& i" _5 N' y! v接着设置语言和本地化设置：1 @# G; O: _ S$ H Y# ~/ x( V7 h
export LC_ALL=en_US.UTF-8
4 f! l2 T- _: b0 t- x& Xexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。3 o. P0 p; G4 P8 |+ y9 P
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？) g+ v, Q/ C; u" \) @$ m
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：8 N4 L4 j0 m4 h# R6 @1 P
Recipe: gitlab::database_migrations* m/ N* C n0 v* w; c
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
: l: w* N% {, D% V * rails_migration[gitlab-rails] action run' z6 R+ R, N! m$ O% f
* bash[migrate gitlab-rails database] action run4 i8 L1 H# j1 w, e" f
+ d" \6 q0 `7 A1 q
================================================================================
/ c. ]* O& P8 I) F; J4 g( l/ ~$ d Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
, P% V& Y f; n/ w) I+ g* l# C ================================================================================( i9 W2 k& a$ D- ]8 D- d! D* B
% h& L. |% M) y3 @6 S Mixlib::ShellOut::ShellCommandFailed
) j/ j# `6 n7 [) R% {# x ------------------------------------
- U1 B7 B m9 c/ Y4 K Command execution failed. STDOUT/STDERR suppressed for sensitive resource
% d1 |. e& B" q9 s" m1 S0 E 0 F6 o( [, z0 r$ k" D, X5 h& G+ U
Resource Declaration:* a* j( p0 Q4 F' j+ _: ]. d- @# V
---------------------# {. T) P7 f' }. U
suppressed sensitive resource output& Y, I5 q. x4 H* F7 W2 x
# [" c, w8 k3 W6 ?) F& M
Compiled Resource:/ j. H+ N/ ?# ]
------------------% V9 B4 B5 m, r1 ~6 K
suppressed sensitive resource output' X" h2 ^, K, Q; A# Y
& \1 D9 r }; h, y7 w! S
System Info:; O# U) g! ^/ i0 m' R: f, b+ q I
------------' V' Q4 f3 A( a% g# H' b$ ]9 B
chef_version=15.17.4* K% } v& q# s% v' @: J3 Q8 H
platform=ubuntu
1 y" Y2 Y6 P$ y0 ]* b2 F0 ^ platform_version=20.04
1 |5 ?0 Q y) d; O6 ? ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
1 s$ o; v+ g8 }) Q; V+ L) ] program_name=/opt/gitlab/embedded/bin/chef-client
; B) q- I( @3 c+ Y executable=/opt/gitlab/embedded/bin/chef-client
7 o# w* z7 }' H5 A7 l6 P+ y * L. V* V+ p$ y; ?
$ a3 X; R L7 V ================================================================================
5 `% j4 g' ^' s. j3 s Error executing action `run` on resource 'rails_migration[gitlab-rails]'
) U, }4 @# `+ j( @& s ================================================================================) b2 f6 O3 p( s: a7 ?- E- G
2 U r7 h4 O0 ^: s. g. `2 r
Mixlib::ShellOut::ShellCommandFailed
) [" m& j' w$ x1 d5 m; e1 c ------------------------------------
# H. \( e K# ~' K4 p9 K6 H- I bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource" [8 Q K4 Y5 i$ s3 `2 ]; [
9 k, T+ m1 H m) O4 c Resource Declaration:
' r0 M* h& i# f5 i5 _: ` ---------------------
" n! P$ g* `. G, Z # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
4 N. o! z9 t# w/ t% T4 B: G
; p: O. [5 O! ]0 Q: ` 51: rails_migration "gitlab-rails" do) U) j4 Y* G$ v8 o) G
52: rake_task 'gitlab:db:configure'
: u0 } E$ W1 P3 O ]4 I) u 53: logfile_prefix 'gitlab-rails-db-migrate'
8 G* Y7 W/ _3 N5 j# b+ l 54: helper migration_helper
* T" a1 Z: q u3 R 55:
3 o0 Y1 l, V7 {2 x, l. E L R0 ^2 N 56: environment env_variables) r. w: Y- F! t6 n' l
57: dependent_services dependent_services7 t* i3 X9 p9 b3 z5 B5 t1 Y) r. w0 e
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
; ~; o x1 S4 r, t 59: notifies :run, "ruby_block[check remote PG version]", :immediately
9 S0 C0 Q2 C# U# l& P# S. ] 60: % z! r3 c' m0 x# a) Z5 K3 T6 G( G0 y
61: only_if { migration_helper.attributes_node['auto_migrate'] }
4 Z3 T7 H" i& n$ a0 L 62: end
; h& b8 k# u. S. t; d% ?$ C1 ~ 2 P- N- G& r2 l x
Compiled Resource:
6 L5 T8 d0 M5 g& z d ------------------
5 J' e7 [4 ~$ ~8 A # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
4 ?# N5 i4 b; w
# Y2 Y/ f$ w' L% s7 @1 O rails_migration("gitlab-rails") do$ T7 u1 S0 ^5 K5 a% g5 P
action [:run]
* n0 I& u7 U- K- T0 _+ p0 Y2 R default_guard_interpreter :default( C% o* D# `$ y. {% B ^6 h
declared_type :rails_migration
2 J' T5 C) I) Z7 i/ X4 v9 y cookbook_name "gitlab"
* i: U8 W% y$ ]$ u recipe_name "database_migrations"
5 }/ e& ]3 ^$ P1 m rake_task "gitlab:db:configure"
3 K3 t- ]) I Y% Q) B logfile_prefix "gitlab-rails-db-migrate"
# ^! j& a% Z. R8 J" j helper "*sensitive value suppressed*") y; ^) x2 B9 W) B6 J5 A- R' u
environment "*sensitive value suppressed*"
, @! k2 f+ c1 W- x# F! j& { dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]4 S+ z; v$ V* e. s5 R% J
only_if { #code block }
# p3 \( G" p( {4 u8 z end$ s* w6 b" K( J! \
/ F' n/ U: w$ s( w" H
System Info:+ `4 ]: m& J$ f( q1 _
------------" p# m1 H$ n( V# {+ T
chef_version=15.17.4. [, a2 H2 n& \- e, n
platform=ubuntu+ l1 x# I. [" n9 y- N3 Q8 V
platform_version=20.043 R+ @0 w6 J' O+ w" f/ \3 T2 O
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]3 l4 ]1 Z R2 h0 i B
program_name=/opt/gitlab/embedded/bin/chef-client
$ R$ |3 P2 Y' n( ] executable=/opt/gitlab/embedded/bin/chef-client
/ ^; Z2 l4 {4 L1 ^7 \" R
: d' B$ F( |; g9 Q& [Running handlers:
$ `; T' @+ i, F; g& ~There was an error running gitlab-ctl reconfigure:
4 r) K: `4 h. ^3 R/ r% [: K" T( Brails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
) U# e" Y; I) P4 x4 eRunning handlers complete) ?& X- V3 a" C: L: Z# P6 D
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：) R$ m! S1 u/ h* t; Y
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；- |+ {0 I+ |9 G S2 I
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
3 v, l z9 o& C( q$ I: [/ U3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
# P$ }9 D* P J+ E/ d4. 运行 gitlab-ctl reconfigure;6 P( b7 F: T0 o. b2 X2 }6 J
5. 重启Gitlab：gitlab-ctl restart。- M, O7 J& b) t& D; R* ^
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。& q* j5 Z/ P8 { }4 n
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated4 @ p1 u. [: a% L% W# y
2. gitlab-ctl reconfigure failing due to migration issue5 N. I' |* s1 }
AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
