|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。# g5 B0 ~1 u% k5 @7 {
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。1 s2 Y0 M8 F+ p5 p3 O: R/ L) m
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。2 U4 m- r, w2 s; [- }) q
接着设置语言和本地化设置：* u$ L) c6 H2 f5 V# P
export LC_ALL=en_US.UTF-8: ]$ D; E' c- [$ c
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。! ~# w( J7 q: d
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？/ o: }8 |$ `* K( k: n3 u7 [
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：6 c. _$ B4 r" z( M9 j3 K1 H
Recipe: gitlab::database_migrations
- k$ d8 d: o) |5 N6 i * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)- s: ]6 p7 P0 u6 R+ z( y! V [ }
* rails_migration[gitlab-rails] action run, K% E1 D$ `3 s
* bash[migrate gitlab-rails database] action run
4 @% w# h' R" w2 P' n! K6 c" k Y* y# Z; I+ M! t
================================================================================6 @5 H3 O* P1 \. z
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'+ {) X! h+ Y' t
================================================================================* G8 B$ h) G$ ] \ o/ U: D1 g
# E3 M9 K, F- T. t
Mixlib::ShellOut::ShellCommandFailed( j- z) w( ~8 l3 k
------------------------------------7 ]7 z. V* p/ s& \
Command execution failed. STDOUT/STDERR suppressed for sensitive resource
( m# C, B! E+ F
* U* w* k7 R. q! C p, v Resource Declaration:
5 F! Z% \, U7 c8 J ---------------------
8 F1 N" }! L7 T, A: H suppressed sensitive resource output, ^ E+ e: V" l# R
3 S1 L5 D5 n1 s8 e6 H
Compiled Resource:
+ ?/ g3 x7 L' j; `4 B3 W ------------------! O# S/ y) J* H, q* o
suppressed sensitive resource output
3 i3 T9 r7 B% m5 o
l3 N! u9 @+ P5 X! c System Info:
. X3 `4 V5 p7 j6 W! _# [ r ------------
: w- v2 _6 k" ~/ W w chef_version=15.17.4
6 s! S1 p4 ?1 }4 t: R7 L platform=ubuntu) d* H; V Q: D* [) l' i
platform_version=20.04: o$ q9 R7 r. m- o
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]8 T( o/ P) V. s' h
program_name=/opt/gitlab/embedded/bin/chef-client( C3 C9 ?0 N- ?
executable=/opt/gitlab/embedded/bin/chef-client
) E$ x+ p7 Y2 q/ d
6 G5 N( C4 N7 K Y # X- U, T; `, r- h
================================================================================
, L8 g8 L' p1 `/ o! W' F Error executing action `run` on resource 'rails_migration[gitlab-rails]'. E9 f: x/ w/ w4 y6 ~5 v5 g5 [
================================================================================' ]+ }. Y. C7 u& s9 e
8 M1 O# r) a4 _% h( Y' V
Mixlib::ShellOut::ShellCommandFailed
& ?* ^; \( d3 a2 R: _5 J0 z" u2 D6 K ------------------------------------" \( X2 g2 F/ C$ g# e. T2 z
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
$ U& Y5 ?: K# {9 c0 Y# w) z
+ H8 t" d- [0 k/ {6 N% D1 M Resource Declaration:
( @7 ~. N8 }1 v ---------------------8 d- j3 I' N* K/ W! }
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb( Z+ ^; s6 D5 a* B) W
& g) j; L: @6 o+ D, P: _: e% `
51: rails_migration "gitlab-rails" do: O9 l4 U+ I' f: ~: k6 S
52: rake_task 'gitlab:db:configure'* \4 f$ e5 d$ J1 _
53: logfile_prefix 'gitlab-rails-db-migrate'
7 g6 g4 ?# s) F& O+ @ 54: helper migration_helper4 a1 [( p" A1 L, a( f
55: " r9 W7 T2 T) l0 O2 h
56: environment env_variables
4 K. [- A3 }% i0 q0 j 57: dependent_services dependent_services
4 w+ J0 U7 q9 Z7 t0 w* Q 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
1 G. z9 M) B# q' o 59: notifies :run, "ruby_block[check remote PG version]", :immediately
6 B7 f; x5 \: I, Y3 A 60: % a2 T3 C+ p$ a6 g! C
61: only_if { migration_helper.attributes_node['auto_migrate'] }* W3 \- A1 o' \" J+ {
62: end% O, \: a. m& C2 K- d" p
1 ^6 Y Q) `* x; W P2 c+ o3 F4 n' t# Q Compiled Resource:
7 R L; h, {" U1 f ------------------
' ^* f* F" } A# o # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
L* m5 U* L4 |, N& | ) \( }; F; }2 C; M* K
rails_migration("gitlab-rails") do( D1 \: Z8 K: E: C7 e6 B- [
action [:run]8 p4 z5 w- S" u2 m5 n* u" P a" Y
default_guard_interpreter :default
7 M: B2 H- M9 ^ declared_type :rails_migration
8 F( q% p" O- P cookbook_name "gitlab"7 v4 H' N% C$ [% A
recipe_name "database_migrations"' _: o/ w% o% {
rake_task "gitlab:db:configure"1 I8 e+ Z: Z; J! a9 F
logfile_prefix "gitlab-rails-db-migrate"# n& M( r0 n# B' D* j! s! r' K/ s+ k
helper "*sensitive value suppressed*"0 _6 Q& T' ~ ^
environment "*sensitive value suppressed*"7 V3 k- \2 D* G: Z+ }
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
0 Y- s7 p) B K! ^/ e7 n# w only_if { #code block }# B u' A3 Q* ^& }7 o: l5 j) G
end
z9 c. n' B0 Z" H5 r+ Z
O' p q/ ~ A" d: A! E5 v' y& y0 l System Info:
: `' @" u+ A0 ]' o" Y+ _ ------------
8 F3 x7 y+ {1 W1 s9 G chef_version=15.17.4
7 |, x: M3 J& t h8 w) J* h platform=ubuntu( f- K+ F8 ]1 P* ^" B
platform_version=20.04
1 H- T7 e M" `4 i4 s ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
- h1 c# h% W3 I program_name=/opt/gitlab/embedded/bin/chef-client
, y+ C2 a9 h3 ~& O. ` executable=/opt/gitlab/embedded/bin/chef-client
2 s! K- k8 E+ _# a ( [; H. D! v1 Y6 o' X% c/ g' I0 i
Running handlers:
5 \8 A$ F) |) h7 P8 h$ EThere was an error running gitlab-ctl reconfigure:" }0 q" A8 A% t* m- X ~
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
4 R4 j7 h7 x3 q2 P, ]Running handlers complete+ [, A7 \. W/ n- j: G4 N
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
, `1 s( f- Y, S% A. o1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；
: L, s1 H% P( I: E2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
; H: K' r; r1 i/ |6 p3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
% {2 I" z$ b8 b! S& G1 f4. 运行 gitlab-ctl reconfigure;
5 c' P' m& [5 }5. 重启Gitlab：gitlab-ctl restart。5 N, w8 Q x' Q& R, x& x
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
& w9 j1 t. t( Q0 |1 n$ ^; R! j参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated1 T) ?0 I# N' x& X9 d1 x" l1 l3 t
2. gitlab-ctl reconfigure failing due to migration issue
5 n+ f" }( a3 e/ CAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
