|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
: F9 r+ Q" L6 o4 m登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。3 R( M. \, E% N$ s/ b
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
* K4 x I) @% @' p接着设置语言和本地化设置：
$ l' v9 e. Z% n7 m& ?) l& @export LC_ALL=en_US.UTF-8
* z6 F$ d5 @! z, u' m" wexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
! m/ n( g' e# u6 q但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
3 O4 B8 T9 Q- j4 M于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：4 j3 v J0 _+ A. L
Recipe: gitlab::database_migrations
6 Y! g" l/ V' C) _ * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)! V& M6 D+ \; S, H; ]
* rails_migration[gitlab-rails] action run
) {9 r9 b4 v, f2 Y( L0 d) x * bash[migrate gitlab-rails database] action run
# ~/ K& c, j. A" a7 x% I& q 0 Q' y; L; E5 I9 ]
================================================================================# N/ `, b6 z4 M2 ^% O1 b1 I
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
; ]+ ?* b, @$ e4 I# H% G ================================================================================
' b% c# C( h$ X% i+ ]( ~$ @, e
3 Q$ ~* |% M+ k6 x4 x! V) h; C Mixlib::ShellOut::ShellCommandFailed
7 y" L4 Q/ H# U& Q9 L2 w ------------------------------------+ W; P( A( | o3 G
Command execution failed. STDOUT/STDERR suppressed for sensitive resource; ~0 ~1 l4 U4 S) T- z
5 e! e& U4 D; ]6 h Resource Declaration:1 `# M D3 u+ N$ D. r$ I
---------------------
5 ?2 M: L; c7 \ suppressed sensitive resource output
5 y4 n( _2 N* k. F+ y) _7 I% S
' x- F7 O9 h: R6 B) V Compiled Resource:9 A% V: X- L2 r
------------------
9 @7 L8 H! H a3 ]1 ?' n suppressed sensitive resource output, h) R8 {" Y4 x" Q! z+ i
}2 g+ @7 b) |6 a
System Info:- w2 x5 P) C% J* N8 u5 G
------------3 U; ]+ _! V0 r1 ^" c; T
chef_version=15.17.4
$ `2 R: E* J! A( j. v1 q1 i platform=ubuntu
' X$ v+ T$ G& j0 i o platform_version=20.04
0 A, ]1 r% M' @6 X, U2 Q; E6 _4 i ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
6 {& k' W0 C6 P1 c- M" k program_name=/opt/gitlab/embedded/bin/chef-client7 K! D& m$ B5 p2 C
executable=/opt/gitlab/embedded/bin/chef-client0 r. C% H2 C3 {& @2 _1 {
; c* E8 M% ^$ x: x
) ?! a# o+ `) F( I* p
================================================================================
1 ?1 ^2 g5 h& K- o6 ] Error executing action `run` on resource 'rails_migration[gitlab-rails]'
4 H3 T, e. m! K: V J ================================================================================$ w) O+ d( B+ z$ f
- Q8 M4 E# z, P; n- o
Mixlib::ShellOut::ShellCommandFailed% q+ p, _! J1 z7 ]1 K. @ Y
------------------------------------; S5 r- n9 p, Y9 c# R4 X' i; Z$ W
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource/ i$ ]/ U1 X3 ?: Y$ z8 }
- S3 b6 M/ `! ]) v" |! ]- q Resource Declaration:, x( d% V# L, m3 k& ~# |. P
---------------------4 @% k( V- K6 |, v9 _
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb" n8 ~- `7 n- q. F- U( E1 J
9 U! b6 s6 s) s1 h
51: rails_migration "gitlab-rails" do
I# }2 h) g8 u4 n1 a 52: rake_task 'gitlab:db:configure': n. D( O: q' @8 H3 H# S" M! ?
53: logfile_prefix 'gitlab-rails-db-migrate'& n; H# R: O1 K4 a6 V; ^
54: helper migration_helper
; S6 _3 D/ y. i/ K 55: , F( K( k4 N) h3 \% Y/ f
56: environment env_variables* M, n0 S) M6 z" L) W
57: dependent_services dependent_services. G# y1 r0 |4 ]1 z
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
' Y" x( ]/ v7 x) ^; F 59: notifies :run, "ruby_block[check remote PG version]", :immediately/ o, c. T" A2 j7 E) D8 d
60:
7 T* B, U% v5 X( G 61: only_if { migration_helper.attributes_node['auto_migrate'] }
! z$ e4 C r+ v" a5 f2 |5 X 62: end- l3 @1 [9 ?" O: Y8 C
, b% O5 e3 l% P4 D; \0 r! V6 J
Compiled Resource:& e; W( _/ p3 L4 v' m
------------------5 _8 S/ K* l0 C% Y9 [
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
% b+ ^$ \0 K( n( Q3 H % m. f5 z0 f8 }4 j
rails_migration("gitlab-rails") do9 \$ a/ J: K6 u$ Z7 t3 i; ?
action [:run]
3 q: l7 I0 y$ w5 Q default_guard_interpreter :default
4 b0 G' C# z8 Z. v declared_type :rails_migration
8 P4 ]2 p; p7 {3 m/ _+ H$ S cookbook_name "gitlab"
^. G, M1 ~/ p# J7 X& G recipe_name "database_migrations"
# H, T6 G& y T e7 I8 P6 ]7 e rake_task "gitlab:db:configure"% H; F# J0 p" Z5 | s
logfile_prefix "gitlab-rails-db-migrate"
' ~- ~5 U0 G* N$ y+ o$ B7 E, |* T helper "*sensitive value suppressed*"
/ [) V# a9 @+ B @6 A environment "*sensitive value suppressed*"
* o. J2 m- s% n8 J& e, e8 e5 ? dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
' ?# W8 ~: o3 O" e3 I- s' e7 i only_if { #code block }* q' J1 V5 X v/ M: k
end N7 G$ a! B& q1 j4 H4 I- \
4 ~( r+ Q( f x. r" Z0 t" y- b System Info:; v# r5 h' X) _6 \/ I( N- S5 ], T" }. d
------------* a3 v6 H2 L8 ?2 @6 H- T
chef_version=15.17.4% c$ v# l: m* c+ [# c& N0 e
platform=ubuntu
$ C- i; |: Z( a( {# ^& y6 J" t platform_version=20.04
" X) t: b9 ~: w, g ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]. R) M& P& m3 f) U' z1 {9 f
program_name=/opt/gitlab/embedded/bin/chef-client7 }2 [5 Z" ^+ U+ [+ o9 R4 V
executable=/opt/gitlab/embedded/bin/chef-client+ G4 j4 r. j2 h! G, W, Y( J- E
9 W" Q5 K0 Y( u) M `! Y
Running handlers:
5 W. u i/ H, eThere was an error running gitlab-ctl reconfigure:
- s9 s @9 D) @. Z1 A3 zrails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
: E o7 ~+ \1 qRunning handlers complete' X. a: f# \; @' k- d% {( x/ M
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：" p! u9 p+ O/ g3 B0 A8 ]7 k" s
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；4 ^' a" R1 T l8 Z7 \
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
% `1 M }& Q6 h& I! d3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
# f2 @5 _! @0 D! _ l3 y" I; M- |4. 运行 gitlab-ctl reconfigure;0 o l6 Q9 C0 d5 a: Z
5. 重启Gitlab：gitlab-ctl restart。5 }" K" j8 s: S# u: w
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
% ]- Z& s4 E4 j参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated2 ^& s H- H" L& t
2. gitlab-ctl reconfigure failing due to migration issue
; S3 ]+ @; Y* qAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
