gitlab 서버 관리 일지 - 1

(포스팅 2022.05.30)

 

 

- 관련 리스트
gitlab 서버 관리 일지 - 5
gitlab 서버 관리 일지 - 4
gitlab 서버 관리 일지 - 3
gitlab 서버 관리 일지 - 2
gitlab 서버 관리 일지 - 1

 

2017회사에서는 자체적으로 gitlab 서버를 운용하고 있습니다.

서버의 관리 기록을 포스팅합니다.

 

이번 포스팅에서는 Gitlab의 상태가 점점 악화되고 있는 내용입니다.

- 목차
2020년 6월 15일 첫 이상 발생
2022년 1월 3일. 502 에러 재발함. DB 리셋함.
2022년 1월 10일. 502 에러 재발함. DB 리셋함.
2022년 1월 17일. 502 에러 네번째 발생.
2022년 1월 23일. 502 에러 다섯번째 발생
2022년 2월 16일. 502 에러 여섯번째 발생
2022년 3월 16일. 502에러 일곱번째 발생. 503 에러발생
2022년 3월 28일. GIT 동작 이상 발생

---

2020년 6월 15일 첫 이상 발생

 

gitlab web 접속시 아래의 에러가 발생하였습니다.

502 Error
Whoops, GitLab is taking too much time to respond.

 

재부팅 후 회복하였습니다.

---

2022년 1월 3일. 502 에러 재발함. DB 리셋함.

 

저번 에러와 동일합니다. gitlab web 접속시 아래의 에러가 발생하였습니다.

502 Error
Whoops, GitLab is taking too much time to respond.

점검 시작.

aaaaa@aaaaa-git:~$ gitlab-rake gitlab:check
/usr/bin/gitlab-rake error: could not load /opt/gitlab/etc/gitlab-rails/gitlab-rails-rc
Either you are not allowed to read the file, or it does not exist yet.
You can generate it with:   sudo gitlab-ctl reconfigure

수정 시도.

aaaaa@aaaaa-git:~$ sudo gitlab-ctl reconfigure
((중략))
Running handlers:
There was an error running gitlab-ctl reconfigure:

service[sidekiq] (dynamically defined) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/sidekiq ----
STDOUT: timeout: run: /opt/gitlab/service/sidekiq: (pid 4090) 1894s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/sidekiq ----
Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/sidekiq returned 1

Running handlers complete
Chef Client failed. 66 resources updated in 01 minutes 02 seconds

문제 관련하여 SO(StackOverflow) 사이트 방문

URL : https://stackoverflow.com/questions/36790965/gitlab-reconfigure-error
looks not good. Are all services online of gitlab? You can only reconfigure if all services especially the DB is on. You can check the status of all gitlab services by executing gitlab-rake gitlab:check ?

DB 리셋을 시도

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake cache:clear
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-ctl reconfigure
Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
(중략)

 

이후 gitlab 서버는 정상동작하였으나, 각 user 의 ssh 로그인이 실패하였습니다.
개발자 전체 공지를 하고, 각자 ssh key 를 다시 등록하라고 요청할 수 밖에 없었습니다.

ㅇ 카톡 공지
2022년 1월 3일 월요일
[ㅁㅁㅁ] [오후 3:03] 오늘 아침에 GITLAB 서버가 나가서 다시 셋팅했습니다.
이 때 모든 유저의 SSH Key 가 지워진 것으로 추정됩니다.
각자 key 재등록을 부탁드립니다.

 

---

2022년 1월 10일. 502 에러 재발함. DB 리셋함.

 

Gitlab 서버에서, 문제의 502에러가 일주일만에 재발하였습니다. 문제가 지속적으로 발생할 것으로 예상되어 서버 점검을 시작하였습니다.

 

처음 보는 에러 발생.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake gitlab:check
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Cannot load `Rails.application.database_configuration`:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:check => gitlab_environment => environment
(See full trace by running task with --trace)

 

디스크 용량 점검 ㅡ> df 커맨드

시스템 디스크 체크 ㅡ> touch /enforcefsck 후 재부팅

 

gitlab:check 에서 표시하는 파일에 대한 점검.

aaaaa@aaaaa-git:~$ lrwxrwxrwx 1 root root 45  5월 28  2019 /opt/gitlab/embedded/service/gitlab-rails/config/database.yml -> /var/opt/gitlab/gitlab-rails/etc/database.yml
-bash: /var/opt/gitlab/gitlab-rails/etc/database.yml: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ cat /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
cat: /opt/gitlab/embedded/service/gitlab-rails/config/database.yml: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ cat /var/opt/gitlab/gitlab-rails/etc/database.yml
cat: /var/opt/gitlab/gitlab-rails/etc/database.yml: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ touch /var/opt/gitlab/gitlab-rails/etc/database.yml
touch: '/var/opt/gitlab/gitlab-rails/etc/database.yml'를 touch할 수 없음: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ sudo touch /var/opt/gitlab/gitlab-rails/etc/database.yml
touch: '/var/opt/gitlab/gitlab-rails/etc/database.yml'를 touch할 수 없음: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ cd /var/opt/gitlab/gitlab-rails/etc/
-bash: cd: /var/opt/gitlab/gitlab-rails/etc/: 그런 파일이나 디렉터리가 없습니다
aaaaa@aaaaa-git:~$ cd /var/opt/gitlab/git
git-data/         gitaly/           gitlab-ci/        gitlab-monitor/   gitlab-rails/     gitlab-shell/     gitlab-workhorse/
aaaaa@aaaaa-git:~$ cd /var/opt/gitlab/gitlab-rails/     ****> 이것이 비어있다.
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ ls -al
합계 8
drwxr-xr-x  2 git  root 4096  1월  9 05:38 .
drwxr-xr-x 20 root root 4096  1월  3 12:26 ..

 

아무것도 해결안됨.

지난번에 했던 방법과 동일하게, DB를 리셋하고 ssh-key 를 재등록하도록 하였습니다.

 

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake cache:clear
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-ctl reconfigure
    
ㅇ 전체 공지
GITLAB 서버가 일주일 전에 502 에러가 발생했다가, 오늘 다시 발생하였습니다. (모두 월요일)
제가 아는 선에서 해결방법이 없어, gitlab 을 reconfigure 하였습니다.

접속은 됩니다만, ssh-key 를 재등록하셔야 할 것으로 예상됩니다.

 

---

2022년 1월 17일. 502 에러 네번째 발생.

 

gitlab:check 로 확인시 이상이 없었습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...
((중략))
Init script exists? ... skipped (omnibus-gitlab has no init script)             ****> 문제 아니라고 함.
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)             ****> 문제 아니라고 함.
Projects have namespace: ...
((중략))
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 12

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

두개의 사이트(첫번째, 두번째)를 참고하여 gitlab을 restart -> reconfigure 하여 문제가 해결되었습니다.

URL : https://minddong.tistory.com/47
aaaaa@aaaaa-git:~$ sudo rm /var/opt/gitlab/gitaly/gitaly.pid

URL : https://harryp.tistory.com/1300
aaaaa@aaaaa-git:~$ sudo gitlab-ctl restart
aaaaa@aaaaa-git:~$ sudo gitlab-ctl reconfigure

---

2022년 1월 23일. 502 에러 다섯번째 발생

에러의 원인을 찾지 못함.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Cannot load `Rails.application.database_configuration`:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:check => gitlab_environment => environment
(See full trace by running task with --trace)

ㅇ 부팅시 fsck 가 돌도록 하기 위해 touch /enforcefsck
aaaaa@aaaaa-git:~$ sudo touch /enforcefsck
aaaaa@aaaaa-git:~$ sudo reboot

ㅡ> 차이 없음

 

시간이 촉박하여, 기존에 사용된 해결방법을 한꺼번에 모두 진행하기로 함.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ sudo gitlab-rake cache:clear
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ gitlab-rake gitlab:check
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ sudo gitlab-ctl reconfigure
aaaaa@aaaaa-git:~$ sudo rm /var/opt/gitlab/gitaly/gitaly.pid
aaaaa@aaaaa-git:~$ sudo gitlab-ctl restart

복구완료.

---

2022년 2월 16일. 502 에러 여섯번째 발생

기존의 방법을 빠르게 반복하는 식으로 해결하였습니다.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ sudo gitlab-rake cache:clear
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ gitlab-rake gitlab:check
aaaaa@aaaaa-git:/var/opt/gitlab/gitlab-rails$ sudo gitlab-ctl reconfigure
aaaaa@aaaaa-git:~$ sudo rm /var/opt/gitlab/gitaly/gitaly.pid
aaaaa@aaaaa-git:~$ sudo gitlab-ctl restart

 

---

2022년 3월 16일. 502에러 일곱번째 발생. 503 에러발생

에러의 양상은 동일합니다.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
[sudo] aaaaa의 암호:
Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Cannot load `Rails.application.database_configuration`:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
File "/opt/gitlab/embedded/service/gitlab-rails/config/database.yml" is a symlink that does not point to a valid file
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:6:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:check => gitlab_environment => environment
(See full trace by running task with --trace)

 

기존의 방법을 모두 사용후에 에러가 503에러로 바뀌었습니다.

503 Whoops, GitLab is currently unavailable.
Try refreshing the page, or going back and attempting the action again.

이후 git push 등이 불가능해졌습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-rnd-server:~/WORK/ㅁㅁ/autobuild_ㅁㅁㅁㅁㅁㅁ$ git push
> GitLab: API is not accessible
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-rnd-server:~/WORK/ㅁㅁ/autobuild_ㅁㅁㅁㅁㅁㅁ$ git push
> GitLab: API is not accessible
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

 

gitlab 의 restart 또는 reconfigure 도 실패하기 시작하였습니다.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED: Failed to connect to internal API      ****> 문제 있음
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.
((중략))

aaaaa@aaaaa-git:~$ sudo gitlab-ctl reconfigure
((중략))
runit_service[gitaly] (gitaly::enable line 86) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[reload_log_service] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 77) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitaly/log ----
STDOUT: kill: run: /opt/gitlab/service/gitaly/log: (pid 1623) 104983s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitaly/log ----
Ran /opt/gitlab/embedded/bin/sv force-reload /opt/gitlab/service/gitaly/log returned 1

Running handlers complete
Chef Client failed. 57 resources updated in 58 seconds

 

Gitlab의 restart / recongure 를 반복하여 문제가 해결되었습니다.

---

2022년 3월 28일. GIT 동작 이상 발생

 

gitlab서버에 git pull 명령 사용시 에러가 발생하는 빈도가 높아지고 있습니다.

aaaaa@vmware-sdm660:~/WORK$ ./git_fetch.sh
Connection to 192.168.0.2 closed by remote host.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
~/WORK
~/WORK/aaaaa_mcu01 ~/WORK
Failed to read config, falling back to gitlab-shell-ruby
/opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-shell/config.yml (Errno::ENOENT)
	from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `open'
	from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `load_file'
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_config.rb:7:in `initialize'
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `new'
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `<top (required)>'

 

Gitlab 프로그램에 문제가 발생한 것이 아닌, 서버에 문제가 발생하여 Gitlab이 영향받는다는 추정을 하게 되었습니다.

서버를 점검하기 시작합니다.

 

정체를 알 수 없는 프로세스들이 다수 존재하고 있습니다.

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 6506 git       20   0 1122980 456580  21404 S   1.3  1.4  70:02.72 bundle
10664 git       20   0     220     68      0 S   1.0  0.0  37:11.24 KKveTTgaAAsecNN	****> 이거
20761 git       20   0     220     72      0 S   1.0  0.0  30:38.99 KKveTTgaAAsecNN	****> 이거
 4595 gitlab-+  20   0   44488   7864   3232 S   0.7  0.0 114:15.64 redis-server
25830 git       20   0     220     64      0 S   0.7  0.0  30:03.80 KKveTTgaAAsecNN	****> 이거
  967 lightdm   20   0  768548  51420  44176 S   0.3  0.2  27:13.26 slick-greeter
 4288 gitlab-+  20   0 2100392  22524  15896 S   0.3  0.1  15:35.06 alertmanager
 4702 gitlab-+  20   0 1710960  11812   7240 S   0.3  0.0  16:09.00 redis_exporter
24573 git       20   0 2802604  72264  16044 S   0.3  0.2   4:11.32 ruby
24574 git       20   0 2879072  72152  16044 S   0.3  0.2   4:17.26 ruby
25843 git       20   0     220     68      0 S   0.3  0.0  30:02.30 KKveTTgaAAsecNN	****> 이거
26463 git       20   0   33628   8116   3500 S   0.3  0.0   7:22.44 usr/bin/git2
27643 aaaaa     20   0   45516   4240   3428 R   0.3  0.0   0:00.03 top

 

몇가지 테스트 후 에러의 양상이 바뀌었습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Traceback (most recent call last):
        11: from /opt/gitlab/embedded/service/gitlab-shell/bin/check:8:in `<main>'
        10: from /opt/gitlab/embedded/service/gitlab-shell/bin/check:8:in `require_relative'
         9: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_net.rb:8:in `<top (required)>'
         8: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_net.rb:8:in `require_relative'
         7: from /opt/gitlab/embedded/service/gitlab-shell/lib/http_helper.rb:2:in `<top (required)>'
         6: from /opt/gitlab/embedded/service/gitlab-shell/lib/http_helper.rb:2:in `require_relative'
         5: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `<top (required)>'
         4: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `new'
         3: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_config.rb:7:in `initialize'
         2: from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `load_file'
         1: from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `open'
/opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-shell/config.yml (Errno::ENOENT)
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

aaaaa@aaaaa-git:~$ sudo gitlab-ctl reconfigure
Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab"]
((중략))
Running handlers:
There was an error running gitlab-ctl reconfigure:

execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] (gitlab::gitlab-shell line 86) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
STDOUT: error: could not open /var/opt/gitlab/.ssh/authorized_keys: Permission denied @ rb_sysopen - /var/opt/gitlab/.ssh/authorized_keys       ****> error. 이 파일에 모든 권한을 주었다. a+rwx
-r-------- 1 git git 3860  3월 27 04:57 /var/opt/gitlab/.ssh/authorized_keys
STDERR:
---- End output of /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions ----
Ran /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions returned 1

Running handlers complete
Chef Client failed. 2 resources updated in 05 seconds

 

Gitlab 의 버전을 확인하였습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-rake gitlab:env:info

System information
System:         LinuxMint 19.1
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        11.11.3
Revision:       e3eeb779d72
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            http://192.168.0.2:9999
HTTP Clone URL: http://192.168.0.2:9999/some-group/some-project.git
SSH Clone URL:  git@192.168.0.2:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        9.1.0
Repository storage paths:
- default:      /data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

 

아래의 방법으로 일시적으로 gitlab이 정상동작하였으나, 어느정도 시간이 지나니 문제가 다시 발생하였습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo chmod a+rwx /var/opt/gitlab/.ssh/authorized_keys
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-ctl reconfigure

 

gitlab:check 를 하니, gitaly에 문제가 있다는 확인이 나왔습니다.

aaaaa@aaaaa-git:~$ sudo gitlab-rake gitlab:check
[sudo] aaaaa의 암호:
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Traceback (most recent call last):
        11: from /opt/gitlab/embedded/service/gitlab-shell/bin/check:8:in `<main>'
        10: from /opt/gitlab/embedded/service/gitlab-shell/bin/check:8:in `require_relative'
         9: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_net.rb:8:in `<top (required)>'
         8: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_net.rb:8:in `require_relative'
         7: from /opt/gitlab/embedded/service/gitlab-shell/lib/http_helper.rb:2:in `<top (required)>'
         6: from /opt/gitlab/embedded/service/gitlab-shell/lib/http_helper.rb:2:in `require_relative'
         5: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `<top (required)>'
         4: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_logger.rb:118:in `new'
         3: from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_config.rb:7:in `initialize'
         2: from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `load_file'
         1: from /opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `open'
/opt/gitlab/embedded/lib/ruby/2.5.0/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-shell/config.yml (Errno::ENOENT)
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... FAIL: 14:Connect Failed		****> 이거

Checking Gitaly ... Finished

 

reconfigure 후 restart 하여 문제가 해결되었습니다.

ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo chmod a+rwx /var/opt/gitlab/.ssh/authorized_keys
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-ctl reconfigure
ㅁㅁㅁㅁㅁ@ㅁㅁㅁㅁㅁ-git:~$ sudo gitlab-ctl restart

 

다음 페이지로 이어집니다.

https://mycatdid0.tistory.com/36

gitlab 서버 관리 일지 - 2