L4 스위치의 부하 분산 기능을 리눅스 서버를 통해 진행하고자 한다. 

서버의 구성은 아래와 같이 하면 될 것 같다. 

VIP : 121.254.175.70LVS1 : 121.254.175.77LVS2 : 121.254.175.78WEB1 : 121.254.175.75WEB2 : 121.254.175.74

임의 설정한 호스트네임 정보를 각 LVS 서버에 등록을 해야 한다.

/etc/hosts /etc/sysconfig/network
/proc/sys/kernel/hostname 
[Hostname]LVS Active: lvsactive.gunnm.xyzLVS Standby: lvsstandby.gunnm.xyz 

1.  LVS Active 서버에서의 설정 : 121.254.175.77

# echo ‘lvsactive.gunnm.xyz’ > /proc/sys/kernel/hostname 
# vi /etc/sysconfig/network
NETWORKING=yesHOSTNAME=lvsactive.gunnm.xyz 
# vi /etc/hosts
121.254.175.77 lvsactive.gunnm.xyz121.254.175.78 lvsstandby.gunnm.xyz 
# vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
# 하단에 추가net.ipv4.vs.secure_tcp=3net.ipv4.vs.drop_packet=0net.ipv4.vs.drop_entry=0net.ipv4.vs.am_droprate=10net.ipv4.vs.amemthresh=1024
# sysctl -p
heartbeat + ldirectord 설치
# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm# yum install heartbeat* -y
# yum -y install ipvsadm# yum -y install perl-IO-Socket-INET6# yum -y install perl-Email-Send# yum -y install perl-MailTools# yum -y install perl-Net-SSLeay# yum -y install perl-libwww-perl# yum install ldirect*
빨간색은 실패하는 설치 패키지이다. 아래의 링크를 통해 ldirectord 파일 다운로드 및 설치를 진행해야 한다. 
– 32bit# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/i686/ldirectord-3.9.5-3.1.i686.rpm
– 64bit# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/ldirectord-3.9.5-3.1.x86_64.rpm
# rpm -ivh ldirectord-3.9.5-3.1.x86_64.rpm 
HA 설정 (Standby 의 상태 체크)/usr/share/doc/heartbeat-3.0.4/ha.cf 를 복사해와도 되지만 그냥 새로 만들고 아래 내용을 입력한다. 
# vi /etc/ha.d/ha.cf
debugfile /var/log/ha-debug.loglogfile /var/log/ha.logkeepalive 2deadtime 10udpport 694ucast eth0 121.254.175.78node lvsactive.gunnm.xyznode lvsstandby.gunnm.xyzping 121.254.175.65auto_failback onrespawn hacluster /usr/lib64/heartbeat/ipfailapiauth ipfail gid=haclient uid=hacluster 
가상IP 를 설정한다. 
# vi /etc/ha.d/haresources
lvsactive.gunnm.xyz \ldirectord::ldirectord.cf \IPaddr2::121.254.175.70/32/eth0 
authkey 를 생성한다. # vi /etc/ha.d/authkeys
auth 22 sha1 gunnmlvs
root 만 읽을수 있도록 권한을 설정한다.
# chmod 600 /etc/ha.d/authkeys
-rw——-. 1 root root 23 2018-08-08 13:07 /etc/ha.d/authkeys 
 분배 설정을 진행한다.# vi /etc/ha.d/ldirectord.cf
checktimeout=10checkinterval=10autoreload=nologfile=”/var/log/ldirectord.log”quiescent=no# Web Server LVSvirtual=121.254.175.70:80real=121.254.175.75:80 gate 10real=121.254.175.74:80 gate 10service=httprequest=”gunnm.lvs”receive=”gunnm”scheduler=wlcpersistent=3600protocol=tcpchecktype=negotiate
세션 동기화를 위한 방화벽 룰셋 추가
# iptables -t nat -A OUTPUT -d 224.0.0.81 -p udp –dport 8848 -j DNAT –to 121.254.175.78:8848# iptables -t nat -A PREROUTING -s 121.254.175.78 -p udp –dport 8848 -j DNAT –to 224.0.0.81:8848# iptables -t mangle -A PREROUTING -p tcp -s 0.0.0.0/0 -d 121.254.175.70/32 –dport 80 -j MARK –set-mark 1# iptables -t mangle -A PREROUTING -p tcp -s 0.0.0.0/0 -d 121.254.175.70/32 –dport 443 -j MARK –set-mark 1 
# service iptables save # service iptables restart

2.  LVS Standby 서버에서의 설정 : 121.254.175.78

# echo ‘lvsstandby.gunnm.xyz’ > /proc/sys/kernel/hostname 
# vi /etc/sysconfig/network
NETWORKING=yesHOSTNAME=lvsstandby.gunnm.xyz 
# vi /etc/hosts
121.254.175.77 lvsactive.gunnm.xyz121.254.175.78 lvsstandby.gunnm.xyz 
# vi /etc/sysctl.conf
net.ipv4.ip_forward = 1
# 하단에 추가net.ipv4.vs.secure_tcp=3net.ipv4.vs.drop_packet=0net.ipv4.vs.drop_entry=0net.ipv4.vs.am_droprate=10net.ipv4.vs.amemthresh=1024
# sysctl -p
heartbeat + ldirectord 설치
# rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm# yum install heartbeat* -y
# yum -y install ipvsadm# yum -y install perl-IO-Socket-INET6# yum -y install perl-Email-Send# yum -y install perl-MailTools# yum -y install perl-Net-SSLeay# yum -y install perl-libwww-perl# yum install ldirect*
빨간색은 실패하는 설치 패키지이다. 아래의 링크를 통해 ldirectord 파일 다운로드 및 설치를 진행해야 한다. 
– 32bit# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/i686/ldirectord-3.9.5-3.1.i686.rpm
– 64bit# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/ldirectord-3.9.5-3.1.x86_64.rpm
# rpm -ivh ldirectord-3.9.5-3.1.x86_64.rpm 
HA 설정 (Active 의 상태 체크)/usr/share/doc/heartbeat-3.0.4/ha.cf 를 복사해와도 되지만 그냥 새로 만들고 아래 내용을 입력한다. 
# vi /etc/ha.d/ha.cf
debugfile /var/log/ha-debug.loglogfile /var/log/ha.logkeepalive 2deadtime 10udpport 694ucast eth0 121.254.175.77node lvsactive.gunnm.xyznode lvsstandby.gunnm.xyzping 121.254.175.65auto_failback onrespawn hacluster /usr/lib64/heartbeat/ipfailapiauth ipfail gid=haclient uid=hacluster 
가상IP 를 설정한다. 
# vi /etc/ha.d/haresources
lvsstandby.gunnm.xyz \ldirectord::ldirectord.cf \IPaddr2::121.254.175.70/32/eth0 
authkey 를 생성한다. # vi /etc/ha.d/authkeys
auth 22 sha1 gunnmlvs
root 만 읽을수 있도록 권한을 설정한다.
# chmod 600 /etc/ha.d/authkeys
-rw——-. 1 root root 23 2018-08-08 14:12 /etc/ha.d/authkeys
 분배 설정을 진행한다.# vi /etc/ha.d/ldirectord.cf
checktimeout=10checkinterval=10autoreload=nologfile=”/var/log/ldirectord.log”quiescent=no# Web Server LVSvirtual=121.254.175.70:80real=121.254.175.74:80 gate 10real=121.254.175.75:80 gate 10service=httprequest=”gunnm.lvs”receive=”gunnm”scheduler=wlcpersistent=3600protocol=tcpchecktype=negotiate
세션 동기화를 위한 방화벽 룰셋 추가
# iptables -t nat -A OUTPUT -d 224.0.0.81 -p udp –dport 8848 -j DNAT –to 121.254.175.77:8848# iptables -t nat -A PREROUTING -s 121.254.175.77 -p udp –dport 8848 -j DNAT –to 224.0.0.81:8848# iptables -t mangle -A PREROUTING -p tcp -s 0.0.0.0/0 -d 121.254.175.70/32 –dport 80 -j MARK –set-mark 1# iptables -t mangle -A PREROUTING -p tcp -s 0.0.0.0/0 -d 121.254.175.70/32 –dport 443 -j MARK –set-mark 1 
# service iptables save # service iptables restart

3.  LVS Active 및 Standby 서버에서 Heartbeat 구동

# service heartbeat start
Starting High-Availability services: ERROR: Cannot locate resource script \ldirectord/usr/share/heartbeat/ResourceManager: line 209: ldirectord.cf: command not foundDone.
서비스 구동이 실패한다! 실패 원인을 확인해 본다.  
 /etc/ha.d/ldirectord.cf 의 구문 띄어쓰기에 문제가 있었음이 확인됐다. (별 말도 안되는 걸로 오류가 남)
checktimeout=10checkinterval=10autoreload=nologfile=”/var/log/ldirectord.log”quiescent=no# Web Server LVSvirtual=121.254.175.70:80        real=121.254.175.75:80 gate 10        real=121.254.175.74:80 gate 10        service=http        request=”gunnm.lvs”        receive=”gunnm”        scheduler=wlc        persistent=3600        protocol=tcp        checktype=negotiate
이렇게 수정하고 나니깐 정상적으로 작동 된다. 

LVS Active 에서 # ip addr을 통해서 아이피 두 개가 활성화 된 것을 확인 할 수 있다. 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 5e:df:dd:9f:3b:e0 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.77/28 brd 121.254.175.79 scope global eth0    inet 121.254.175.70/32 brd 121.254.175.79 scope global eth0    inet6 fe80::5cdf:ddff:fe9f:3be0/64 scope link       valid_lft forever preferred_lft forever
LVS Standby 에서는 아래와 같이 확인 된다. 
# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 2a:8b:f9:ad:f0:30 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.78/28 brd 121.254.175.79 scope global eth0    inet6 fe80::288b:f9ff:fead:f030/64 scope link       valid_lft forever preferred_lft forever

4.  강제 failover 발생시 구동 상태 확인

LVS Active 에서 heartbeat 를 중지한다. 
#  service heartbeat stopStopping High-Availability services: Done.
# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 5e:df:dd:9f:3b:e0 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.77/28 brd 121.254.175.79 scope global eth0    inet6 fe80::5cdf:ddff:fe9f:3be0/64 scope link       valid_lft forever preferred_lft forever
# vi /var/log/ha.logResourceManager(default)[10507]:        2018/08/08_16:33:14 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stopAug 08 16:33:15 lvsactive.gunnm.xyz heartbeat: [10494]: info: All HA resources relinquished.Aug 08 16:33:16 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing /usr/lib64/heartbeat/ipfail process group 9816 with signal 15Aug 08 16:33:17 lvsactive.gunnm.xyz heartbeat: [9790]: WARN: 1 lost packet(s) for [lvsstandby.gunnm.xyz] [542:544]Aug 08 16:33:17 lvsactive.gunnm.xyz heartbeat: [9790]: info: No pkts missing from lvsstandby.gunnm.xyz!Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing HBFIFO process 9793 with signal 15Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing HBWRITE process 9794 with signal 15Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing HBREAD process 9795 with signal 15Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing HBWRITE process 9796 with signal 15Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: killing HBREAD process 9797 with signal 15Aug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: Core process 9797 exited. 5 remainingAug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: Core process 9794 exited. 4 remainingAug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: Core process 9795 exited. 3 remainingAug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: Core process 9793 exited. 2 remainingAug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: Core process 9796 exited. 1 remainingAug 08 16:33:18 lvsactive.gunnm.xyz heartbeat: [9790]: info: lvsactive.gunnm.xyz Heartbeat shutdown complete.

LVS Standby 에서 상태 확인 한다. 
# ip addr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 2a:8b:f9:ad:f0:30 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.78/28 brd 121.254.175.79 scope global eth0    inet 121.254.175.70/32 brd 121.254.175.79 scope global eth0    inet6 fe80::288b:f9ff:fead:f030/64 scope link       valid_lft forever preferred_lft forever
# vi /var/log/ha.logAug 08 16:33:16 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Received shutdown notice from ‘lvsactive.gunnm.xyz’.Aug 08 16:33:16 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Resources being acquired from lvsactive.gunnm.xyz.Aug 08 16:33:16 lvsstandby.gunnm.xyz heartbeat: [3108]: info: acquire all HA resources (standby).ResourceManager(default)[3135]: 2018/08/08_16:33:16 info: Acquiring resource group: lvsstandby.gunnm.xyz ldirectord::ldirectord.cf IPaddr2::121.254.175.70/32/eth0Aug 08 16:33:16 lvsstandby.gunnm.xyz heartbeat: [3109]: info: Local Resource acquisition completed.ResourceManager(default)[3135]: 2018/08/08_16:33:16 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start/usr/lib/ocf/resource.d//heartbeat/IPaddr2(IPaddr2_121.254.175.70)[3224]:       2018/08/08_16:33:17 INFO:  Resource is stoppedResourceManager(default)[3135]: 2018/08/08_16:33:17 info: Running /etc/ha.d/resource.d/IPaddr2 121.254.175.70/32/eth0 startIPaddr2(IPaddr2_121.254.175.70)[3356]:  2018/08/08_16:33:17 INFO: Adding inet address 121.254.175.70/32 with broadcast address 121.254.175.79 to device eth0IPaddr2(IPaddr2_121.254.175.70)[3356]:  2018/08/08_16:33:17 INFO: Bringing device eth0 up

Active 다운시 자동으로 Standby 가 역할 전담하는 것을 확인했다. 
다시 Active 활성화시의 상태를 확인한다. 
Active log 
Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10859]: info: Pacemaker support: falseAug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10859]: WARN: Logging daemon is disabled –enabling logging daemon is recommendedAug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10859]: info: **************************Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10859]: info: Configuration validated. Starting heartbeat 3.0.4Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: heartbeat: version 3.0.4Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: Heartbeat generation: 1533705889Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: bound send socket to device: eth0Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: set SO_REUSEPORT(w)Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: bound receive socket to device: eth0Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: set SO_REUSEPORT(w)Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ucast: started on port 694 interface eth0 to 121.254.175.78Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: glib: ping heartbeat started.Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: G_main_add_TriggerHandler: Added signal manual handlerAug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: G_main_add_TriggerHandler: Added signal manual handlerAug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: G_main_add_SignalHandler: Added signal handler for signal 17Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: Local status now set to: ‘up’Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: Link 121.254.175.65:121.254.175.65 up.Aug 08 16:37:54 lvsactive.gunnm.xyz heartbeat: [10860]: info: Status update for node 121.254.175.65: status pingAug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10860]: info: Link lvsstandby.gunnm.xyz:eth0 up.Aug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10860]: info: Status update for node lvsstandby.gunnm.xyz: status activeharc(default)[10869]:   2018/08/08_16:37:56 info: Running /etc/ha.d//rc.d/status statusAug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10860]: info: Comm_now_up(): updating status to activeAug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10860]: info: Local status now set to: ‘active’Aug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10860]: info: Starting child client “/usr/lib64/heartbeat/ipfail” (496,495)Aug 08 16:37:56 lvsactive.gunnm.xyz heartbeat: [10886]: info: Starting “/usr/lib64/heartbeat/ipfail” as uid 496  gid 495 (pid 10886)Aug 08 16:37:57 lvsactive.gunnm.xyz heartbeat: [10860]: info: remote resource transition completed.Aug 08 16:37:57 lvsactive.gunnm.xyz heartbeat: [10860]: info: remote resource transition completed.Aug 08 16:37:57 lvsactive.gunnm.xyz heartbeat: [10860]: info: Local Resource acquisition completed. (none)Aug 08 16:37:57 lvsactive.gunnm.xyz heartbeat: [10860]: info: lvsstandby.gunnm.xyz wants to go standby [foreign]Aug 08 16:37:58 lvsactive.gunnm.xyz heartbeat: [10860]: info: standby: acquire [foreign] resources from lvsstandby.gunnm.xyzAug 08 16:37:58 lvsactive.gunnm.xyz heartbeat: [10890]: info: acquire local HA resources (standby).ResourceManager(default)[10903]:        2018/08/08_16:37:58 info: Acquiring resource group: lvsactive.gunnm.xyz ldirectord::ldirectord.cf IPaddr2::121.254.175.70/32/eth0ResourceManager(default)[10903]:        2018/08/08_16:37:58 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start/usr/lib/ocf/resource.d//heartbeat/IPaddr2(IPaddr2_121.254.175.70)[10967]:      2018/08/08_16:37:59 INFO:  Resource is stoppedResourceManager(default)[10903]:        2018/08/08_16:37:59 info: Running /etc/ha.d/resource.d/IPaddr2 121.254.175.70/32/eth0 startIPaddr2(IPaddr2_121.254.175.70)[11098]: 2018/08/08_16:37:59 INFO: Adding inet address 121.254.175.70/32 with broadcast address 121.254.175.79 to device eth0IPaddr2(IPaddr2_121.254.175.70)[11098]: 2018/08/08_16:37:59 INFO: Bringing device eth0 upIPaddr2(IPaddr2_121.254.175.70)[11098]: 2018/08/08_16:37:59 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-121.254.175.70 eth0 121.254.175.70 auto not_used not_used/usr/lib/ocf/resource.d//heartbeat/IPaddr2(IPaddr2_121.254.175.70)[11072]:      2018/08/08_16:37:59 INFO:  SuccessAug 08 16:37:59 lvsactive.gunnm.xyz heartbeat: [10890]: info: local HA resource acquisition completed (standby).Aug 08 16:37:59 lvsactive.gunnm.xyz heartbeat: [10860]: info: Standby resource acquisition done [foreign].Aug 08 16:37:59 lvsactive.gunnm.xyz heartbeat: [10860]: info: Initial resource acquisition complete (auto_failback)Aug 08 16:38:00 lvsactive.gunnm.xyz heartbeat: [10860]: info: remote resource transition completed.Aug 08 16:38:06 lvsactive.gunnm.xyz ipfail: [10886]: info: Telling other node that we have more visible ping nodes.Aug 08 16:38:12 lvsactive.gunnm.xyz heartbeat: [10860]: info: lvsstandby.gunnm.xyz wants to go standby [all]Aug 08 16:38:13 lvsactive.gunnm.xyz heartbeat: [10860]: info: standby: acquire [all] resources from lvsstandby.gunnm.xyzAug 08 16:38:13 lvsactive.gunnm.xyz heartbeat: [11183]: info: acquire all HA resources (standby).ResourceManager(default)[11196]:        2018/08/08_16:38:13 info: Acquiring resource group: lvsactive.gunnm.xyz ldirectord::ldirectord.cf IPaddr2::121.254.175.70/32/eth0ResourceManager(default)[11196]:        2018/08/08_16:38:14 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf start/usr/lib/ocf/resource.d//heartbeat/IPaddr2(IPaddr2_121.254.175.70)[11254]:      2018/08/08_16:38:14 INFO:  Running OKAug 08 16:38:14 lvsactive.gunnm.xyz heartbeat: [11183]: info: all HA resource acquisition completed (standby).Aug 08 16:38:14 lvsactive.gunnm.xyz heartbeat: [10860]: info: Standby resource acquisition done [all].Aug 08 16:38:15 lvsactive.gunnm.xyz heartbeat: [10860]: info: remote resource transition completed.
Standby log
Aug 08 16:37:55 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Heartbeat restart on node lvsactive.gunnm.xyzAug 08 16:37:55 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Link lvsactive.gunnm.xyz:eth0 up.Aug 08 16:37:55 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Status update for node lvsactive.gunnm.xyz: status initAug 08 16:37:55 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Status update for node lvsactive.gunnm.xyz: status upharc(default)[3761]:    2018/08/08_16:37:55 info: Running /etc/ha.d//rc.d/status statusharc(default)[3777]:    2018/08/08_16:37:55 info: Running /etc/ha.d//rc.d/status statusAug 08 16:37:57 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Status update for node lvsactive.gunnm.xyz: status activeharc(default)[3793]:    2018/08/08_16:37:57 info: Running /etc/ha.d//rc.d/status statusAug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [1415]: info: remote resource transition completed.Aug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [1415]: info: lvsstandby.gunnm.xyz wants to go standby [foreign]Aug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [1415]: info: standby: lvsactive.gunnm.xyz can take our foreign resourcesAug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [3809]: info: give up foreign HA resources (standby).Aug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [3809]: info: foreign HA resource release completed (standby).Aug 08 16:37:58 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Local standby process completed [foreign].Aug 08 16:38:00 lvsstandby.gunnm.xyz heartbeat: [1415]: WARN: 1 lost packet(s) for [lvsactive.gunnm.xyz] [12:14]Aug 08 16:38:00 lvsstandby.gunnm.xyz heartbeat: [1415]: info: remote resource transition completed.Aug 08 16:38:00 lvsstandby.gunnm.xyz heartbeat: [1415]: info: No pkts missing from lvsactive.gunnm.xyz!Aug 08 16:38:00 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Other node completed standby takeover of foreign resources.Aug 08 16:38:12 lvsstandby.gunnm.xyz heartbeat: [1415]: info: lvsstandby.gunnm.xyz wants to go standby [all]Aug 08 16:38:13 lvsstandby.gunnm.xyz heartbeat: [1415]: info: standby: lvsactive.gunnm.xyz can take our all resourcesAug 08 16:38:13 lvsstandby.gunnm.xyz heartbeat: [3849]: info: give up all HA resources (standby).ResourceManager(default)[3862]: 2018/08/08_16:38:13 info: Releasing resource group: lvsstandby.gunnm.xyz ldirectord::ldirectord.cf IPaddr2::121.254.175.70/32/eth0ResourceManager(default)[3862]: 2018/08/08_16:38:13 info: Running /etc/ha.d/resource.d/IPaddr2 121.254.175.70/32/eth0 stopIPaddr2(IPaddr2_121.254.175.70)[3923]:  2018/08/08_16:38:13 INFO: IP status = ok, IP_CIP=/usr/lib/ocf/resource.d//heartbeat/IPaddr2(IPaddr2_121.254.175.70)[3897]:       2018/08/08_16:38:13 INFO:  SuccessResourceManager(default)[3862]: 2018/08/08_16:38:13 info: Running /etc/ha.d/resource.d/ldirectord ldirectord.cf stopAug 08 16:38:13 lvsstandby.gunnm.xyz heartbeat: [3849]: info: all HA resource release completed (standby).Aug 08 16:38:13 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Local standby process completed [all].Aug 08 16:38:15 lvsstandby.gunnm.xyz heartbeat: [1415]: WARN: 1 lost packet(s) for [lvsactive.gunnm.xyz] [26:28]Aug 08 16:38:15 lvsstandby.gunnm.xyz heartbeat: [1415]: info: remote resource transition completed.Aug 08 16:38:15 lvsstandby.gunnm.xyz heartbeat: [1415]: info: No pkts missing from lvsactive.gunnm.xyz!Aug 08 16:38:15 lvsstandby.gunnm.xyz heartbeat: [1415]: info: Other node completed standby takeover of all resources.

Active 와 Standby 가 알아서 역할을 잘 설정한다. 

5.  실시간 모니터링 및 ldirectord 설정 변경시 적용 파일 생성

# vi /root/lvs.sh
#!/bin/shwatch -n1 -t “echo ; ip addr ; echo ; ipvsadm -Ln ; ipvsadm -Ln –rate ; ipvsadm -Lnc”


# vi /root/ldirect.sh (/etc/ha.d/ldirectord.cf 파일 내용 수정 후, reload 하는 파일)
#!/bin/sh/etc/ha.d/resource.d/ldirectord /etc/ha.d/ldirectord.cf reload 

Active 에서# sh lvs.sh 입력시 접속 세션 확인 가능 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 5e:df:dd:9f:3b:e0 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.77/28 brd 121.254.175.79 scope global eth0    inet 121.254.175.70/32 brd 121.254.175.79 scope global eth0    inet6 fe80::5cdf:ddff:fe9f:3be0/64 scope link       valid_lft forever preferred_lft forever
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  121.254.175.70:80 wlc persistent 3600  -> 121.254.175.74:80            Route   10     0          1  -> 121.254.175.75:80            Route   10     5          2IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port                 CPS    InPPS   OutPPS    InBPS   OutBPS  -> RemoteAddress:PortTCP  121.254.175.70:80                   0        1        0       85        0  -> 121.254.175.74:80                   0        0        0       38        0  -> 121.254.175.75:80                   0        0        0       47        0IPVS connection entriespro expire state       source             virtual            destinationTCP 48:09  NONE        92.31.1.6:0        121.254.175.70:80  121.254.175.75:80TCP 14:37  ESTABLISHED 222.231.27.246:6060 121.254.175.70:80  121.254.175.75:80TCP 05:11  ESTABLISHED 222.231.27.246:6005 121.254.175.70:80  121.254.175.75:80TCP 14:36  ESTABLISHED 222.231.27.246:6064 121.254.175.70:80  121.254.175.75:80TCP 01:08  FIN_WAIT    222.231.27.246:6061 121.254.175.70:80  121.254.175.75:80TCP 01:36  FIN_WAIT    222.231.27.246:6062 121.254.175.70:80  121.254.175.75:80TCP 05:11  ESTABLISHED 222.231.27.246:6020 121.254.175.70:80  121.254.175.75:80TCP 00:00  CLOSE       118.69.170.35:60980 121.254.175.70:80  121.254.175.74:80TCP 14:47  ESTABLISHED 121.254.171.226:52493 121.254.175.70:80  121.254.175.75:80TCP 59:49  NONE        118.69.170.35:0    121.254.175.70:80  121.254.175.74:80TCP 59:17  NONE        121.254.171.226:0  121.254.175.70:80  121.254.175.75:80TCP 59:36  NONE        222.231.27.246:0   121.254.175.70:80  121.254.175.75:80


Standby 에서는 아무것도 연결이 되어 있지 않다. 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 2a:8b:f9:ad:f0:30 brd ff:ff:ff:ff:ff:ff    inet 121.254.175.78/28 brd 121.254.175.79 scope global eth0    inet 121.254.175.70/32 brd 121.254.175.79 scope global eth0    inet6 fe80::288b:f9ff:fead:f030/64 scope link       valid_lft forever preferred_lft forever
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  121.254.175.70:80 wlc persistent 3600  -> 121.254.175.75:80            Route   10     3          0IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port                 CPS    InPPS   OutPPS    InBPS   OutBPS  -> RemoteAddress:PortTCP  121.254.175.70:80                   0        0        0        0        0  -> 121.254.175.75:80                   0        0        0        0        0IPVS connection entriespro expire state       source             virtual            destinationTCP 11:27  ESTABLISHED 222.231.27.246:5992 121.254.175.70:80  121.254.175.75:80TCP 56:27  NONE        222.231.27.246:0   121.254.175.70:80  121.254.175.75:80TCP 11:27  ESTABLISHED 222.231.27.246:5991 121.254.175.70:80  121.254.175.75:80TCP 11:27  ESTABLISHED 222.231.27.246:5993 121.254.175.70:80  121.254.175.75:80

이번 테스트는 동일 아이피 대역의 서버들끼로 진행을 했으니깐 간단하게 설정이 되었으나, 대역이 다른 서버들간의 구성시에는 라우트 추가가 진행되어야 한다. 

출처: https://gunnm.tistory.com/category/리눅스 공부 [독까의 이야기]