Haproxy LBR Troubleshooting

vivek.karakandy · March 5, 2020, 12:57pm

I was not able to isolate the issue with Haproxy LBR Troubleshooting . could you please someone give some light on this . I just need the tips for haproxy.cfg file

[root@stlb01 haproxy]# haproxy -c -f haproxy.cfg
[ALERT] 064/122301 (127) : parsing [haproxy.cfg:57] : ‘timeout’ supports ‘client’, ‘server’, ‘connect’, ‘check’, ‘queue’, ‘http-keep-alive’, ‘http-request’, ‘tunnel’, ‘tarpit’, ‘client-fin’ and ‘server-fin’ (got ‘checking’)
[ALERT] 064/122301 (127) : Error(s) found in configuration file : haproxy.cfg
[ALERT] 064/122301 (127) : Fatal errors found in configuration.

[root@stlb01 haproxy]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2020-03-05 12:28:55 UTC; 27s ago
Process: 142 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=1/FAILURE)
Main PID: 142 (code=exited, status=1/FAILURE)

Mar 05 12:28:55 stlb01 haproxy-systemd-wrapper[142]: [ALERT] 064/122855 (143) : parsing [/etc/haproxy/haproxy.cfg:57] : ‘timeout’…king’)
Mar 05 12:28:55 stlb01 haproxy-systemd-wrapper[142]: [ALERT] 064/122855 (143) : Error(s) found in configuration file : /etc/hapro…xy.cfg
Mar 05 12:28:55 stlb01 haproxy-systemd-wrapper[142]: [ALERT] 064/122855 (143) : Fatal errors found in configuration.
Mar 05 12:28:55 stlb01 haproxy-systemd-wrapper[142]: haproxy-systemd-wrapper: exit, haproxy RC=1
Mar 05 12:28:55 stlb01 systemd[1]: Child 142 belongs to haproxy.service
Mar 05 12:28:55 stlb01 systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Mar 05 12:28:55 stlb01 systemd[1]: haproxy.service changed running → failed
Mar 05 12:28:55 stlb01 systemd[1]: Unit haproxy.service entered failed state.
Mar 05 12:28:55 stlb01 systemd[1]: haproxy.service failed.
Mar 05 12:28:55 stlb01 systemd[1]: haproxy.service: cgroup is empty
Hint: Some lines were ellipsized, use -l to show in full.
[root@stlb01 haproxy]# vi haproxy.cfg
[root@stlb01 haproxy]# cat haproxy.cfg
#---------------------------------------------------------------------

Example configuration for a possible web application. See the

full configuration options online.

http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout checking 10s
maxconn 3000

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------
frontend main *:5000
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js

use_backend static          if url_static
default_backend             app

#---------------------------------------------------------------------

static backend for serving up images, stylesheets and such

#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
backend app
balance roundrobin
server app1 127.0.0.1:5001 check
server app2 127.0.0.1:5002 check
server app3 127.0.0.1:5003 check
server app4 127.0.0.1:5004 check

moby04 · March 5, 2020, 1:50pm

If you check full result from systemctl status haproxy you will find information that there’s a problem with one section of haproxy configuration (it says that it’s something with timeouts but section name is more important)… look closely at this section in haproxy.cnf and I bet you’ll spot a typo.

vivek.karakandy · March 5, 2020, 3:28pm

Thank you …

yes systemctl status haproxy has the clue.
I was dealing with the Haproxy config file for the first time.

[root@stlb01 haproxy]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2020-03-05 15:16:27 UTC; 12s ago >>>>>>>
Main PID: 138 (haproxy-systemd)

santash.jena · March 5, 2020, 6:29pm

Can you please explain what the problem was?

reg · March 10, 2020, 7:02pm

please can you explain how you were able to get into haproxy. Do you ssh it or do cd? thanks

informixids · March 11, 2020, 10:40am

Hi,

It’s a tricky question, in reality they put an error in the configuration file,and the log is indicating the error, you need only to concentrate and you will find the answer.
Read carefully the error, because you have the answer in it.

NarendraSaiS · March 13, 2020, 6:22am

Correct the below timeout parameter…

akchakravarthi08 · March 13, 2020, 9:47am

the timeout parameter is correct. see the service status below and give me clues to solve the error.
loki@stlb01 ~]$ systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: inactive (dead)

Mar 13 07:31:56 stlb01 systemd[1]: Collecting haproxy.service
Mar 13 07:33:29 stlb01 systemd[1]: Collecting haproxy.service

The service is inactive. suggest me the solution.

Tej-Singh-Rana · March 13, 2020, 2:58pm

first start the haproxy service and then see error logs by systemctl status haproxy. Read error properly and move into that file. Just a small error.

yogendra · March 27, 2020, 2:16pm

Hi @Inderpreet, @peterwhite and other techies. Hope you guys are doing well.

Here are the workarounds of my ongoing “Haproxy LBR Troubleshooting” task:

Step1: Installed Haproxy service and started. Service in ruunning status.

Step2: Checked the backend app servers for load balancing and found below details in Haproxy file. I need guidance here to verify is it correct or something i need to tweak here.
HAproxy2

Step3: Checked the port of Haproxy and found it as 405. Task says it should run on default port i.e 32700 (correct me if m wrong). Please guide me in changing the port.

Step4: Tried with port 80 as per task info and found below. Not sure its correct or wrong.

Request others as well to have an eye on my issues and help in learning. Sharing is caring

Thanks
Yogendra_Beginner

peterwhite · March 29, 2020, 1:06am

Hi @yogendra,

Without a screenshot of the task description, I’d have to guess at the requirements of this task. That said, IIRC from when I last go this task, you were supposed to configure haproxy as a “reverse proxy” with the “frontend” listening on port 80. I followed the instructions found at The Four Essential Sections of an HAProxy Configuration when I was assigned this task…

Hope this helps,
Peter

aina_dipupo · March 29, 2020, 2:21pm

I just completed this task and failed… Can someone please explain why?

Here is the final haproxy.cfg

#---------------------------------------------------------------------

Example configuration for a possible web application. See the

full configuration options online.

http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#---------------------------------------------------------------------

#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
# # 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------
frontend main *:5000
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static          if url_static
default_backend             app
#---------------------------------------------------------------------

static backend for serving up images, stylesheets and such

#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
backend app
balance roundrobin
server app1 127.0.0.1:5001 check
server app2 127.0.0.1:5002 check
server app3 127.0.0.1:5003 check
server app4 127.0.0.1:5004 check

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------
frontend app_on_http_port_80 *:80
default_backend app_backend
#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
backend app_backend
balance roundrobin
server stapp01 172.16.238.10:80 check
server stapp02 172.16.238.11:80 check
server stapp03 172.16.238.12:80 check

HAProxy was also successfully started:

And listening on port 80 - confirmed with

ss -nltp

Inderpreet · April 1, 2020, 5:41pm

@aina_dipupo

What is your email address or KKE username ?

devops503 · April 19, 2020, 3:31pm

haproxy -c -f /path/to/haproxy.cfg this command help you check your config is ok or not , and tell you where is likely the problem to be.
Hope it helps !

peterwhite · April 29, 2020, 4:38am

Hi @aina_dipupo,

HAProxy may have started but that doesn’t confirm that it’s configured properly, or that all required dependencies are, for this specific task. For example, all of your app_backend hosts route to port 80 but I don’t recall those app servers ever being configured to run on port 80 on Kode Kloud.

You can easily validate these types of tasks by using curl against every service in the chain (sometimes apache + nginx + haproxy) to determine exactly where the problem is. For example, if you can’t successfully ‘curl 172.16.238.10:80’ you’ll know your haproxy configuration is invalid.

Generally speaking, I always work backwards for these types of tasks - I verify that apache is running, and fix it if it isn’t, before looking at the nginx and/or haproxy config.

Hope this helps,
Peter

swaroopcs88 · May 8, 2020, 7:46am

hi,
not sure if I follow you. I am unable to find any typo in this section?
please advise.

swaroopcs88 · May 8, 2020, 1:55pm

@peterwhite @Inderpreet @akshayyw @Salim

thanks guys for pointing out to typo hint. fixed my issue.

akshayyw · May 8, 2020, 3:10pm

What solved your issue? Did you use this command for checking syntax errors in the file?

haproxy -c -f haproxy.cfg

Aman1011 · May 8, 2020, 4:07pm

haproxy -c -f /etc/haproxy/haproxy.cfg
timeout checking 10s —> timeout check 10s
systemctl start haproxy.service

swaroopcs88 · May 8, 2020, 11:04pm

Hi Akshay,
The status of the haproxy service was refering to timeout section of client, connect and server. there was the typo of client in the configuration section. I just corrected the same.
Again restarted the haproxy services. The services was up and running.
I didn’t try below steps given by you.
Thanks
Swaroop