12. 10. 2023 Reinhold Trocker Log Management, Log-SIEM

stunnel TCP keepalive Settings Preventing Firewall from Blocking Log Traffic

Infrastructure Scenario

An image says more than 1000 words 😉

Basically, the log source continuously sends log messages encrypted via TLS to the NetEye server. ¹

TLS is handled by stunnel and then content is internally forwarded unencrypted to an Elastic Agent Integration “Custom TCP Logs” inside the NetEye server.²

Cause: Logs lost due to firewall settings

In this example, a firewall in the communication path was configured to silently drop TCP connections after 10 minutes if there was no traffic.

Solution: Don’t let the TCP connection be idle

Telling stunnel to do TCP keepalives solved the issue. Please see the following stunnel configuration as an example³:

# cat /etc/stunnel/stunnel.conf
cert = /etc/stunnel/tls/certs/cert.crt
key = /etc/stunnel/tls/private/key.key

[logsource1]
accept = 5020
connect = 5021
TIMEOUTclose = 0
socket = l:SO_KEEPALIVE=1
socket = r:SO_KEEPALIVE=1
socket = a:TCP_KEEPINTVL=10
socket = a:TCP_KEEPIDLE=15

SO_KEEPALIVE is set to 1 for stunnel input and output, so keepalive is used.
SO_KEEPIDLE=15 means that stunnel will send keepalive packets to the client every 15 seconds
SO_KEEPINTVL=10 means that if keepalive packets fail for some reason, stunnel will retry sending keepalive packets after 10 seconds

The logging client in this case was a cloud service where it was not possible to configure keepalive behavior. ↩︎
At the time of implementation, the integration “Custom TCP Logs” used was not able to process transport encryption via TLS. This feature has been introduced with version 1.3.0 of the integration, see Custom TCP Logs | Documentation (elastic.co) Nevertheless, the integration’s tcp_keep_alive setting will use the default OS settings for keepalive. If you don’t want to set the general NetEye default OS settings, stunnel is still the best choice. ↩︎
This blog is not the full stunnel documentation, it only refers to the socket settings; see the last 4 lines of the stunnel configuration example shown. ↩︎