Linux Kernel Hardening via sysctl.conf. Protection Against SYN Flood Attacks and TCP Tuning

Introduction

In this tutorial, you will harden the Linux kernel using sysctl.conf, protect your server against SYN flood attacks, and optimize TCP stack behavior. Kernel-level tuning improves resilience against network-based denial-of-service (DoS) attacks and enhances overall TCP performance. You will also simulate a SYN flood attack in a controlled environment and verify that your protection mechanisms work correctly.

Prerequisites

Target Audience: Intermediate system administrators
Estimated Time: 45–60 minutes

System Requirements

Operating System: Ubuntu 24.04 LTS (tested); compatible with Debian 12/13 and other modern Linux distributions
Kernel version: 5.x or later
Minimum 1 GB RAM
Root or sudo privileges
Network access between test machines

Required Tools

Install the following utilities:

sudo apt update

sudo apt install -y hping3 net-tools procps iproute2

Verify installation: hping3 --version
Expected output: version information confirming successful installation.

Tip

Perform SYN flood simulation only in a controlled lab environment. Never test against systems you do not own or have permission to test.

Step 1: Back Up Current sysctl Configuration

Before making changes, create a backup.

sudo cp /etc/sysctl.conf /etc/sysctl.conf.backup

Result: A backup file /etc/sysctl.conf.backup is created.

Step 2: Configure Kernel Parameters for SYN Flood Protection

Open the configuration file:

sudo nano /etc/sysctl.conf

Add or modify the following parameters.

Enable SYN Cookies

net.ipv4.tcp_syncookies = 1

Purpose: Enables SYN cookies when the SYN backlog queue overflows.
Values:

0 – Disabled
1 – Enabled when SYN backlog overflow occurs (recommended)
2 – Always enabled (rarely used)

Why: SYN cookies prevent memory exhaustion by avoiding allocation of resources until handshake completion.

Tip

In modern distributions this option is enabled by default. You can check it with the following command: sysctl -a | grep net.ipv4.tcp_syncookies

Increase SYN Backlog Queue

net.ipv4.tcp_max_syn_backlog = 4096

Purpose: Maximum number of queued connection requests awaiting acknowledgment.
Why: Larger backlog allows more half-open connections before kernel starts dropping them.

Reduce SYN-ACK Retries

net.ipv4.tcp_synack_retries = 3

Purpose: Number of retransmissions for SYN-ACK packets.
Why: Lowering this reduces time spent waiting on malicious half-open connections.

Enable TCP Time-Wait Reuse

net.ipv4.tcp_tw_reuse = 1

Purpose: Allows reuse of TIME-WAIT sockets for new outgoing connections.
Values:

0 – Disabled
1 – Enabled

Why: Reduces socket exhaustion under heavy load.

Reduce FIN Timeout

net.ipv4.tcp_fin_timeout = 15

Purpose: Time a socket remains in FIN-WAIT-2 state.
Why: Reduces resource usage from stale connections.

Enable Reverse Path Filtering

net.ipv4.conf.all.rp_filter = 1

Purpose: Validates source IP addresses against routing table.
Values:

0 – Disabled
1 – Strict mode
2 – Loose mode

Why: Helps mitigate IP spoofing attacks.

Increase Maximum Open Files

fs.file-max = 2097152

Purpose: Maximum number of file handles system-wide.
Why: Prevents file descriptor exhaustion during high connection load.

Step 3: Apply sysctl Changes

Apply changes without rebooting:

sudo sysctl -p

Expected output: each parameter displayed with its new value.
Verify specific parameter: sysctl net.ipv4.tcp_syncookies
Expected output: net.ipv4.tcp_syncookies = 1

Result: Kernel parameters are active.

Step 4: Simulate a SYN Flood Attack

Info

Important: Perform this only in a lab environment.

From a separate test machine, run:

sudo hping3 -S --flood -V <YOUR_SERVER_IP> -p 80

Explanation:

-S sends SYN packets
--flood sends packets as fast as possible
-p 80 targets HTTP port

Result: High volume of SYN packets sent to server.

Step 5: Monitor Server During Attack

On the target server, monitor connections:

netstat -ant | grep SYN_RECV | wc -l

Result: Displays number of half-open connections.

Check SYN cookies counter:

nstat -az | grep Syncookies

You should observe increasing counters such as TcpExtSyncookiesSent. This confirms SYN cookies are being used.

Step 6: Compare Behavior Before and After Hardening

If you test without hardening:

SYN_RECV count grows rapidly
Server may become unresponsive
Connection queue overflows

After hardening:

SYN cookies activate
Fewer dropped connections
Server remains responsive

Install a web server on the target server:

sudo apt install nginx

To test responsiveness:

curl http://localhost

Result: Server responds normally even during simulated attack.

Step 7: TCP Performance Tuning (Optional Advanced Optimization)

Add additional tuning parameters to the sysctl config. Run nano /etc/sysctl.conf and add:

net.core.somaxconn = 8192

net.core.netdev_max_backlog = 5000

net.ipv4.tcp_max_tw_buckets = 2000000

Explanation:

net.core.somaxconn – maximum listen queue length for applications.
net.core.netdev_max_backlog – maximum packets queued on input side before processing.
net.ipv4.tcp_max_tw_buckets – maximum TIME-WAIT sockets. This option is relevant for servers with high load and sufficient RAM.

These improve performance under heavy traffic.

Apply changes:

sudo sysctl -p

Verification Checklist

sysctl -p runs without errors
tcp_syncookies equals 1
SyncookiesSent counter increases during attack
Server remains responsive
No kernel warnings in logs: dmesg | grep -i syn

Reverting Changes

Restore original configuration:

sudo cp /etc/sysctl.conf.backup /etc/sysctl.conf

sudo sysctl -p

Result: System returns to original kernel parameters.

Optionally remove test tools:

sudo apt remove hping3 -y

Troubleshooting

Syncookies Not Increasing

Check: sysctl net.ipv4.tcp_syncookies
If value is 0, reapply configuration.

Server Still Drops Connections

Increase backlog: net.ipv4.tcp_max_syn_backlog = 8192
Then apply changes:

sudo sysctl -p

Conclusion

You hardened the Linux kernel using sysctl.conf, enabled SYN flood protection mechanisms, optimized TCP stack behavior, and verified protections through controlled attack simulation. These changes improve resilience against SYN flood attacks and increase overall network stability.