The VPN market is a circus. Every provider claims military-grade encryption, no-log policies, and complete anonymity. None of that tells you what actually matters: which protocol they’re using, how the key exchange works, and what happens to your traffic between your device and their server. The protocol is the machine underneath the marketing. IPSec has been around for decades and is the backbone of corporate VPNs. OpenVPN is the open-source workhorse. WireGuard is the lean newcomer that threw out everything it didn’t need. And L2TP/IPSec is the legacy option that should have been retired years ago. Here’s what’s actually happening inside the tunnel.
The TLDR
VPN protocols create encrypted tunnels between your device and a remote endpoint. IPSec operates at the network layer with two sub-protocols (AH for authentication, ESP for encryption) and two modes (transport and tunnel), negotiated through IKE (Internet Key Exchange). OpenVPN uses SSL/TLS for key exchange and runs over UDP or TCP, offering flexibility at the cost of complexity. WireGuard is the modern option — minimal codebase, fixed cryptographic choices, excellent performance, built on the Noise protocol framework. L2TP/IPSec and SSTP are legacy options with specific drawbacks. The protocol determines your actual security posture, performance, and attack resistance. Choose based on those factors, not on a VPN provider’s landing page.
The Reality
Most folks pick a VPN provider based on price, server count, or which YouTuber is shilling it this week. The protocol is an afterthought. That’s like choosing a bank based on the lobby decor without checking whether they have a vault. The protocol is the vault.
And here’s the uncomfortable truth about commercial VPN services: you’re shifting trust, not eliminating it. Without a VPN, your ISP sees your DNS queries and connection metadata. With a VPN, the VPN provider sees that instead. You’ve traded one set of eyes for another. The protocol determines whether anyone between you and the VPN server can read your traffic, but it says nothing about what the VPN provider does with it on their end.
NIST SP 800-77 Rev. 1 (Guide to IPSec VPNs) is the authoritative reference for IPSec deployments. But the landscape has shifted — WireGuard’s inclusion in the Linux kernel mainline in 2020 signaled a generational change in what “good enough” looks like for encrypted tunnels.
How It Works
IPSec — The Enterprise Standard
IPSec (Internet Protocol Security) is a protocol suite that operates at the network layer (Layer 3). It’s been around since the mid-1990s, standardized in multiple RFCs, and is the default choice for site-to-site VPNs and corporate remote access.
Two Sub-Protocols:
AH (Authentication Header): Provides authentication and integrity for IP packets — verifies the packet came from who it claims to come from and wasn’t modified in transit. But AH does not encrypt the payload. The data is readable. AH is rarely used alone because authentication without encryption isn’t enough for most use cases.
ESP (Encapsulating Security Payload): Provides encryption, authentication, and integrity. ESP is what people actually mean when they say “IPSec VPN.” It encrypts the packet payload and authenticates the entire ESP packet. In practice, ESP is used for almost everything, and AH is either unused or combined with ESP for additional header authentication.
Two Modes:
Transport Mode: Encrypts only the payload of the IP packet, leaving the original IP header intact. Used for host-to-host communication. The source and destination IPs are visible — an observer knows who’s talking to whom, just not what they’re saying.
Tunnel Mode: Encrypts the entire original IP packet and wraps it in a new IP packet with new headers. Used for site-to-site VPNs and remote access. The original source and destination are hidden inside the encrypted tunnel. This is the mode that provides actual traffic confidentiality.
IKE — The Key Exchange:
Before IPSec can encrypt anything, the two endpoints need to agree on algorithms, exchange keys, and establish a Security Association (SA). That’s what IKE does.
IKEv1 uses two phases: Phase 1 establishes a secure channel (the ISAKMP SA), Phase 2 negotiates the actual IPSec SAs for data transfer. It’s complex, chatty, and has known issues with NAT traversal.
IKEv2 (RFC 7296) is the modern replacement: fewer round trips, built-in NAT traversal, MOBIKE support for mobile devices that switch networks, and better resistance to denial-of-service. IKEv2 combined with IPSec ESP in tunnel mode is the current recommended configuration for enterprise VPNs.
The SA negotiation includes: encryption algorithm (AES-256), integrity algorithm (SHA-256/SHA-384), DH group for key exchange (Group 14 minimum, Group 20/21 for ECC), and authentication method (certificates, PSK, or EAP).
OpenVPN — The Open-Source Workhorse
OpenVPN uses SSL/TLS for key exchange and channel setup, then encapsulates encrypted traffic over UDP (preferred) or TCP (fallback). It’s been around since 2001, is open-source, and runs on essentially everything.
Strengths: Highly configurable. Runs on any port (can disguise itself as HTTPS traffic on port 443 to bypass restrictive firewalls). Mature, extensively audited. Supports both tun (Layer 3, routed) and tap (Layer 2, bridged) interfaces. Uses the OpenSSL library for cryptography — same battle-tested crypto as the web.
Weaknesses: Complexity. The configuration file for OpenVPN is verbose, and misconfiguration is common. Performance overhead from running in userspace (not in the kernel). The OpenSSL dependency means it inherits OpenSSL’s vulnerabilities — Heartbleed affected OpenVPN servers too. The codebase is large (over 100,000 lines), making it harder to audit than simpler alternatives.
Typical configuration: AES-256-GCM for data encryption, RSA or ECDSA certificates for authentication, ECDHE for key exchange with forward secrecy, HMAC-SHA256 for packet authentication. The tls-crypt directive adds an additional layer that encrypts and authenticates control channel packets, providing defense against fingerprinting and active probing.
WireGuard — The Modern Minimalist
WireGuard is what happens when someone looks at IPSec and OpenVPN and asks “what if we started over?” Created by Jason Donenfeld, WireGuard takes a radically different approach:
Fixed cryptographic choices: No cipher negotiation. WireGuard uses ChaCha20-Poly1305 for symmetric encryption, Curve25519 for key exchange, BLAKE2s for hashing, and SipHash for hashtable keys. You don’t choose algorithms. The protocol chooses them for you. This eliminates an entire class of downgrade attacks and misconfiguration.
Noise Protocol Framework: Instead of TLS or IKE, WireGuard uses the Noise protocol framework for its handshake. Specifically, it implements the Noise_IKpsk2 pattern — a one-round-trip handshake with a pre-shared symmetric key for post-quantum defense. The handshake is simpler, faster, and formally verified.
Minimal codebase: Approximately 4,000 lines of code in the Linux kernel implementation. Compare that to OpenVPN’s 100,000+ or strongSwan’s (IPSec) code footprint. Smaller codebase means fewer bugs, easier auditing, and faster security reviews.
Kernel-level performance: WireGuard runs in the Linux kernel, avoiding the context switches and buffer copies of userspace VPNs. The result is measurably better throughput and lower latency — often within a few percent of raw line speed.
Cryptokey routing: Each peer is identified by a public key and associated with a list of allowed IP addresses. It’s a routing table where the routing key is the cryptographic key. If a packet arrives encrypted with a known peer’s key and its source IP is in that peer’s allowed list, it’s accepted. Otherwise, it’s dropped. No state machine, no negotiation, no session management. Elegant and brutal.
The tradeoff: WireGuard’s minimal design means it lacks some features. There’s no built-in mechanism for dynamic IP allocation (you need an external tool), no TCP mode (problematic on networks that block UDP), and no obfuscation (it’s identifiable as WireGuard to deep packet inspection). The fixed cipher suite means if ChaCha20 or Curve25519 are ever broken, the entire protocol needs to be replaced — there’s no algorithm negotiation to fall back on. The pre-shared key option provides a hedge against quantum threats, but it’s manual key management.
L2TP/IPSec — The Legacy Option
Layer 2 Tunneling Protocol (L2TP) by itself provides no encryption. It creates the tunnel; IPSec provides the encryption around it. The combination — L2TP/IPSec — was built into Windows, macOS, and iOS for years, making it the default “easy” VPN option.
The problems: L2TP uses UDP port 500 and protocol 50, which are commonly blocked. It double-encapsulates packets (L2TP inside IPSec), adding overhead. The cryptographic agility depends entirely on the IPSec configuration underneath, and many default implementations use weak settings. There is no compelling reason to deploy new L2TP/IPSec infrastructure. Use IKEv2/IPSec or WireGuard instead.
SSTP — Microsoft’s TLS Tunnel
Secure Socket Tunneling Protocol wraps PPP traffic inside HTTPS. It uses port 443, making it excellent at bypassing firewalls — it looks like normal web traffic. But it’s proprietary to Microsoft, closed-source, and limited to Windows-centric environments. It works, but it locks you into an ecosystem.
How It Gets Exploited
Split Tunneling Risks
Split tunneling routes some traffic through the VPN and some directly to the internet. It improves performance for non-sensitive traffic but creates a security gap: a compromised machine with split tunneling can route traffic from the protected network to the internet, bypassing the VPN’s security controls entirely. MITRE ATT&CK T1090 documents proxy/tunnel-based techniques that exploit this configuration.
Weak IKE Configuration
IPSec VPNs configured with aggressive mode IKEv1 transmit the identity in cleartext and are vulnerable to offline dictionary attacks against the pre-shared key. Weak DH groups (Group 1 using 768-bit, Group 2 using 1024-bit) are breakable. The NSA’s exploitation of VPN infrastructure, discussed in the Snowden documents, likely included attacks on weak Diffie-Hellman parameters.
VPN Provider Compromise
If the VPN provider’s infrastructure is compromised, the attacker can observe all decrypted traffic passing through their servers. This has happened — HideMyAss provided logs to the FBI (despite “no-log” claims), and several free VPN services have been caught injecting ads or selling browsing data. The protocol can be perfect and the provider can still be the weak link.
Protocol Fingerprinting and Blocking
Censorship-heavy regimes actively identify and block VPN protocols. IPSec, OpenVPN, and WireGuard all have distinctive traffic patterns. Deep packet inspection can identify them even when running on non-standard ports. OpenVPN on TCP port 443 is harder to distinguish from HTTPS but not impossible with statistical analysis. WireGuard’s fixed packet sizes and handshake patterns are particularly identifiable.
What You Can Do
Choose WireGuard for new deployments where possible. It’s faster, simpler, and has a smaller attack surface than IPSec or OpenVPN. The Linux kernel, Windows, macOS, iOS, and Android all support it natively.
For enterprise/corporate VPN, use IKEv2/IPSec with modern parameters. DH Group 20 or 21 (ECC), AES-256-GCM, SHA-384, certificate-based authentication. Follow NIST SP 800-77 Rev. 1 for the full configuration guidance.
Disable split tunneling unless you have a specific, justified reason to use it. Full tunnel is more secure. If you must split, define the split explicitly — only route specific, non-sensitive subnets outside the tunnel.
Evaluate VPN providers on protocol support, jurisdiction, and audit history — not marketing. Look for providers that have undergone independent security audits, publish the results, and support WireGuard or OpenVPN with strong configurations.
If you need censorship resistance, consider pluggable transports. Tools like obfs4 (used by Tor), Shadowsocks, or V2Ray can disguise VPN traffic as innocuous HTTPS. This is beyond standard VPN protocol scope but critical in restrictive environments.