Everything you need to know about the IPsec protocol for enhanced network security
The IPsec protocol, a fundamental pillar of modern network security, ensures the confidentiality and integrity of data exchanged on the Internet.
The IPsec protocol is one of the essential cybersecurity standards. It protects data exchanged on the Internet through encryption and authentication, ensuring that only the right people can access sensitive information.
For a company, IPsec is not just a technical brick: it is a lever for cyber governance. Whether you are RSSI, network administrator or CIO, understanding IPsec is essential to configure a VPN, compare its modes (transport/tunnel) and choose between IKEv1 and IKEv2. Indeed, properly configured, it makes it possible to reduce the risks of data leaks, to meet regulatory obligations (RGPD, NIS2, DORA) and to strengthen the trust of customers and partners.
What is the IPsec protocol?
The IPsec protocol (Internet Protocol Security) is a set of protocols standardized by theIETF (Internet Engineering Task Force) that secures network communications at the layer 3 level of the OSI model. In concrete terms, this means that it protects all IP traffic, without requiring applications to be adapted.
This transparent approach is an asset for organizations: all you need to do is configure network equipment (routers, firewalls, VPN gateways) to secure data exchanges, without touching business software.
Why is this important for your business?
- IPsec protects all of your network flows, not just your web applications.
- It reduces the risk of data interception (e.g. sensitive files, internal e-mails).
- It's a building block for a secure corporate VPN.
What is the main purpose of the IPsec protocol?
Provide a service of security complete, guaranteeing the confidentiality, integrity,authentication From the source of data and protection against replay attacks. To achieve this, it relies on a architecture modular and a set of complementary protocols that work together.
The three pillars of IPsec security
The robustness of the IPsec protocol is based on three essential security services that address the main threats to network communication.
Data confidentiality through encryption
The first service is the confidentiality. IPsec uses powerful encryption algorithms like AES, now the reference, to make information unreadable to anyone who would intercept data packets on the network (old algorithms like 3DES are still supported, but now considered obsolete). The content of the payload is thus protected, preventing any eavesdropping on traffic.
→ This encryption is crucial for protecting your company's sensitive information, whether it's trade secrets, personal data, or strategic communications. Your data then remains unreadable for any attacker.
Information integrity and source authentication
The second service guarantees the integrity of the data and the authentication of their origin. IPsec ensures that packets have not been modified while being transported over the Internet. To do this, it uses message authentication codes (MACs) based on hashing algorithms (mainly SHA-2, more rarely SHA-1 or MD5, which are currently not recommended) and shared secret keys.
→ This mechanism also verifies that packets come from the expected source, thus preventing identity theft and malicious data injections. Only authorized senders can initiate communication. Mutual authentication at both ends of the communication is a prerequisite to establish a secure connection and to be certain that the data has not been altered..
Protection against replay (Anti-Replay)
The third service is protection against replay attacks. This type of attack involves capturing legitimate packets and sending them back later to disrupt service or gain unauthorized access. IPsec assigns a unique sequence number to each packet in a session. The receiver verifies that each sequence number is unique and arrives within an expected reception window, rejecting any duplicate or too old packets.
→ This process ensures that communication is not only secure but also reliable by preventing a hacker from reusing old intercepted packets.
What are the key components of the IPsec architecture?
The IPsec architecture is composed of several protocols and mechanisms that work together to establish and maintain secure connections. Understanding these components is critical to effectively deploying and managing an IPsec VPN.
Authentication Header (AH)
The protocol Authentication Header provides data origin authentication, packet integrity, and anti-replay protection. However, it does not offer encryption and therefore does not guarantee the confidentiality of information. It protects the entire IP packet, including IP headers that don't change in transit. On the other hand, this mechanism is not compatible with NAT, as any modification of the IP header invalidates the authentication.
Encapsulating Security Payload (ESP): The most commonly used protocol.
It offers all HA services (authentication, integrity, anti-replay) and adds payload encryption (Payload) of the package. ESP is therefore the solution of choice to guarantee the confidentiality and integrity of data.
Security Association (SA)
An SA is a unidirectional security contract between two communicating entities. It defines the security settings to be used, such as encryption and authentication algorithms, encryption keys, and key lifespan. For two-way communication, two SAs are required, one for each direction of traffic.
Internet Key Exchange (IKE)
The IKE protocol (currently in version 2, IKEv2) is a hybrid protocol that automates key management and SA negotiation. It authenticates both parties of the communication and generates the encryption and authentication keys that will be used by AH and ESP. IKE dramatically simplifies IPsec deployment by dynamically managing the complex key exchange process.
These components may seem technical, but their purpose is simple: to automate end-to-end communications security. Understanding these elements is a first step towards better risk management. To go further, make a cybersecurity audit can help identify faults specific to your infrastructure.
Setting up IPsec is complex. For a CISO, the challenge is to ensure that the parameters chosen (algorithms, keys, modes) meet best practices and regulatory requirements.
With Egerie, you can model your IPsec flows, visualize the risk scenarios associated with your VPNs and verify their compliance (NIS2, DORA, ISO 27001). Do not hesitate to request a demo.
How does the IPsec protocol work in practice?
The operation of IPsec may seem complex, but in reality it is based on a fairly clear logic: before exchanging data, both parties establish a security agreement, then the communications are protected automatically.
The key negotiation and exchange process (IKE)
Before securing the traffic, both hosts (for example a VPN client and a server) must agree on the security settings and generate the secret keys. This is the role of the IKE protocol that manages authentication and key generation.
This process generally takes place in two phases:
Phase 1: setting up a secure channel
- Both parties authenticate each other, either with a shared key (PSK) or in a more robust way, with digital certificates. They then negotiate the algorithms to be used (e.g. AES encryption, SHA-256 hash) and exchange Diffie-Hellman keys to generate a common session key.
Phase 2: Defining IPsec parameters
- Once this channel is established, the two hosts negotiate the precise rules that will protect the traffic: protocol chosen (AH or ESP), encryption algorithms and network flows concerned (e.g. all traffic between two sites, or only certain applications). New session keys are then generated to secure exchanges.
Result: at the end of these steps, the connection is ready and the traffic can flow in an encrypted and authenticated manner.
The two modes of operation of IPsec: transport mode vs tunnel mode
The IPsec protocol can operate in two distinct modes, each meeting a specific network security need. The choice between transport mode and tunnel mode depends on the architecture of the network and the security objectives.
Transport mode: end-to-end protection
- Only the packet payload is encrypted and authenticated, with the IP header still visible. This mode is used to directly secure communications between two hosts (e.g. between a user station and a server). It has the advantage of a limited technical cost, but it is not the most common.
Tunnel mode: the basis of VPNs
- In this mode, the entire IP packet (header + data) is encapsulated and protected. A new IP header is added with the addresses of the security gateways (e.g. both ends of a VPN). This is the most common mode, especially to connect remote sites or to allow nomadic employees to access the company network. This operation hides the original addresses and offers a reinforced level of security.
Governance and risk management
Configuring IPsec involves numerous technical choices: algorithms, key lengths, security policies... A bad configuration can considerably reduce the expected protection.
On the governance side, IPsec is not only a technical tool: it is a brick that must be integrated into a broader risk management and compliance approach. With a platform like EGERIE, it becomes possible to:
- visualize flows protected by IPsec,
- model risk scenarios associated with VPNs,
- and ensure compliance with regulations such as DORA or NIS2.
The concrete use cases of the IPsec protocol
Thanks to its flexibility and robustness, the IPsec protocol is at the heart of many network security strategies. Its best known application is the creation of VPNs, but its uses extend well beyond that, and directly affect corporate governance and business continuity. A good strategy of management of risks associated with third parties often relies on IPsec connections to secure exchanges with partners.
IPsec VPN: the flagship solution for secure access
The most emblematic use case of the IPsec protocol is undoubtedly the virtual private network (VPN). An IPsec VPN creates an encrypted tunnel on an unsecured network like Internet, making it possible to extend a private network in a secure manner.
Remote Access VPN
This scenario allows individual users (remote workers, employees on the go) to connect securely to their company network.
- How it works : A VPN client software is installed on the user's laptop or mobile device. This client establishes an IPsec tunnel in tunnel mode with a VPN gateway located at the edge of the corporate network.
- Benefits : the user has the impression of being connected directly to the company's local network. He can access file servers, internal applications, and other resources as if he were in the office, with a high level of security. All data exchanged is protected against interception and modification.
- Governance challenge : ensure that remote accesses are controlled and in accordance with internal security policies.
Site-to-site VPN
This scenario connects several offices or subsidiaries via the Internet in a secure manner.
- How it works : security gateways (IPsec-compatible routers or firewalls) are deployed on each site. These gateways establish a permanent IPsec tunnel between them, usually in tunnel mode.
- Benefits : it is an economical alternative to expensive leased links (MPLS). All traffic between sites is automatically encrypted, creating a unified and secure wide area network (WAN). Employees at one site can communicate transparently with resources and collaborators at other sites.
- Governance challenge : control the interconnections between sites and partners, which are often critical in the context of DORA or NIS2 compliance.
Protection within the local area network (LAN)
Although IPsec is often associated with the protection of Internet traffic, it is also very effective in securing communications within a corporate network.
- How it works : in transport mode, it secures communications between critical servers or between workstations and sensitive applications.
- Benefits : This contributes to strengthening a “zero trust” approach, by adding a layer of protection within the LAN. However, IPsec is only one part of a larger strategy, which also includes identity, segmentation, and access control. Even if an attacker were able to penetrate the network perimeter, they would not be able to intercept or manipulate communications between systems protected by IPsec. This protects against internal threats and lateral movements by attackers.
- Governance challenge : limit the risks of lateral movements in the event of intrusion into the network.Example: in a hospital, some critical communications — such as between the radiological information system (RIS) and the medical records database (DMP) — can be secured via IPsec in transport mode. Other environments use TLS or SSL VPNs instead, depending on technical and regulatory constraints. This ensures that sensitive health data is encrypted and authenticated, even within the hospital network, in accordance with confidentiality requirements.
Protection of specific applications
IPsec can be configured in a very granular way to only protect traffic for certain applications or services.
- How it works : apply IPsec rules on specific ports or applications (e.g. SSH connections, access to an SQL database).
- Benefits : reduce the load while protecting the most sensitive data.
- Governance challenge: prioritizing critical resources in a risk management approach.
The implementation of these security solutions must be preceded by a compliance audit comprehensive to identify the most at risk data flows and define an appropriate protection strategy.
Governance and risk management related to IPsec uses
In each of these cases, the The main risk is not the protocol itself, but its configuration and management over time (updates, key renewal, security policies).
- A poorly configured VPN can expose the business to massive leaks.
- Poor internal segmentation can facilitate an attacker's move.
With a platform like Egerie, you can map these flows, identify the risk scenarios associated with your VPNs, and ensure that your technical choices comply with current standards and regulations.
IPsec FAQ
This section answers the most frequently asked questions about the IPsec protocol, how it works, and its place in the cybersecurity ecosystem.
What is the difference between IPsec and SSL/TLS?
IPsec and SSL/TLS (now primarily TLS) are two fundamental security protocols, but they operate at different levels of the OSI model and meet different needs.
- Level ofoperation : IPsec operates at the network layer (layer 3), allowing it to secure all IP traffic transparently for applications. SSL/TLS operates at a higher layer, between the transport layer and the application layer (often considered layer 5 or 6).
- Scope: IPsec can protect all traffic on a device or between two networks. SSL/TLS is designed to secure the communication of a specific application, most often web traffic (HTTPS), but also emails (SMTPS, IMAPS) or other services.
- Complexity: The implementation of IPsec is generally considered to be more complex due to its numerous components (IKE, AH, ESP, SA) and its modes of operation. SSL/TLS is easier to integrate directly into an application.
- VPN : Both protocols can be used to create VPNs. IPsec VPNs are traditionally heavy clients, while SSL VPNs are often accessed directly through a web browser, making them popular for thin client access to specific web applications.
Is IPsec vulnerable?
Like any technology, IPsec is not infallible, but it is considered to be a very robust and mature protocol. Its safety depends on several factors:
- Implementation : Errors in the software implementation of the protocol can create flaws. It is crucial to use products from reputable suppliers and to keep systems up to date.
- Setup : Misconfiguration is the most common source of vulnerability. Using weak encryption algorithms (like DES), short or easy to guess pre-shared keys (PSKs), or security policies that are too permissive can compromise the security of the connection. An IT security audit is essential to validate the robustness of the configuration.
- Key management : The security of the entire system is based on the protection of the keys. Poor key management can destroy the benefits of the protocol. The use of digital certificates and the IKEv2 protocol is highly recommended.
What is NAT-Traversal (NAT-T)?
NAT (Network Address Translation) is a very common technology that allows multiple devices on a private network to share a single public IP address. However, NAT changes the headers of IP packets, which can “break” the operation of IPsec, in particular the integrity checks of the AH protocol and some ESP headers.
NAT-Traversal (NAT-T) is an IPsec feature designed to address this problem. When both peers detect a NAT device between them, NAT-T encapsulates IPsec ESP packets into UDP packets (typically on port 4500). Because NAT is designed to handle UDP packets, IPsec communication can pass through the NAT device without interruption. This allows VPN clients behind a home internet box, for example, to connect to a corporate VPN server.
Why use IKEv2 over IKEv1?
IKEv2 (Internet Key Exchange version 2) is an improved and simplified version of its predecessor, IKEv1. It is highly recommended to use IKEv2 for all new IPsec VPN implementations. The benefits include:
- Simplicity and efficiency: IKEv2 reduces the number of messages needed to establish an SA, making the connection faster.
- Robustness: It integrates liveness checks to ensure that the other peer is still active.
- Mobility: IKEv2 includes the MOBIKE protocol, which allows a VPN to maintain its connection transparently even if the user changes networks (for example, by switching from Wi-Fi to 4G/5G). This is a major advantage for mobile users.
- Reliability and security: IKEv2 simplifies the exchange (fewer messages than IKEv1), fixes some design flaws in the first version, and integrates more robust configuration and error management.
