from Wikipedia, the free encyclopedia
IPsec in the TCP / IP protocol stack :
application HTTP IMAP SMTP DNS ...
transport TCP UDP
Internet IPsec
Network access Ethernet Token
FDDI ...

Internet Protocol Security ( IPsec ) is a protocol suite that is intended to enable secure communication over potentially insecure IP networks such as the Internet .

IPsec works directly on the network layer (" Internet Layer" , corresponds to OSI Layer 3) of the DoD model and is a further development of the IP protocols. The goal is to provide encryption-based security at the network level. IPsec offers this possibility through the connectionless integrity as well as the access control and authentication of the data. In addition, IPsec guarantees the confidentiality and authenticity of the packet sequence through encryption.

Since data packets are forwarded from one computer to the next on the Internet, every computer can read and even change the content of a data packet on the way. A computer can also send data packets on behalf of another computer by entering its address as the "sender" ( IP spoofing ). IPsec should make it possible to meet the protection goals of confidentiality , authenticity and integrity in such an IP network . Various mechanisms are used for this, such as the encryption of individual IP packets and the insertion of an additional packet header with a message authentication code . IPsec can be used to set up virtual private networks (VPN) or to protect against replay attacks .

The Internet Engineering Task Force proposes the architecture of IPsec as the standard in RFC 2401 and the more recent RFC 4301 . These RFCs refer to the RFCs listed below, which describe essential parts of IPsec: the Authentication Header (AH) , Encapsulated Security Payload (ESP) and Internet Key Exchange (IKE) protocols for exchanging keys.

Connection establishment


IPsec manages connections and can guarantee both encryption and data integrity upon request. It uses one of two modes for this: the transport mode establishes point-to-point communication between two end points, while the tunnel mode connects two networks via two routers. Both modes are quite similar in terms of the security associations to be created . The following illustration only considers the transport mode.

If an IP packet is to be sent, two local databases are used:

SPD (security policy database)
In the SPD, for example, it is stored how the connection between the communication endpoints, which are identified by their IP addresses, is to be secured. AH, ESP or both are then used as security procedures. IKE is mostly used to generate the keys. Compared to the SAD (see below), the SPD is more of a static nature, as an entry in the SPD is "stateless".
SAD (security association database)
SAs between two hosts
Security Associations (SA) are administered in the SAD . These have a state (English stateful ) and change quite often compared to entries in the SPD. SA entries contain, among other things, the key for the protocol used, and they have a limited validity. There are separate SA entries in the SAD for AH and ESP. An SA is usually created using the IKE protocol and is only used in one direction: It specifies the encryption method and key for the sender and the appropriate decryption method for the recipient. When using symmetric encryption, decryption takes place with the same key that was used for encryption. If two hosts use AH and ESP, four SA entries are required for each host. This is illustrated in the picture.

With IPsec, all endpoints must be preconfigured, otherwise no trust relationship can be established.

In order for two endpoints to be able to establish a trust relationship, a procedure for exchanging the keys is required. The exchange can be done manually or automatically.

Manual key management

The keys that are used for IPsec are exchanged in advance during "manual keying" and permanently configured on both endpoints.

Automatic key management via IKEv1

The Internet Key Exchange protocol is used for automatic key management for IPsec. It uses the Diffie-Hellman key exchange for a secure exchange of keys over an insecure computer network and is probably the most complex part of IPsec. Internet Key Exchange was originally specified in RFC 2409 and was based on the Internet Security Association and Key Management Protocol ( ISAKMP , RFC 2408 ), the IPsec Domain of Interpretation ( DOI , RFC 2407 ), OAKLEY ( RFC 2412 ) and SKEME . The RFCs 2407, 2408 and 2409 are combined in the current version IKEv2 in RFC 4306 and in the current RFC 5996 and are therefore obsolete.

IKE defines how security parameters are agreed and shared keys are exchanged. What is exchanged is the responsibility of a DOI document.

Before the actual start of an encrypted connection with IPsec, both sides need to authenticate each other and to the to use key - algorithms some. This is what IKE is for. The Pre Shared Keying (PSK) and Certificate processes are used for authentication . IPsec works with various symmetrical and asymmetrical keys.

IKE is based on UDP and uses port 500 as the source and destination port by default. However, if IKE and IPsec are operated behind a masquerading firewall, most IPsec implementations use UDP port 4500 in this case. In order to solve the problem with IPsec connections behind masquerading firewalls, several proposals have been submitted. However, none of the proposals has been recognized as a standard, which is why running an IPsec connection from a host across a firewall is very unreliable. The best solution is a non-masquerading firewall with an attached demilitarized zone (DMZ). The end point of the IPsec connection is then in the DMZ.

IKE works in two phases:

  1. Negotiation of an SA ( Security Association ) for ISAKMP either via Main Mode (main mode, preferred) or Aggressive Mode (aggressive mode)
  2. Generating an SA for IPsec in Quick Mode (Quick mode)

A Security Association (SA) is an agreement between the two communicating sides and consists of the following points:

  1. Identification (either by PSK or certificate)
  2. Definition of the key algorithm to be used for the IPsec connection
  3. from which (IP) network the IPsec connection is made
  4. to which (IP) network the connection should exist
  5. Periods in which re-authentication is required
  6. Period after which the IPsec key must be renewed

Phase 1 (main mode variant)

The main mode is used in the first phase of the agreement encryption and authentication (Internet Key Exchange). It should be preferred to the aggressive mode .

In main mode, the initiator (the person who wants to establish the connection) and the respondent (the responder) negotiate an ISAKMP SA with one another. This negotiation takes place in the following steps:

  1. The initiator sends one or more proposals with authentication and encryption algorithms.
  2. The responder selects the most secure from the intersection of the algorithms offered and those supported by it and sends the selection result to the initiator.
  3. The initiator sends its public part of the Diffie-Hellman key exchange and a random value (the nonce ).
  4. The responder also sends its public part of the Diffie-Hellman key exchange and a random value. This value is used for authentication in step 5.

Since both (the responder and the initiator) now know the public parts for the Diffie-Hellman key exchange, this method is used to calculate the secret key. This is then used for the encryption according to the agreed key procedure for the following steps. The calculated (Diffie-Hellman) key is also used to generate a further key that is used for authentication.

Step 5 is authentication. Both parties involved must identify themselves as authorized to access. Two different methods are used here:

  1. Authentication by means of an agreed secret (in English pre-shared keys , PSK) or
  2. certificate-based.

Certificate- based authentication uses X.509 certificates and is essentially a public key infrastructure, as it is also used for SSL and S / MIME . PGP certificates are a different approach and cannot be used for this.

The authentication methods differ, but the basic procedure is always the same: A hash value is always created, encrypted and sent about the secret generated with the Diffie-Hellman key exchange, the identity, the negotiated crypto procedures and the messages sent so far. ( Cookies are sometimes mentioned in the literature : a hash value about a generated secret, IP address and time stamp.) The key that is used here for encryption is not that from the Diffie-Hellman key exchange, but a hash value about these as well as the messages sent.

PSK authentication (Pre Shared Keying):

In this process, authentication is based on a single shared secret. It can be used when a manageable number of participants is connected to the IPsec VPN. The main disadvantage is: If someone gains unauthorized access to this key, the keys must be exchanged on all participating hosts in order to restore security. If a computer network is to grow, this procedure should be rejected even if only a few nodes are initially involved. The additional work for certificate-based authentication usually pays for itself after a short time.

Certificate-based authentication:

This authentication has a different approach. X.509 certificates are used for this. This system is based on trustworthy CAs (Certification Authorities, e.g. with eTrust) or a hierarchy of these. The principle here is that each individual endpoint knows its CAs (trust centers) and recognizes all certificates that are signed by these trust centers as valid. In practice, this means that all certificates from trustworthy CAs are imported and thus all certificates issued by these CAs have access. Certificates can be obtained from well-known CAs ( Verisign , eTrust and many more). This ensures that unknown VPN partners can also be authenticated. Unfortunately, this is not so easy in practice because other parameters (e.g. computer network addresses) play a role and these can collide with existing VPN connections. It has therefore established itself to use a private PKI (Public Key Infrastructure). With their own PKI, only known and trustworthy hosts should have access to the VPN.

Certificate-based authentication is the same as PSK authentication, with one difference: Depending on the connection, a different certificate can be used, and anyone who does not publish their CA certificate can specifically control who is allowed access.

Another advantage of certificate-based authentication: the CA can revoke individual certificates. In the so-called CRL (Certificate Revocation List), all certificates that have somehow become invalid are revoked. With PSK authentication, however, all keys have to be exchanged.

Phase 1 (aggressive mode variant)

In aggressive mode , the above steps are summarized in three. The encryption of the fifth step above is then omitted. Instead, the hash values ​​of the pre-shared keys are transmitted in clear text. The security of the process is closely linked to the strength of the pre-shared key and the hash process used. Since in practice strong keys are often not used for convenience, this mode should be used with caution.

One reason for using this mode can be if the address of the initiator is not known to the responder in advance and both sides want to use pre-shared keys for authentication.

Further application scenarios exist when a faster connection is required and the policies of the responder are sufficiently known.

Example: An employee wants to access the company network remotely. The guidelines (e.g. encryption with AES , hashing with SHA and authentication with RSA signatures signed by the company's certification authority) are known.

Phase 2

In the second phase of IKE, Quick Mode is used (protection by the IKE SA). All communication in this phase is encrypted. As in the first phase, a proposal is made and transmitted together with a hash value and the nonce . The keys are recalculated later and no information from the previously generated SAs is incorporated. This ensures that no one can infer the new keys from the previously generated keys ( Perfect Forward Secrecy ). This is achieved by an additional Diffie-Hellman exchange taking place. The key building secrets are discarded once the exchange is complete.

Several "Quick Modes" can take place at the same time and be protected by the same IKE SA. The message ID field of the ISAKMP header is used to distinguish between the various changes. The status of such an exchange is identified by the cookies.

Difference between IKEv1 and IKEv2

Since IKEv1 is quite complex, many implementations of IPsec became incompatible with each other. While IKEv1 is still specified in several RFCs, IKEv2 is completely described in RFC 7296 . This offers fewer opportunities for misunderstandings and is therefore less error-prone than the first version.

IKEv1 is not very suitable when using dynamic IP addresses, as are common with DSL connections in the private sector. IKEv2 addresses these issues. At the same time, the most widespread DSL routers from the German manufacturer AVM (Fritz-Box) so far only support IKEv1 and not IKEv2 (as of July 2020).

With IKEv2, the phases known from IKEv1 have been fundamentally changed. To create a Security Association (SA), you now only need four UDP messages instead of nine. This enables a connection to be established more quickly, which can be particularly beneficial in the event of malfunctions. In addition, the number of possible combinations for authentication in phase 1 of IKEv1 is reduced. Instead of eight options, only authentication using signatures or MACs is allowed. For more efficient runs, a pair of SAs is also created during phase 1 with IKEv2 during the initial IKE exchange. In particular, if only one pair is required, the exchange is accelerated.

In addition, IKEv2 does not use a preventive exchange of cookies, as problems with denial-of-service attacks against VPN gateways have only occurred in isolated cases in recent years.

While IKEv1 did not regulate responsibilities in the event of packet loss, IKEv2 defined the responsibilities of the peers more clearly. As a result, the connection stability could be improved. In addition, NAT traversal is an integral part of IKEv2, which means that connections can also be established across NAT routers.

Authentication Header (AH)

The authentication header (AH) is intended to ensure the authenticity and integrity of the transmitted packets and to authenticate the sender. It also protects against replay attacks . AH protects the invariant parts of an IP datagram; IP header fields that can be changed by routers on the way through an IP network (e.g. TTL ) are not taken into account. If routers with activated Network Address Translation (NAT) are passed on the way through the network , they change the actually invariant parts of an IP datagram, consequently authentication is no longer possible - NAT and AH are consequently incompatible due to design - just a combination NAT and ESP (see RFC 3948 - UDP Encapsulation of IPsec ESP Packets) is possible. The user data is not encrypted at AH and can therefore be read by everyone. AH is based directly on IP and uses IP protocol number 51.

An AH package looks like this:

Byte 0 Byte 1 Byte 2 Byte 3
Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th
Next header User data length reserved
Security Parameters Index (SPI)
Sequence number field

Authenticity data (variable)

Meaning of the fields:

Next header
identifies the protocol of the data transmitted in the packet. Contains the value which is specified in the IP header for unprotected IP datagrams, but which has been replaced by the value 51 (= AH) when using AH.
Payload length
Size of the AH header
reserved (RESERVED)
reserved for future use
Security Parameters Index (SPI)
Identifies the Security Association (SA) in connection with the IP address and the security protocol
Sequence number field
increasing number, which is set by the sender, should provide protection against replay attacks
Authentication data
Contains the value of the integrity check (ICV), which results from a hash of the rest of the packet

Protection against replay attacks

To protect against replay attacks , the recipient of an AH packet cannot rely on the fact that the sequence number is always higher than that of the previous packet. IP packets can also have been swapped en route. Therefore - based on the highest received sequence number to date - a fixed set of smaller sequence numbers is also accepted. If a sequence number is received for the second time within this set, the corresponding packet is discarded. This also applies to packets with a sequence number that is too small (i.e. below the specified set of smaller sequence numbers).

Encapsulating Security Payload (ESP)

Encapsulating Security Payload (ESP) provides mechanisms to ensure the authenticity, integrity and confidentiality of the transmitted IP packets. In contrast to the AH, the header of the IP packet is not taken into account by the ICV (Integrity Check Value), but the user data is transmitted in encrypted form. ESP is based directly on IP and uses internet protocol number 50.

An ESP package looks like this:

Byte 0 Byte 1 Byte 2 Byte 3
Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th Bit 0 1 2 3 4th 5 6th 7th
Security Parameters Index (SPI)
Sequence number

User data * (variable)

  Filling (0-255 bytes)
    Length filling Next header

Authenticity data (variable)

Meaning of the fields:

Security Parameters Index (SPI)
Identifies the Security Association (SA) in connection with the IP address and the security protocol
Sequence numbers
increasing number, which is set by the sender, should provide protection against replay attacks
Payload data
contains the data packets
Filling (padding)
is used for the block cipher used to fill data up to the full size of the block
Pad length
contains number of inserted bytes for padding
Next header
identifies the protocol of the data transmitted in the packet. Contains the value that is specified in the IP header for unprotected IP datagrams, but which has been replaced by the value 50 (= ESP) when using ESP.
Authentication data
contains the value of the integrity check value (ICV).

Protection against replay attacks corresponds to the AH mechanism.

Comparison of transport and tunnel mode

Comparison between transport and tunnel mode

In transport mode, IPsec connects two endpoints directly with each other ( point-to-point ), for example via software installed on the endpoints.

In tunnel mode, however, two IP networks are connected to one another. The end points are formed by two routers or VPN gateways, between which the tunnel is established.

IPsec in transport mode

IPsec AH header in transport and tunnel mode
IPsec ESP header in transport and tunnel mode

In transport mode, the IPsec header is inserted between the IP header and the user data. The IP header remains unchanged and is still used to route the packet from the sender to the recipient. The transport mode is used when the “cryptographic endpoints” are also the “communication endpoints”. After receiving the IPsec packet, the original user data (TCP / UDP packets) are unpacked and passed on to the higher layer. Transport mode is mainly used for host-to-host or host-to-router connections, e.g. B. for network management .

IPsec in tunnel mode

In tunnel mode, the original packet is encapsulated and the security services of IPsec are applied to the entire packet. The new (outer) IP header is used to address the tunnel ends (i.e. the cryptographic endpoints), while the addresses of the actual communication endpoints are in the inner IP header. The original (inner) IP header only represents a payload for routers etc. on the way between the tunnel ends and is only used again when the receiving security gateway (the tunnel end on the receiving end) has removed the IP encapsulation and deliver the package to the actual recipient.

Gateway-to-gateway or peer-to-gateway connections are possible in tunnel mode. Since the end of the tunnel and the communication end point can coincide on the same computer on either side, peer-to-peer connections are also possible in tunnel mode. One advantage of the tunnel mode is that for the gateway-to-gateway connection, IPsec only needs to be implemented and configured in the gateways (tunnel ends). In this way, attackers can only determine the tunnel endpoints of the IPsec tunnel, but not the entire path of the connection.


To ensure that the connection between the endpoints has not been interrupted in the meantime, they exchange keepalive messages at regular intervals , which can also be used to automatically re-establish a tunnel that has been lost due to the connection interruption.

Dead peer detection

Dead Peer Detection (DPD) was passed in February 2004. By using DPD it can be recognized whether an IPsec connection (especially the ISAKMP tunnel) was unintentionally and unexpectedly aborted. In the event of an error, the remote stations dismantle the SAs (Security Associations) in order to enable the ISAKMP tunnel and the ESP / AH tunnel to be re-established. Without the use of DPD, an end point with a tunnel that is still in place will defend itself against the reconstruction, as the SPIs (Security Payload Identifier) ​​no longer fit. Otherwise, the tunnel can only be rebuilt after the re-keying timer has expired.

DPD is transmitted as a notify message in the ISAKMP protocol (UDP: 500) (message values: RU-THERE - 36136 / RU-THERE-ACK - 36137). The DPD function, on the other hand, ensures that the connection to the remote station is continuously checked and automatically re-establishes itself in the event of an unintentional break in the connection. The specification is defined in RFC 3706 and is also called ISAKMP keepalive.

UDP keepalive

It usually automatically prevents (for NAT traversal) NAT initiated timeout for longer time delays in data entry. The specification is specified in RFC 3519 and is also called NAT keepalive.

Criticism of IPsec


“IPsec was a great disappointment to us. Given the quality of the people that worked on it and the time that was spent on it, we expected a much better result. "

“IPsec was a huge disappointment for us. Given the qualifications of the people who worked on it and the time it took to do it, we expected a much better result. "

The cryptographers Niels Ferguson and Bruce Schneier evaluated the IPsec protocol several times and found several points of criticism. In addition to the way in which it was created, the high level of complexity and thus susceptibility to errors is criticized. However, both also found that IPsec was the best way to secure the original IP at the time of their investigation.

Norms and standards

IPsec was created in the course of the development of IPv6 and is specified in various current RFCs .

According to RFC 5406 (Guidelines for Specifying the Use of IPsec):

  • IPsec Version 1: RFC 1825 (obsolete from 1995).
  • IPsec Version 2: RFC 2401 (obsolete from 1998).
  • IPsec Version 3: RFC 4301 (from 2005).

Other relevant RFCs:

  • RFC 2367 - PF_KEY interface
  • RFC 2403 - The Use of HMAC-MD5-96 within ESP and AH
  • RFC 2405 - The ESP DES -CBC Cipher Algorithm With Explicit IV
  • RFC 2410 - The NULL Encryption Algorithm and Its Use With IPsec
  • RFC 2411 - IP Security Document Roadmap
  • RFC 2412 - The OAKLEY Key Determination Protocol
  • RFC 2451 - The ESP CBC-Mode Cipher Algorithms
  • RFC 2857 - The Use of HMAC-RIPEMD-160-96 within ESP and AH
  • RFC 3526 - More Modular Exponential (MODP) Diffie-Hellman groups for Internet Key Exchange (IKE)
  • RFC 3602 - The AES-CBC Cipher Algorithm and Its Use with IPsec
  • RFC 3686 - Using Advanced Encryption Standard (AES) Counter Mode With IPsec Encapsulating Security Payload (ESP)
  • RFC 3706 - A Traffic-Based Method of Detecting Dead Internet Key Exchange (IKE) Peers
  • RFC 3715 - IPsec- Network Address Translation (NAT) Compatibility Requirements
  • RFC 3947 - Negotiation of NAT-Traversal in the IKE
  • RFC 3948 - UDP Encapsulation of IPsec ESP Packets
  • RFC 4106 - The Use of Galois / Counter Mode (GCM) in IPsec Encapsulating Security Payload (ESP)
  • RFC 4301 - Security Architecture for the Internet Protocol
  • RFC 4302 - IP Authentication Header
  • RFC 4303 - IP Encapsulating Security Payload (ESP)
  • RFC 4304 - Extended Sequence Number (ESN) Addendum to IPsec Domain of Interpretation (DOI) for Internet Security Association and Key Management Protocol (ISAKMP)
  • RFC 4306 - Internet Key Exchange (IKEv2) Protocol
  • RFC 4307 - Cryptographic Algorithms for Use in the Internet Key Exchange Version 2 (IKEv2)
  • RFC 4308 - Cryptographic Suites for IPsec
  • RFC 4309 - Using Advanced Encryption Standard (AES) CCM Mode with IPsec Encapsulating Security Payload (ESP)
  • RFC 4478 - Repeated Authentication in Internet Key Exchange (IKEv2) Protocol
  • RFC 4543 - The Use of Galois Message Authentication Code (GMAC) in IPsec ESP and AH
  • RFC 4555 - IKEv2 Mobility and Multihoming Protocol (MOBIKE)
  • RFC 4621 - Design of the IKEv2 Mobility and Multihoming (MOBIKE) Protocol
  • RFC 4718 - IKEv2 Clarifications and Implementation Guidelines
  • RFC 4806 - Online Certificate Status Protocol (OCSP) Extensions to IKEv2
  • RFC 4809 - Requirements for an IPsec Certificate Management Profile
  • RFC 4835 - Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH)
  • RFC 4945 - The Internet IP Security PKI Profile of IKEv1 / ISAKMP, IKEv2, and PKIX
  • RFC 7296 - Internet Key Exchange Protocol Version 2 (IKEv2)


  • Daniel Bachfeld, Andreas Steffen: VPN etiquette. VPN protocols and standards. In: c't . No. 7 , 2006, p. 114–121 ( heise.de [accessed on July 20, 2019] IPSec Version 2, also slightly abbreviated free of charge ).
  • Naganand Doraswamy, Dan Harkins: IPSec. The new security standard for the internet, intranets, and virtual private networks. 2nd edition. Prentice Hall PTR, Upper Saddle River NJ 2003, ISBN 0-13-046189-X .
  • Christoph Sorge, Nils Gruschka, Luigi Lo Iacono: Security in communication networks. Oldenbourg Wissenschaftsverlag, Munich 2013, ISBN 978-3-486-72016-7 .

Web links

Individual evidence

  1. heise: IPSec VPNs are becoming easier and more flexible thanks to IKEv2. January 3, 2008, accessed March 26, 2009 .
  2. ↑ Connect the FRITZ! Box with a company VPN , under Requirements / Restrictions , extract from the AVM knowledge database, accessed on July 11, 2020
  3. ^ Sorge, Gruschka, Lo Iacono: Security in communication networks 2013, pp. 153f.