Secure Protocols

These protocols typically perform a security function or inherently protect the information being transmitted with cryptographic algorithms. An example of this is the IPSec protocols.

security (IPSec) is a suite of protocols for communicating securely with IP by providing mechanisms for authentication and encryption.

Standard IPSec only authenticates hosts with each other. If an organization requires users to authenticate, they must employ a nonstandard proprietary IPSec implementation, or use IPSec over Layer 2 Tunneling Protocol (L2TP).

The latter approach uses L2TP to authenticate the users and encapsulate IPSec packets within an L2TP tunnel. Because IPSec interprets the change of IP address within packet headers as an attack, NAT does not work well with IPSec.

To resolve the incompatibility of the two protocols, NAT-Transversal (NAT-T) encapsulates IPSec within UDP port 4500 (see RFC 3948 for details).

Authentication Header (AH)

The Authentication Header (AH) is used to prove the identity of the origin node and ensure that the transmitted data has not been tampered with. Before each packet (headers + data) is transmitted, a hash value of the packet's contents (except for the fields that are expected to change when the packet is routed), that is based on a shared secret, is inserted in the last field of the AH.

The endpoints negotiate which hashing algorithm to use and the shared secret when they establish their security association. To help thwart replay attacks (when a legitimate session is retransmitted to gain unauthorized access), each packet transmitted during a security association has a sequence number that is stored in the AH. In transport mode, the AH is inserted between the packet's IP and TCP header. The AH helps ensure authenticity and integrity, not confidentiality.

Encapsulating Security Payload (ESP)

The Encapsulating Security Payload (ESP) encrypts IP packets and ensures their integrity. ESP contains four sections:

• Header: Contains information showing which security association to use and the packet sequence number. Like the AH, the ESP sequences every packet to thwart replay attacks.
• Payload: The payload contains the encrypted part of the packet. If the encryption algorithm requires an initialization vector (IV), it is included with the payload. The endpoints negotiate which encryption to use when the security association is established. Because packets must be encrypted with as little overhead as possible, ESP typically uses a symmetric encryption algorithm.
• Trailer: May include padding (filler bytes) if required by the encryption algorithm or to align fields.
• Authentication: If authentication is used, this field contains the integrity check value (hash) of the ESP packet. As with the AH, the authentication algorithm is negotiated when the endpoints establish their security association.

A security association (SA) defines the mechanisms that an endpoint will use to communicate with its partner. All SAs cover transmissions in one direction only. A second SA must be defined for two-way communication.

Mechanisms that are defined in the SA include the encryption and authentication algorithms and whether to use the AH or ESP protocol. Deferring the mechanisms to the SA, as opposed to specifying them in the protocol, allows the communicating partners to use the appropriate mechanisms based on situational risk.

Endpoints communicate with IPSec using either transport or tunnel mode.

In transport mode, the IP payload is protected. This mode is mostly used for end-to-end protection, for example, between client and server.

In tunnel mode, the IP payload and its IP header are protected. This difference between the two modes is critical to their use. When a tunnel mode packet is processed by a network address translation (NAT) device, the source header is replaced with the NAT's external interface, but the original source and destination are properly protected as well.

If transport mode were used, this change of address would cause the receiving host to reject the packet as altered, since the integrity of the packet had been changed since source header is different.

Consequently, tunnel mode is often used between networks, such as with firewall-tofirewall VPNs.

Internet key exchange (IKE) is one of the most widely used methods that allows two devices to "exchange" symmetric keys for the use of encrypting in AH or ESP. IKE relies primarily on the use of either DiffieHellman-Merkle key exchange or the use of public key certificates.

The specific use would determine the best method of key exchange/negotiation. DH would be used between devices like routers. Public key certificates would be used in an end user VPN connection. Besides IKE, protocols like Kerberized Internet Negotiation of Keys (KINK) or pre-shared keys could also be used to share keys between devices.