ESP: Encapsulating Security Payload

What is ESP: Encapsulating Security Payload

In order to understand ESP, we will have to go through a little scenario. As shown in below figure we have two routers R1 and R2 in branches of an organization. When R1 LAN wants to communicate with R2 LAN, it will generate a packet with source IP 192.168.101.X and use a destination IP of 192.168.102.X.

Recommended Read: 

• What is Virtual Private Network-VPN?
• Types of VPN
• 7 Elements to Consider while Selecting a Networking Training Institute for your Cisco Certification.

When this packet arrives at R1, R1 will route it based on its destination IP. R1 has no information about 192.168.102.0 network, so it uses a default route to route the packet to the the internet. A router or Layer 3 devices use its default route to forward those packets which don’t match with other routes.

When packet will arrive at exit interface, the router will check the packet with crypto-map. The packet will match with crypto-map access-list, so the router will forward it to R2 because it is VPN traffic not normal traffic. If router forwards this packet as it is, the internet will drop it because it contains private IP address as a destination and ISP doesn’t maintain information about private IP addresses. So we cannot forward it as it is.

The other thing that we have to keep in mind is that this is our confidential packet we cannot forward it without any security. To secure it we have to encrypt it using encryption algorithms, also will need to add an extra IP header with this encrypted data so that we can forward it to our peer R2.

But before actual transmission, both peers will perform IKE Phase1 & IKE Phase2 negotiation. Here both peers will use IPSec protocols like IKE, ESP and AH.

Internet Key Exchange provides a framework to exchange the security parameters & policies. ESP and AH provide features mentioned in Table.

IKE is a management protocol, it use ISAKMP for key exchange. For Data Origin Authentication we use Pre-Shred Key (PSK).

Now main ESP working will start. There are three things Key, Data and external IP header as shown in Figure-1-4

ESP working Steps:-

Encapsulating Security Payload will use two things for hashing key and data. After hashing ESP will get a hash product called Integrity Check Value (ICV). Below mentioned steps will take place:

1. ESP will put key in hash algorithm.
2. ESP will put data in hash algorithm.
3. ESP will get a hash product which is called ICV (Integrity Check Value).
4. ESP will take the same data which was used for hashing and it will create a PDU (packet).
5. Packet will be encapsulated with ESP header which will contain ICV.
6. ESP header will be encapsulated with External IP header.
7. This packet will be delivered to ISP from R1.
8. ISP will check external IP header’s destination IP address i.e. 102.1.1.100. ISP knows this public IP address very well. ISP will forward that packet to R2 after decreasing 1 in the TTL.
9. R2 will receive the packet.
10. R2 will separate External IP header, ESP header & Data.
11. R2 will put own key in hash algorithm.
12. R2 will put data in hash algorithm.
13. R2 will get a hash product (ICV).
14. R2 will match own ICV with R1’s ICV. Which is in ESP header
15. If hash matches to each other, data packet will be accepted.
16. If it does not match then data packet will be dropped.

For a more clear understanding refer to graphic images in Figure-1-5, Figure-1-6, Figure-1-7 and Figure-1-8.

As illustrated, ESP does not include external IP header for hashing. Now assuming there is a NAT device in the mid of R1 and ISP then it will change the source IP address of external IP header then packet will be forward to ISP. ISP will forward the packet to its destination R2. R2 will still be able to verify the hash. Thus one can conclude that IP address translation has no effect on integrity check value (ICV), because ESP doesn’t include external IP header for ICV.

ESP encrypts entire packet using encryption algorithms and then it is encapsulated with external IP header. Now there will be a problem when packet arrives at a NAT device and it tries to translate the packet. Any NAT device requires Layer 3 information as well as Layer 4 information to translate a packet. External IP header contains Layer 3 information but layer 4 information of packet is encrypted. This absence of Layer 4 information will lead to NAT device not translating the packet, so in this case VPN will not establish. To solve this problem NAT-Traversal is used which is discussed later in this chapter.

ESP can be summarized as:

• It provides all IPSec features
• It uses IP protocol number 50.
• It works with NAT
• It use NAT-T
• It doesn’t include external IP for Integrity Check Value (ICV).

Authentication Header

For ease of understanding, lets use the same scenario which we used for ESP. The problem is similar to what we had earlier before implementation of ESP. AH is a bit different in its working as compared to ESP. AH provides integrity, Data Origin Authentication & Anti-Replay. It cannot provide confidentiality because it does not use encryption algorithms like (DES, 3DES, AES).

AH will work in the below-mentioned manner:

1. AH will put key in hash algorithm.
2. AH will put data in hash algorithm.
3. AH will put external IP header in hash algorithm.
4. AH will get a hash product called as ICV.
5. AH will generate a PDU with same data.
6. AH will add AH header to PDU.
7. AH header will be encapsulated by external IP header.
8. This entire PDU will we forwarded to ISP.
9. ISP will receive the packet & it will check the destination IP address which will be 102.1.1.100. It will then forward the packet to R2 after decreasing Time To Live (TTL).
10. R2 will receive the packet.
11. R2 will separate External IP, AH header and Data.
12. AH will put external IP in hash algorithm.
13. AH will put data in hash algorithm.
14. AH will put key in hash algorithm.
15. AH will get a hash (ICV) product.
16. AH will verify own ICV with sender ICV.
17. If ICV matches packet will be accepted
18. If ICV does not match then the packet will be discarded.

For a more clear understanding refer to graphic images in Figure-1-9, Figure-1-10, Figure-1-11 and Figure-1-12.

As illustrated AH includes external IP header for hashing. Now if there is a NAT device in the mid of R1 & ISP, it will translate source IP address of external IP header and then packet will be forwarded to ISP. ISP will route the packet to its destination R2. R2 will verify the hash which will not match because IP header has been changed. So AH does not work with NAT.

One key point is that when AH uses external IP header for ICV it does not include TTL. To understand the reason for this let us take an example. Assuming R1 is sending a packet to R2 then TTL value at R1 will be 255. So when ICV was generated at R1 TTL value in external IP header was 255. The processed packet will be delivered to ISP. Before routing the packet ISP will decrease 1 in TTL. After TTL decrement packet will be delivered to R2, now TTL value is 254. So when R2 will verify ICV, it will mismatch and the packet will be dropped.

This is the reason why AH when calculating ICV will add 20 bytes IP header but will exclude 8 bits of TTL. This process will be done on both R1 and R2.

AH can be summarized as:

• It does not provide confidentiality, because it does not use encryption algorithms.
• It uses IP protocol number 51.
• It cannot work with NAT.
• It cannot use NAT-T.
• It does include external IP for ICV.
• It does not include TTL value for ICV