Dealing with TLS Connection Resets Caused by a Huawei Core Firewall

This is the debugging archive of a TLS connection reset problem encountered by the SUSTech mirror site in April 2021. TL;DR: the first HTTPS connection initiated from outside campus to the mirror site’s public IPv4 address (116.7.234.220) on TCP port 443 is answered ahead of the real server and blocked by a middlebox; the mirror host never even receives the packets associated with that TCP connection.

Introduction

The first HTTPS connection initiated from outside campus to the mirror site’s public IPv4 address (116.7.234.220) on TCP port 443 is answered ahead of the real server and blocked by a middlebox; the mirror host never even receives the packets associated with that TCP connection.

Between 09:00 and 24:00 Beijing time this problem can be reproduced with very high probability; in the early morning it seems less likely to occur. We speculate that this is related to the load on the middlebox or to utilization of the egress bandwidth.

The middlebox seems to let the IP’s subsequent TCP connections to port 443 through once it has intercepted a single TLS Client Hello. The timeout of this allow rule seems to vary, roughly from 30 minutes to an hour and a half.

Scope of Impact

It was first observed on TCP port 443 of the mirror site’s public IPv4 address. The HTTP service provided on TCP port 80 of the public IPv4 address, as well as the IPv6 address, are not affected.

Subsequent testing found that port 443 of some IPs in the 116.7.234.0/24 subnet of the school’s telecom egress is likewise affected, though apparently at a lower frequency than the mirror site’s 116.7.234.220. One IP in the 110.65.147.0/24 education network egress appeared to be affected, but this could not be reproduced on the education network IPs obtained afterwards.

How to Reproduce

Start a packet capture program on the client. Reference command: tcpdump -i eth0 -n -w /tmp/test.pcap host 116.7.234.220

Then run the command below repeatedly and observe the capture records.

curl http://116.7.234.220:443/ --max-time 5 -v -o /dev/null

The TCP handshake proceeds normally, but the HTTP request sent by the client is never answered: although the client’s TCP stack keeps retransmitting the PSH,ACK packet carrying it, the other side never sends back an ACK.

Capture on the mirror machine at the same time to confirm that the packets associated with this TCP connection do not arrive at the mirror machine on the internal network. From this we infer that the connection was intercepted and answered ahead of the real server by a middlebox.

Afterwards, run curl https://116.7.234.220/ -v -o /dev/null

The local capture shows that after the client sends the TLS Client Hello, the other side directly returns an RST packet terminating this TCP connection. No related packets are seen on the mirror machine.

According to our observations, once the RST has been successfully triggered, the middlebox stops interfering with communication between the outside world and the mirror’s port 443 for a period of time, and a TLS connection can be established normally. An HTTP request to port 443 then receives the nginx 400 Bad Request response normally.

Summary of Debugging Conclusions

The border firewall used by the school is provided by Huawei (USG9500). During debugging with the vendor’s technical staff, we found from the ACL rule match records that when the problem occurs, packets from outside destined for the public 443 port seem unable to complete NAT and be forwarded to the corresponding internal IP.

The ACL counters for port 80. The number of packets from the client to the mirror’s public IP equals the number destined for its internal IP (corresponding to rule 5 and rule 15 respectively).

Advanced ACL 3001, 4 rules ( Reference counter 1 )
Acl's step is 5
 rule 5 permit tcp source <client-public-ip> 0 destination 116.7.234.220 0 destination-port eq www (16 times matched)
 rule 10 permit tcp source 116.7.234.220 0 source-port eq www destination <client-public-ip> 0 (16 times matched)
 rule 15 permit tcp source <client-public-ip> 0 destination <mirror-intra-ip> 0 destination-port eq www (16 times matched)
 rule 20 permit tcp source <mirror-intra-ip> 0 source-port eq www destination <client-public-ip> 0 (16 times matched)

The statistics for port 443, however, show an anomaly: it looks as if the firewall received the packet on the public side but did not send it on to the intended mirror internal IP (see the difference in hit counts between rule 5 and rule 15).

Advanced ACL 3001, 4 rules ( Reference counter 1 )
Acl's step is 5
 rule 5 permit tcp source <client-public-ip> 0 destination 116.7.234.220 0 destination-port eq https (3 times matched)
 rule 10 permit tcp source 116.7.234.220 0 source-port eq https destination <client-public-ip> 0 (0 times matched)
 rule 15 permit tcp source <client-public-ip> 0 destination <mirror-intra-ip> 0 destination-port eq https (0 times matched)
 rule 20 permit tcp source <mirror-intra-ip> 0 source-port eq https destination <client-public-ip> 0 (0 times matched

Once the reset behaviour on port 443 is triggered by the TLS Client Hello sent by the client, the statistics for port 443 on the firewall return to normal as well. This phenomenon seems strongly correlated with the timing of the port 443 reset problem.

At this point (the afternoon of 25 March), Huawei’s on-site engineer had already recognized the problem and fed it back internally.

On the morning of 31 March, Huawei’s on-site engineer said that while checking the firewall logs he had found that the behaviour of one mechanism looked suspicious, and that after turning it off tests showed the port 443 problem was gone. After consulting with the teachers at the school’s information center, he decided to raise the threshold of this feature, and after some testing the mirror site’s current traffic volume no longer seems to trigger the man-in-the-middle problem.

The problem seems to have been solved, but we do not yet know the specific name of this feature or its intended behaviour (for example, why it would attack TLS connections by intercepting the TLS Client Hello). It is still unclear whether the problem will recur if the mirror site’s external bandwidth is increased in the future.

Miscellaneous Debugging Notes

Running traceroute from outside to 116.7.234.220, the last router that returns an ICMP Time Exceeded message for both ICMP and UDP packets has the IP 183.56.64.2 or 183.56.64.10. We infer this hop to be the router on the Shenzhen Telecom access side; call its hop count N.

If a traceroute is performed against TCP port 80 and TCP port 443 in its normal state using TCP SYN packets, the responding SYN ACK can be measured as coming from hop N+3, and neither hop N+1 nor N+2 returns an ICMP Time Exceeded message.

When the problem occurs, a SYN packet with TTL N+1 alone is enough to trigger the middlebox to return a SYN ACK. This can also be confirmed by specifying the TTL value of the socket.

The nping tool provided by nmap can be used to craft TCP packets and traceroute, for example: nping --tcp --traceroute --dest-ip 116.7.234.220 -p 443 --flags syn

In TCP Probe mode (rather than TCP connect mode), nping sends the crafted TCP packets directly through a raw socket, and the kernel’s TCP stack does not take part. To keep the kernel’s TCP stack from sending RST packets and disturbing the middlebox’s state machine, an iptables rule can be used to drop RST packets destined for the IP under test before starting:

iptables -I OUTPUT -p tcp -d 116.7.234.220 --tcp-flags RST RST -j DROP

In addition, socket options can be adjusted to craft a TLS Client Hello with a specific TTL, so as to precisely trigger the reset behaviour of the middlebox in question. An example in Python:

import socket
import ssl

hostname = '116.7.234.220'
context = ssl.create_default_context()
N=17

with socket.create_connection((hostname, 443)) as sock:
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_TTL, N+1)
    with context.wrap_socket(sock, server_hostname=hostname) as ssock:
        print(ssock.version())

It is worth mentioning that, because carriers may use load-balancing strategies such as ECMP on intermediate routes, packets with different five-tuples may differ in hop count, so when trying to locate the problem with TTL one should first confirm that the intermediate routing topology is stable.

We carried out the tests described above on 21 March (requesting port 443 of all services SUSTech exposes to the public internet using the http protocol; the expected result should be 400 bad request, and if the reported problem occurs the request will time out indefinitely — but since we set a 5-second timeout, any request in the results taking longer than 5 seconds can be considered to have been disturbed by the school’s firewall equipment):

The results are shown in the table below (tests were run every 10 minutes, initiated respectively from Alibaba Cloud Shenzhen, Shenzhen Telecom and Hong Kong Broadband Network; about 700 tests were run on 21 March):

According to Huawei’s on-site engineer:

Because you only have one public IP and everything points to port 443, once the traffic gets large it triggers the firewall’s attack prevention mechanism, exceeding the default value of the attack prevention mechanism, which is what caused the dropping

The engineer has now raised the threshold of the attack prevention mechanism, and the situation of connections being reset should be significantly improved.

We will continue to follow this issue.