TL;DR: we have discovered XMPP (Jabber) instant messaging protocol encrypted TLS connection wiretapping (Man-in-the-Middle attack) of jabber.ru (aka xmpp.ru) service’s servers on Hetzner and Linode hosting providers in Germany.
The attacker has issued several new TLS certificates using Let’s Encrypt service which were used to hijack encrypted STARTTLS connections on port 5222 using transparent MiTM proxy. The attack was discovered due to expiration of one of the MiTM certificates, which haven’t been reissued.
There are no indications of the server breach or spoofing attacks on the network segment, quite the contrary: the traffic redirection has been configured on the hosting provider network.
The wiretapping may have lasted for up to 6 months overall (90 days confirmed). We believe this is lawful interception Hetzner and Linode were forced to setup.
Last update: 03 Nov 2023
oxpa, experienced UNIX administrator who is in charge for the oldest Russian XMPP service
jabber.ru established in year 2000, was puzzled by “Certificate has expired” message upon connecting to the service on 16 October 2023. All the certificates seemed to be up-to-date on the server, but the connection to port 5222 (XMPP STARTTLS) was presenting an outdated certificate to the client.
During all kinds of software, network and configuration checks with the help of other professionals and
ejabberd software author, it was discovered that:
acme.shcertificate issuing script
The affected machines are dedicated server on Hetzner and two virtual servers on Linode, all hosted in Germany.
Traffic dump on port 5223, all the data intact as it should be.
Traffic dump on port 5222, the connection is hijacked on application level (L7), the server receives replaced ClientHello message from the client.
It was apparent that the connections on port 5222 are under Man-in-the-Middle attack which intercepts encrypted communications.
At first we had 2 hypothesis to check: whether the servers have been hacked and whether the traffic is altered due to some kind of local network (neighbor) spoofing.
For the server, we tested:
LD_PRELOADhijacking libraries with statically compiled
But nothing stood out. We noticed the only uncommon record in the kernel log on Hetzner dedicated machine, it lost the physical network link for 19 seconds on 18 July 2023:
[Tue Jul 18 12:58:29 2023] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Down [Tue Jul 18 12:58:48 2023] igb 0000:04:00.0 enp4s0: igb: enp4s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
In addition, Linode machines are only used as a tunnel to Hetzner server as an alternative route, and are running only bare-bones basic services like SSH and NTP.
Yet we still see unaltered ServerHello packet from the Hetzner server in Hetzner↔Linode tunnel on Linode machine, which is alerted upon sending it to the client over Linode network interface. In other words, the encrypted connection interception of the same kind happens both on Hetzner dedicated and Linode VPS machines.
Note about kernel memory dump
Linode VPS are running Linode-compiled kernel which is booted by the hypervisor and is not present in the VM image. The source code of the kernel is not distributed by Linode. Luckily, there’s embedded build configuration in
/proc/config.gz, and the kernel is built with full symbols map
CONFIG_KALLSYMS_ALL=y, which could be grabbed from
/proc/kallsyms and is sufficient for dump analysis.
From the network perspective, we tested:
No sights of any hijacking by the network segment neighbors on either servers, nothing out of ordinary in the network configuration.
While it’s still not uncommon to see insufficient spoofing protection on physical networks for dedicated servers, it’s hard to believe that such sophisticated stealth spoofing is possible on a virtual infrastructure.
On the Linode machines there aren’t any neighbors at all except the router. Affected Linode machines are unable to ARP-discover or ping any adjacent IP addresses in the same network segment which are perfectly functioning from the outside network.
"That’s weird", I thought.
The unaffected (third-party) Linode VM can ping its neighbors. It is connected to the Cisco systems router:
# ip neigh 188.8.131.52 dev eth0 lladdr 00:00:0c:9f:f0:05 REACHABLE OUI lookup: 00:00:0C Cisco Systems, Inc
However the affected VPS are connected to some unidentified MAC manufacturer:
# ip neigh 184.108.40.206 dev eth0 lladdr 90:de:01:49:76:ae REACHABLE OUI lookup: (no matches)
It became clear that the affected Linode VM have uncommon network setup compared to regular Linode instances, possibly have been isolated into separate VLAN.
After checking crt.sh certificate transparency database, rogue certificates have been discovered which were not issued by any of jabber.ru servers.
There are two genuine certificates used in XMPP service: the one issued for
xmpp.ru, *.xmpp.ru and the other one is for
The maliciously-issued certificates are slightly different from the regular ones for these domains: either the wildcard Subject Alternative Name is missing or a single certificate is issued for both
jabber.ru, xmpp.ru. Moreover, MiTM configuration on xmpp.ru domain (which points to Linode servers) was slightly misconfigured: it serves only
xmpp.ru certificate, yet the original server is configured to serve both
xmpp.ru certificates depending on requested XMPP domain.
List of rogue certificates:
|Domain||Serial||Not Before||Not After||Used in MiTM|
|xmpp.ru||03:f3:68:ee:36:30:80:6a:07:81:17:81:04:0c:e3:d9:10:b1||Jul 18 12:49:03 2023 GMT||Oct 16 12:49:02 2023 GMT||+|
|xmpp.ru||04:9c:2d:af:cc:61:88:d6:67:9f:8b:97:99:ce:ad:c9:b7:e0||Oct 13 11:21:30 2023 GMT||Jan 11 11:21:29 2024 GMT||+|
|jabber.ru||03:43:75:1f:3d:80:20:7d:11:f5:61:98:5b:87:a7:37:81:c6||Apr 18 10:23:29 2023 GMT||Jul 17 10:23:28 2023 GMT||?|
|jabber.ru||04:4c:1c:8a:f4:37:a0:5a:dd:83:9c:54:74:89:bd:b9:97:90||Jul 18 12:38:51 2023 GMT||Oct 16 12:38:50 2023 GMT||+|
|jabber.ru, xmpp.ru||04:d1:d2:5d:09:95:48:9b:d6:14:cc:81:91:df:ac:7f:ec:c6||Apr 25 05:46:23 2023 GMT||Jul 24 05:46:22 2023 GMT||?|
|jabber.ru, xmpp.ru||04:b7:85:83:9a:fd:df:81:26:48:5b:34:28:08:53:d9:e6:79||Apr 25 06:04:19 2023 GMT||Jul 24 06:04:18 2023 GMT||+|
18 July 2023 issuing time is about the same when Hetzner server has lost network link for several seconds.
We have a confirmation from the external network scanner that Linode servers started to serve
04:b7:85… certificate on port 5222 since at least 21 July 2023. Unfortunately, this scanner doesn’t process Hetzner ranges.
We decided to run more network tests of Linode VPS from outside of the machine.
All adjacent Linode hosts are reachable by hop 13 from my laptop’s internet connection.
# lft -d 22 220.127.116.11 Tracing ............*T TTL LFT trace to 172-104-226-23.ip.linodeusercontent.com (18.104.22.168):22/tcp 1 _gateway (192.168.100.1) 31.2ms … 9 a23-210-52-59.deploy.static.akamaitechnologies.com (22.214.171.124) 67.0ms 10 10.210.32.1 62.1ms ** [neglected] no reply packets received from TTL 11 12 10.210.3.93 67.5ms 13 [target open] 172-104-226-23.ip.linodeusercontent.com (126.96.36.199):22 67.4msAs well as hijacked port 5222 on
# lft -d 5222 dawn.jabber.ru Tracing ............**.T TTL LFT trace to dawn.jabber.ru (188.8.131.52):5222/tcp 1 _gateway (192.168.100.1) 29.3ms … 9 a23-210-52-57.deploy.static.akamaitechnologies.com (184.108.40.206) 64.9ms 10 10.210.32.2 64.7ms ** [neglected] no reply packets received from TTL 11 12 10.210.1.235 60.7ms 13 [target open] dawn.jabber.ru (220.127.116.11):5222 60.2msHowever all other non-hijacked ports, such as SSH port 22, is reachable only via additional non-disclosed hop and now reachable only on hop 14:
# lft -d 22 dawn.jabber.ru Tracing ............?*?**.?T TTL LFT trace to dawn.jabber.ru (18.104.22.168):22/tcp 1 _gateway (192.168.100.1) 28.8ms … 9 a23-210-52-57.deploy.static.akamaitechnologies.com (22.214.171.124) 65.4ms 10 10.210.32.2 66.2ms ** [neglected] no reply packets received from TTL 11 12 10.210.1.235 61.7ms ** [neglected] no reply packets received from TTL 13 14 [target open] dawn.jabber.ru (126.96.36.199):22 61.1msThis means that the IP address of Linode VM have been placed behind additional machine which handles TCP port 5222 by itself and routes other ports to the real destination.
From inside the VPS it’s not possible to establish new connections using source port 5222. The router does not reply for the packets originating from this source port, as well as doesn’t send any ICMP error or Time-to-Live Exceeded for TTL=0 or TTL=1 outgoing packets.
# curl -v -4 --max-time 10 --local-port 5222 ifconfig.co tcpdump: 23:37:46.116991 IP 188.8.131.52.xmpp-client > 184.108.40.206.http: Flags [S], seq 4385521, win 64240, options [mss 1360,sackOK,TS val 758380820 ecr 0,nop,wscale 7], length 0 23:37:47.135972 IP 220.127.116.11.xmpp-client > 18.104.22.168.http: Flags [S], seq 4385521, win 64240, options [mss 1360,sackOK,TS val 758381839 ecr 0,nop,wscale 7], length 0 23:37:49.189189 IP 22.214.171.124.xmpp-client > 126.96.36.199.http: Flags [S], seq 4385521, win 64240, options [mss 1360,sackOK,TS val 758383892 ecr 0,nop,wscale 7], length 0
Using testssl.sh SSL/TLS testing script, it was discovered that the intercepting proxy accepts connections with anonymous ciphers on both Linode (xmpp.ru) and Hetzner (jabber.ru).
This is a rare (mis)configuration which could be used as one of the XMPP MiTM detection fingerprint. Regular SSL/TLS libraries are usually compiled without anonymous ciphers support, you won’t be able to configure your service to use anonymous ciphers even if you explicitly allow it in configuration file most of the time, and it is confirmed to be not configured on the original server.
$ ./testssl.sh --starttls xmpp --xmpphost jabber.ru --standard --openssl-timeout 15 --connect-timeout 15 jabber.ru:5222 ########################################################### testssl.sh 3.0.8 from https://testssl.sh/ (abdd51d 2022-09-28 09:19:37) This program is free software. Distribution and modification under GPLv2 permitted. USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK! Please file bugs @ https://testssl.sh/bugs/ ########################################################### Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~179 ciphers] on val:./bin/openssl.Linux.x86_64 (built: "Sep 1 14:03:44 2022", platform: "linux-x86_64") Start 2023-10-18 23:39:14 -->> 188.8.131.52:5222 (jabber.ru) <<-- rDNS (184.108.40.206): utro.jabber.ru. Service set: STARTTLS via XMPP (XMPP domain=\'jabber.ru\') Testing cipher categories NULL ciphers (no encryption) not offered (OK) Anonymous NULL Ciphers (no authentication) offered (NOT ok) ←←← …
We have tested about 20 other popular XMPP services for the same MiTM method with the same
testssl.sh command and discovered no anomalies (no support for Anonymous NULL Ciphers).
All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.
End-to-end encrypted communications, such as OMEMO, OTR or PGP, are protected from the interception only if both parties have validated the encryption keys. The users are asked to check their accounts for new unauthorized OMEMO and PGP keys in their PEP storage, and change passwords.
We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.
As of 20 Oct 2023, we’re still waiting for the adequate reply from Hetzner and Linode to our inquiries.
There are several indications which could be used to discover the attack from day 1:
We haven't received any proper reply from both Hetzner and Linode regarding this incident since 20 Oct 2023. The inquiries were sent as a dedicated tickets with all the evidence information, written in a clear and easy for comprehension manner to every support level staff, with the following wording (snippet):
I'm kindly asking to urgently investigate the network of XXX IP address and especially your network premises.I have strong belief that your router or other network device have been reconfigured to intercept connections to XXX, either maliciously (by the hackers) or by (hosting provider) itself due to legal requirements.If you can't comment on this matter any more, I will assume that the interception was set up by (hosting provider) due to legal requirements which can't be disclosed. Otherwise you should take action immediately and investigate your network.
The only reply received by Hetzner is:
I refer to Ticket# XXX. Unfortunately, we cannot suggest any further solutions here. I understand from your reply that the problem has been solved.Please be aware, that for dedicated root and Cloud servers, we only provide the hardware, network access and necessary infrastructure.Thank you for understanding.
And from Linode:
Hello,Thank you for providing these logs. At this time, we have observed no illegal activity impacting your services and are closing out this ticket.Regards,Akamai