OpenSSL DoS Flaw Lets Unauthenticated Attackers Trigger Massive Memory Allocation

A newly disclosed critical weakness in OpenSSL lets an attacker crash a server without ever completing a handshake or proving their identity. Documented by the Okta Red Team, named “HollowByte,” the flaw takes advantage of how OpenSSL reserves memory during the earliest stage of a TLS connection, allowing a payload of just 11 bytes to […] The post OpenSSL DoS Flaw Lets Unauthenticated Attackers Trigger Massive Memory Allocation appeared first on Cyber Security News.

A newly disclosed critical weakness in OpenSSL lets an attacker crash a server without ever completing a handshake or proving their identity. Documented by the Okta Red Team, named “HollowByte,” the flaw takes advantage of how OpenSSL reserves memory during the earliest stage of a TLS connection, allowing a payload of just 11 bytes to force a target into allocating far more memory than the request actually warrants. The TLS handshake begins with a ClientHello message wrapped in a record. Each handshake message carries a 4-byte header that indicates the size of the incoming message body. OpenSSL DoS Flaw Older OpenSSL versions allocate a receive buffer based on that attacker-declared length before any data actually arrives. When the malicious 11-byte payload lands, the TLS state machine reads the header and triggers an unvalidated pre-allocation based on the length field: Read Header → grow_init_buf() → OPENSSL_clear_realloc() → malloc(attacker_size) Because there’s no payload validation at this stage, malloc() allocates up to 131 KB purely on the packet’s claim. The worker thread then blocks indefinitely, waiting for data that will never arrive. Holding connections open to exhaust threads is a classic trick, similar to Slowloris. HollowByte adds a nastier compounding effect tied to how glibc manages memory. When a connection drops, OpenSSL frees the buffer, but glibc doesn’t immediately return small-to-medium allocations to the OS, it holds them for reuse. By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing freed chunks. The heap fragments heavily, and the server’s Resident Set Size (RSS) continues to climb. Even after the attacker disconnects, the server stays permanently bloated. The only fix is killing the process. Testing unpatched versus patched OpenSSL under NGINX revealed the severity: In a 1 GB RAM environment, the unpatched server was OOM-killed after freezing 547 MB of fragmented memory. In a 16 GB RAM setup, the attack locked up 25% of total system memory while staying under typical connection limits, meaning standard connection-throttling defenses won’t stop it. Because OpenSSL is embedded across the ecosystem, this flaw touches web servers (Apache, NGINX), language runtimes (Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL). OpenSSL resolved the issue by switching to incremental buffer growth, merged via PRs #30792, #30793, and #30794. The fix shipped silently in OpenSSL v4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. Instead of trusting the header outright, OpenSSL now grows the buffer only as bytes actually arrive on the wire, so an empty claim costs the server nothing. Notably, OpenSSL treated this as a hardening fix rather than issuing a formal CVE security advisory. Regardless, administrators should upgrade their distribution’s OpenSSL packages immediately, particularly on internet-facing TLS endpoints where unauthenticated exposure makes this trivially exploitable at scale. Prevent critical incidents and financial loss with stronger proactive defense. Integrate a live threat feed from 15K SOCs The post OpenSSL DoS Flaw Lets Unauthenticated Attackers Trigger Massive Memory Allocation appeared first on Cyber Security News.