See through Walls: Detecting Malware in SGX Enclaves with SGX-Bouncer

April 4, 2021 at 7:34 am

PDF

Intel Software Guard Extensions (SGX) offers strong confidentialityand integrity protection to software programs running in untrustedoperating systems. Unfortunately, SGX may be abused by attackersto shield suspicious payloads and conceal misbehaviors in SGXenclaves, which cannot be easily detected by existing defense solu-tions. There is no comprehensive study conducted to characterizemalicious enclaves. In this paper, we present the first systematicstudy that scrutinizes all possible interaction interfaces betweenenclaves and the outside (i.e., cache-memory hierarchy, host vir-tual memory, and enclave-mode transitions), and identifies sevenattack vectors. Moreover, we proposeSGX-Bouncer, a detectionframework that can detect these attacks by leveraging multifariousside-channel observations and SGX-specific features. We conductempirical evaluations with existing malicious SGX applications,which suggestsSGX-Bouncercan effectively detect various abnor-mal behaviors from malicious enclaves.