Hiding The Encryption Entropy of a Shellcode

Hiding The Encryption Entropy of a Shellcode


3 min read

In red team exercises, a key challenge in the “initial access” phase is bypassing the detection and response capabilities (EDR) on enterprise endpoints which have been increasing deployments of these types of solutions that have distrupted more simple C2 frameworks like Metasploit and more advanced frameworks like CobaltStrike without significant beacon crafting.

Shellcode encryption is one of the most common method to obfuscate C2 payloads. The method works by encrypting a shellcode to a lightweight encryption standard like AES and attach a decrypt function like AESDecrypt to store the decrypted shellcode for execution. Systems like this have been widespread, and scripts are easily obtained in the wild like san3ncrypt3d’s AESShellcodeEncryptor.

The problem is that this increases the entropy of a file. In shannon entropy, the closer the enthropy value is to the max value of 8 can represent a encrypted file, while a low entropy around 3-4 can suggest a un-encrypted file. Many EDRs use this as a way to detect obfuscated payloads, making the shellcode encryptor above being possibly flagged as malicious by common EDR solutions.

Below is a regular stageless CobaltStrike beacon shellcode, which is 265728 bytes in length. We can then include the raw shellcode, allocate executable memory, copy the shellcode into the executable memory and execute inline:

    if (mem) {
        memcpy(mem, shellcode, shellcode_length);
        ((void (*)(void))mem)();

Once compiled we can analyze the entropy of the file using using CyberChef, we can see that the results are quite high with a score of 7.983542615531063 which is near the maximum score of 8. Food for thought.

Screenshot 2022-11-17 at 20.58.17.png

Shellcode is simply an array of bytes with values ranging from 0x00 to 0xff, so rather than including encrypted shellcodes we can instead use something like English words of the same length then we could instead map the sequence of English words based on the original shellcode values index like a hashmap.

ascii_strings = ['bank', 'base', 'bear', 'beat']

output_file = "shellcode.h"
raw = open(sys.argv[1], "rb").read()
arc4 = ARC4("vine")
buf = arc4.encrypt(raw)
encoded = "".join([ascii_strings[x] + "\x00" for x in buf])

Now that we have the shellcode in an encoded format, we can unencode, unencrypt and inline execute it again

int main() {
    int string_length = 5;
    unsigned char* ciphertext = (unsigned char*)malloc(length);
    unsigned char* plaintext = (unsigned char*)malloc(length);

    if (ciphertext && plaintext && mem){
        for (int i = 0; i < length; i++) {
            *(ciphertext + i) = get_index((char*)(encoded + (i * string_length)));

        RC4_decrypt(ciphertext, plaintext, length);
        memcpy(mem, plaintext, length);
        ((void (*)(void))mem)();

    return 0;

Running the compiled program through CyberChef reveals that the program now has a lower entropy rating of 4.582389507921654 which falls between the scores of common english text. The combination of shellcode encryption and entropy obfuscation can also result in more inconclusive test results, with many vendors dishing out generic names due to the CobaltStrike beacon signature is no longer evident in the file.

Screenshot 2022-11-17 at 20.59.53.png

With the same inline execution process and final shellcode behaviour, fewer vendors can identify the lower entropy shellcode as suspicious even though the end execution result of the CobaltStrike beacon was the same in terms of execution time and effectiveness.

It's mindful however to note that common EDRs use many combination of tricks to detect malicious behavior. While runtime detection may not be detected, other suspicious behaviors might be detected through inline hooks or ETW-TI. But the fact that this method is quick to do and has no impact to the performance and operation of the shellcode proves that this is a low risk method with a high reward matrix if a payload was successfully executed.