There is no dark magic in computing—computers have no flesh and blood.
Anyone who believes in computer dark magic is a muggle.
—Me

cdqe

The Symptom

While debugging with a colleague, we encountered what seemed like “computer sorcery”:

    syscallnr = rec->nr;
    printf("syscallnr=%d\n", syscallnr);
    printf("In renderxxx\nsyscall table address: %p\n", syscalls);

    // causes seg fault
    entry = get_syscall_entry(syscallnr, false);

    // executes normally
    // entry = syscalls[syscallnr].entry;

    printf("entry: %p\n", entry);
    printf("syscall name: %s\n", entry->name);
    printf("arg1name: %s\n", entry->arg1name);

In the code above, get_syscall_entry causes a segmentation fault. But commenting out that line and using the direct array access instead works perfectly. Yet the function implementation is identical to the commented-out line:

struct syscallentry * get_syscall_entry(unsigned int callno, bool do32)
{
    printf("In getxxx\nsyscall table address: %p\n", syscalls);
    printf("In getxxx\nentry: %p\n", syscalls[callno].entry);

    return syscalls[callno].entry;
}

Tracing the Root Cause

The output showed:

Comparing the two entry addresses: the arrow marks the value after the return. The return itself changed the value. This seemed like sorcery—return syscalls[callno].entry shouldn’t alter the value. The direct assignment works fine. Let’s debug.

At the point of return, get_syscall_entry hasn’t modified the return value (stored in rax):

But after the return—something goes wrong! Notice that rax has been mangled. The culprit? A mysterious cdqe instruction!

Checking the documentation: cdqe performs a signed extension of EAX into RAX. This means the upper bits of RAX get sign-extended (potentially zeroed or set), exactly matching the observed behavior.

Digging Deeper

Now that we know cdqe is the culprit, the question becomes: why is the compiler emitting an instruction I never wrote?

After extensive research, I found an Intel blog post describing various pitfalls in 64-bit Linux development. Example 7 closely matches our issue.

In C, you can use functions without preliminary declaration.

So C lets you use functions without declaring them first. But since the compiler doesn’t know the function’s return type, it defaults to int and emits a signed extension. The fix is simple: properly declare the function prototype in a header file.

How do you permanently prevent this kind of issue? Stop using C.