Arxiv Paper

Using LLM (GPT3.5) to discover logic bugs in smart contracts

Questions

• How to reduce false positives? Vulnerability confirmation.
• Does temperature = 0 really increase reliability? To my understanding, it just makes the answer deterministic. They do “mimic-in-thebackground” prompting, is that a good method? Any evaluations?
• How are the Filtering Types determined for different types of vulnerabilities?

Highlights

Importance of Logic Vulnerabilities

The third group of vulnerabilities requires highlevel semantical oracles for detection and is closely related to the business logic. Most of these vulnerabilities are not detectable by existing static analysis tools. This group comprises six main types of vulnerabilities: (S1) price manipulation, (S2) ID-related violations, (S3) erroneous state updates, (S4) atomicity violation, (S5) privilege escalation, and (S6) erroneous accounting.

Use of Cheaper GPT-3.5

Methodology

Challenges

• Big code base cannot be directly feed into GPT as a whole => narrow down the scope

Can we break down vulnerability types in a manner that allows GPT, as a generic and intelligent code understanding tool, to recognize them directly from code-level semantics?

• Unreliable answers (FP?) => confirm vulnerabilities

Some Gadgets

GPT's Disadvantages (may not be general to LLMs)

However, we found that GPT struggles to comprehend the concept of “before,”

Missing Context Can Cause FP

For these two types, the main reason for the false alarms is that these vulnerabilities require specific triggering conditions involving other related logic, which may not be contained within a single function and its callers or callees.

New Vulnerabilities

GPTScan successfully discovered 9 vulnerabilities from 3 different types, which did not appear in the audit reports of Code4rena

GPT4

We also conducted a preliminary test using GPT-4, but we did not observe a notable improvement, while the cost increased 20 times.

Prompts

Prompt for scenario and property matching

System: You are a smart contract auditor. You will be asked questions related to code properties. You can mimic answering them in the background five times and provide me with the most frequently appearing answer. Furthermore, please strictly adhere to the output format specified in the question; there is no need to explain your answer.

Scenario Matching

Given the following smart contract code, answer the questions below and organize the result in a json format like {"1": "Yes" or "No", "2": "Yes" or "No"}.

"1": [%SCENARIO_1%]?

"2": [%SCENARIO_2%]?

[%CODE%]

Property Matching Does the following smart contract code "[%SCENARIO, PROPERTY%]"? Answer only "Yes" or "No".

[%CODE%]

Prompt for finding related variables/statements.

In this function, which variable holds the value of total minted share or amount? Please answer in a section starts with "VariableA:". In this function, which variable or function holds the total supply/liquidity AND is used by the conditional branch to determine the supply/liquidity is 0? Please answer in a section starts with "VariableB:". In this function, which variable or function holds the value of the deposit/mint/add amount? Please answer in a section starts with "VariableC:". Please answer in the following json format: {"VariableA":{"Variable name":"Description"}, "VariableB":{"Variable name":"Description"}, "VariableC":{"Variable name":"Description"}}

[%CODE%]