Significant developments in the artificial intelligence sector for the cryptocurrency market are making headlines, and research in this area is drawing attention. Accordingly, researchers from Salus Security, a blockchain security company with offices in North America, Europe, and Asia, have published a report showcasing the capabilities of GPT-4 when it comes to parsing and auditing smart contracts.
Research Report on GPT-4 Released
It seems that the artificial intelligence sector is quite good at generating and parsing code, but you wouldn’t want to use it as a security auditor. The article contains striking statements on this topic:
“GPT-4 could be a useful tool for assisting in smart contract auditing, especially in parsing code and providing security vulnerability clues. However, considering its limitations in detecting vulnerabilities, it cannot fully replace professional auditing tools and experienced auditors at this time.”
Salus researchers used a dataset known as the SolidiFI-benchmark vulnerability library, consisting of 35 smart contracts containing a total of 732 security vulnerabilities, to evaluate the ability of artificial intelligence to detect potential security weaknesses in seven common types of security vulnerabilities.
Research Results Are Exciting
According to the results, ChatGPT is successful in detecting true positives, which are real security vulnerabilities that could be worth investigating outside of the test environment. The researchers were intrigued by the accuracy rate exceeding 80% in the tests. However, there is a significant issue with producing false negatives. This is expressed with a statistic called the recall rate, and in Salus team’s experiments, GPT-4’s recall rate dropped to only 11%.
The researchers concluded that this indicates a deficiency in GPT-4’s ability to detect security vulnerabilities, with the highest accuracy rate being only 33%. Therefore, the researchers recommend using specialized auditing tools and old-fashioned human knowledge to audit smart contracts until artificial intelligence systems like GPT-4 gain momentum:
“In summary, GPT-4 could be a useful tool for assisting in smart contract auditing, especially in parsing code and providing security vulnerability clues. When using GPT-4, it should be combined with other auditing methods and tools to increase the overall accuracy and efficiency of the audit.”
