Phishing Email Detection Based on Large Language Models

Abstract

As artificial intelligence technology advances, highly realistic phishing emails generated by Large Language Models (LLMs) have become a severe cybersecurity challenge. To counter this threat, this study proposes and implements a phishing email detection system based on LLMs. The core methodology of this research lies in adopting a resource-efficient "Prompt Engineering" strategy, leveraging the models' intrinsic classification capabilities through zero-shot learning without the need for fine-tuning. We extract multi-dimensional features from emails—including text, URLs, Optical Character Recognition (OCR) from images, and attachment filenames—to construct structured prompts that guide the model's judgment. This study specifically conducts a comparative performance analysis of two prominent open-source, lightweight models: Mistral-7B and LLaMA3-8B. Experimental results demonstrate that Mistral-7B significantly outperforms LLaMA3-8B across all metrics, including accuracy, precision, recall, and F1-score. We further analyze that this performance disparity may stem from Mistral-7B's streamlined, efficiency-oriented architecture, which allows for more stable performance in well-defined classification tasks. In contrast, the architectural complexity and larger vocabulary of LLaMA3-8B might lead to an over-interpretation of features in a zero-shot context, resulting in a higher false positive rate. This research not only validates the feasibility of using prompt engineering for phishing detection but also provides concrete empirical evidence and insights for model selection in specific cybersecurity application scenarios.