Abstract—Quantum computing holds tremendous potential for various applications, but its security remains a crucial concern. Quantum circuits need high-quality compilers to optimize the depth and gate count to boost the success probability on current noisy quantum computers. There is a rise of efficient but unreliable/untrusted compilers; however, they present a risk of tampering, such as malicious insertion of Trojans, which can degrade circuit performance and reliability. This work focuses on analyzing the impact of Trojans in Quantum Approximate Optimization Algorithm (QAOA) circuits, which are widely used for solving combinatorial optimization problems. We propose a methodology to reveal vulnerable locations and adversarial gate types for Trojan insertion that maximizes the negative impact on QAOA's approximation ratio in solving Max-Cut problem. By disrupting critical paths and altering qubit states, the strategic insertion of additional gates degrade the approximation ratio by up to 50% based on evaluations on benchmark graphs. These insights on plausible attack mechanisms advance the under- standing of optimization-oriented Trojan vulnerabilities specific to quantum computing. Additionally, a Convolutional Neural Network (CNN) model, referred to as QTrojanNet, is presented to detect the presence of Trojans in compiled QAOA circuits by learning inherent features that indicate malicious modifications. Experimental results showcase an average accuracy of 98.80% and an average F1-score of 98.53% in effectively detecting and classifying Trojan-inserted QAOA circuits.