The increasing use of artificial intelligence (AI) in clinical decision-making requires rigorous validation, especially in critical scenarios such as antimicrobial prescribing guided by antibiogram. This study aimed to compare agreement proportions of a rules-based deterministic algorithm with those of three large language models (LLMs), using the decisions of experienced infectious disease specialists as reference.

Cross-sectional validation study conducted with 87 real anonymized urine culture antibiograms associated with simulated clinical cases. Cases included relevant variables such as severity, pregnancy, feasibility of oral administration, and infection site (upper or lower urinary tract). Antibiotic decisions were defined blindly and independently by five sources: an algorithm created by infectious disease physicians, three LLMs (Gemini 2.5 Pro, ChatGPT-4.1 and DeepSeek 03-0324), and two independent infectious disease specialists who did not participate in creating the algorithm. The primary outcome was agreement of each system with at least one of the two specialists. Agreement proportions were calculated. Statistical significance of differences was assessed using the Chi-Square (χ²) test, with p<0.05 considered significant.

For the primary outcome, the rules-based algorithm achieved 95.40% agreement with at least one specialist. In contrast, the LLMs showed substantially lower agreement: Gemini 64.37%, ChatGPT 58.62%, and DeepSeek 49.43%. Differences in agreement between the algorithm and all LLMs were highly statistically significant (p<0.001). In other words, the algorithm showed robust superiority over the tested language models in antimicrobial selection.

A rules-based algorithm showed significantly higher agreement with infectious disease specialists than LLMs for antimicrobial selection. This suggests that, for certain clinical decisions, deterministic pathways may still provide recommendations more similar to specialists’ than LLMs. Additionally, although rapid evolution of LLMs is notable, these results indicate the need to explore hybrid strategies combining the interpretability of rule-based algorithms with the rapidly scalable generative power of LLMs.