Establishing a performance benchmark for artificial intelligence in pediatric urology: an expert evaluation of ChatGPT and Gemini on circumcision
Establecimiento de un punto de referencia de rendimiento para la inteligen-cia artificial en urología pediátrica: evaluación experta de ChatGPT y Gemini en la circuncisión

1. Introduction

Artificial intelligence-based large language models (LLMs) are attracting attention for their potential to answer patient and guideline-based clinical questions in medicine. Compared to competitors such as Gemini, ChatGPT offers higher accuracy and shorter responses in some medical fields; however, competitors may perform better in certain specialized areas, such as emergency situations [1]. In clinical cases, ChatGPT-4 can make diagnoses with a level of accuracy similar to that of specialist physicians [2]. In some medical licensing exams, ChatGPT-4 has achieved an accuracy rate of up to 81%, outperforming many medical students and surpassing previous models [3]. The consistency and repeatability of responses depend on the version of the model and whether the question is open- or closed-ended [4]. ChatGPT may sometimes provide incorrect or incomplete information, especially when it comes to citing references and sources [5]. The model’s lack of specialized medical data in its training creates limitations in terms of accuracy and timeliness [6].

Circumcision is the surgical removal of the foreskin, the piece of skin that covers the tip of the penis. The practice dates back to ancient times. It is one of the most commonly performed surgical procedures worldwide for medical, religious, cultural, and social reasons [7]. Today, about one-third of men worldwide are circumcised. It is particularly common in Muslim and Jewish communities as a religious ritual. In some countries, such as the United States, Canada, and Australia, circumcision is often performed on newborns for medical or social reasons [8]. Circumcision is medically indicated for cases of phimosis (inability to retract the foreskin), recurrent infections, and certain urological conditions [7]. Medical research shows that circumcision can reduce the risk of urinary tract infections (UTIs), Human Immunodeficiency Virus (HIV), certain sexually transmitted infections, penile cancer [9]. But circumcision is not without risks. The most common problems that can happen are bleeding, infection, rarely injury to the penis. The complication rate is about 0.5% [10]. Therefore, circumcision should be performed by specialists when appropriate medical conditions exist.

The current literature does not contain a specific evaluation of the responses provided by ChatGPT or Gemini to patient or guideline-based questions directly related to circumcision. The primary aim of this study is not merely to compare the performance of two leading artificial intelligence (AI) models, but to provide a preliminary benchmark regarding the proficiency of state-of-the-art LLMs in the specific subdomain of circumcision. By evaluating their responses to a comprehensive set of 30 guideline-based and 20 patient-focused questions, we intend to establish a performance baseline for future AI evaluations in pediatric urology and determine the readiness of these tools for clinical integration.

2. Material and methods

The comparative study was conducted from August to October of 2025. Because the study did not involve human participants or patient records and only used publicly available AI-generated data, it was exempt from formal Institutional Review Board approval. The research was conducted in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines [11]. We evaluated the performance of the ChatGPT-5 and Gemini 2.5 Flash models using two sets of questions related to circumcision.

Guideline-Based and Academic Questions (n = 30) (Supplementary material): Questions were derived by identifying strong recommendations within the European Association of Urology (EAU) and American Urological Association (AUA) guidelines and converting them into interrogative sentences to ensure content validity. Additionally, specific academically significant clinical questions regarding circumcision were included to cover areas where guidelines might be less explicit. These questions cover recommendations for indications of circumcision, management of complications, pain control, and patient education. Circumcision is not a main topic in the EAU guidelines. Rather, it is presented as a treatment recommendation under various headings, particularly in pediatric urology for the treatment of conditions such as phimosis and urinary tract infections. In oncology guidelines, circumcision is included as a treatment recommendation in the penile cancer section.

Patient-Focused FAQs (Frequently Asked Questions) (n = 20) (Supplementary material): Questions most frequently asked by parents and patients were compiled from reputable international urology associations and patient education portals. The questions are written in plain language and reflect common concerns encountered in clinical practice.

The ChatGPT-5 (OpenAI, San Francisco, CA, USA) and Gemini 2.5 Flash (Google DeepMind, Mountain View, CA, USA) models were presented with all questions in English. A clean chat session was opened for each question to eliminate the influence of previous responses. Responses were recorded in their original form without any intervention. To ensure the evaluation reflected typical user behavior, a “zero-shot” prompting strategy was used. All questions were entered into the models verbatim in English without prior examples, role-playing instructions, or specific formatting constraints. Additionally, the prompts did not explicitly direct the models to cite literature or provide a bibliography. This approach allowed us to assess the models’ intrinsic propensity to offer evidence-based support. Consequently, any references provided were generated spontaneously as part of the models’ default response mechanisms.

Two independent reviewers selected guideline- and evidence-based questions, matching each one with a clear guideline recommendation or evidence-based statement from the scientific literature. Two board-certified urologists evaluated the responses generated by the models independently. During the evaluation, the reviewers were blinded to the specific AI model (ChatGPT or Gemini) generating the response to prevent assessment bias. Any conflicting scores were resolved through direct discussion until full consensus was achieved.

The evaluation utilized two scoring systems as follows [12]:

Binary Accuracy Scoring (BAS): 1 = Correct; 0 = Incorrect.

Detailed Accuracy Scoring (DAS): 1 = Completely correct; 2 = Correct but inadequate; 3 = Mixed correct and misleading information; 4 = Completely incorrect.

Statistical analysis was performed using IBM SPSS Statistics 26.0 (IBM Corp., Armonk, NY, USA). The Wilcoxon signed-rank test was used to compare paired ordinal data (detailed accuracy scores) between ChatGPT and Gemini for the same set of questions. The Mann-Whitney U test was used to compare the distributions of detailed accuracy scores between the guideline-based and patient-focused question groups. Descriptive statistics were presented as frequencies and percentages. The results are reported as z and p values, with a p value of < 0.05 considered statistically significant. Inter-rater reliability was assessed using Cohen’s Kappa coefficient to quantify the consistency between the two independent urologists. A formal a priori power analysis was not conducted. Instead, the sample size was determined by the availability of relevant content. This reflects the limited number of high-level recommendations in the EAU and AUA guidelines, as well as specific, academically significant clinical questions regarding circumcision. To provide an estimate of the precision of the accuracy proportions, 95% confidence intervals (CI) were calculated using the Clopper-Pearson exact method.

3. Results

A total of 50 questions were analyzed: 30 guideline-based/academic and 20 patient-focused FAQs. For the guideline-based questions, both ChatGPT and Gemini received DAS of 93.3% (95% CI: 77.9–99.2%), indicating that they provided completely correct responses. One ChatGPT response and two Gemini responses were rated as “correct but inadequate”, and one ChatGPT response contained mixed or misleading information. No completely incorrect answers were identified. There was no statistically significant difference between ChatGPT and Gemini (p = 0.705, z = −0.378; Wilcoxon signed-rank test) (Table 1).

For the patient-focused FAQs, Gemini had a slightly higher proportion of correct answers (85%) (95% CI: 62.1–96.8%) than ChatGPT (75%) (95% CI: 50.9–91.3%). ChatGPT provided more mixed responses (15% vs. 5%), but this difference was not statistically significant (p = 0.315, z = −0.933; Wilcoxon signed-rank test) (Table 2).

A qualitative assessment of the responses revealed distinct patterns in the limitations of the AI models. Responses classified as “Correct but Inadequate” (DAS 2) provided accurate general information but lacked specific clinical details. For example, they omitted particular reduction maneuvers for paraphimosis or failed to specify preferred anesthetic agents. Conversely, responses labeled as “Mixed/Misleading” (DAS 3) were predominantly observed in patient-focused questions about subjective or controversial topics, such as the potential for psychological trauma or long-term regret. In these instances, the models attempted to balance conflicting viewpoints, occasionally resulting in ambiguous answers that lacked clear guidance for lay readers.

In BAS, both models correctly answered all guideline-based and patient-focused questions.

The inter-rater reliability was assessed using Cohen’s Kappa coefficient. For DAS, the reviewers demonstrated almost perfect agreement for ChatGPT (Cohen’s κ = 0.850) and substantial agreement for Gemini (Cohen’s κ = 0.782). Scoring discrepancies were observed in only 2 out of 50 questions for both ChatGPT and Gemini. These differences were resolved through direct discussion between the reviewers until full consensus was reached. For BAS, there was 100% absolute agreement between the raters.

Fig. 1 shows comparisons of DAS distributions between guideline-based and patient-focused question groups for ChatGPT (Panel A) and Gemini (Panel B). According to the Mann-Whitney U test, there was no significant difference between the two groups for either model (p = 0.68 for ChatGPT and p = 0.322 for Gemini).

Fig. 1.Detailed accuracy score comparison for guideline-based and patient-focused frequently asked questions provided by ChatGPT (Panel A) and Gemini (Panel B) (p = 0.68 & p = 0.322).

4. Discussion

According to EAU guidelines, circumcision reduces the incidence of penile cancer and is recommended as an additional treatment for superficial noninvasive disease (PeIN) and invasive disease limited to the glans (categories T1 and T2) [13, 14]. Secondary phimosis is the sole absolute indication; however, it is also indicated for primary phimosis refractory to medical treatment, recurrent balanoposthitis, and UTIs associated with urinary tract abnormalities [15]. Circumcision is further proposed as a preventive measure due to its role in reducing the incidence of sexually transmitted infections (STIs), including HIV, syphilis, and herpes simplex virus type 2 [16], and its inverse relationship with genital Human Papillomavirus (HPV) prevalence [17]. The AUA guidelines affirm that neonatal circumcision is generally safe when performed by an experienced surgeon, noting a minor complication rate of approximately 3%. The well-established medical benefits and low morbidity of circumcision may encourage anxious parents and health-conscious individuals to seek more information through AI-driven resources. Therefore, the accuracy of medical advice obtained from these platforms is clinically relevant.

AI and chatbots, like ChatGPT, are being used more and more to provide patients and physicians with information about urological diseases. Studies in recent years have shown these systems have advantages and limitations regarding response quality, accuracy, and patient safety. Importantly, the absence of a statistically significant difference between ChatGPT and Gemini in this dataset does not prove equivalence. Rather, it demonstrates the considerable proficiency of current generative AI technologies in this domain. Our findings suggest that both models have reached a level of competence that allows them to serve as useful adjuncts for patient education with expert supervision. Therefore, this study provides a preliminary performance benchmark, shifting the focus of future research from determining “superiority” to optimising the safe integration of these tools into clinical practice.

To effectively integrate these tools into daily urology practice, we propose a “physician-in-the-loop” workflow [18]. In this framework, LLMs can rapidly draft responses to common patient inquiries or generate discharge instructions. However, these responses must be rigorously vetted by the urologist. This supervision is critical not only for identifying obvious errors but also for recognizing “correct but inadequate” advice, a risk that our study highlighted when models occasionally omitted specific drug dosages or procedural nuances. We hypothesize that the frequent occurrence of “correct but inadequate” responses to clinical management questions stems from an LLM optimization strategy. Models prioritize general, globally safe information while omitting specific, high-stakes, context-dependent details (e.g., drug dosages or procedural steps) to mitigate the risk of providing unsafe or region-specific misinformation. Treating LLMs as “drafting engines” rather than “decision-makers” allows clinicians to leverage AI’s speed while mitigating the risks of contextual misunderstanding and incomplete guidance.

Our results regarding the accuracy of LLMs in circumcision align with broader trends observed in pediatric urology. In a comparable study evaluating pediatric urology guidelines and FAQs using ChatGPT-3.5, the responses were found to be completely accurate 92% of the time, with high consistency [19]. AI-based models can accurately diagnose, treat, and predict the prognosis of urinary tract infections, kidney stones, and urological cancers. For instance, artificial neural networks have demonstrated an accuracy rate of 98.3% in diagnosing UTIs [20]. Similarly, our study found high accuracy rates for both models. However, distinctions emerge when compared to complex oncological topics. While a study of GPT-3.5 reported accurate answers regarding urological cancers (prostate, bladder, and kidney), previous studies have noted its moderate-to-low applicability and comprehensibility issues [21]. In contrast, our study suggests that for a specific, widely understood procedure like circumcision, the models generate information that is generally more accurate and actionable.

Although models can occasionally match human performance, they consistently underperform in terms of clinical reasoning and visual interpretation. Success rates vary significantly across medical specialties [22]. The comparative performance of AI models varies across urological subspecialties. Previous research has shown that Gemini and ChatGPT have similar accuracy rates for bladder-related diseases [23], while ChatGPT has been found to be more comprehensive for bladder cancer [24]. On the contrary, ChatGPT had a lower rate of poor-quality responses than Gemini for erectile dysfunction [25]. Our study contributes to this comparative landscape by demonstrating that, in the specific domain of circumcision, the performance gap narrows significantly, with no statistically significant difference observed between the two models. These findings suggest that, for standard surgical procedures, the choice of model may be less critical than the implementation of expert oversight.

ChatGPT has the potential to transform urology in the areas of education, research, and clinical practice. Although it functions as a powerful complement to human expertise, deploying this AI technology requires a steadfast commitment to ethical and responsible standards [26]. The data sets used to train the models may be outdated and therefore may not reflect new medical developments [27]. LLMs can sometimes generate information that is not based in reality, which is sometimes referred to as a “hallucination” [27, 28]. Despite the high accuracy observed in our dataset, the potential for misinformation remains a critical concern for clinical implementation. A survey of urologists revealed that 74.9% believe LLMs can disseminate misinformation. The most commonly expressed ethical concerns were artificial hallucinations (46.1%) and plagiarism (51.9%). Since nearly half of users identified inaccuracy as a primary limitation, urologists must be vigilant in verifying AI outputs to prevent the dissemination of misleading information [29]. Although ChatGPT improves access to information, the risk of incomplete responses underscores the importance of human expertise [30]. While we did not identify hallucinations in our dataset, the prevalence of “correct but inadequate” answers confirms the need for caution and validates the conclusion that AI cannot yet replace clinical judgment in diagnostic and therapeutic decision-making [31].

It is important to note that this study used English-language prompts. Since LLM performance can vary significantly across languages, future research should prioritize multilingual testing to determine if these high levels of accuracy are maintained for non-English-speaking patient populations. Additionally, given the rapid pace of AI development, longitudinal studies are necessary to monitor the influence of future model updates on the consistency and reliability of medical advice over time. The accuracy rate drops significantly in languages other than English; success in multilingual medical exams can decrease by up to 10% [27]. Since LLMs are typically trained on data collected within a specific timeframe, they may be inadequate for current events or evolving information. Studies have shown that models trained with timestamps not only have better recall of information from the training period but also update more efficiently with new data [32]. Recent studies have revealed that variations in prompts significantly impact model performance, and existing methods are insufficient for predicting this sensitivity. Additionally, interactive and visual prompt engineering tools can help users develop more effective prompts [33].

Models generally do not cite sources in their responses and lack transparency [28]. For instance, only 43% of citations provided by ChatGPT-4 have been verified as accurate, with 57% being false or entirely hallucinated [34]. ChatGPT-4o achieved a higher accuracy rate in answering medical questions than its predecessors, ChatGPT-3.5 and 4, reaching an average score of 4.46 out of 5 while lowering the false response rate to 3.7%. However, its reproducibility, or consistency in responding to repeated queries, is reportedly lower [35]. Overall, newer iterations of ChatGPT (specifically versions 4 and 4o) demonstrate a significant improvement in the accuracy of the information they provide. While the AI models were not instructed to cite sources in our study, ChatGPT-5 provided references when answering questions about the cost-effectiveness of circumcision and penile cancer. Upon investigation, we confirmed the accuracy of these citations; they were not the result of hallucinations.

This study has several limitations. First, the data were collected through simulations and do not reflect real patient interactions. The questions were predefined and modeled only in English, which limits the study’s generalizability to different languages or cultural contexts. Additionally, AI models’ knowledge base may not fully cover the current literature, and some responses may lack verifiable sources. Since the study focused solely on questions related to circumcision, the results cannot be generalized directly to other urological or medical fields.

5. Conclusions

In general, AI chatbots can provide accurate and balanced information about urological diseases, often performing at near-human levels. However, they have significant limitations, such as hallucinations, poor reasoning, a lack of citations, and issues with timeliness. In this study, ChatGPT and Gemini demonstrated high accuracy in answering 30 guideline-based and 20 patient-focused questions; their performance showed no statistically significant difference within this specific question set. These results provide a preliminary performance benchmark, suggesting that current LLMs have the potential to support clinical practice when used under expert supervision. To safely integrate this potential into daily workflows, we recommend a “physician-in-the-loop” model that utilizes LLMs as drafting assistants while ensuring rigorous oversight to mitigate risks such as “correct but inadequate” advice. Ultimately, with expert supervision, these models can serve as effective adjuncts for patient education and clinical efficiency.

Availability of data and materials

The data supporting the findings of this study are ethically restricted and are available from the corresponding author (YA) upon reasonable request.

Author contributions

YA—writing, data curation, original draft. AT—data curation, conceptualization. Both authors read and approved the final manuscript.

Ethics approval and consent to participate

Not applicable. Because the study did not involve human participants or patient records and only used publicly available AI-generated data, it was exempt from formal Institutional Review Board approval.

Acknowledgment

The authors would like to express their gratitude to all the dedicated urology professionals for their hard work.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at https://files.intandro.com/files/article/2071860267858182144/attachment/Supplementary%20material.docx.