Can artificial intelligence chatbots provide reliable patient education for penile curvature treatment?

¿Pueden los chatbots de inteligencia artificial proporcionar una educación confiable al paciente para el tratamiento de la curvatura del pene?

Background: This study evaluated the reliability and quality of information provided by four artificial intelligence (AI) chatbots regarding penile curvature treatment. As patients increasingly seek medical information online, particularly for sensitive urological conditions such as penile curvature, AI chatbots have demonstrated potential for addressing these inquiries. However, the accuracy, comprehensiveness, and readability of their responses remain uncertain. The primary objective of this study was to evaluate the reliability and quality of information provided by AI chatbots on penile curvature treatments. Methods: The question “How is penile curvature treated?” was asked to four different AI chatbots: ChatGPT, Perplexity, Gemini and Copilot. Responses were independently evaluated by five urologists based on readability, understandability, actionability, reliability and transparency. The DISCERN score, PEMAT-P (Patient Education Materials Assessment Tool for Print Materials) test, WRR (Web Resource Rating) scale, Coleman-Liau index and Likert scale were used for assessment. Results: The DISCERN score evaluation showed that Gemini provided poor-quality information, ChatGPT and Copilot offered moderate-quality information, and Perplexity provided good-quality information (Total DISCERN scores: 31, 41, 42 and 51, respectively). PEMAT-P Understandability scores were 45% for Gemini, 55% for Copilot, 64%for ChatGPT and 73% for Perplexity. PEMAT-P Actionability scores were 40% for ChatGPT, Gemini and Copilot, and 60% for Perplexity. According to the Coleman-Liau index, readability levels were required at least at the university-level education. Conclusions: AI chatbots can be useful tools for obtaining information on penile curvature. Given their growing utilization, educating patients on effective interactions with AI chatbots to enhance response accuracy is crucial. With continuous updates and professional oversight, these tools are expected to evolve to become more effective in the future.

Resumen

Antecedentes: Este estudio evaluó la fiabilidad y la calidad de la información proporcionada por cuatro chatbots de inteligencia artificial (IA) en relación con el tratamiento de la curvatura del pene. Dado que cada vez más pacientes buscan información médica en línea, en particular para afecciones urológicas delicadas como la curvatura del pene, los chatbots de IA han demostrado potencial para abordar estas consultas. Sin embargo, la precisión, exhaustividad y legibilidad de sus respuestas siguen siendo inciertas. El objetivo principal de este estudio fue evaluar la fiabilidad y la calidad de la información proporcionada por los chatbots de IA sobre los tratamientos de la curvatura del pene. Métodos: La pregunta “¿Cómo se trata la curvatura del pene?” se formuló a cuatro chatbots de IA diferentes: ChatGPT, Perplexity, Gemini y Copilot. Cinco urólogos evaluaron las respuestas de forma independiente, basándose en su legibilidad, comprensión, viabilidad, fiabilidad y transparencia. Para la evaluación se utilizaron la puntuación DISCERN, la prueba PEMAT-P (Patient Education Materials Assessment Tool for Print Materials), la escala WRR (Web Resource Rating), el índice Coleman-Liau y la escala Likert. Resultados: La evaluación de la puntuación DISCERN mostró que Gemini proporcionaba información de baja calidad, ChatGPT y Copilot ofrecían información de calidad moderada, y Perplexity proporcionaba información de buena calidad (puntuaciones DISCERN totales: 31, 41, 42 y 51, respectivamente). Las puntuaciones de comprensión de PEMAT-P fueron del 45 % para Gemini, del 55 % para Copilot, del 64 % para ChatGPT y del 73 % para Perplexity. Las puntuaciones de procesabilidad de PEMAT-P fueron del 40 % para ChatGPT, Gemini y Copilot, y del 60 % para Perplexity.Según el índice Coleman-Liau, los niveles de legibilidad exigidos eran al menos en el nivel de educación universitario. Conclusiones: Los chatbots de IA pueden ser herramientas útiles para obtener información sobre la curvatura del pene. Dado su creciente uso, es fundamental educar a los pacientes sobre interacciones efectivas con chatbots de IA para mejorar la precisión de las respuestas. Con actualizaciones continuas y supervisión profesional, se espera que estas herramientas evolucionen para ser más eficaces en el futuro.

Artificial intelligence; Chatbot; Quality assessment; Treatment; Penile curvature

Palabras Clave

Inteligencia artificial; Chatbot; Evaluación de la calidad; Tratamiento; Curvatura del pene

[1] Nyirády P, Kelemen Z, Bánfi G, Rusz A, Majoros A, Romics I. Management of congenital penile curvature. The Journal of Urology. 2008; 179: 1495–1498.

[2] Hsieh JT, Liu SP, Chen Y, Chang HC, Yu HJ, Chen CH. Correction of congenital penile curvature using modified tunical plication with absorbable sutures: the long-term outcome and patient satisfaction. European Urology. 2007; 52: 261–267.

[3] Appiah Boakye S, Obeng F. Clinical outcomes of circumcisions and prevalence of complications of male circumcisions: a five-year retrospective analysis at a teaching hospital in Ghana. PLOS Global Public Health. 2025; 5: e0003788.

[4] Vorobev V, Beloborodov V, Sherbatykh A, Kalyagin A, Baklanova O, Popov S, et al. The effectiveness of penile curvature treatment by cavernous body rotation and plication of the tunica albuginea. Basic and Clinical Andrology. 2023; 33: 10.

[5] Sasso F, Vittori M, D’Addessi A, Bassi PF. Penile curvature: an update for management from 20 years experience in a high volume centre. Urologia Journal. 2016; 83: 130–138.

[6] Paulis G, De Giorgio G, Paulis A. Clinical presentation of peyronie’s disease: a retrospective study of 564 cases. Diagnostics. 2024; 14: 1125.

[7] Howell S, Green T, Khalafalla K. Current insights and future directions in Peyronie’s disease management: a narrative review. UroPrecision. 2024; 3: 3–16.

[8] Gul M, Diri MA. YouTube as a source of information about premature ejaculation treatment. The Journal of Sexual Medicine. 2019; 16: 1734–1740.

[9] Naik N, Hameed BMZ, Nayak SG, Gera A, Nandyal SR, Shetty DK, et al. Telemedicine and telehealth in urology—what do the ‘patients’ think about it? Frontiers in Surgery. 2022; 9: 863576.

[10] Erkan A, Koc A, Barali D, Satir A, Zengin S, Kilic M, et al. Can patients with urogenital cancer rely on artificial intelligence chatbots for treatment decisions? Clinical Genitourinary Cancer. 2024; 22: 102206.

[11] Plambeck B, Jiang J, Perez J, Wordekemper BE, Fu D, Colvin A, et al. Online health information for penile prosthesis implants lacks quality and is unreadable to the average US patient. Cureus. 2023; 15: e34240

[12] Talyshinskii A, Naik N, Hameed BMZ, Juliebø-Jones P, Somani BK. Potential of AI-driven chatbots in urology: revolutionizing patient care through artificial intelligence. Current Urology Reports. 2024; 25: 9–18.

[13] Goodman RS, Patrinely JR, Stone CA, Zimmerman E, Donald RR, Chang SS, et al. Accuracy and reliability of chatbot responses to physician questions. JAMA Network Open. 2023; 6: e2336483.

[14] Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health. 1999; 53: 105–111.

[15] Agency for Healthcare Research and Quality. The patient education materials assessment tool (PEMAT) and user’s guide. Available at: https://www.ahrq.gov/es/health-literacy/patient-education/pemat1.html (Accessed: 11 February 2025).

[16] Dobbins M, Watson S, Read K, Graham K, Nooraie RY, Levinson AJ. A tool that assesses the evidence, transparency, and usability of online health information: development and reliability assessment. JMIR Aging. 2018; 1: e3.

[17] EAU Patient Information. Penile curvature. Available at: https://patients.uroweb.org/condition/penile-curvature/ (Accessed: 11 February 2025).

[18] Warren CJ, Sawhney R, Shah T, Behbahani S, Sadeghi-Nejad H. YouTube and men’s health: a review of the current literature. Sexual Medicine Reviews. 2021; 9: 280–288.

[19] Carneiro B, Dizon DS. Prostate cancer social media: in YouTube we trust? European Urology. 2019; 75: 568–569.

[20] Selvi I, Baydilli N. An analysis of misleading YouTube videos on urological conditions: what to do about the danger of spreading misinformation of the YouTube videos? World Journal of Urology. 2022; 40: 885–886.

[21] Meyer AND, Giardina TD, Spitzmueller C, Shahid U, Scott TMT, Singh H. Patient perspectives on the usefulness of an artificial intelligence-assisted symptom checker: cross-sectional survey study. Journal of Medical Internet Research. 2020; 22: e14679.

[22] Warren CJ, Edmonds VS, Payne NG, Voletti S, Wu SY, Colquitt J, et al. Prompt matters: evaluation of large language model chatbot responses related to Peyronie’s disease. Sexual Medicine. 2024; 12: qfae055.

[23] Charnock D. The DISCERN Handbook: Quality criteria for consumer health information on treatment choices. 1st edn. Radcliffe Medical Press: Abingdon. 1998.

[24] Shoemaker SJ, Wolf MS, Brach C. Development of the patient education materials assessment tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Education and Counseling. 2014; 96: 395–403.

[25] Cocci A, Pezzoli M, Lo Re M, Russo GI, Asmundo MG, Fode M, et al. Quality of information and appropriateness of ChatGPT outputs for urology patients. Prostate Cancer and Prostatic Diseases. 2024; 27: 103–108.

[26] Liu Y, Ji M, Lin SS, Zhao M, Lyv Z. Combining readability formulas and machine learning for reader-oriented evaluation of online health resources. IEEE Access. 2021; 9: 67610–67619.

[27] Coskun S, Bagcivan G. Associated factors with treatment adherence of patients diagnosed with chronic disease: Relationship with health literacy. Applied Nursing Research. 2021; 57: 151368.

[28] Severance SJ, Cohen KB. Measuring the readability of medical research journal abstracts. BioNLP 15. Beijing, China and July 2015. Association for Computational Linguistics: Beijing. 2015.

[29] MacLennan S, Duncan E, Skolarus TA, Roobol MJ, Kasivisvanathan V, Gallagher K, et al. Improving guideline adherence in urology. European Urology Focus. 2022; 8: 1545–1552.