Can ChatGPT help patients understand their andrological diseases?
¿ChatGPT puede ayudar a los pacientes a comprender sus enfermedades andrológicas?

1. Introduction

Andrological problems, with their broad subcategories, impact a significant portion of the male population [1, 2, 3, 4]. When we consider subcategories such as male hypogonadism, erectile dysfunction (ED), ejaculation disorders, infertility, penis size abnormalities, penile curvature, and priapism, it becomes evident that they collectively affect a substantial segment of society [2, 3, 4]. Epidemiological studies published to date have consistently indicated a high worldwide incidence and prevalence of ED [1]. For example, a study conducted in Boston reported that the overall prevalence of ED in men aged 40 to 70 was 52% [2]. Regarding Premature Ejaculation (PE), epidemiological data indicates that approximately 5% of the general population has an ejaculation latency of less than 2 minutes, both for acquired PE and lifetime PE [3]. An article on penile deformity stated that the prevalence of definite and probable cases of Peyronie’s disease in the USA was 0.7% and 11%, respectively [4]. Approximately 15 percent of couples seek medical assistance because they cannot achieve pregnancy in the first year [5]. Fifty percent of couples have a factor associated with male infertility, usually abnormal semen parameters [6]. As can be understood from the data above, andrological diseases constitute common health problems in society and are frequently encountered in urology practice.

The use of the internet and artificial intelligence-supported information systems is increasing in many areas. Since andrological problems are typically sensitive issues that patients may hesitate to discuss, they often prefer to seek preliminary information and satisfy their curiosity through the internet and artificial intelligence-supported information systems before consulting a healthcare professional. Medical students can also utilize these systems to enhance their education due to their speed and ease of accessing information.

Chat Generative Pre-training Transformer (ChatGPT) is an artificial intelligence program created by OpenAI that provides instant answers to questions using an Internet database [7]. It can be accessed through a smartphone application or an Internet browser. This program, which helps users access information by instantly synthesizing data on various subjects, is rapidly gaining popularity. Several articles have been published investigating ChatGPT’s ability to provide accurate information in the medical field. In a study conducted by Gilson et al. [8], it was determined that ChatGPT provided satisfactory answers to 60% of medical school exam questions. In another study, despite its limitations, Caglar et al. [9] found that ChatGPT provided satisfactory answers to questions related to pediatric urology. Although there are many articles published so far investigating the accuracy and reliability of ChatGPT’s answers in various medical fields [7, 8, 9, 10], we noticed that there is no such study in the field of andrology. Therefore, in this study, we investigated the accuracy and validity of ChatGPT’s answers on andrology and the usability of these answers in terms.

2. Material-methods

All questions frequently asked by patients on health websites, urology association websites, and social media (Facebook, Instagram, and Twitter) were recorded. Health website selection was made as follows: When the sub-topic to be evaluated was searched on Google, the frequently asked questions on the patient information sites that first appeared were evaluated. Strongly recommended recommendations on each topic of the sexual and reproductive health section of the EAU were transformed into questions [11]. Inappropriate questions, questions related to personal health, and repetitive questions were excluded from the study. The questions obtained from the internet and the questions obtained from the EAU Guidelines were categorized as follows: Male Hypogonadism, Erectile Dysfunction and Sexual Desire Disorder, Disorders of Ejaculation, Penile Curvature and Penile Size Abnormalities, Priapism, Male Infertility.

Questions were answered using free ChatGPT-3.5 version and subscription paid version 4.0 on internet browsers with cookies and history cleared. The answers provided by ChatGPT were evaluated and scored by two urologists. The authors assigned a score to each answer on a scale of 1 to 4: If the answer correctly included all the information that a urologist should provide to a patient, it was rated as 1 (completely correct); if it was correct but insufficient, it was rated as 2; if it contained a mix of correct and misleading information, it was rated as 3; and if it contained completely incorrect answers, it was rated as 4.

Discrepancies in the scoring between the two auditors were resolved by taking the arithmetic average, and consecutively scored answers were reviewed collaboratively by the auditors, evaluating them based on primary sources.

ChatGPT’s responses to all questions were generated after confirmation by reposing the same questions from different IP addresses, thus confirming the repeatability of the answers. We asked ChatGPT about the difficulty level of each question and classified them based on its responses. According to anwers, all questions were categorized as easy, medium or difficult.

3. Statistical analysis

The data collected from the answers were imported into Excel (Version 16, Microsoft, USA). The accuracy of the answers was assessed, and the accuracy percentages were calculated based on andrology topics. All data from both groups were compared using SPSS for Windows (Version 25.0, IBM Corp., Armonk, NY, USA).

4. Results

A total of 120 questions were initially included in the study. However, 28 questions were excluded based on exclusion criteria. Among these excluded questions, 13 were repetitive, 11 were grammatically inadequate, and 4 pertained to personal health. After applying the exclusion criteria, 92 questions remained and were included in the study. Fig. 1 illustrates the questions included and excluded in the research.

Fig. 1.Study inclusion flow chart.

The answer rates provided by ChatGPT for all questions are presented in Table 1. Among the questions, 50.0% received a rating of “completely correct” (grade 1) (57.6% for 4.0 version). Approximately 33.6% of the questions had missing information (grade 2) (32.2% for 4.0 version), 13.3% were a mix of accurate and misleading information (grade 3) (11.1% for 4.0 version), and 3.3% were answered completely incorrectly (grade 4) (2.2% for 4.0 version). Out of the 92 questions, 77 received ratings of “completely correct” or “correct but insufficient” (80 for 4.0 version).

The topic with the fewest completely correct answers in both versions was “Ejaculation Disorders” 33.3% for 3.5 version (44.4% for 4.0 version). Questions that received completely incorrect answers were related to “Penile Curvature and Size Abnormalities” and “Priapism” issues.

The grading of answers provided for frequently asked questions and questions obtained from the EAU Guidelines is presented in Table 2. The rate of completely correct answers to frequently asked questions on the subjects of Male Hypogonadism, Erectile Dysfunction and Sexual Desire Disorder, Disorders of Ejaculation, Penile Curvature and Size Abnormalities, Priapism, and Male Infertility was 80.0%, 60.0%, 42.8%, 50.0%, 44.4% and 38.4%, respectively. These rates were 50%, 66%, 0%, 30%, 77.7% and 42.8%, respectively, in the questions generated from the EAU guidelines. For version 4.0, the rates of answers to the EAU guidelines questions were the same, but the rates of answers to the frequently asked questions were different. The rate of completely correct answers were 80.0%, 80.0%, 57.1%, 60.0%, 44.4% and 46.1% respectively.

All questions were categorized as easy, medium, and difficult. Among the frequently asked questions, 38 questions were categorized as easy, and 16 questions were at the medium level. There were no difficult questions among the frequently asked questions.

Examples of questions are shown in Table 3. Among the questions derived from the guidelines, 13 questions were easy, 15 questions were medium, and 12 were considered difficult.

The rates of correct answers to the questions (grades 1 and 2 combined) were 85.2% for frequently asked questions and 81.5% for questions obtained from the guidelines (88.9% for 4.0 version). The rate of completely incorrect answers (grade 4) for the questions was 1.8% in the group of frequently asked questions and 5.2% in the group of guideline-based questions (0% and 5.2% for 4.0 version). Despite the guideline questions being more challenging and detailed, the success rate in answering the questions was reasonably similar.

5. Discussion

Artificial intelligence (AI) is increasingly being used widely today. The ChatGPT program, which can be easily used by patients in their own language, is an evaluation by the OpenAI company. While the source of information is not specified, previous studies have highlighted its high success rates, particularly in providing accurate responses to health-related questions.

Numerous studies have previously been published, indicating the prevalence of controversial and inaccurate health information shared on social media platforms such as YouTube, Facebook, Twitter, Reddit and Pinterest [12]. For instance, in a study conducted by Alsyouf et al. [13] that assessed articles shared on social media regarding urological malignancies, it was found that the quantity of false and misleading articles was notably higher than the number of accurate articles. However, such negative findings have not been reported for systems utilizing Artificial Intelligence (AI). In a study by Bulck and Moons, it was observed that the patient information articles selected were summarized and presented at a level that patients could easily comprehend [14].

Our article represents the first study to assess the accuracy and reliability of ChatGPT’s responses to andrology-related inquiries. In our study, we observed that ChatGPT provided responses rated as completely correct or correct but insufficient (grade 1 and grade 2) for 83.6% (89.8% for 4.0 version) of the andrology-related questions. Unlike other internet resources such as social media, ChatGPT acquires and presents comprehensive information from numerous sources. It is evident that the breadth of the database it utilizes contributes to the higher percentage of accurate responses to the questions.

Guidelines provide physicians with universally applicable and typically detailed information in their respective fields. It was initially thought that ChatGPT might face challenges in accessing such information, but in the study conducted by Cakır et al. [15], it was discovered that ChatGPT provided satisfactory answers to 80% of the questions based on the EAU urolithiasis guidelines. In our study, we observed that ChatGPT provided 83.6% correct answers (89.8% for 4.0 version). This indicates the high effectiveness of ChatGPT in addressing questions that require detailed information, such as those derived from strong recommendations in guidelines.

In the present study, we observed that the answer success rates for frequently asked questions and guideline-based questions were quite similar, a finding consistent with other studies [8]. This indicates that ChatGPT is adept at retrieving specific information with a high degree of success. However, it’s worth noting that the Grade 2 response category (33.6%), which also includes missing information, is relatively substantial. As a result, we believe that ChatGPT may not be entirely sufficient for medical education on its own but could serve as a valuable supplementary tool in the learning process.

The program offers advantages in various aspects. We believe that such easy access to information will enhance the health literacy of society. When patients search for their symptoms on the internet, they often encounter topics related to malignancies, which can increase their anxiety levels. We anticipate that artificial intelligence methods can provide more accurate information, thus reducing anxiety levels.

Although the use and popularity of ChatGPT continue to rise, it is essential to be mindful of its potential drawbacks. For instance, in a study conducted by Rafeh et al. [16], it was found that ChatGPT increased the perceived urgency of the disease in 44% of the cases queried. This increase in urgency may lead to higher healthcare costs and heightened patient anxiety levels [16]. In another study by Hofmman et al. [17], it was determined that the responses provided by ChatGPT 3.5 and 4.0 versions to orthopedic board exam questions exhibited statistically significant differences in success rates. In their study, the 4.0 version’s answers were found to be on par with those of a 3-year orthopedic specialist, whereas the 3.5 version’s responses were comparable to those of an orthopedic intern [17].

An advantage of our study was that the paid version of ChatGPT 4.0 was used. We had the opportunity to evaluate both versions. The fact that versions 3.5 and 4.0 answered the questions obtained from the EAU 2023 guideline with the same success rate was due to the fact that both versions used the EAU 2023 guideline as a database. Another advantage of our study is that it addressed a deficiency seen in previous studies, which were predominantly conducted in English and lacked research in other languages. The fact that we obtained similar results to studies conducted with English questions indicates that we could readily access information with ChatGPT’s high translation success.

However, a limitation of our study is the number of the evaluator urologists. It may result in observer fatigue bias.

6. Conclusions

In conclusion, ChatGPT offered satisfactory responses to frequently asked questions and questions derived from the EAU Guidelines. However, the category of responses with missing information was substantial. Therefore, we suggest that the implementation of ChatGPT under the supervision of urologists in urology clinics could enhance patients’ understanding of andrology issues. Additionally, we believe it can serve as a valuable educational support tool for medical students.

Availability of data and materials

The data presented in this study are available on reasonable request from the corresponding author.

Author contributions

İEE and AS—designed the research study. İEE—performed the research; analyzed the data. AS—wrote the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Ethical approval was not obtained for this study because the use of ChatGPT and European Association of Urology (EAU) guideline information is publicly accessible.

Acknowledgment

Not applicable.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.