Artificial intelligence applications as a source of information on penis-lengthening and penis girth enhancement: ChatGPT 4o Plus vs. Gemini advanced
Aplicaciones de inteligencia artificial como fuente de información sobre alargamiento de pene y aumento del grosor del pene: ChatGPT 4o Plus vs. Gemini advanced

1. Introduction

The internet has become a frequently used tool for accessing health-related information, due to ease of access. Patients, particularly those who feel embarrassed to consult a doctor, often turn to online sources. Platforms, such as YouTube, Facebook, Instagram and Twitter, are commonly used as health information sources. Numerous studies have explored the accuracy of information provided by these platforms [1, 2, 3].

Today, artificial intelligence (AI) and large language models (LLMs) have rapidly transformed the healthcare sector, introducing both promising opportunities and notable challenges, particularly in the realms of patient education and early diagnosis [4]. Among the most widely used LLM systems is Chat Generative Pre-training Transformer (ChatGPT), launched by OpenAI (San Francisco, CA, USA) in November 2022. ChatGPT is a deep learning-based AI chatbot built on natural language processing models and designed to provide logical informative answers instantly by mimicking human language patterns [5]. Released in March 2023, Gemini AI, formerly known as Google Bard and based on the Pathways Language Model 2, is another chatbot similar to ChatGPT but distinguished by its real-time internet access and LaMDA (Language Model for Dialogue Applications) communication model [6].

The reliability and accuracy of AI-generated medical information play a crucial role in shaping patients’ decisions and their engagement with healthcare providers [7]. Advanced language models like ChatGPT and Gemini have the potential to offer valuable guidance in the management of urological disease [8, 9, 10], yet their effectiveness requires thorough evaluation. Assessing AI’s role in patient guidance, comparing its responses to established medical knowledge, and ensuring the dissemination of accurate information before clinical consultations are essential steps. Nevertheless, the growing integration of AI in medicine raises concerns regarding the credibility of its outputs and the influence these automated recommendations may have on patient expectations and clinical outcomes [11].

Although male sexual dysfunction, including hypogonadism, low sexual desire, hypoactive sexual desire disorder, erectile dysfunction, ejaculatory disorders, penile curvature, and priapism have also been investigated using ChatGPT [10, 12, 13, 14], penis-lengthening and penis girth enhancement have not been studied with AI. YouTube and social media videos addressing these topics show that the quality and reliability of such information are extremely poor [3, 15, 16].

This study aimed to determine the quality, reliability and readability of information provided by AI applications, specifically ChatGPT 4o Plus and Gemini Advanced, on penis-lengthening and penis girth enhancement for the first time in the literature.

2. Materials and methods

2.2 Evaluation of the responses

The responses provided by the AI were stripped of any identifiers such as AI name, font style, numbering or any other distinguishing markers. They were then converted into plain text without any indicators and presented to the evaluator in a completely random order. The evaluation was conducted blindly by an andrologist (EA) with more than 10 years of clinical experience and expertise in andrology, serving as secretary-general of the National Andrology Association and editor of its peer-reviewed journal. After the evaluation of all responses, another researcher (MG) recorded the evaluation results under main headings (Fig. 1).

Fig. 1.The GQS-R and GQS-U scores assigned to the responses provided by ChatGPT-4o Plus and Gemini advanced for questions related to penis size. GQS-R: Global Quality Score reliability; GQS-U: Global Quality Score usefulness.

2.4 Assessment of readability

The Flesch formula, most commonly used readability metric for evaluating the readability of written health information materials, was used. The Flesch Reading Ease (FRE) and the Flesch-Kincaid Reading Grade Level (FKRGL) calculate readability based on average sentence length (in terms of word count) and average word length (in terms of syllable count) (Fig. 2) [21]. Readability scores were calculated using https://readable.com, an online tool that allows the computation of FRE and FKGL scores. The AI-generated responses were stripped of headings and unnecessary spaces, converting them into plain text before the analysis. Scores obtained from the FRE range from 0 (unreadable) to 100 (very easy to read). The FKRGL provides a score corresponding to a grade level. In our study, the interpretation of the FRE and FKRGL scores was made according to Table 2. A lower FKRGL level and a higher FRE score were considered better readability. To reach individuals with low literacy levels, the standard difficulty level was determined as FRE scores of 61–70 points and FKRGL grade levels 8–9, following the literature [22].

Fig. 2.Calculation of the Flesch Reading Ease (FRE) and the Flesch-Kincaid Reading Grade Level (FKRGL).

Table 2.Interpretation of flesch reading ease scores.

Reading Difficulty	FRE score	Estimated FKRGL reading grade level
Very easy	91–100	Grade 5 or 11 year old
Easy	81–90	Grade 6
Fairly easy	71–80	Grade 7
Standard	61–70	Grade 8–9 or 13–15 year old
Fairly difficult	51–60	Grade 10–12
Difficult	31–50	Grade 13–16
Very difficult	0–30	College graduate
FRE: Flesch Reading Ease; FKRGL: Flesch-Kincaid Reading Grade Level.

3. Results

A total of 62 questions were asked of both AI applications. Of these, 17 were general questions about penis size, 24 were related to penis lengthening, and 21 were about penis girth enhancement. As a result, the GQS-R (reliability) and GQS-U (usefulness) scores of ChatGPT 4o Plus were significantly higher in all three categories of general penis size, penis-lengthening, and penis girth enhancement compared to Gemini Advanced (GQS-R, p < 0.001; GQS-U, p < 0.001) (Table 3). Additionally, ChatGPT 4o Plus tended to display greater consistency in providing the same answers when the questions were repeated, compared to Gemini Advanced (p = 0.044). A Comparison of the FKGRL, the FRE, reproducibility and, response times for penis-related question types between ChatGPT 4o Plus and Gemini Advanced is presented in Table 4. Regarding response times, ChatGPT 4o Plus was faster than Gemini Advanced, and this difference was significant (p = 0.021). Gemini Advanced had higher FKGRL scores than ChatGPT 4o Plus. This difference was significant for penis-lengthening questions and across all questions (p < 0.001). However, Gemini Advanced tended to show higher FRE scores in categories other than penis-lengthening, although this difference was not significant. In conclusion, while the readability of responses provided by both AI bots was generally challenging, Gemini Advanced had a slightly better readability level than ChatGPT 4o Plus. Both applications require a reading level equivalent to someone above the age of 15. The response time of ChatGPT 4o Plus was significantly shorter than that of Gemini Advanced (p = 0.021).

4. Discussion

The present study is the first to demonstrate that AI provided high-quality and useful information on penis-lengthening and penis girth enhancement, yet the readability and comprehensibility of this information remained challenging. Furthermore, this study highlights the superiority of ChatGPT 4o Plus over Gemini Advanced in terms of reliability, usefulness, reproducibility and response speed, while Gemini Advanced outperformed ChatGPT 4o Plus in readability and comprehensibility.

ChatGPT, trained on extensive textual data, effectively captures the nuances and complexities of human language. It provides instant, logical, informative, contextually relevant and appropriate responses to queries [5]. Given the ever-increasing body of medical knowledge, it is inevitable that the healthcare field will embrace this technology, as it is characterized by large amounts of textual data and complex clinical applications. Indeed, physicians and patients have shown interest in ChatGPT due to its ability to deliver accurate and prompt answers on a wide range of topics [23]. Gemini AI, formerly known as Google Bard, is based on the Pathways Language Model 2 and similarly engages in conversational interactions in response to human input [6].

The quality and reliability of medical information provided by Gemini AI and ChatGPT have been investigated, and the two systems have been compared [24, 25, 26]. Andrological disorders, encompassing male reproductive and sexual health, are among the topics that have been explored. Şahin et al. [14] entered the most frequently searched terms related to premature ejaculation (PE) into ChatGPT, evaluating the generated responses for readability. They reported that the text produced by ChatGPT was of questionable quality, significantly difficult to read and comprehend and of a literary level that could only be understood by highly educated individuals. Similarly, in our study, the readability and comprehensibility of the ChatGPT responses were challenging. However, our findings differed by highlighting the high quality and reliability of the responses.

In terms of readability, Gemini Advanced had higher FKRGL scores, making its responses more complex and suited for highly literate users, it also showed better FRE scores in some areas, improving accessibility. In contrast, ChatGPT 4o Plus had lower FKRGL scores, making its responses easier to understand for a broader audience but potentially reducing comprehensiveness. These findings highlight the need for AI models to balance readability with medical depth to serve diverse user populations effectively. The fact that readability differences were significant in some categories but not in others suggests that the generalizability of these findings may be limited. The variations across different types of questions indicate that AI models do not perform uniformly across all topics, which should be considered when interpreting the results.

Yigman et al. [10] identified the four most searched keywords for sexual dysfunction, erectile dysfunction and PE on Google and grouped them under headings, such as definitions, causes, pharmacological treatments and general treatments. They reported that ChatGPT was an acceptably useful and reliable source for acquiring information about sexual dysfunction and its most common causes, erectile dysfunction and PE. The scales used to characterize the usefulness and reliability of information provided by ChatGPT were consistent and reliable. However, they noted questionable consistency among evaluators regarding the usefulness of PE information. Ergin et al. [13] recorded frequently asked questions on health websites, urology association websites, and social media platforms regarding topics, such as male hypogonadism, erectile dysfunction (ED), ejaculatory disorders, penile curvature, penile size abnormalities, priapism and male infertility. They converted the strongly recommended guidelines of the EAU sexual and reproductive health section into questions and presented them to ChatGPT-3.5 and 4o. The accuracy rates for the responses were 85.2% for frequently asked questions in ChatGPT-3.5 and 88.8% in ChatGPT 4o. The accuracy rates for questions derived from the guidelines were 81.5% for ChatGPT-3.5 and 88.9% for ChatGPT 4o. However, the evaluation of penile size abnormalities was limited to questions like “What is the normal length and girth of the penis?” and “Is penis size important to women?”. Our study went further by asking 14 questions about penile size, allowing for a more in-depth assessment. Similar findings were reported by Caglar et al. [12], who entered frequently asked questions from hospital websites, YouTube, and Instagram into ChatGPT using the EAU guideline recommendations on andrological topics, such as male hypogonadism and ED. They reported that ChatGPT answered 87.9% of questions accurately and sufficiently, 9.3% accurately but insufficiently, and no questions incorrectly. The highest accuracy rates were observed for ejaculatory disorders, penile curvature, and male hypogonadism.

From ancient times, the penis has been associated with power and viewed as a symbol of masculinity. The societal idealization of larger penises can lead to significant psychological distress for patients. Numerous surgical and non-surgical methods have been developed to improve penile length and girth [17, 18, 19]. While patients often research this topic on social media, studies evaluating the quality and reliability of these platforms are limited. Seranio et al. [3] assessed the top 100 most-viewed YouTube videos on penis-lengthening, and most videos discussed non-surgical methods, such as penile traction devices (19.2%) and surgical methods (65.1%). However, the overall quality and reliability of the videos were poor. Videos created by physicians were rated higher in quality. Another study investigating the relationship between the quality of YouTube videos on penis-lengthening surgery and the academic profiles of surgeons reported a significant correlation between video quality and the surgeon’s h-index and total publication count [27]. Videos containing academic knowledge were of higher quality. In our study, we tested the grasp of AI for academic knowledge by asking questions derived from the EAU guidelines and review articles. Çağlayan et al. [28] analyzed the reliability of Instagram posts with the hashtag #penislengthening and the impact of such posts on young adult men’s perception of their penis size. As results, only six of 1000 posts (1%) contained reliable information. During the survey phase of the study, participants reported a significant decline in genital self-image (p < 0.001) and a significant increase in media exposure (p < 0.001).

The growing presence of AI in scientific research has sparked concerns about its potential to generate inaccurate or misleading information. While AI offers numerous advantages, the risk of producing incorrect diagnoses or recommendations must be carefully considered [11]. The performance of chatbots is influenced by various factors, including data availability, linguistic structures, and contextual differences [4]. To ensure accuracy and reliability, multilingual chatbots must adapt to these variations when processing and delivering medical information.

If AI models are trained with outdated or incomplete datasets, they may generate incorrect medical advice and fail to identify rare diseases, leading to significant health risks. AI-powered chatbots cannot replace professional medical evaluations, as they rely on user-input symptoms rather than comprehensive clinical assessments. A major limitation is their inability to account for individual health conditions such as chronic illnesses, allergies, or psychiatric disorders [7].

To enhance the reliability of AI-driven healthcare tools, continuous updates with the latest medical literature and clinical research are essential. Additionally, establishing oversight mechanisms that verify the accuracy of medical information in chatbot responses can significantly improve their effectiveness. Implementing such safeguards will help address existing quality concerns and ensure that AI applications in healthcare remain a valuable and trustworthy resource.

This study had some limitations. The findings were based solely on data available during the study period. As AI continues to learn and evolve, responses to the same questions may vary over time, due to updates and changes in training data. The evaluation of responses from ChatGPT inherently introduced subjectivity to the study, as it relied on human judgment. Additionally, questions were asked only in one language, and the quality of responses may have differed in other languages.

5. Conclusions

The findings of this study suggest that LLMs have the potential to provide reliable information in the field of penile lengthening and penile girth enhancement. Artificial intelligence models can serve as an easily accessible source of information for patients, yet the balance between the accuracy, reliability, and readability of their responses must be carefully assessed. Our study provides valuable insights for healthcare professionals, patients and AI developers regarding the evaluation of AI-generated health information.

Notably, ChatGPT 4o Plus demonstrated superiority in reproducibility, quality, reliability, usefulness and response speed, making it a potential tool for clinical information support. In contrast, Gemini Advanced exhibited better readability and comprehensibility, suggesting its potential use in patient education materials. However, both models need further development to improve the comprehensibility of their responses. For AI systems to be more widely adopted in the healthcare field, their content must be optimized for better readability. Future research should focus on integrating AI models into medical information dissemination processes and ensuring that patients can easily understand the information provided.

Availability of data and materials

The data supporting the findings of the present study may be found at https://doi.org/10.5281/zenodo.14645247.

Author contributions

ÇÖ and EA—designed the research study; wrote the manuscript. ÇÖ—performed the research. AA—provided help and advice on methodology and resources. MG—analyzed the data. All authors read and approved the final manuscript. All authors contributed to editorial changes in the manuscript.

Ethics approval and consent to participate

Not applicable.

Acknowledgment

Not applicable.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at https://files. intandro.com/files/article/1972591395783753728/attachment/Supplementary%20material.docx.