Using sentiment analysis to differentiate bots and humans in the dissemination of scientific publications on COVID-19 on social media X

a study with ChatGPT 3.5 and Gemini 1.5 Flash

Authors

DOI:

https://doi.org/10.5195/biblios.2025.1297

Keywords:

Sentiment analysis, ChatGPT, Gemini, X (Twitter), Bots

Abstract

Objective. This study aims to investigate the application of sentiment analysis to differentiate between automated accounts (bots) and human users in the dissemination of scientific publications about COVID-19 on the social network X. To achieve this, the research compares the effectiveness of Large Language Model (LLM) tools, specifically ChatGPT 3.5 and Gemini 1.5 Flash, in classifying the sentiment polarity expressed in posts about a scientific article. The study seeks to understand the performance differences between these tools, evaluate their effectiveness in polarity classification, and identify sentiment patterns that best distinguish bots from human users in the context of scientific dissemination. Method. This study begins with the collection of a sample of posts from X that mentioned the analyzed publication. The posts were collected using Python 3.12 and the Beautiful Soup 4.12 library within the Google Colab environment, resulting in a dataset of 9,792 posts and 5,601 unique profiles.  In the second stage, these profiles were compared to a previously classified dataset of bots and human users. To enhance classification reliability, a subsequent manual reclassification was performed on 41 accounts that had posted more than four times (third stage), identifying 20 as bots and 21 as human users. These accounts generated a total of 3,493 posts, which were then subjected to sentiment polarity classification (fourth stage) using ChatGPT 3.5 and Gemini 1.5 Flash. The classification followed a standardized prompt to categorize sentiments as positive, negative, or neutral and was applied in batches of 100 posts due to token limitations of the tools. In the fifth stage, 315 of the analyzed posts was manually classified for validation. Results. The analysis of 3,493 posts about the scientific article on X revealed a predominance of negative sentiments (92.3%), with neutral posts (6.2%) and positive ones (0.6%) being less frequent (0.9% were unidentified). The AI tools ChatGPT 3.5 and Gemini 1.5 Flash showed similar performance in classifying negative sentiments, but discrepancies emerged in 315 posts, with ChatGPT achieving 85% accuracy in posts that Gemini failed to classify. Bots exhibited greater emotional variability and were more critical of scientific dissemination, whereas human posts tended to be more neutral and consistent, highlighting relevant differences for bot detection. Conclusions. Sentiment analysis performed by ChatGPT and Gemini highlights the capability of these tools to classify social media posts related to scientific articles, revealing distinct patterns between bots and human accounts. Bots tend to generate more polarized and predominantly negative content, while humans exhibit a greater diversity of sentiments, with a balance between negative, neutral, and some positive posts. Although ChatGPT proved to be more effective in scenarios with limited contextual data or metadata to accurately assess a text’s emotional polarity, the study suggests that a more comprehensive analysis is needed to refine these tools and deepen the understanding of interactions between human and bot accounts.

Author Biographies

Danielle Pompeu Noronha Pontes, University of Brasília

She has a degree in Data Processing from the Federal University of Amazonas (1996) and a master's degree in Digital Systems from the University of São Paulo (2012). She completed a lato sensu postgraduate course in Computer Science at the Federal University of Ceará (2002). She is currently a professor at the Amazonas State University. She is a student in the PhD program in Information Science at the University of Brasilia in the areas of Altmetrics and Artificial Intelligence.

João de Melo Maricato, University of Brasília

Ph.D. in Information Science from the School of Communications and Arts of the University of São Paulo - ECA/USP (2010). BA in Library and Information Science from the Federal University of São Carlos - UFSCar (2002). Professor of Library Science at the Faculty of Information Science (FCI) of the University of Brasilia (UnB). Professor in the Postgraduate Program in Information Science at UnB (PPGCinf).

References

Anwar, A., & Yaqub, U. (2020). Bot detection in Twitter landscape using unsupervised learning. In The 21st Annual International Conference on Digital Government Research 2020 (pp. 329–330). ACM. https://dl.acm.org/doi/10.1145/3396956.3401801

Arredondo, L. (2018). A study of altmetrics using sentiment analysis. Honors Capstones, 70, 1–6. https://huskiecommons.lib.niu.edu/studentengagement-honorscapstones/70/

Babur, Z., Bekdemir, U., Sen, A., Carkit, S. O., Genc, O., Gulcu, A., Gumustas, C., & Soyak, E. G. (2023). Unmasking Twitter bots: Feature engineering and machine learning for bot account identification. In Innovations in Intelligent Systems and Applications Conference, ASYU 2023 (pp. 16–21). IEEE. https://doi.org/10.1109/ASYU58738.2023.10296706

Bär, D., Calderon, F., Lawlor, M., Licklederer, S., Totzauer, M., & Feuerriegel, S. (2023). Analyzing social media activities at Bellingcat. In Proceedings of the 15th ACM Web Science Conference 2023 (pp. 163–173). ACM. https://dl.acm.org/doi/10.1145/3578503.358360

Barreto, S., Moura, R., Carvalho, J., Paes, A., & Plastino, A. (2023). Sentiment analysis in tweets: An assessment study from classical to modern word representation models. Data Mining and Knowledge Discovery, 37(1), 318–380. https://doi.org/10.1007/s10618-022-00853-0

Bello, B. S., & Heckel, R. (2019). Analyzing the behaviour of Twitter bots in post-Brexit politics. In 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS) (pp. 61–66). IEEE. https://doi.org/10.1109/SNAMS.2019.8931874

Bello, B. S., Heckel, R., & Minku, L. (2018). Reverse engineering the behaviour of Twitter bots. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS) (pp. 27–34). IEEE. https://doi.org/10.1109/SNAMS.2018.8554675

Breve, B., Caruccio, L., Cirillo, S., Deufemia, V., & Polese, G. (2024). Analyzing the worldwide perception of the Russia-Ukraine conflict through Twitter. Journal of Big Data, 11(1). https://doi.org/10.1186/s40537-024-00921-w

Buscemi, A., & Proverbio, D. (2024). ChatGPT vs Gemini vs LLaMA on multilingual sentiment analysis. arXiv, 1–11. https://doi.org/10.48550/arXiv.2402.01715

Cai, M., Luo, H., Meng, X., Cui, Y., & Wang, W. (2023). Network distribution and sentiment interaction: Information diffusion mechanisms between social bots and human users on social media. Information Processing & Management, 60(2), 103197. https://doi.org/10.1016/j.ipm.2022.103197

Çetinkaya, Y. M., Toroslu, İ. H., & Davulcu, H. (2020). Developing a Twitter bot that can join a discussion using state-of-the-art architectures. Social Network Analysis and Mining, 10(1), 1–21. https://doi.org/10.1007/s13278-020-00665-4

Chawla, V., & Kapoor, Y. (2023). A hybrid framework for bot detection on Twitter: Fusing digital DNA with BERT. Multimedia Tools and Applications, 82(20), 30831–30854. https://doi.org/10.1007/s11042-023-14730-5

Chen, H., Zhu, Z., Qi, F., Ye, Y., Liu, Z., Sun, M., & Jin, J. (2021). Country image in COVID-19 pandemic: A case study of China. IEEE Transactions on Big Data, 7(1), 81–92. https://doi.org/10.1109/TBDATA.2020.3023459

Darling, E., Shiffman, D., Côté, I., & Drew, J. (2013). The role of Twitter in the life cycle of a scientific publication. Ideas in Ecology and Evolution, 6, 32–43. https://doi.org/10.4033/iee.2013.6.6.f

Galgoczy, M. C., Phatak, A., Vinson, D., Mago, V. K., & Giabbanelli, P. J. (2022). (Re)shaping online narratives: When bots promote the message of President Trump during his first impeachment. PeerJ Computer Science, 8, 1–26. https://doi.org/10.7717/peerj-cs.947

Gatkal, S., Panjwani, D., Barhate, S., Mangla, R., & Kazi, F. (2021). Community detection and impact of bots on sentiment polarity of Twitter networks. 2021 Asian Conference on Innovation in Technology (ASIANCON 2021), 1–6. https://doi.org/10.1109/ASIANCON51346.2021.9544691

Graham, T., Bruns, A., Angus, D., Hurcombe, E., & Hames, S. (2021). #IStandWithDan versus #DictatorDan: The polarised dynamics of Twitter discussions about Victoria’s COVID-19 restrictions. Media International Australia, 179(1), 127–148. https://doi.org/10.1177/1329878X20981780

Halevi, G., & Schimming, L. (2018). An initiative to track sentiments in altmetrics. Journal of Altmetrics, 1(1), 1–7. https://doi.org/10.29024/joa.1

Hassan, S. U., Aljohani, N. R., Tarar, U. I., Safder, I., Sarwar, R., Alelyani, S., & Nawaz, R. (2023). Exploiting tweet sentiments in altmetrics large-scale data. Journal of Information Science, 49(5), 1229–1245. https://doi.org/10.1177/01655515211043713

Hassan, S. U., Saleem, A., Soroya, S. H., Safder, I., Iqbal, S., Jamil, S., Bukhari, F., Aljohani, N. R., & Nawaz, R. (2021). Sentiment analysis of tweets through Altmetrics: A machine learning approach. Journal of Information Science, 47(6), 712–726. https://doi.org/10.1177/0165551520930917

Ibrahim, M., Abdillah, O., Wicaksono, A. F., & Adriani, M. (2015). Buzzer detection and sentiment analysis for predicting presidential election results in a Twitter nation. In 2015 IEEE International Conference on Data Mining Workshop (ICDMW) (pp. 1348–1353). IEEE. https://doi.org/10.1109/ICDMW.2015.113

Inuwa-Dutse, I., Bello, B. S., & Korkontzelos, I. (2018). Lexical analysis of automated accounts on Twitter. In Proceedings of the International Conferences on WWW/Internet 2018 and Applied Computing 2018 (pp. 75–82). https://doi.org/10.48550/arXiv.1812.07947

Kraaijeveld, O., & De Smedt, J. (2020). The predictive power of public Twitter sentiment for forecasting cryptocurrency prices. Journal of International Financial Markets, Institutions and Money, 65, 101188. https://doi.org/10.1016/j.intfin.2020.101188

Kudugunta, S., & Ferrara, E. (2018). Deep neural networks for bot detection. Information Sciences, 467, 312–322. https://doi.org/10.1016/j.ins.2018.08.019

Liu, H., Chatterjee, I., Zhou, M., Lu, X. S., & Abusorrah, A. (2020). Aspect-based sentiment analysis: A survey of deep learning methods. IEEE Transactions on Computational Social Systems, 7(6), 1358–1375. https://doi.org/10.1109/TCSS.2020.3033302

Liu, X. (2019). A big data approach to examining social bots on Twitter. Journal of Services Marketing, 33(4), 369–379. https://doi.org/10.1108/JSM-02-2018-0049

Mouronte-López, M. L., Gómez Sánchez-Seco, J., & Benito, R. M. (2024). Patterns of human and bots behaviour on Twitter conversations about sustainability. Scientific Reports, 14(1), 1–14. https://doi.org/10.1038/s41598-024-52471-z

Mustafaraj, E., & Metaxas, P. T. (2011). What edited retweets reveal about online political discourse. In Proceedings of the 5th AAAI Conference on Analyzing Microtext 2011 (pp. 38–43). AAAI Press. https://dl.acm.org/doi/10.5555/2908630.2908637

Pastor-Galindo, J., Zago, M., Nespoli, P., Bernal, S. L., Celdran, A. H., Perez, M. G., Ruiperez-Valiente, J. A., Perez, G. M., & Marmol, F. G. (2020). Spotting political social bots in Twitter: A use case of the 2019 Spanish general election. IEEE Transactions on Network and Service Management, 17(4), 2156–2170. https://doi.org/10.1109/TNSM.2020.3031573

Pontes, D., & Maricato, J. (2023a). Dataset with classification of accounts that disclose scientific articles on Twitter between bot, human and cyborg through Botometer [Dataset]. Zenodo. https://zenodo.org/records/8137161

Pontes, D., & Maricato, J. (2023b). Classificação de contas do Twitter através do Botometer. In Encontro Nacional de Pesquisa em Informação 2023, Aracaju, SE. Anais [...]. Aracaju, SE. https://enancib.ancib.org/index.php/enancib/xxxiiienancib/paper/view/1957

Rane, N., Choudhary, S., & Rane, J. (2024). Gemini versus ChatGPT: Applications, performance, architecture, capabilities, and implementation. Journal of Applied Artificial Intelligence, 5(1), 69–93. https://doi.org/10.48185/jaai.v5i1.1052

Silva, M. E. M., & Serrano, P. H. S. M. (2023). Análise de sentimentos em textos de redes sociais: Uma comparação entre o ChatGPT e métodos tradicionais. https://doi.org/10.5902/2316882X84828

Suarez-Lledo, V., & Alvarez-Galvez, J. (2022). Assessing the role of social bots during the COVID-19 pandemic: Infodemic, disagreement, and criticism. Journal of Medical Internet Research, 24(8), 1–12. https://doi.org/10.2196/36085

Tavazoee, F., Buscaldi, D., Mola, F., & Conversano, C. (2020). Empowering detection of malicious social bots and content spammers on Twitter by sentiment analysis. Electronic Journal of Applied Statistical Analysis, 13(2), 375–389. https://doi.org/10.1285/i20705948v13n2p375

Uyheng, J., & Carley, K. M. (2020). Bot impacts on public sentiment and community structures: Comparative analysis of three elections in the Asia-Pacific. In Lecture Notes in Computer Science (Vol. 12268, pp. 12–22). Springer. https://doi.org/10.1007/978-3-030-61255-9_2

Varol, O., Ferrara, E., Davis, C. A., Menczer, F., & Flammini, A. (2017). Online human-bot interactions: Detection, estimation, and characterization. Proceedings of the 11th International Conference on Web and Social Media (ICWSM 2017), 280–289. https://doi.org/10.1609/icwsm.v11i1.14871

Velayutham, T., & Tiwari, P. K. (2017). Bot identification: Helping analysts for right data in Twitter. In 2017 3rd International Conference on Advances in Computing, Communication & Automation (ICACCA) (Fall) (pp. 1–5). IEEE. https://doi.org/10.1109/ICACCAF.2017.8344722

Wu, J., Teng, E., & Cao, Z. (2022). Twitter bot detection through unsupervised machine learning. In Proceedings - 2022 IEEE International Conference on Big Data (Big Data 2022) (pp. 5833–5839). IEEE. https://doi.org/10.1109/BigData55660.2022.10020983

Yousefinaghani, S., Dara, R., Mubareka, S., Papadopoulos, A., & Sharif, S. (2021). An analysis of COVID-19 vaccine sentiments and opinions on Twitter. International Journal of Infectious Diseases, 108, 256–262. https://pubmed.ncbi.nlm.nih.gov/34052407/

Zhang, L., Gou, Z., Fang, Z., Sivertsen, G., & Huang, Y. (2023). Who tweets scientific publications? A large-scale study of tweeting audiences in all areas of research. Journal of the Association for Information Science and Technology, 74(13), 1485–1497. https://doi.org/10.1002/asi.24830

Downloads

Published

2025-07-03

How to Cite

Pontes, D. P. N., & Maricato, J. de M. (2025). Using sentiment analysis to differentiate bots and humans in the dissemination of scientific publications on COVID-19 on social media X: a study with ChatGPT 3.5 and Gemini 1.5 Flash. Biblios Journal of Librarianship and Information Science, (esp.), e001. https://doi.org/10.5195/biblios.2025.1297

Issue

No. esp. (2025)

Section

Original

License

Copyright (c) 2025 Danielle Pompeu Noronha Pontes, João de Melo Maricato

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

  1. The Author retains copyright in the Work, where the term “Work” shall include all digital objects that may result in subsequent electronic publication or distribution.
  2. Upon acceptance of the Work, the author shall grant to the Publisher the right of first publication of the Work.
  3. The Author shall grant to the Publisher and its agents the nonexclusive perpetual right and license to publish, archive, and make accessible the Work in whole or in part in all forms of media now or hereafter known under a Creative Commons Attribution 4.0 International License or its equivalent, which, for the avoidance of doubt, allows others to copy, distribute, and transmit the Work under the following conditions:
    1. Attribution—other users must attribute the Work in the manner specified by the author as indicated on the journal Web site;
    with the understanding that the above condition can be waived with permission from the Author and that where the Work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.
  4. The Author is able to enter into separate, additional contractual arrangements for the nonexclusive distribution of the journal's published version of the Work (e.g., post it to an institutional repository or publish it in a book), as long as there is provided in the document an acknowledgement of its initial publication in this journal.
  5. Authors are permitted and encouraged to post online a prepublication manuscript (but not the Publisher’s final formatted PDF version of the Work) in institutional repositories or on their Websites prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. Any such posting made before acceptance and publication of the Work shall be updated upon publication to include a reference to the Publisher-assigned DOI (Digital Object Identifier) and a link to the online abstract for the final published Work in the Journal.
  6. Upon Publisher’s request, the Author agrees to furnish promptly to Publisher, at the Author’s own expense, written evidence of the permissions, licenses, and consents for use of third-party material included within the Work, except as determined by Publisher to be covered by the principles of Fair Use.
  7. The Author represents and warrants that:
    1. the Work is the Author’s original work;
    2. the Author has not transferred, and will not transfer, exclusive rights in the Work to any third party;
    3. the Work is not pending review or under consideration by another publisher;
    4. the Work has not previously been published;
    5. the Work contains no misrepresentation or infringement of the Work or property of other authors or third parties; and
    6. the Work contains no libel, invasion of privacy, or other unlawful matter.
  8. The Author agrees to indemnify and hold Publisher harmless from Author’s breach of the representations and warranties contained in Paragraph 6 above, as well as any claim or proceeding relating to Publisher’s use and publication of any content contained in the Work, including third-party content.

Revised 7/16/2018. Revision Description: Removed outdated link. 

Similar Articles

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 > >> 

You may also start an advanced similarity search for this article.