Examen Rétrospectif

Laurence-Olivier M. Foisy

Université Laval

Question 1

La reproductibilité

Introduction

  • Contexte : Émergence des GML en recherche
  • Promesse : Analyse rapide et facile de données massives
  • Problème : Tension entre efficacité et reproductibilité

Question

Un élément fondamental de la recherche est la reproductibilité des analyses. Expliquez comment ce principe s’applique à la recherche proposée, en précisant les limites potentielles de la contribution de ces études.

Contexte Théorique

Réplicabilité dans différents contextes

Standardisation

Complexité & pluralisme

Défis Majeurs à la réplicabilité

  1. Complexité Contextuelle et Temporelle
  2. Ambiguïté Catégorielle
  3. Biais de Données et de Représentation
  4. Causalité Complexe

Défi 1 : Complexité Contextuelle

Défi 2 : Ambiguïté Catégorielle

Exemple Concret

Q : “Quel enjeu est le plus important pour vous?” R : “Que le gouvernement gère mieux le système de santé”

  • Catégorie Santé?
  • Catégorie Gouvernance?
  • Impact sur la reproductibilité des analyses

Défi 3 : Biais de Données

Performe moins bien dans les contextes non occidentaux

Défi 4 : Causalité Complexe

  • Identification des tendances != Compréhension causale (Yamin et al. 2024)
  • Rôle du chercheur essentiel dans l’interprétation (Fearon 1991)

Solutions Proposées

Satisfaire les standardisateurs

Satisfaire les pluralistes

Conclusion

  • Approche hybride nécessaire
  • GML comme outils d’augmentation, non de remplacement
  • Transparence dans le processus

Discussion

Question 2

Les biais dans les revues de littérature

Introduction

  • La science s’appuie sur les travaux antérieurs
  • Les revues de littérature sont essentielles
  • Problème: Biais systématiques dans la sélection et l’analyse

Question

La recherche en science politique nécessite une bonne compréhension des travaux antérieurs. On sait que cet effort de recensement est souvent associé à des biais. En quoi les GML peuvent-ils éviter ces écueils?

Défi Méthodologique - Biais de sélection

Angles Morts Persistants

  1. Biais intersectionnels (Weldon 2006)
  2. Biais occidentaux (O’Donnell 1993)
  3. Savoirs pratiques difficiles à formaliser (Scott 1999)
  4. Domination de l’anglais (Daoust, Gagnon, and Galipeau 2022)

Solution: Revues systématiques

Apport des GML

Automatisation

Aide au chercheur

Limites des GML

Solutions Proposées

  • Critères de sélection prédéfinis
  • Protocoles rigoureux
  • Approche hybride humain-machine
  • Diversification des sources de données
  • Utilisation de modèles open-source

Conclusion

  • GML comme outil d’amélioration, non de remplacement
  • Nécessité d’une supervision humaine
  • Vers des revues de littérature plus inclusives et rigoureuses

Discussion

Conclusion

  • La science est imparfaite.
  • Les méthodes sont imparfaites.
  • Le but est de les améliorer.

Bibliographie

Examen Rétrospectif

Abdin, Marah, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, et al. 2024. “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.” May 23, 2024. https://doi.org/10.48550/arXiv.2404.14219.
Alshami, Ahmad, Moustafa Elsayed, Eslam Ali, Abdelrahman EE Eltoukhy, and Tarek Zayed. 2023. “Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions.” Systems 11 (7): 351. https://www.mdpi.com/2079-8954/11/7/351.
Altena, A.j. van, R. Spijker, and S.d. Olabarriaga. 2019. “Usage of Automation Tools in Systematic Reviews.” Research Synthesis Methods 10 (1): 72–82. https://doi.org/10.1002/jrsm.1335.
Anderson, Benedict R. O’G. 1983. Imagined Communities: Reflections on the Origin and Spread of Nationalism. Rev. and extended ed., 13. impression. London: Verso.
Baack, Stefan. 2024. “A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl.” In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2199–2208. FAccT ’24. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3630106.3659033.
Boix, Carles. 1999. “Setting the Rules of the Game: The Choice of Electoral Systems in Advanced Democracies.” American Political Science Review 93 (3): 609–24. https://doi.org/10.2307/2585577.
Brady, Henry E., and David Collier, eds. 2010. Rethinking Social Inquiry. Diverse Tools, Shared Standards. 2nd ed. Maryland: Rowman & Littlefield Publishers.
Clark, Justin, Paul Glasziou, Chris Del Mar, Alexandra Bannach-Brown, Paulina Stehlik, and Anna Mae Scott. 2020. “A Full Systematic Review Was Completed in 2 Weeks Using Automation Tools: A Case Study.” Journal of Clinical Epidemiology 121 (May): 81–90. https://doi.org/10.1016/j.jclinepi.2020.01.008.
Cox, Gary W. 1997. Making Votes Count: Strategic Coordination in the World’s Electoral Systems. Political Economy of Institutions and Decisions. Cambridge, U.K. ; New York: Cambridge University Press.
Daoust, Jean-François, Alain-G. Gagnon, and Thomas Galipeau. 2022. “Language Matters: The Study of Canadian Politics Through an Exploration of Syllabi and Comprehensive Exams.” Canadian Journal of Political Science 55 (4): 897–915. https://doi.org/10.1017/S0008423922000749.
Della Porta, Donatella. 2001. “Social movements and new challenges to représentative democracy. . A perspective from Italy.” Politique européenne 4 (3): 73–103. https://doi.org/10.3917/poeu.004.0073.
Della Porta, Donatella, and Michael Keating, eds. 2008. Approaches and Methodologies in the Social Sciences: A Pluralist Perspective. Cambridge , New York: Cambridge University Press.
Downs, Anthony. 1957. An Economic Theory of Democracy. 1st ed. New York: Harper & Row.
Fearon, James D. 1991. “Counterfactuals and Hypothesis Testing in Political Science.” World Politics 43 (2): 169–95. https://doi.org/10.2307/2010470.
Geddes, Barbara. 1990. “How the Cases You Choose Affect the Answers You Get: Selection Bias in Comparative Politics.” Political Analysis 2 (January): 131–50. https://doi.org/10.1093/pan/2.1.131.
Gruber, Johannes B., and Maximilian Weber. 2024. “Rollama: An R Package for Using Generative Large Language Models Through Ollama.” April 11, 2024. http://arxiv.org/abs/2404.07654.
Hartmann, Jochen, Jasper Schwenzow, and Maximilian Witte. 2023. “The Political Ideology of Conversational AI: Converging Evidence on ChatGPT’s Pro-Environmental, Left-Libertarian Orientation.” SSRN Electronic Journal.
Horton, Nicholas J., Rohan Alexander, Micaela Parker, Aneta Piekut, and Colin Rundel. 2022. “The Growing Importance of Reproducibility and Responsible Workflow in the Data Science and Statistics Curriculum.” Journal of Statistics and Data Science Education 30 (3): 207–8. https://doi.org/10.1080/26939169.2022.2141001.
Jiang, Albert Q., Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, et al. 2023. “Mistral 7B.” October 10, 2023. https://doi.org/10.48550/arXiv.2310.06825.
Johnson, Rebecca L., Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. “The Ghost in the Machine Has an American Accent: Value Conflict in GPT-3.” March 15, 2022. http://arxiv.org/abs/2203.07785.
King, Gary, Robert O. Keohane, and Sidney Verba. 1994. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, N.J: Princeton University Press.
Marshall, Iain J., and Byron C. Wallace. 2019. “Toward Systematic Review Automation: A Practical Guide to Using Machine Learning Tools in Research Synthesis.” Systematic Reviews 8 (1): 163. https://doi.org/10.1186/s13643-019-1074-9.
Meskó, Bertalan. 2023. “Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial.” Journal of Medical Internet Research 25 (1): e50638. https://doi.org/10.2196/50638.
Meta. 2024. “Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date.” Meta AI. 2024. https://ai.meta.com/blog/meta-llama-3/.
Muka, Taulant, Marija Glisic, Jelena Milic, Sanne Verhoog, Julia Bohlius, Wichor Bramer, Rajiv Chowdhury, and Oscar H. Franco. 2020. “A 24-Step Guide on How to Design, Conduct, and Successfully Publish a Systematic Review and Meta-Analysis in Medical Research.” European Journal of Epidemiology 35 (1): 49–60. https://doi.org/10.1007/s10654-019-00576-5.
Naveed, Humza, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2023. “A Comprehensive Overview of Large Language Models.” December 27, 2023. http://arxiv.org/abs/2307.06435.
Neveu, Erik. 1996. Sociologie des mouvements sociaux. 4. éd. Repères 207. Paris: La Découverte.
Nori, Harsha, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. “Capabilities of GPT-4 on Medical Challenge Problems.” April 12, 2023. https://doi.org/10.48550/arXiv.2303.13375.
North, Douglass C., and Barry R. Weingast. 1989. “Constitutions and Commitment: The Evolution of Institutions Governing Public Choice in Seventeenth-Century England.” The Journal of Economic History 49 (4): 803–32. https://doi.org/10.1017/S0022050700009451.
O’Donnell, Guillermo. 1993. “On the State, Democratization and Some Conceptual Problems: A Latin American View with Glances at Some Postcommunist Countries.” World Development 21 (8): 1355–69. https://doi.org/10.1016/0305-750X(93)90048-E.
Olson, Mancur. 1965. The Logic of Collective Action: Public Goods and the Theory of Groups. 21. printing. Harvard Economic Studies 124. Cambridge, Mass.: Harvard Univ. Press.
OpenAI. 2022. “Is ChatGPT Biased? | OpenAI Help Center.” 2022. https://help.openai.com/en/articles/8313359-is-chatgpt-biased.
Parthasarathy, Venkatesh Balavadhani, Ahtsham Zafar, Aafaq Khan, and Arsalan Shahid. 2024. “The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities.” August 23, 2024. https://doi.org/10.48550/arXiv.2408.13296.
Sarmah, Bhaskarjit, Mingshu Li, Jingrao Lyu, Sebastian Frank, Nathalia Castellanos, Stefano Pasquali, and Dhagash Mehta. 2024. “How to Choose a Threshold for an Evaluation Metric for Large Language Models.” December 10, 2024. https://doi.org/10.48550/arXiv.2412.12148.
Scott, James C. 1999. Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed. Veritas paperback edition. Yale Agrarian Studies. New Haven, CT London: Yale University Press.
Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. 2023. “LLaMA: Open and Efficient Foundation Language Models.” February 27, 2023. http://arxiv.org/abs/2302.13971.
Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, et al. 2023. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” July 19, 2023. http://arxiv.org/abs/2307.09288.
Tsafnat, Guy, Paul Glasziou, Miew Keen Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera. 2014. “Systematic Review Automation Technologies.” Systematic Reviews 3 (1): 74. https://doi.org/10.1186/2046-4053-3-74.
Tsebelis, George. 1995. “Decision Making in Political Systems: Veto Players in Presidentialism, Parliamentarism, Multicameralism and Multipartyism.” British Journal of Political Science 25 (3): 289–325. https://doi.org/10.1017/S0007123400007225.
Weldon, S. Laurel. 2006. “The Structure of Intersectionality: A Comparative Politics of Gender.” Politics & Gender 2 (02). https://doi.org/10.1017/S1743923X06231040.
Yamin, Khurram, Shantanu Gupta, Gaurav R. Ghosal, Zachary C. Lipton, and Bryan Wilder. 2024. “Failure Modes of LLMs for Causal Reasoning on Narratives.” October 31, 2024. https://doi.org/10.48550/arXiv.2410.23884.
Zhao, James Xu, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Michael Qizhe Xie. 2023. “Automatic Model Selection with Large Language Models for Reasoning.” October 23, 2023. https://doi.org/10.48550/arXiv.2305.14333.