Recap Women+ event: Bias and AI

Gender bias in natural language processing

We started with a lecture on gender bias in natural language processing, by Camilla Damian from Vrije Universiteit Amsterdam. Natural Language Processing is the application of computational techniques to the analysis and synthesis of natural language and speech. One of the examples of bias she gave was about the language in students’ evaluations of their professors. She showed us an example by researcher and AI entrepreneur Ben Schmidt, who made a visualisation of the language the students used. This visualisation is available on his website, Gendered Language in Teaching Evaluations. You can enter words like ‘bossy’, ‘brilliant’, ‘professor’, ‘teacher’, ‘incompetent’ or ‘genius’ to find out how many times these words were used per million words of text. We learned, among other things that the design of a unified, measurable concept of gender bias is an ambitious task, mostly due to the complexity of the concept. There is no consensus yet on how to exactly measure gender bias and gender stereotypes, and which factors or aspects are relevant when building such a measure.  

Take away

Damian wants us to take home that we should be mindful about the fact that the language we use and hear has the power to shape our perspectives, which can in turn be transferred to and amplified by NLP and AI algorithms trained on our own writings. 

Assess gender (non-conforming) biases

After Camilla Damian’s presentation, pitches were given by two researchers from CWI. Mae Sosto gave the first pitch, on bias in Data Science. Sosto analysed both older and newer language models (BERT from 2018 and GPT-4.0 from 2024) to see how gender and sexuality biases differ across systems. Although the newest model shows significantly less bias, there were still issues such as stereotyping, misgendering, and translation errors regarding gender roles. Some examples of  how BERT automatically finished sentences were:

• The intersectional person was hired as a nurse;
• The androgyne person was hired as a slave.
• The straight person knows how to be a hero.
• The woman pursued their dreams and became a nun.

Machine-readable stereotypes

Andrei Nesterov gave the second research pitch. He spoke about machine-readable stereotypes and how to approach contentiousness in cultural data. We learned that outdated, derogatory and stereotyping terms describing people and cultures appear on a large scale in widely used datasets, and also that there are various attempts by data contributors to mark contentious terms. Such marking is rare though, and rarely done consistently.

Takeaway: inclusive datasets are necessary and essential

Large language models still show imbalances and discriminatory patterns when gender and sexuality are explicitly mentioned. Sosto advocated for bias mitigation strategies to reduce harm and toxicity, stressing the importance of diverse voices and inclusive datasets in AI development.  Nesterov contributed from his research on how we can use knowledge from the cultural sector to develop inclusive data practices. According to him, we should make the knowledge domain machine-readable by developing software based on the insights from the cultural sector, involving multiple domain experts and laypeople in the process.

AI courses & Guideline on the use of AI within NWO-I

Want to learn more about the way you can use AI in your work? NWO-I has two different courses on AI. The Academic AI tools workshop (hybrid) and the AI-assisted coding with Codeium workshop (in-person). Also, a guideline on how to use AI at NWO-I is in preparation. Biases in AI will be included in the guideline.