Chatbots, search engines, and the sealing of knowledges

AI algorithms based on machine learning (ML) have already been used in search engines for years. They determine for example in which order search results are arranged and which advertisements are shown to which users in which order. A newer development is the integration of AI algorithms for language processing, so called language models (LMs) or large language models (LLMs), which are trained on large quantities of text and consist of a neural network with a very large number of parameters, into search engines. They have been employed to process user inquiries and to extract which information a user is looking for (Nayak 2019).² The recent development of integrating LLMs for language generation, i.e. for producing direct textual responses to online search queries goes an important step further as it leads to LMs being used not only for query understanding but also for answer generation. This development, of LLMs being used to auto-generate textual answers to user input in search engines, is the current focus of search engine development and the focus of this article. To indicate that this paper specifically refers to the automated generation of textual answers based on user input—and not only the sense-making of user input—the term ‘chatbot’ is used.

Another term is central to this paper: knowledges. By using the term knowledges (in plural), I situate my research in a specific theoretical tradition. Since the beginning of the 1980s, feminist theorists, philosophers, writers and activist increasingly turned to questions of what constitutes knowledges, how knowledges are produced, and which and whose knowledges are considered to be knowledge (Anderson 2020). These questions have a political dimension as they are intertwined with the question who has—and who has not—the power to define common paradigms and knowledges in a society. Feminist epistemologists and theorists, with this political understanding of knowledge, divert from dominant conception of knowledge as impartial, neutral and detached from its context of ‘discovery’ (Harding 1986; McLaren 2002; Stanley and Wise 2013; Traweek 1988; Waldby 1995). More than just diverting from it, they pose a critique on visions of knowledge as ‘objective’, which posit that knowledge is independent from the person formulating knowledge claims and therefore disembodied (Anderson 2020). This, critical epistemologists claim, lacks an awareness of the inherent political nature of knowledge, as the question of which and whose knowledge is considered knowledge is always also a question of power. It matters who produces knowledges and whose knowledge is regarded as knowledges.

Donna Haraway (1988) prominently wrote about knowledges and possibilities of knowing. In her seminal paper Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective from 1988, Haraway formulated her theory of situated knowledges. The theory holds that knowledges are always situated and partial. There are no knowledges that are detached from the person who formulates them, and no knowledges that are independent of the respective cultural, historical, and social context in which they are produced. Turning against dominant narratives and understandings of objective knowledge as independent from the person making knowledge claims, Haraway (1988, p. 583) positions herself and her theory in opposition to what she terms the god trick. The god trick is the claim to see “everything from nowhere” (Haraway 1988, p. 581), the already mentioned dominant idea that there are impartial, de-situated, disembodied, whole truths and knowledges. Instead, Haraway argues that knowledges are always produced by humans, may it be through scientific experiments, through try and error, through thought, or through experience. We see, hear and experience with our bodies. Bodies shape which experiences we make and thus what we know. Only with our bodies can we come to formulate knowledges. We are perceived by others through our bodies and the knowledge that is accepted as knowledge within society often depends on which body a person has. A person who is Black will be perceived differently when making knowledge claims than a white person (and is less likely to be believed), just as there are differences in credibility assigned to people of different genders and different bodily abilities. At the same time, people who are racialized and gendered make different experiences within society than people who pass as unmarked, leading to them carrying different knowledges. With that, knowledges are always essentially partial, as humans have different experiences, different pre-knowledges, different bodies. Feminist objectivity, then, in Haraway’s understanding, “is about limited location and situated knowledge, not about transcendence and splitting of subject and object”.

With this, she moves away both from ideas of knowledge as independent of the person who carries it and from previous ideas of subjugated knowledges which Harding (1986, p. 28) proposed. Theories of subjugated knowledge claim that it is especially the knowledges of subjugated and marginalized people that need to be appreciated and listened to. While Haraway agrees that subjugated knowledges should be acknowledged and listened to, she does not consider them as inherently more important than others. Rather, it is crucial is to appreciate and acknowledge the value of partial perspectives which are grounded in their own situatedness. This necessarily includes an appreciation of a “view from a body, always a complex, contradictory, structuring, and structured body, versus the view from above, from nowhere, from simplicity” (Haraway 1988, p. 589).

While this may at first sound like there are only individual knowledges, this is not was Haraway has in mind. Situated Knowledges, on the very contrary, allow for “the joining of partial views and halting voices into a collective subject position that promises a vision of the means of ongoing finite embodiment, of living within limits and contradictions—of views from somewhere” (Haraway 1988, p. 590). Knowledges are situated and there is a plurality of different knowledges. However, importantly, they can and do join into knowledge collectives. This allows for a solidarity and collective knowledges while avoiding the god trick.

I use the term knowledges in the plural to show the always already existing plurality of (possible) knowledges which are produced by embodied subjects, always partial, always situated, yet joining and forming coalitions. They develop in certain conditions and structure realities in powerful ways. Knowledges are something a person knows. It entails an understanding of a phenomenon and the ability to apply an information to something else or to act on it. It has always an embodied character as a person’s knowledge is tied to their embodied experience. Information, in contrast, is a something that is e.g. written down somewhere and can be accessed by a person to gain knowledge. It is broken down so that it can be transferred and processed. With that, it is lacking context and application. Importantly, information in that sense is not de-situated either as it has been written down as a knowledge someone previously gained. I consider knowledges and information as always situated. The predominant usage of the term knowledges in this paper signifies its theoretical rooting in Haraway’s concept of Situated Knowledges.

The usage of chatbots providing direct answers to search queries profoundly changes how and which knowledges users can find in internet searches. It leads to what I call sealed knowledges. The terminology and metaphor of sealing derives from the concept of sealed surfaces, which Rainer Mühlhoff (2018) introduced to refer to.

the observation that the design of digital devices often conceals the technical nature of the device beneath closed surfaces, which manifest as user interfaces. The design of human-machine interfaces is increasingly involving the use of design features that hinder users from gaining insights into the technical workings of devices and applications. (Mühlhoff Forthcoming, p. 50)

The term is both a description and critique on the current trend in UX (user experience) design which conceals the inner functioning of (digital) devices. This trend has two aspects: First, “users are not supposed to—or assumed not to want to—use a (digital) device instrumentally, meaning as a tool with a generally indefinite list of purposes, which the user can operate through conscious reflection and knowledge of the device’s technical characteristics” (Mühlhoff Forthcoming, p. 55). In UX design, users are not expected to use electronic devices as purpose open tools whose usability can be understood through knowledges about their technical characteristics. This goes hand in hand with a very specific image of a user who acts on the stimuli presented by the intuitive interfaces and does not aim to act on it because of a comprehension of its function. This is related to the second aspect of the sealed surfaces, namely that it is not considered reasonable that users have the will, desire, or ability to structurally understand the functions of a device (Mühlhoff 2018, p. 568).

For example, Microsoft has changed the Windows folder structure over the years in such a way that the inner workings of the system became increasingly obscure in newer versions. The critique of sealed surfaces contents that in UX design trends of the last decades, the rationale seems to be that non-experts do not need to access deeper folder levels and thus to understand how the system works (Mühlhoff 2018, p. 566f.). As a result, users must put more and more effort into familiarizing themselves with how the system works to use it instrumentally, while the interfaces are increasingly designed in such a way that it is unattractive for users to do so. While knowledgeable users can still get through the “sealed surfaces” and e.g. understand how the devices whose surface is sealed functions internally, it is less and less common knowledge of average users. This growing inaccessibility is the starting point of Mühlhoff’s critique on UX design and his concept of sealed surfaces. While the surfaces are sealed, thus difficult to get through, the seal can be broken, and the surfaces can be permeated by knowledgeable users who are willing and able to put effort into it. This is why the metaphor of sealing is so apt. Mühlhoff (2018) argues that this trend of sealed surfaces is part of UX design and its call for usability, effectivity, efficiency and user satisfaction without the aim that users understand the systems. As I will show in this paper, the metaphor of sealing is also helpful to understand the impact that chatbots used to assemble direct answers in search engines have on knowledges.

When being portrayed with a direct answer to a search query, users read an information that is broken down into a discernible text. For example, Fig. 3 shows the answer to the question “what is racism?” in Bing Copilot.

The output of the algorithm appears highly structured, with highlighted headings (in bold) and a list which points to several different aspects concerning racism. Without going into the details of the content of the output, this algorithmically assembled text provides a certain view on what racism is. As an important, long-standing, and contested concept, the answer the LLM produces is necessarily only one of several possible answers that could be given to the search query “what is racism?”. This, arguably, is mostly the case when looking up information. If I would type the same search query into a ‘traditional’ search engine and visit a website on which a person has written an explanation of racism, that information would also be partial. What is new and different through the usage of chatbots is that the information does not result from a situated knowledge which is broken down to data/text that can followingly be transferred. As already explained in Sect. 2.2., information describes the text in itself and the content it carries, which lacks context and application. It is processed data which can be transferred. In traditional search, this information is assembled by a human who holds it as knowledge which is embodied in a situated subjectivity. It is a knowledge which is embedded in the lived experience of a person who can abstract it to write it down as transferrable information in a text. Through using chatbots in search engines, this changes. The text is assembled by a disembodied and de-situated algorithm (see below). The algorithm has never lived in a racist society, has never been subjected to racist slurs and has never engaged in a racist activity. An embodiment, situatedness and lived experience is exactly what the language model does not have. Thus, the use of chatbot for direct answers changes how information and the concurrent knowledge is created.

The de-situatedness of the algorithmically produced text, as a non-situatedness in a certain context, discourse, and original space of writing, is reinforced by the inherent disembodiment of the language model. As Bender and Koller (2020) write, LLMs miss any kind of bodily situatedness. If a person writes a text which is then published online, for example a Wikipedia entry, they have the capacity to reflect on their writings and the sources which they use to write the text. They also have an embodied experience of being in the world. Language models, on the other hand, while necessitating a material infrastructure of servers, etc., are disembodied and have no (embodied) understanding of the world. Therefore, they lack even the possibility of giving an embodied account of knowledges. Bender and Koller (2020) point out that while LLMs may be very good at simulating the form of human language, they do not have any way of ‘learning’ meaning from it (Bender and Koller 2020). This is due to their technological functioning: they are programmed to output statistically relevant and likely words and texts. They do this by imitating patterns of words and texts with which they are trained. The algorithms imitate the form of the human text without any possibility for understanding meaning it. Thus, using chatbots for the provision of direct answers in online search presupposes that knowledge is conceived of as disembodied, as it is assembled by a disembodied AI system.

Secondly, the implementation of chatbots into search engines relies on a very specific, imagined user interacting with it. This, through the following sealing of knowledges, leads to user subjectivities which adhere to the previously imagined user. First to the imagined user. Metzler et al. (2021), a team of Google researchers, wrote an opinion paper about the future of the Google search in 2021. In it, they lay out the idea to use LLMs for the generation of direct answers and proclaim that this mode of search will be the way forward, foreshadowing the current development of online search. The reasons they name for why chatbots should be implemented in online search are telling about the image they have on the users engaging with the search engine. They proclaim that ‘traditional’ web search “induces a rather significant cognitive burden on the user” which would be resolved if chatbots provided one simple, clear and authoritative answer to user inputs (Metzler et al. 2021, p. 1). The formulation of searches placing a “cognitive burden” on users is quite remarkable. It implies that users cannot, and do not want to, choose between different websites. Even more, choosing between different answers is framed as negatively impacting users and as potentially even being harmful for them (a burden is quite negatively connotated term). This negative aspect of online search, so Metzler et al., can be alleviated through the integration of chatbots into search engines. In contrast to ‘traditional’ search, the integration of chatbots into search engines is framed as “user-friendly”, “convenient” and “time-saving” (Shah and Bender 2022, p. 222). The authors’ idea of what users want in this proposal (which has now become reality) is, therefore, to receive a simple and clear answer in order to save time.

This imagination of users has several implications for the implementation of chatbots into online search. First, the necessity of providing users with the information of how the LLM works, or of the sealing of knowledges taking place through it, is not a matter of debate. Even more, it is not mentioned (Metzler et al. 2021; Pichai 2023). In the Bing chat function, there is no attempt to clarify to users why they are shown a certain output or how the algorithm works. Due to the imperative of saving time and maximal convenience, users are not expected to question results they receive or the way an answer to their search query is assembled by the language model. The user is not expected to want to understand how the system functions—very much in line with the theory of sealed surfaces by Mühlhoff. This is closely connected to the specific interface design of the search engines. Figure 4 shows the start page of the Bing chat feature. The sleek interface does not invite a critical engagement with the system but rather seems to spread an aura of trustworthiness. There is no tab telling the user how the system works or cautioning against taking the output of the bot as the only possible answer (which it is often not, as we saw in Figs. 1 and 2). The chat interface reads “ask me anything” (Fig. 4). The chatbot is anthropomorphised (“me”) and at the same time has an appearance of omniscience (“anything”). It promises to provide better answers than ‘traditional’ search. The whole appearance, both content-wise and design wise does not tell how the bot functions and how answers are produced. The sealed surface (Mühlhoff Forthcoming) of the interface design of the LLM integrated into the online search engine feeds into the sealing of knowledges.

Secondly, this imagined idea of users and their wishes, and the concurrent implementation of the search based on this as described above, impacts users and shapes their subjectivities. I understand subjects and subjectivation based on theories by Foucault here. Very briefly put, subjects, in this theoretical understanding, are constituted by power relations and emerge in and through them (see Kammler et al. 2014, p. 294). Subjects are in constant subjectivation processes through which they are constituted and produced. Subjectivation processes can be understood as the interplay between the power relations that produce the subject and the formation of the subject’s own power relations (Vogelmann 2017, pp. 11–12). The integration of chatbots into search plays into the subjectivation processes of the users engaging with them. It creates a user subjectivity in which the display of a singular answer to a search query is taken as the norm and passed unquestioned. It becomes increasingly difficult to question how the system works as it becomes the norm to accept the answer of a bot without a question. It is convenient and fast to use the bot and there is nothing which shows the potentially problematic implication of taking its answer as authoritative. The user subjectivity is centered around mere acceptance of, not critical engagement with, the answers of the bot. This, in a larger realm, can lead to subjects which increasingly accept singular answers instead of questioning the information they find to build their own knowledge. The imagined user, through the actual implementation of this new mode of search, shapes the subjectivation and subjectivities of actual users and thus creates lived realities.

The implementation of chatbots in searches increasingly takes away users’ control over what information they can find in a search. Users do not have the possibility to click on different search results and concurrently do not find different opinions on an issue by chance. While the search results may state several different points concerning an issue, it is still a less diverse picture than in ‘traditional’ search, where different opinions are voiced quite strongly on different websites by different people. It may, at times, be unpleasant for users to engage in certain discourses by being thrown into these types of discussions, but it is important to get an overview over complex and ambiguous issues. Having to decide which websites to visit and having to put effort into finding certain information is pivotal to critically build an own opinion and to form informed personal knowledges.

The provision of a direct answer by chatbots and the co-occurring sealing of knowledges leads to a reduction of possibilities of doubt. At the same time, doubt can be a crucial way of resisting the sealing of knowledges. In her book “Cloud Ethics”, the political geographer Louise Amoore (2020) describes how algorithms change what comes into perception. She argues that algorithms reduce and distill certain outputs from vast amounts of data, thus bringing certain, actionable outputs to the forefront of attention. Other possible actions which could take place are disregarded. Algorithms thereby control which structures in data become visible—and which ones not. They determine for example who is flagged as a possible suspect at a border control. In the moment of flagging, the singled-out person is brought to the attention and into conscious perception of the border control. Therefore, algorithms promise to reduce uncertainty by providing an actionable output in situations of uncertainty about what to do and how to act. They reduce the doubt which otherwise accompanies decisions. The opaque algorithms seem to produce clarity by giving a singular answer to complex questions based on vast amounts of data.

While Amoore mainly refers to automated border control and surveillance algorithms, some of her claims can also be applied to automated text-generation in the usage of chatbots giving direct answers to search queries. Through LLMs, certain knowledges are brought to attention of users, while other possible knowledges out of the complex knowledge space are disregarded. The algorithm determines what becomes perceptible, findable knowledges and opens spaces to act on these, and only these, algorithmically filtered knowledges. Through the automatically assembled output, the underlying information selection mechanisms and text-generation processes are opaque and invisible. This reduces the room for doubt, as doubting the algorithms presupposes and necessitates the knowledge that doubting the algorithm is possible. This, as I wrote in Sect. 3.2., becomes increasingly difficult through the creation of specific user subjectivities which do not include a desire for understanding how the LLM works and is aimed at an unquestioned acceptance of the output the bot produces.

However, even though chatbots as search engines may reduce doubt through the creation of specific user subjectivities, they do not elimante the possibility of it. In reference to Amoore, I suggest that we need to start and keep doubting the algorithm and the output of automated answers to search inquiries. Amoore (2020, p. 24) argues that.

cutting against the grain of the dominance of definiteness as algorithms act on doubt, I seek to reinstate doubtfulness […] within the calculative architecture of the algorithm. Though at the point of optimized output, the algorithm places action beyond doubt, there are multiple branching points, weights, and parameters in the arrangements of decision trees and random forest algorithms, branching points at which doubt flourishes and proliferates.

This doubtfulness works in manifold ways. It is a doubt of oneself and how oneself knows, as well as a doubt of how the algorithm works and why it displays certain answers. It is a doubt of the singularity of the answer the algorithm provides and of the idea of a securable future which it promises (Amoore 2020, p. 148). It is a doubting of data points and a doubtful accounting of the calculations of the algorithm as well as a doubting of the perspectives which are shown through the algorithms and of their inherent partiality.

In the specific context of chatbots providing direct answers to search queries, doubting means to question the validity and adequacy of the output of the search answer as being the one and only correct—or at least most relevant—one. Even though the sealing of knowledges makes it more difficult to access the diverse possible answer space to a search query, it is still there. The sleek, intuitive user interfaces may seal knowledges from view. But it does not eliminate those knowledges. It just makes it harder to find them. A seal can be broken. Doubting requires looking for cracks in the seal, to widen them, to build a sense for them. It means the will to unseal knowledges, to dig behind shiny surfaces, to look for the knowledges flourishing behind it. It involves doubting the generalization of the portrayed knowledges and an awareness for the inherent situatedness of the portrayed knowledges which were taken from somewhere and are now newly arranged, de-situated, nowhere in particular. It means to be aware that knowledges are always partial and stand in certain contexts—and that the output of a LLM is therefore also always partial and context dependent. To refer back to Haraway (1988): acknowledging and cherishing the situatedness and partiality of knowledges is what we need to do to avoid the god trick. Doubting thus involves a conscious re-situating of knowledges.

Situating knowledges is a strategy to become aware of the dominance of certain knowledges and views of knowledge over others. It allows to see contingencies in portrayed knowledges. It means, to draw on the famous quote by Haraway (2016), to stay with the trouble, to choose the way of leaving the future open by being aware of the multiplicity of knowledges in the present instead of “making an imagined future safe” (Haraway 2016, p. 1). It means to take the troublesome way of doubting the knowledges and outputs portrayed by chatbots and to question what it sealed. This means to reinstall an understanding of knowledges in their situatedness and partiality, to imbed knowledges in discourses and power-relational emergences. Doubting the outputs of LLMs can function as a feminist intervention to resist the sealing of knowledges, especially of marginalized and non-dominant knowledges (see e.g. Ahmed 2008; Haraway 2016). Writing about feminist futures, Sara Ahmed (2008, p. 251) stresses the importance of hope, “a hope that things can be different, and that the world can take different forms”. This idea of open, non-determined futures, allowing for hope, different imaginaries, and political action in the present, crucially includes a future that is not pre-determined by algorithmically given answers which filter which knowledges come to be known.