Visual impairment is a relatively common condition that is affecting millions of people worldwide [1]. According to the World Health Organization, at least 2.2 billion people worldwide have near or distant vision impairment and 36 million people are blind [2]. This statistic indicates that 1 out of every 5 individuals in the world has a visual impairment, and 1 out of every 200 people is blind. Despite the significant number of people with blindness, very few assistive technologies are commercially available for these people to conveniently interact with digital web content. One of the most predominant assistive technologies for people with severe vision impairments, including blindness, is a screen reader, such as NVDA (https://www.nvaccess.org/ (accessed on 1 October 2023)), JAWS (https://www.freedomscientific.com/products/software/jaws/ (accessed on 1 October 2023)), or VoiceOver (https://www.apple.com/accessibility/mac/vision/ (accessed on 1 October 2023)).

A screen reader, as the name suggests, narrates the content on the screen and allows blind users to navigate the content using special keyboard shortcuts (e.g., ‘H’ for the next heading). This one-dimensional mode of interaction has been shown to create a plethora of accessibility and usability issues for blind users while interacting with computer applications. These challenges encompass the absence of well-defined structural elements, labels, and descriptions within web elements, hindering the screen reader’s ability to identify and interpret content ^[3][4][5][3,4,5]. Moreover, the inclusion of dynamic and interactive features, like animations, pop-ups, and videos, may not be screen reader-compatible, resulting in potential interference and incomplete experiences [6]. Additionally, the inconsistency and complexity of web design and layouts can be disorienting and overwhelming for blind users [7]. Furthermore, the lack of feedback and guidance within web applications or websites leaves blind users uncertain and lost, impeding their ability to make informed decisions and navigate effectively ^[8][9][8,9].

Existing works to improve accessibility and usability predominantly focus on general web navigation and visual content, such as images ^[10][11][10,11], and videos [12]; extraneous content, such as advertisements and promotions, however, is still an uncharted research territory. Advertisements and promotions are widely present on most modern websites, serving as a crucial means of generating revenue and frequently offering utility to users. Research indicates that people engage with advertisements by clicking on them and, in general, consider ads as beneficial [13]. However, such extraneous content is, by design, visually rich, and as such, primarily intended for sighted consumption; therefore, it is unclear how these extraneous elements impact the browsing experience and engagement of blind users who can only listen to content using their screen readers. To fill this knowledge gap, we conducted an interview study with 18 blind participants familiar with web browsing.

The study uncovered many insights, with the most notable one being that blind users are often ‘deceived’ by ads, specifically the ones that are contextually integrated into the web page content, e.g., native promotion ads that are similar to surrounding web page content, as shown in Figure 1 (https://www.kayak.com/flights/ORF-WAS/2023-12-01/2024-01-11?sort=bestflight_a&attempt=3&lastms=1697423402586&force=true (accessed on 1 October 2023)). The participants further stated that such deceptions often resulted in interacting with content that did not align with their intended objectives or interests, and sometimes the consequences were serious, such as unintentionally installing viruses, revealing personal information, and buying unintended items on shopping websites. Another informative observation from the study was that only a small fraction of the participants used ad blockers; many participants stated that ad blockers were hard to install and configure with screen readers. Moreover, a few participants also expressed that they were unable to access some of their favorite websites with an active ad blocker, as these websites prohibited access to their content on detecting an ad blocker. To address the problem of contextually deceptive ads, we need an algorithm that can accurately identify such ads on web pages. With such an algorithm, countermeasures (e.g., feeding a screen reader with additional contextual information) can be developed. To devise such an algorithm, we first built an ad dataset comprising both deceptive and non-deceptive ads sampled from a diverse set of web pages across multiple domains, including travel, e-commerce, retail, news, hotel booking, and tourism. Leveraging this dataset, we trained a custom multi-modal classification model that uses a combination of both handcrafted and automatically retrieved features as input. Evaluation of the model on a representative test dataset yielded an F1 score of 0.86, thereby demonstrating its effectiveness in distinguishing deceptive ads from non-deceptive ads. We then used this trained classification model in our algorithm to identify all deceptive ads (if any) on any given web page. An ‘in-the-wild’ evaluation of the algorithm on 20 randomly selected websites yielded an F1 score of 0.88. 

2. Non-Visual Web Interaction Using Screen Readers

A screen reader (e.g., JAWS, NVDA, or VoiceOver) is a special-purpose software that enables visually impaired users to interact with computer applications by listening to their speech output. The prevalence of screen readers among blind users is due to the fact that they are cheaper and simpler to use than hardware devices, like braille displays ^[14][15][16][14,15,16]. Screen readers provide many shortcuts that facilitate efficient website navigation and content accessibility (https://www.freedomscientific.com/training/jaws/hotkeys/ (accessed on 1 October 2023)) (https://www.nvaccess.org/files/nvdaTracAttachments/455/keycommands%20with%20laptop%20keyboard%20layout.html (accessed on 1 October 2023)). For instance, in NVDA, pressing the ‘D’ key allows users to cycle through and focus on different landmarks, such as headings, navigation menus, or main content areas, providing a quick overview of the page’s organization. Similarly, in JAWS, the ‘D’ key, used in conjunction with a JAWS-specific modifier key, like insert or caps lock, serves the same purpose, aiding users in navigating web pages with ease. Nevertheless, the abundance of shortcuts presents a significant challenge to visually impaired users since it may be arduous to remember and effectively employ each shortcut. For example, NVDA and JAWS offer various complex multi-key combinations for advanced tasks. To trigger these shortcuts, users often need to press multiple keys simultaneously, which can be challenging for individuals with dexterity or motor control issues. An example is the “Insert + 3” key combination in JAWS, which activates the JAWS cursor for advanced navigation and interaction. Remembering and executing such combinations can be daunting, particularly for users with certain physical disabilities or limited dexterity. In general, individuals tend to depend on a limited number of fundamental shortcuts in order to navigate websites [17]. Due to limited shortcut vocabulary, blind users often encounter difficulties navigating modern web pages due to intricate and extensive HTML document object models (DOMs) underlying web pages [18]. In order to tackle these concerns, continuous investigations have been undertaken in the realm of web usability, and the pain points of blind screen-reader users on the web have been identified and addressed ^{[19][20][21][22][23][24][25][26][27][28]}[19,20,21,22,23,24,25,26,27,28]. Apart from navigation-related issues, web content itself contains a plethora of visually rich elements, such as videos, images, and memes. Blind users face problems interacting with these elements as they are primarily designed for visual consumption, and often there are no proper textual alternatives for these elements (e.g., alt-text), despite the availability of standard web content accessibility guidelines (WCAG) ^[29][30][29,30]. For example, in the case of images, it has been found that although alt-texts are present, they do not convey the full information equivalent to what a sighted person perceives by looking at these images [31]. To address this issue, AI-based solutions have been recently proposed to automatically generate informative descriptions or captions for visual elements [32]. For instance, Singh et al. [33] built Accessify using machine learning as a means to offer alternative text for every image included on a website, operating inside a non-intrusive framework. Accessify does not necessitate any initial configuration and is compatible with both static and dynamic websites. Apart from images, video accessibility has also gained attention in recent years; e.g., Siu et al. [34] developed a system to automatically generate descriptions for videos and answer blind and low-vision users’ queries about the videos.

3. Dark Patterns and Deceptive Web Content

Brignull first introduced the term “dark patterns ” (https://www.deceptive.design/ (accessed on 1 October 2023)), where he described certain user interface designs as “tricks used in websites and apps that trick you into doing things you didn’t intend to, such as purchasing or signing up for something”. Brignull’s initial work sparked a flurry of academic research that attempted to define and describe dark patterns. Dark patterns—insidious design choices and techniques that manipulate user behavior or deceive users—have become increasingly widespread across various websites ^{[35][36][37][38][39]}[35,36,37,38,39]. In recent years, the prevalence of deceptive practices and dark patterns on numerous digital platforms, including social media, travel websites, e-commerce, apps, and mobile activities, has led to concerns about user trust and autonomy ^{[40][41][42][43][44]}[40,41,42,43,44]. These patterns leverage cognitive biases, create a sense of urgency, or hide essential information, leading users to unintended actions or decisions [35]. While many extant works have focused on different types of dark patterns, there have been relatively fewer specific efforts in the literature regarding deceptive online ads ^[45][46][45,46]. For instance, Toros et al. [47] focused on deceptive online advertising tactics within e-commerce platforms to shed light on how companies and marketers employ misleading strategies to persuade consumers. They found that companies used tricks to affect the purchasing behaviors of users by incorrectly representing the core products by mimicking, inventing, and relabeling them. Nonetheless, all existing works on dark patterns and deceptive ads ^{[37][39][48][49]}[37,39,48,49] have been conducted under the premise of sighted interaction; thus, they do not account for the unique aspects associated with the audio-based interactions of blind users. As ads often contain many visual elements, there is potential for even legitimate ads to be contextually deceptive to blind users in case the screen reader cannot properly communicate these visual elements to the user. For instance, if an ad lacks proper alternative text (alt-text) or fails to communicate the presence of a Google ad symbol, visually impaired users may not receive the necessary auditory feedback from screen readers to distinguish between regular content and ads. This lack of visual context can lead to a situation where users may inadvertently interact with or be misled by advertisements that they did not intend to engage with, highlighting the critical importance of ensuring web content is properly labeled and described for accessibility. Therefore, the range of potentially deceptive ads on web pages is wider for blind screen-reader users than for sighted users viewing the same pages, thereby warranting a separate focused analysis to understand the challenges blind users face with ads.

4. Web Ad Filtering and Blocking

Numerous ad detection browser add-ons (e.g., AdGuard (https://adguard.com/en/welcome.html (accessed on 1 October 2023)), AdBlock (https://getadblock.com/en/ (accessed on 1 October 2023)), and AdBlock Plus (https://adblockplus.org/ (accessed on 1 October 2023))) exist to help users avoid ads on web pages. They can be downloaded and installed as browser extensions. They work by blocking communications to ad servers and hiding them from the HTML DOM [50]. They perform this blanket filtering operation by referring to a filter list containing the addresses of all known ad servers along with their pattern-matching rules. However, they do not eliminate any internal promotions or ads [51], most of which are usually deceptive [36]. Internal promotions or ads contribute significantly to the income of many websites and form an integral part of their content. Blocking them could potentially jeopardize the sustainability of these websites and disrupt their layouts. In addition to commercially available ad blockers, some academic works have proposed ad detection algorithms ^[52][53][52,53]. For instance, Lashkari et al. [54] developed CIC-AB, which is an algorithm that employs machine learning methodologies to identify advertisements and classify them as non-ads, normal ads, and malicious ads, thereby eliminating the need to regularly maintain a filter list (as with earlier rule-based approaches) [51]. CIC-AB was developed as an extension for common browsers (e.g., Firefox and Chrome). Similarly, Bhagavatula et al. [55] developed an algorithm using machine learning for ad blocking with less human intervention, maintaining an accuracy similar to hand-crafted filters (e.g., [51]), while also blocking new ads that would otherwise necessitate further human intervention in the form of additional handmade filter rules. Nonetheless, increasing numbers of websites are now discouraging ad blocking due to the loss of associated ad revenue. Numerous websites have incorporated techniques to identify the existence of ad blockers, potentially leading to the denial of access to content or services upon detection [56]. Complementary to ad blocking, efforts are underway to raise awareness and advocate for the inclusion of accessibility guidelines in the design and implementation of online advertising practices so as to make ads more palatable for all users ^[57][58][57,58]. However, these efforts are still at a nascent stage and may probably require a long time to yield positive outcomes, as in the case of any other accessibility efforts in general, e.g., WCAG guidelines. Moreover, the creators of deceptive ads may not follow ad-related accessibility guidelines for obvious reasons, so there is a need for approaches that detect deceptive ads and communicate their presence and/or details to screen-reader users.