Google (Canada)

Meaningful governance of any system requires the system to be assessed and monitored effectively. In the domain of Artificial Intelligence (AI), global efforts have established a set of ethical principles, including fairness, transparency, and privacy upon which AI governance expectations are being built. The computing research community has proposed numerous means of measuring an AI system’s normative qualities along these principles. Current reporting of these measures is principle-specific, limited in scope, or otherwise dispersed across publication platforms, hindering the domain’s ability to critique its practices. To address this, we introduce the Responsible AI Measures Dataset, consolidating 12,067 data points across 791 evaluation measures covering 11 ethical principles. It is extracted from a corpus of computing literature (n = 257) published between 2011 and 2023. The dataset includes detailed descriptions of each measure, AI system characteristics, and publication metadata. An accompanying, interactive visualization tool supports usability and interpretation of the dataset. The Responsible AI Measures Dataset enables practitioners to explore existing assessment approaches and critically analyze how the computing domain measures normative concepts.

How the brain recovers the three-dimensional structure of surfaces and objects from 2D retinal images remains mysterious. Shading patterns provide one of the most powerful-yet least understood-visual depth cues. Most theories assume the brain infers surface normals from luminance values. However, this seems unlikely as visual neurons are broadly insensitive to luminance. To identify alternative cues, we measured responses of model orientation-selective cell populations to images of shaded objects. We found a surprising statistical relationship between image orientations and surface curvature properties, suggesting a way to estimate shape from shading. We find that the orientation-based cues not only predict striking illusions of shape perception when lighting varies, but also the impressive robustness of shape perception when large image modifications are introduced to directly pit luminance and image orientation cues against one another. The findings resolve the longstanding question of which image measurements drive shape from shading perception.

Graphs are a powerful tool for representing and analyzing complex relationships in real-world applications. Large Language Models (LLMs) have demonstrated impressive capabilities by advancing state-of-the-art on many language-based benchmarks. Their ability to process and understand natural language open exciting possibilities in various domains. Despite the remarkable progress in automated reasoning with natural text, reasoning on graphs with LLMs remains an understudied problem that has recently gained more attention.This tutorial builds upon recent advances in expressing reasoning problems through the lens of tasks on graph data. The first part of the tutorial will provide an in-depth discussion of techniques for representing graphs as inputs to LLMs. The second, hands-on, portion will demonstrate these techniques in a practical setting. As a learning outcome of participating in the tutorial, participants will be able to analyze graphs either on free-tier Colab or their local machines with the help of LLMs.

Understanding how communities experience algorithms is necessary to mitigate potential harmful impacts. This paper presents folk theories of text-to-image (T2I) models to enrich understanding of how artist communities experience creative machine learning systems. This research draws on data collected from a workshop with 15 artists from 10 countries who incorporate T2I models in their creative practice. Through refexive thematic analysis of workshop data, we highlight artist folk theories of T2I use, harm, and harm reduction. Folk theories of use envision T2I models as an artistic medium, a mundane tool, and locate true creativity as rising above model afordances. Theories of harm articulate T2I models as harmed by engineering eforts to eliminate glitches and product policy eforts to limit functionality. Theories of harm-reduction orient towards protecting T2I models for creative practice through transparency and distributed governance. We examine how these theories relate, and conclude by discussing how folk theorization informs responsible AI eforts.

Abstract We conduct a large-scale fine-grained comparative analysis of machine translations (MTs) against human translations (HTs) through the lens of morphosyntactic divergence. Across three language pairs and two types of divergence defined as the structural difference between the source and the target, MT is consistently more conservative than HT, with less morphosyntactic diversity, more convergent patterns, and more one-to-one alignments. Through analysis on different decoding algorithms, we attribute this discrepancy to the use of beam search that biases MT towards more convergent patterns. This bias is most amplified when the convergent pattern appears around 50% of the time in training data. Lastly, we show that for a majority of morphosyntactic divergences, their presence in HT is correlated with decreased MT performance, presenting a greater challenge for MT systems.

We study the relationship between acceptability/annoyance rating and traditional MOS quality ratings of UGC videos. Acceptability/annoyance is a key concept for evaluating services, as it classify whether the delivered service quality falls into acceptable, annoying but acceptable, or not acceptable. This relates to the willingness of users to use those services. While audiovisual quality estimation models have a long research history, the translation of these quality scores to acceptability and willingness to use the services has only been weakly studied. In this work, a new dataset was then created to evaluate both quality and the acceptability/annoyance of videos. Different state-of-the-art quality prediction models videos were evaluated at predicting quality of UGC videos. Furthermore, performance at predicting acceptability/annoyance of videos was also tested.

Computing demand continues to grow exponentially, largely driven by "big data" processing on hyperscale data stores. At the same time, the slowdown in Moore's law is leading the industry to embrace custom computing in large-scale systems. Taken together, these trends motivate the need to characterize live production traffic on these large data processing platforms and understand the opportunity of acceleration at scale.

Beyond self-report data, we lack reliable and non-intrusive methods for identifying flow. However, taking a step back and acknowledging that flow occurs during periods of focus gives us the opportunity to make progress towards measuring flow by isolating focused work. Here, we take a mixed-methods approach to design a logs-based metric that leverages machine learning and a comprehensive collection of logs data to identify periods of related actions (indicating focus), and validate this metric against self-reported time in focus or flow using diary data and quarterly survey data. Our results indicate that we can determine when software engineers at a large technology company experience focused work which includes instances of flow. This metric speaks to engineering work, but can be leveraged in other domains to non-disruptively measure when people experience focus. Future research can build upon this work to identify signals associated with other facets of flow.