Google Pixel 9 Uses AI to Instantly Find Information Hidden in Your Screenshots

Google's new Pixel 9 smartphone introduces an AI-powered feature that allows users to effortlessly search through their screenshots to find crucial information like WiFi passwords and event details. Unveiled at the company's Bay View campus, this tool aims to simplify daily tasks by enhancing screenshot management.

Google’s Pixel 9 AI Feature Simplifies Finding Crucial Information in Screenshots, but Only for Those Willing to Make the Switch

Mixed into a phone's photo library are 4,811 screenshots, many of which contain essential information. Capturing those images was easy; finding them when needed was the real challenge.

Google claims to have a solution.

Among the new AI features introduced, Google's Pixel 9 smartphones—unveiled on August 13 at the company's Bay View campus in Mountain View, Calif.—offer a tool to help users locate vital information like WiFi passwords and event details hidden in past screenshots.

According to the company, all required is to ask through a new Pixel Screenshots app or by interacting with the AI-powered Gemini assistant.

According to The Washington Post, this new feature could mean an end to endlessly scrolling through photos to find a crucial image—if one is willing to switch to a Pixel. The app processes the request, ideally pulling the needed information from the screenshot and displaying the corresponding images.

This feature isn’t enabled by default; users must opt-in. Fortunately, none of the screenshots are sent to external servers for processing; Google says all image analysis occurs directly on the Pixel phones, thanks to the new Tensor G4 processors.

Compared to current AI tools integrated into smartphones—like text rewriters, image manipulators, and audio transcribers—this attempt to give phones a kind of visual memory could be surprisingly helpful in daily life. After all, many people take screenshots of things they want to remember.

However, there are some limitations. The feature can extract valuable details from new screenshots, but by default, it can't access older ones stored in services like Google Photos. Users can manually add older images to the Pixel Screenshots app if desired. The app warns that its responses "may be inaccurate," but allows double-checking the answers.

For better or worse, Google plans to keep this tool exclusive to its Pixel 9 smartphones rather than making it available to other Android phone makers, as it has done with some AI features.

Google's Pixel 9 AI Features Aim to Boost Market Share Amid Growing Competition from Other Generative AI Smartphones

This exclusivity might be disappointing for owners of other Android devices capable of running AI models without relying on the cloud. According to IDC research, shipments of these "generative AI smartphones" are expected to more than quadruple this year. However, Google's decision to reserve certain AI features for its devices could be crucial as it aims to capture a more significant share of the smartphone market.

Despite increasing Pixel phone shipments yearly since 2021, Google accounted for just 0.8 percent of the global smartphone market share in the first half of 2024, according to market research firm Canalys.

Other new AI features in Google's latest phones include a more conversational version of its Gemini AI called Gemini Live. This version can engage in long, natural conversations with users—similar to how tools like OpenAI’s ChatGPT operate. Google says Gemini can now also react to and provide additional information about what’s displayed on the phone’s screen, potentially outpacing Apple's upgraded Siri in the race for users' attention.

"Google had a first-mover advantage in the market that helped it differentiate, but the competition is catching up," said Runar Bjorhovde, an analyst at Canalys.