‘Hey Copilot’: Microsoft gives Windows 11 a voice, vision and action with new AI features

As Microsoft bids adieu to Windows 10 and the wait for Windows 12 reached its pinnacle, the Redmond-based tech giant has given its current operating system a major AI push. Microsoft is more deeply integrating its Copilot AI assistant at the center of the Windows experience, which the company claims means every Windows 11 PC will become an AI PC.

Also Read | I used the OnePlus Pad 3 as my laptop for 90 days and there is no switching back

New AI features coming to Windows 11:

‘Hey Copilot’:

Microsoft is making Copilot more conversational, in line with other voice assistants like Google Assistant, Alexa, or Siri. As you summon any of these assistants with a hotword, you’ll now be able to do the same with Copilot. No prizes for guessing the hotword: it’s “Hey Copilot.”

The new voice feature for Copilot is opt-in, meaning users would need to enable it in the settings of the Copilot app before they start talking to the AI assistant. After summoning Copilot with the hotword, users will see a Copilot microphone appear on their screen along with a chime that indicates that Copilot is listening.

The unique feature that Microsoft is bringing here is the use of the “Goodbye” hotword to end a conversation. Users can also end the conversation by tapping on the ‘X’ icon, or Copilot will terminate it on its own if there is no interaction for a few seconds.

Microsoft says using voice with Copilot leads to users engaging twice as much as when using text, while leading to deeper engagement with the chatbot.

Copilot now has ‘vision’:

Apart from adding voice to Copilot, Microsoft is also giving it ‘vision,’ which will be available in all markets where the AI chatbot is already present. With the new Copilot Vision feature, users will be able to share their desktop or applications, and Copilot can help analyze content, provide insights, and answer their questions.

Users can also go beyond just asking questions about the task on their screen and ask the AI assistant to ‘show’ them how to perform a specific task. Copilot can also now give tips while playing games, viewing photos, or reviewing travel itinerary.

Copilot will also be able to see beyond what’s available on the users’ screen when sharing Word, Excel, or PowerPoint files. For instance, when users share a PowerPoint deck with Copilot, it can analyze the entire presentation without needing to flip through each slide.

Microsoft says it is also planning to bring a text-in, text-out feature for Windows Insiders, where users will be able to converse with Copilot Vision using text and get answers the same way.

Copilot in Windows taskbar:

Microsoft is bringing a new “Ask Copilot” option on the taskbar for Windows Insiders. The new option can help users find apps, files, and settings right from the taskbar while having one-click access to Copilot Voice and Vision.

“This experience leverages existing Windows APIs to return your apps, files and settings, like the Windows Search experience, and does not grant Copilot access to your content,” said Yusuf Mehdi – Executive Vice President, Consumer Chief Marketing Officer at Microsoft.

Copilot Actions:

Microsoft is bringing agentic AI abilities to Windows Insiders with the new Copilot Actions feature. Users will be able to describe the task they want done in natural language, and the agent will attempt to complete it by interacting with desktop and web applications.

“While this is happening, you can choose to focus on other tasks. At any time, you can take over the task or check in on the progress of the action, including reviewing what actions have been taken,” Mehdi says.

The new feature can help users accomplish tasks like resizing photos, extracting information from PDFs, and performing bulk edits on documents.