New Microsoft Copilot tricks: Remembering what you tell it, interacting with anything running on your PC

Microsoft is bringing some big updates to its Copilot AI applications for Windows and mobile. Some of the biggest are that Copilot on Windows now runs entirely as a native app rather than a web app. It now has memory and personalization features that can recall things it's learned about you and apply them to future interactions. A new Copilot Vision feature lets the software see what's on your screen and offer contextual suggestions for apps, games, and other activities. And there's support for Actions that allow Copilot to use websites to do things on your behalf like buy tickets, make restaurant reservations, or even buy things.

The new features are starting to roll out today, but it could be a while before they're available to all users in all markets.

Of course some of the features that are potentially the most useful are also the ones that might be the creepiest. Do you really want Microsoft's AI assistant remembering everything you've told it? So the company is stressing that users can "choose which types of information it remembers about you or… opt out entirely."

Likewise Copilot Vision allows you to pull up Copilot "while working across multiple applications, browsers tabs, or files" and allow it to "read the screen and interact with the content" to do things like start a search, adjust settings, and more. But it only does that when you specifically initiate the action.

There's also a mobile version of Copilot Vision that recognizes imagery from a smartphone camera to let you point the phone at something (or use photos saved on your device), allowing you to get answers to questions about the contents of the frame. For example it can tell you what historic building or landmark you're looking at or "improve plant health by asking it to examine your plants and suggest actions."

Microsoft is also bringing Shopping features to Copilot, including notifications about sales and price drops on products that you're interested in, and the ability to make a purchase directly through the Copilot app.

Copilot is also borrowing a page from Google's playbook and adding the ability to "generate AI-powered podcasts that curate and deliver personalized audio content based on your interests." Basically tell Copilot what you'd like to learn about and it will create a fake podcast with fake people talking about it in a conversational way in an attempt to make the information easier to digest without reading source material. I get how that could be useful if you want to listen while cooking, driving, or doing other activities where reading might not be an option, but as a podcast & radio news producer, a little piece of me dies every time I think about these fake podcasts.

Some other new & upcoming Copilot features include:

Pages: Organize all of your stuff from a variety of notes & searches into one simple place.

Deep Research: "Conduct complex, multi-step research tasks more efficiently" with Copilot, which can now "find, analyze and combine information from online sources or large amounts of documents and images."

Copilot Search: With a catchy name like "Copilot Search in Bing," what's not to love? Anyway, this is Microsoft's answer to Google Search's AI Overviews. Microsoft says "it cross-checks information across multiple sites to deliver detailed, comprehensive responses directly in Bing, complete with helpful cited sources and suggestions for further exploration."

That last one… seriously makes me wonder about the future of the internet. Not only do AI overviews (or whatever Microsoft wants to call its version) sometimes present incorrect information, but they do it in a plausible enough way that even though I'm skeptical I often find myself glancing at a result and not bothering to click through to the source material.

But where do Google, Microsoft, OpenAI, and other companies get the material they use to power those generative AI summaries? Websites (like the one you're reading). So what happens when the results get good enough (or searchers get lazy enough) that a vanishingly small number of people click through to a website after conducting a search? Websites make less money, web publishers spend less time creating and updating those sites, and the training data starts to dry up.

It's a problem that I have yet to see any real proposed solution to. Maybe that's because it's still early days and the more noticeable AI + web publishing problem at the moment is the proliferation of AI slop websites that have popped up in recent years.

Anyway, you can read about new Copilot features in Microsoft's announcement.

Liliputing's primary sources of revenue are advertising and affiliate links (if you click the "Shop" button at the top of the page and buy something on Amazon, for example, we'll get a small commission).

But there are several ways you can support the site directly even if you're using an ad blocker* and hate online shopping.

Contribute to our Patreon campaign

or...

Contribute via PayPal * If you are using an ad blocker like uBlock Origin and seeing a pop-up message at the bottom of the screen, we have a guide that may help you disable it. Join 9,541 other subscribers