ChatGPT has been updated again. The Mac version has a new "screen reading programming" function, and the Windows desktop client is officially open to all users.
Mac version of Plus users can now let ChatGPT read code directly from the development tools. Currently, this feature supports five programming tools, including Xcode, VS Code, TextEdit, Terminal and iTerm. Users need to enable the "Accessibility" permission for the ChatGPT client in the system settings to use it. ChatGPT processes the entire code in the frontmost window of the editor (until the window length limit is exceeded) and, if part of the content is selected, focuses on the selection. For terminal tools, ChatGPT processes the last 200 lines in the window. This feature is currently available to Plus and Team subscribers, and will be available to Enterprise and Edu users in a few weeks. The name of the function "Work with Apps" implies that more applications may be supported in the future, perhaps as part of the OpenAI intelligent agent plan.
However, this feature is currently implemented by calling Apple's "Accessibility" interface, rather than based on visual reading of screen content. The newly generated code also needs to be copied and pasted manually, and the user experience still needs to be improved. In contrast, Anthropic and Google already have vision-based screen manipulation capabilities.
At the same time, the long-awaited desktop client for Windows users has also been fully launched and is also available to free users. The new version of the client supports advanced voice mode and the latest search functions, and can be called from any interface through the Alt+spacebar shortcut key. Users can enter text in the pop-up window, upload screenshots, take photos with their camera, or upload local files. Although the Microsoft App Store shows that the client was launched on September 7, the OpenAI official website information shows that the beta version was opened to paying users in October. The Windows client is now available to all users. It should be noted that Windows 10 users need version 17763.0 or higher to use it.
Vinci AI Review
The launch of ChatGPT’s “screen reading programming” function marks the further development of AI programming assistance tools. Although it still relies on interface calls and manual copy and paste, its ability to read program code and generate new program code is of great significance for improving programming efficiency, especially to assist in introductory programming learning. In the future, as technology advances, vision-based code understanding and more seamless code interaction will become a trend. At the same time, combined with the emphasis on STEM in Hong Kong’s education policy, integrating such AI tools into programming teaching may further stimulate students’ interest in learning and cultivate students’ computational thinking abilities, which is worthy of the attention and exploration of educators.
Original link:
https://beebom.com/chatgpt-app-can-see-screen-macos/
