Elon Musk's company, X, previously known as Twitter, has made significant updates to its terms of service, primarily aimed at preventing unauthorized scraping and crawling of its platform. These changes came into effect on September 29 and explicitly prohibit any form of scraping or crawling without obtaining prior written consent.
Previously, the terms allowed crawling if it adhered to the directives set in the robots.txt file. However, under the new terms, any scraping without the platform's explicit approval is expressly forbidden.
NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.
In recent months, Twitter also modified its robots.txt file, essentially instructing web crawlers to focus solely on Google. This suggests a potential shift in their approach, as in 2015, Twitter had a significant deal with Google to feature tweets in search results. It remains to be seen whether this arrangement has undergone any changes with the recent management transition.
X has further revised the robots.txt file to disallow crawlers from accessing information such as likes and retweets linked to specific posts and more.
Earlier this year, Twitter briefly restricted access to posts for logged-out users and later removed the login requirement to view tweets. These actions were prompted by concerns about excessive data scraping, which adversely affected service quality for regular users.
Chanuka Nadun Perera
Elon Musk has been a vocal opponent of companies using Twitter/X data to train AI models. In July, he initiated a scraping lawsuit against unidentified entities.
Notably, X recently updated its privacy policy, indicating its intention to utilize public data for AI model training. Musk had previously mentioned that xAI, founded in July, would employ public data, including tweets, for this purpose.
Chanuka Nadun Perera
Remember, in a previous article, we discussed how Twitter is planning to collect biometric data from its users. So, the revised privacy policy also includes provisions for collecting users' biometric data, educational background, and employment history.
