X's Latest Terms Update - No More Crawling or Scraping

· 2 min read
Elon Musk attend the opening ceremony of the new Tesla Gigafactory for electric cars in Gruenheide, Germany / Patrick Pleul/Pool via REUTERS

Elon Musk's company, X, previously known as Twitter, has made significant updates to its terms of service, primarily aimed at preventing unauthorized scraping and crawling of its platform. These changes came into effect on September 29 and explicitly prohibit any form of scraping or crawling without obtaining prior written consent.

Previously, the terms allowed crawling if it adhered to the directives set in the robots.txt file. However, under the new terms, any scraping without the platform's explicit approval is expressly forbidden.

NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.

In recent months, Twitter also modified its robots.txt file, essentially instructing web crawlers to focus solely on Google. This suggests a potential shift in their approach, as in 2015, Twitter had a significant deal with Google to feature tweets in search results. It remains to be seen whether this arrangement has undergone any changes with the recent management transition.

terms of services-x-twitter
X updates its terms to ban crawling and scraping / Twitter

X has further revised the robots.txt file to disallow crawlers from accessing information such as likes and retweets linked to specific posts and more.

Earlier this year, Twitter briefly restricted access to posts for logged-out users and later removed the login requirement to view tweets. These actions were prompted by concerns about excessive data scraping, which adversely affected service quality for regular users.

Google’s Privacy Policy Update - Implications for User Data and AI Development
Discover how Google’s privacy policy empowers the development of AI through data.

Elon Musk has been a vocal opponent of companies using Twitter/X data to train AI models. In July, he initiated a scraping lawsuit against unidentified entities.

Notably, X recently updated its privacy policy, indicating its intention to utilize public data for AI model training. Musk had previously mentioned that xAI, founded in July, would employ public data, including tweets, for this purpose.

X’s Privacy Policy and AI Training with Public Data?
Uncover the revelations in X’s privacy policy, where public data fuels AI training.

Remember, in a previous article, we discussed how Twitter is planning to collect biometric data from its users. So, the revised privacy policy also includes provisions for collecting users' biometric data, educational background, and employment history.