Walled Culture has already written about the two–pronged attack by the copyright industry against the Internet Archive, which was founded by Brewster Kahle, whose Kahle/Austin Foundation supports this blog. The Intercept has an interesting article that reveals another reason why some newspaper publishers are not great fans of the site: The New York Times tried …
the internet archive doesn’t respect robots.txt:
the only way to stay out of the internet archive is to follow the process they created and hope they agree to remove you. or firewall them.
https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/