I have released the 2.2 version of my Perl script archive.pl for saving URL sets in the Internet Archive. The most important improvement is that there is a Docker image now that contains TOR as service for a every 10 seconds IP rotation and a headless Firefox browser that betters the quality of HTML downloads and analysis. There is a blacklist, too, which spares by time by blocking social media URLs that only provide a login mask. Head over to the project page!
Leave a Reply