MPGHThe Dark Arts
RegisterLog in

web crawler / mirror website

Posts 1–3 of 3 · Page 1 of 1
web crawler / mirror website
Hello,

I wanted to mirror a database to be able to do my research offline too, I tried using HTTrack, which was very slow (two days downloading and one day verifying the files), does anyone know a better way to mirror a database/online encyclopedia? (without crashing their website of course )
#1 ·
you can easily mirror most sites with a scraper... how useful that is questionable
#2 ·
Quote Originally Posted by Dave84311 View Post
you can easily mirror most sites with a scraper... how useful that is questionable
my plan for now would be to a rent cloud server with ubuntu from hetzner and use HTTrack with the setting "Maximum Number of links" on max value (999999999 as example, but be aware that not the wrong websites get downloaded since it scans all links), and then download it from the server to my computer via ftp or I upload it to my Mega account.

- - - Updated - - -

here is my test config from my local vm

I forgot to mention that I wanted to download 100k+ websites, so I want something that can run in the background or on a server since I need to restart my computer due to me testing diffrent drivers on windows 10 and not having a reliable internet connection
#3 · edited ·
Posts 1–3 of 3 · Page 1 of 1

Post a Reply

Similar Threads

PHP ProgrammingCreate a PHP web crawler/scraper in 5 minutes.iHolyElementiHolyElement 4 User Services[Web Design] Creating WebsitesDFDfo hacker13 0 User Services[Web Designing | Editing Websites] I do HTML & CSSNInigel123mc 2 GeneralTHE MOST ANNOYING WEBSITE ON THE WEBCyberGenius 12 User Services[Web Design] Website Support and Help~Shadow 2

Tags for this Thread

None

Need help?