Okay,now that thats off my chest ill explain to you people what i coded in the last month
this is the Ex0rMulti API scraper kit which includes this:
1. reading commands from text file and executing them in a scrapeable matter #will be explained later on
2. scraping as many sites as you want by changing the last digit.
what i mean by that
google.com/ ->
google.com/2
google.com/3
3.Multithreaded workload,good for websites where the loading times are lower the the the time to request to load them. (plain text sites like APIs)
where:
explanation
a single thread:
Request website 3000ms
load site 1000ms
Request website 3000ms
load site 1000ms
2 websites in 8000ms (8 seconds)
2 threads
thread 1 | thread 2
Request website 3000ms Request website 3000ms -> those happen at the same time
|
|
load site 1000ms |load site 1000ms -> those both take 1 second each and will take 2 seconds together because it happens at the same time because of your internets speed
2 websites in 5000ms (5 seconds) 3000+1000+1000
this isnt completely true of course,but it shows the logic behind waiting for something to load and loading it
5. Retrying when failing to download (more on this laster on)
6. comes with thread creator,acctual thread,command file
this is how it looks when decompiled
this is how it looks running
screenshot
As you can see 5 threads are running scraping around 1 page of my api every 3 seconds,EACH
thats 26X5/3 = users pers second (43 in this example,running on a crappy pc and horrible internet connection)
How to use the Ex0rMulti:
DISCLAIMER: this is not a program for people that want a good user experience (yet) this program is very basic and does what it should. it wont correct you. it will spit out an error you will find to be nonsense. feel free to psot a screenshot of it in the thread but dont be suprised if things are still broken
1. Download and freaking THANK me for createing this piece of software
2. create a folder and extract the contents of the rar into it
3. look at the runners.txt text file,this is where you will find the command instructions.
every new line is a new thread
it goes like this:
(thread executable) ("starting page") ("end page") ("url to scrape where the last number that should be a page number is not there") ("milliseconds between retries to scrape a page,necessary to let the other threads continue")
i have some presets inside so you can take a look.
press insert preset,make the page field empty and look how the url looks. http://www.mcbans.com/servers/ <- here you can get server urls
then insert however you want or just leave it at the default beastmc.
now you press "create scraping link" and loko at the url it generated.
you can use it in the url parameter in the runners.txt file
now all you do is make the threads like you like em etc,and then save the runners.txt file and run the Ex0rMulti executable too load all the command prmopts that show you the progress on each on of the threads.
closing the command promts will close the thread,also pressing f11 will close one of them so watch out.
now wait for it to finish,you will get a tray notification that it finished.
disclaimer:
thread no1 sometiems doesnt close by itself at the end,im working on fixing that bug.
now it wrote al the scraped usernames to scraped.txt in the same folder where its located.