Web scraper help
Web scraper help
I am trying to scrape a website that has a large amount of .mp4 videos on it. I have tried a few tools like youtube-dl and you-get with out any luck.
Looking to see if there is any one with some python or programming skills that might be able to help,
Login to web site (Using cookies.txt) or Username /Password
Take a list of base urls from a text file
Spider that web page for any links that start with /player.aspx
Download any urls that have "var VideoSource_HD ="
Take the next base url repeat until all pages have been scraped.
Happy to share all the content scraped.
Looking to see if there is any one with some python or programming skills that might be able to help,
Login to web site (Using cookies.txt) or Username /Password
Take a list of base urls from a text file
Spider that web page for any links that start with /player.aspx
Download any urls that have "var VideoSource_HD ="
Take the next base url repeat until all pages have been scraped.
Happy to share all the content scraped.
are you looking for someone who can do this work for you for free?
Someone who could guide me would be nice .
got most of this now built need help on regex
Any one know regex well ? can you then hit me up on a PM ?
Any one know regex well ? can you then hit me up on a PM ?
re.findall("var VideoSource_HD =(.*?)\n", data)
Can any 1 do uber login bypass fetching script
Originally Posted by JaneD0e
I am trying to scrape a website that has a large amount of .mp4 videos on it. I have tried a few tools like youtube-dl and you-get with out any luck.
Looking to see if there is any one with some python or programming skills that might be able to help,
Login to web site (Using cookies.txt) or Username /Password
Take a list of base urls from a text file
Spider that web page for any links that start with /player.aspx
Download any urls that have "var VideoSource_HD ="
Take the next base url repeat until all pages have been scraped.
Happy to share all the content scraped.
Looking to see if there is any one with some python or programming skills that might be able to help,
Login to web site (Using cookies.txt) or Username /Password
Take a list of base urls from a text file
Spider that web page for any links that start with /player.aspx
Download any urls that have "var VideoSource_HD ="
Take the next base url repeat until all pages have been scraped.
Happy to share all the content scraped.
I tried something, but didn't test it.
import bs4
import requests
res = requests.get('URL')
#res.text can give you the whole page source
soup = bs4.BeautifulSoup(res.text, 'lxml')
for i in soup.select('var VideoSource_HD ='):
print(i.text)
Thanks for all the help I was able to scrape the site in the end just took a lot of trial and error. Shows I need to start learning to code by doing a few online courses :-)
So will you be sharing all content scraped as promised now?
This thread is closed for replies.
