What are the rules associated with web scraping?

[Holmesdr]

I starting learned Beautiful Soup and want to test myself by doing some projects but I found not all websites allow web scraping and somethings about robots.txt. What are the legal things associated with it anyone advise me about what should I do or some projects?

[Holmesdr]

I have found solutions for test my project https://mydataprovider.com/solutions/web-scraping/. It's provide any type of data extraction configuration the needs of me

[Matthew]

You could scrape all external links that the site gives you (does it redirect you to a third party, etc) or be boring and scrape the image links that are embedded in the site's code. I'm not to brushed up on web scraping

[Bababam]

Are you getting a specific error/stack trace?

You should be able to open a standard HTTPS connection to the webpage you want to scrape. The server wont know the difference between your program or a users web browser. If you are having problems in this area then either check that you are setting the correct spoofed HTTP headers such as device type, etc. Or, change your implementation to use an existing browser in headless mode for your HTTPS connection.

[JackGambler]

i cant read and

[PeterBrodbeck]

It is also possible to use a headless browser, such as Puppeteer or Selenium, to scrape web pages. This approach can provide greater flexibility and control over the scraping process, as it allows you to interact with the page as if you were using a real browser.