I starting learned Beautiful Soup and want to test myself by doing some projects but I found not all websites allow web scraping and somethings about robots.txt. What are the legal things associated with it anyone advise me about what should I do or some projects?
You could scrape all external links that the site gives you (does it redirect you to a third party, etc) or be boring and scrape the image links that are embedded in the site's code. I'm not to brushed up on web scraping
You should be able to open a standard HTTPS connection to the webpage you want to scrape. The server wont know the difference between your program or a users web browser. If you are having problems in this area then either check that you are setting the correct spoofed HTTP headers such as device type, etc. Or, change your implementation to use an existing browser in headless mode for your HTTPS connection.