Howdy! I'm currently doing a learn mission on recordsdata science where I in point of fact had been tasked with scraping 40,000 LinkedIn profiles.
I've already written the code to scrape profiles utilizing BS4+Selenium with the Firefox webdriver. The code begins with my hang LinkedIn profile, and keeps adding the URLs of the « Urged Profiles » to the listing of profiles to be scraped. I've made a dummy yarn to witness these profiles. The scraping itself works swish, but my accounts lend a hand getting restricted. I've tried diverse delays in the execution to manufacture it seem extra human-like, on the opposite hand it keeps getting flagged after ~400 profiles.
I'm appropriate a pupil so I will't come up with the money for any delicate APIs in the market on-line. What's the finest manner to head about doing this?