I've been experimenting with Element Click and Pagination as ways to get Web Scraper to initiate a file download from an tag. Both work, but I get a "Page Timeout" error with both during the download and the scrape ends abruptly.
I'm not sure if its timing out waiting for the download to complete or because its waiting for a new page that never comes. Any way to ignore errors or wait forever for the next page? Any suggestions on how to do this another way?
Hi, can you provide more information on how this issue can be reproduced?
Sure. I'll try to explain.
I have a multi-page / pagination scrape that has pages full of links to detail pages. Each detail page contains a bunch of description text elements and a button that when clicked opens a new URL to initiate the file download - The button URL looks something like this...
"https://content-files.website.com/download/z967d860/filename.ext?filename=Name_of_file&download=attachment&expires=1745776413&token=2a9b2829ced336793e68f248a407a308"
As part of the page scrape, I grab this URL data from the details page using a Link selector called "DownloadLink". I click on "DownloadLink" and add an element selector called "ButtonClick" which looks like below.
"id":"ButtonClick","multiple":false,"parentSelectors":["downloadlink"],"selector":".ct_dl_fps_60 .ct_dl_columns_2 li:nth-of-type(1) a","type":"SelectorElementClick"}
Without this ButtonClick element the scrape goes through all the detail page links and grabs the expected details data from each element. When I add the "ButtonClick" element under the "Downloadlink" and run the scrape again it grabs all the detail page links and when it drills into the first detail page, it grabs all the data, initiates the file download but then stops ending with an eventual "Page Timeout" error.
Same here for me! Seems like there is a time-out happening? Can the scraper just not handle too many large docs? Support, you out there somewhere?
Sorry, but I cannot troubleshoot this without being able to see your full sitemap.
Seems like I’m trying to make it do something it wasn’t designed for. I appreciate the assistance however I’ve taken a different approach to file downloading using a powershell script and Selenium.