Why won't this scrape this site for image link?

#1

I just cannot seem to figure out a way to scrape this site to grab the Health Product Declaration images listed. Originally, I tried initially setting it to grab the filters of Manufacturer, by Product Name which would have been nice, but didn't work. Now I am digging into thumbnails pages and beyond to just try to scrape it that way just to get something. It seems to go through the pages but in the end nothing is there. Any insight?

https://hpdrepository.hpd-collaborative.org/repository/HPDThumbnails/Forms/AllItems.aspx#InplviewHasha393bdaa-4646-4640-b915-1dc8fda84d58=

Sitemap:
{"_id":"hpdbasic","startUrl":["https://hpdrepository.hpd-collaborative.org/repository/HPDThumbnails/Forms/AllItems.aspx#"],"selectors":[{"id":"containerdoclinkbasic","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"a.ms-listlink","multiple":true,"delay":0},{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root","pagination"],"selector":"#pagingWPQ2next > a","multiple":true,"delay":"4000","clickElementSelector":"#pagingWPQ2next > a","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"hpdimage","type":"SelectorImage","parentSelectors":["containerdoclinkbasic"],"selector":"body > img","multiple":false,"delay":0}]}

#2

Hi JennyK

not sure if I am getting your questions but I extracted the thumbnails url in below example, try it out.

{"_id":"a002","startUrl":["https://hpdrepository.hpd-collaborative.org/repository/HPDThumbnails/Forms/AllItems.aspx#"],"selectors":[{"id":"item","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"tr.ms-itmHoverEnabled","multiple":true,"delay":2000,"clickElementSelector":"td#pagingWPQ2next","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"name","type":"SelectorText","parentSelectors":["item"],"selector":"a.ms-listlink","multiple":false,"regex":"","delay":0},{"id":"url","type":"SelectorElementAttribute","parentSelectors":["item"],"selector":"a.ms-listlink","multiple":false,"extractAttribute":"href","delay":0},{"id":"picture_size","type":"SelectorText","parentSelectors":["item"],"selector":"span[dir]","multiple":false,"regex":"","delay":0},{"id":"file_size","type":"SelectorText","parentSelectors":["item"],"selector":"td:nth-of-type(6)","multiple":false,"regex":"","delay":0},{"id":"Modified","type":"SelectorText","parentSelectors":["item"],"selector":"span.ms-noWrap","multiple":false,"regex":"","delay":0}]}

#3

lego,
Thank you! I realize now it is a table. Tweaked the pagination a smidge and got them all!

Can I inquire about how you might approach it from this side of that site which displays all the relevant information pertaining to those image?
https://hpdrepository.hpd-collaborative.org/Pages/Results.aspx#

-To pull by each Manufacturer Name filter, and then click through each manufactuer's result to get to the popup image link once again.

This would be helpful to understand: 1) because the manufacturer name individual data preview results I have show a link " javascript {} " error, 2) because the pagination gets stuck.

Here was my sitemap for this type scrape.
{"_id":"hpdstartagainfront","startUrl":["https://hpdrepository.hpd-collaborative.org/Pages/Results.aspx#"],"selectors":[{"id":"manufname","type":"SelectorLink","parentSelectors":["_root"],"selector":"#ctl00_ctl47_g_4a75f100_8ac5_463f_9a59_10d8e03613dd_csr3 .ms-ref-unsel-longList a","multiple":true,"delay":0},{"id":"items","type":"SelectorElementClick","parentSelectors":["manufname"],"selector":"a.ms-srch-item-link","multiple":true,"delay":"3000","clickElementSelector":".ms-srch-Paging li:nth-of-type(n+2) a","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"HPDimage","type":"SelectorImage","parentSelectors":["items"],"selector":"#ctl00_ctl47_g_b2156794_f17b_4175_9696_ac8cc66a6ee0_csr227_item_hoverCommonBody > div.properties > a","multiple":false,"delay":0}]}

This is such a learning experience!

#4

hi Jenny

I see your difficulties to navigate the manufacturer one by one. I spend 10 minutes and I am stuck as well.

try to work around, I tried to search for "*" in the search and it shows 7856 results.

do you see it is a better starting point to do the scrapping?

Lego

#5

Hi Lego,
Thanks for responding. I did try a workaround of the 7856 docs with a pagination range of [1-7856] in the starting URL. It spent a bit over 4 hours looking as if it was scraping but in the end only gave me one partial result. An interesting observation during the process was it appeared to load a new result one at a time at the top of the results list (not a whole page worth). Thinking there is something to that which I should be considering but too inexperienced to know what it is.

#6

Maybe this will simplify the task.

{"_id":"hpdstartagainfront-new","startUrl":["https://hpdrepository.hpd-collaborative.org/Pages/Results.aspx#k=#s=[1-7856:15]"],"selectors":[{"id":"items","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".ms-srch-item","multiple":true,"delay":2000,"clickElementSelector":"a.ms-srch-item-link","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"img-link","type":"SelectorLink","parentSelectors":["items"],"selector":"div.ms-srch-hover-outerContainer > div.ms-srch-hover-innerContainer > div.ms-srch-hover-content > div.instructions > div.properties > a","multiple":false,"delay":0}]}

#7

Hi, evgenylukin
Thanks for the tip with the modification to the range. I will give that a try and see how that goes.