Popup Link not working as it should

How to?
#1

Hello,

I am trying to catch data within a popup using the type "popup link" but it seems that Web Scraper does not want to scrape it. When I start scraping, the popup opens and everything seems to work correctly, but when looking in the results I see only "null"
Any idea what I did wrong? Thanks in advance :slight_smile:

URL: Schmitt und Zerreissen GmbH

Sitemap:
{"_id":"unterkonstruktion_test","startUrl":["https://schmitt-zerreissen.de/preisliste?filter=unterkonstruktion"],"selectors":[{"id":"produkt","type":"SelectorPopupLink","parentSelectors":["_root","produkt"],"selector":"div.col-md-5","multiple":true,"delay":0},{"id":"artikelbezeichnung","type":"SelectorText","parentSelectors":["produkt"],"selector":".modal.show p.lead","multiple":false,"regex":"","delay":0},{"id":"preis","type":"SelectorText","parentSelectors":["produkt"],"selector":".modal.show div.row:nth-of-type(2) div:nth-of-type(2)","multiple":false,"regex":"","delay":0}]}

#2

Hello @wullner
Would you please use the preformatted text option when you paste your sitemap because I was not able to open it, it shows that the JSON is invalid, try to copy your sitemap from here yourself and you will see what I'm talking about.

#3

Hello @ViestursWS
Thanks for the hint. I edited my post und you should now be able to open it.
Thanks and greetings
wullner

1 Like
#4

face same issue in this site: Nibco | Chariot Plumbing Supply and Design - Salt-Lake-City-Utah

Here my sitemap:
{"_id":"chariotwholesale","startUrl":["https://www.chariotwholesale.com/Nibco-v1.htm"],"selectors":[{"id":"ELEMENT CLICK","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.col-xl-3","multiple":true,"delay":2000,"clickElementSelector":".container-fluid .container-fluid i.fa-chevron-right","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"POPUP","type":"SelectorPopupLink","parentSelectors":["ELEMENT CLICK"],"selector":"div.block2-overlay","multiple":false,"delay":0},{"id":"Category","type":"SelectorGroup","parentSelectors":["POPUP"],"selector":"a.crumb","delay":0,"extractAttribute":""},{"id":"Product name","type":"SelectorText","parentSelectors":["POPUP"],"selector":"h4.d-print-none","multiple":false,"regex":"","delay":0},{"id":"sku","type":"SelectorText","parentSelectors":["POPUP"],"selector":"p.d-print-none","multiple":false,"regex":"","delay":0}]}

#5

@wullner You can't use link or pop-up link selector because when you click on the product the web page address stays the same(there is no link change). Thats why the only possible way seems to be element-click selector.

Here's the updated version:
{"_id":"unterkonstruktion_test-edit","startUrl":["https://schmitt-zerreissen.de/preisliste?filter=unterkonstruktion"],"selectors":[{"id":"produkt","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"body:has(h5.lead)","multiple":true,"delay":"1000","clickElementSelector":"div.card-body > div[data-toggle=\"modal\"]","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector","clickActionType":"auto"},{"id":"artikelbezeichnung","type":"SelectorText","parentSelectors":["element-card"],"selector":"p.lead ","multiple":false,"regex":"","delay":0},{"id":"preis","type":"SelectorText","parentSelectors":["element-card"],"selector":"div.no-gutters div:contains(\"Preis\") + div","multiple":false,"regex":"","delay":0},{"id":"element-card","type":"SelectorElement","parentSelectors":["produkt"],"selector":".modal.show div.modal-content","multiple":true,"delay":0},{"id":"close-click","type":"SelectorElementClick","parentSelectors":["element-card"],"selector":"_parent_","multiple":false,"delay":"1000","clickElementSelector":"button.close","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText","clickActionType":"auto"}]}

Hope it helps!

1 Like
#6

@emamulbd This page contains a lot of hidden links. I managed to get to the product details and pagination.

{"_id":"chariotwholesale","startUrl":["https://www.chariotwholesale.com/Nibco-v1.htm"],"selectors":[{"id":"ELEMENT CLICK","type":"SelectorElement","parentSelectors":["_root","PAGINATION"],"selector":"div#content > div > div ","multiple":true,"delay":0},{"id":"PRODUCT-LINK","type":"SelectorLink","parentSelectors":["ELEMENT CLICK"],"selector":"div.position-relative a","multiple":false,"delay":0},{"id":"Category","type":"SelectorGroup","parentSelectors":["PRODUCT-LINK"],"selector":"a.crumb","delay":0,"extractAttribute":""},{"id":"Product name","type":"SelectorText","parentSelectors":["PRODUCT-LINK"],"selector":"h4.d-print-none","multiple":false,"regex":"","delay":0},{"id":"sku","type":"SelectorText","parentSelectors":["PRODUCT-LINK"],"selector":"p.d-print-none","multiple":false,"regex":"","delay":0},{"id":"PAGINATION","type":"SelectorLink","parentSelectors":["_root","PAGINATION"],"selector":"a[rel=\"next\"]:nth(1)","multiple":true,"delay":0}]}

#7

thank you so much. i also solve this isse and able to scrap multipage. Here my sitemap
{"_id":"chariotwholesale","startUrl":["https://www.chariotwholesale.com/Nibco-v1-Page-[1-100].htm?continueSort=true"],"selectors":[{"id":"element click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.col-xl-3","multiple":true,"delay":2000,"clickElementSelector":"> div.d-none .pagination li:nth-of-type(3)","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"link","type":"SelectorLink","parentSelectors":["element click"],"selector":"a","multiple":false,"delay":0},{"id":"Catagory","type":"SelectorGroup","parentSelectors":["link"],"selector":"a.crumb","delay":0,"extractAttribute":""},{"id":"Product Title","type":"SelectorText","parentSelectors":["link"],"selector":"h4.d-print-none","multiple":false,"regex":"","delay":0},{"id":"SKU","type":"SelectorText","parentSelectors":["link"],"selector":"p.d-print-none","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["link"],"selector":".d-print-none .align-items-end h2","multiple":false,"regex":"","delay":0},{"id":"des","type":"SelectorText","parentSelectors":["link"],"selector":".pl-md-5 > p.disclaimer","multiple":false,"regex":"","delay":0},{"id":"short specitifation","type":"SelectorGroup","parentSelectors":["link"],"selector":"#product-details-list-specs li","delay":0,"extractAttribute":""},{"id":"Product Details_sku","type":"SelectorText","parentSelectors":["link"],"selector":"small strong","multiple":false,"regex":"","delay":0},{"id":"Product Details","type":"SelectorText","parentSelectors":["link"],"selector":".col-sm-12 p","multiple":false,"regex":"","delay":0},{"id":"Table 1 text","type":"SelectorText","parentSelectors":["link"],"selector":"div#productdetails2","multiple":false,"regex":"","delay":0},{"id":"Table 1 html","type":"SelectorHTML","parentSelectors":["link"],"selector":"div#productdetails2","multiple":false,"regex":"","delay":0},{"id":"Table 2 text","type":"SelectorText","parentSelectors":["link"],"selector":"div#productdetails3","multiple":false,"regex":"","delay":0},{"id":"image link","type":"SelectorImage","parentSelectors":["link"],"selector":"li img","multiple":false,"delay":0},{"id":"Table 2 html","type":"SelectorHTML","parentSelectors":["link"],"selector":"div#productdetails3","multiple":false,"regex":"","delay":0}]}

1 Like
#8

@ViestursWS Thank you so much, it works perfect. Now I try to understand what you did so that I can solve a problem by myself next time. Thank you again. I appreciate the time you spent to help :slight_smile:

Greetings

1 Like