Solved - Popup menu/Filter tab - How do I select when URL doesn't change?

How to?
#1

Problem:
I am trying to select the Manufacturer tab which has a popup list of manufacturer names and click through to eventually select each manufacturer's displayed products.

The Manufacturer tab doesn't have an URL change but seems to have a popup filter list. Because I wasn't able to figure out how to select the Manufacturer tab properly, I resorted to making the actual popup list of Manufacturer names as my selector starting point. While I can see the elements and data preview of the individual selectors, the actual scraping process results in nothing. So something about that Manufacturer tab and the popup list is not constructed properly. Any guidance on this is much appreciated!

Site:
https://declare.living-future.org/

Sitemap:
{"_id":"declaretwo","startUrl":["Declare - International Living Future Institute > div > div.container.lf-filter-navigation > nav > div > ul > li:nth-child(6) > div > ul > div > li","multiple":true,"delay":0},{"id":"prodpages","type":"SelectorLink","parentSelectors":["manufacturereslist"],"selector":"a.lf-card","multiple":true,"delay":0},{"id":"prodname","type":"SelectorText","parentSelectors":["prodpages"],"selector":".single-page-header h3","multiple":false,"regex":"","delay":0},{"id":"descipriton","type":"SelectorText","parentSelectors":["prodpages"],"selector":"p","multiple":false,"regex":"","delay":0},{"id":"declareid","type":"SelectorText","parentSelectors":["prodpages"],"selector":"div.declare-option:nth-of-type(1) span","multiple":false,"regex":"","delay":0},{"id":"declareexpiration","type":"SelectorText","parentSelectors":["prodpages"],"selector":"div.declare-option:nth-of-type(2) span","multiple":false,"regex":"","delay":0},{"id":"declarationstatus","type":"SelectorText","parentSelectors":["prodpages"],"selector":"div:nth-of-type(3) li:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"prodimage","type":"SelectorImage","parentSelectors":["prodpages"],"selector":"#root > div > div > div:nth-child(2) > div.col-md-4 > div > div.image-primary.lf-featured-image.lf-sidebar-image.fsr-holder","multiple":false,"delay":0},{"id":"declareimage","type":"SelectorImage","parentSelectors":["prodpages"],"selector":".lf-label-image img","multiple":false,"delay":0}]}

#2

Hi,
1.Click on "manufacturer"
2.Click on name from list
3.In the top where its says "SELECTED FILTERS" click on "SHARE FILTERED VIEW"
4. This will give you a proper URL to the page for example:
https://declare.living-future.org/?filter=manufacturer%7C3form
Based on this create all the URLs
5.Web scrape first all the names from the list
6. In excel add the first portion of the URL to all the names: https://declare.living-future.org/?filter=manufacturer%7C + name
7. Bulk extract all the created URL's

#3

Hi!
Thank you for responding. This is all a learning experience for me. Two questions come to mind:

  1. I created a new site map to test just a scrape of the manufacturer names from the list (again I can see the data in the individual selector) resulted in nothing. This was the selector path and was my only selector for that action:

    li:nth-of-type(6) .col-md-4 li (I can view them all and tried just as text selector and various others, all scraped nothing)

  2. Back to URLs: I understand getting the manufacturer's URL from the shared filter/excel modification. Once I have all that excel info where and how does that get incorporated into the actual site map so that I can get my selector for the actual manufacturers' products and accompanying product information to run? (see the remainder of my original post's site map) . Am I going back to metadata section and somehow load all those excel created URLS as "Start URLs"? (hopefully not adding them individually since there are so many of them!)

#4

Hello, I never used the scraper before. I was just surfing and stumbled upon your question and thought to help out. So I really can not help you in depth I'm afraid...But looking at your questions

  1. I think you need .col-md-4 li as a selector
  2. Looking at documentation at LINK it seems that you need to do something like http://example.com/page/[url1-url2-url3-etc]. I would advice to only use 5 URLs in the beginning to see if it works.

Good luck

#5

Interestingly, you did set me on a good path for resolution! So, thank you! I already did a test with a few created URLs and put into the metadata start URLs and it did pull exactly how I wanted it to, all the way through to the product information!! So, thank you for that! Since each manufacturer's URL will be a bit different sadly I can't do a "range". That is ok, worse comes to worse I will paste each start URL in (it would probably take me longer if I tried to do a script/macro for that).

Thank you again for your help!

#6

You're welcome :slight_smile: Glad to be of assistance

Scrape multiple url's
#7

Unfortunately there is still no easy bulk import URL feature. You can manually edit the JSON to add URLs. Refer to

The goal would be get your URLs to look something like this in the sitemap JSON:

2021-01-30_072449
2021-01-30_072449857×793 71.9 KB

#8

Thanks! I did find another post about loading the urls into the sitemap similar to what you are showing and that ultimately worked for me. Mission accomplished!