Describe the problem.
The website (https://www.woolworths.com.au/) is a single page app, written in Angular. The URL in the address bar does not change, and page contents are fetched by Javascript.
I am trying to build a sitemap for Product Categories:
In the navbar there are main categories. When you click on one of the links, it opens a side menu to subcategories. When click on an item in sub-categories, it opens a further menu for sub-sub-categories.
I am trying to capture this categories tree.
The web scraper sitemap looks fine, and "element preview" and "data preview" show results as expected.
However, when I actually run the scraper, I get empty csv file (with only headings but no data).
I am guessing this has to do with the dynamic nature of the website and that somehow I need to "click" each link so that the DOM content can get updated.
Please advise what to do
Sitemap:
{"_id":"wollies","startUrl":["https://www.woolworths.com.au/"],"selectors":[{"id":"topCat","type":"SelectorLink","parentSelectors":["_root"],"selector":"nav.categoryHeader-navigation a.categoryHeader-navigationLink","multiple":true,"delay":0},{"id":"midCat","type":"SelectorLink","parentSelectors":["topCat"],"selector":"nav.categoriesNavigation--category a.categoriesNavigation-link","multiple":true,"delay":0},{"id":"subCat","type":"SelectorLink","parentSelectors":["midCat"],"selector":"nav.categoriesNavigation--subcategory a.categoriesNavigation-link","multiple":true,"delay":0}]}