When I traverse the subcategories one by one, most of the subcategories have multiple pages (because the number of records are more than the website's threshold of records per page) and my pagination SiteMap work perfectly.
But on a few sub categories, there are no page links below because the total number of records in that subcategory is less than the page threshold. By this what I mean is, not even the page 1, link is shown below. In other words, there are no page links. For this subcategory, my webscraper does not scrape any records and it skips to the next subcategory. How can I prevent this?
Thanks
Hard to diagnose without a sitemap, but I'm guessing your sitemap is not structured properly. It probably looks like:
Root -> Go to L1 subcat -> Go to L2 subcat -> Click links -> scrape data
For the sitemap to work as you described, "Click links" need to operate in both L1 and L2 subcat, i.e. must be a child of both. You can set this in the Parent Selectors section.
OK.Thank you Leemeng.