Hey! Just checking if I setup my sitemap with a private directory and I wanted to check if everything is setup correctly. Could you let me know if this will work? There are a lot of pages and I didn't want to start the scraper to only find it breaks after 100 records.
I have the sitemap setup this:
- Profile links are setup to go into each link and get all the details in the profile link.
- Pagination is set to click next because the page numbers weren't being detected.
Here is how the next button is setup in the HTML
<div class="dataTables_paginate paging_bootstrap" id="DataTables_Table_0_paginate">
<ul class="pagination">
<li class="prev disabled"><a href="#">Prev Page</a></li>
<li class="active"><a href="#">1</a></li>
<li><a href="#">2</a></li>
<li><a href="#">3</a></li>
<li><a href="#">4</a></li>
<li><a href="#">5</a></li>
<li class="next"><a href="#">Next Page</a></li>
</ul>
</div>
So I opted for the pagination type "Click multiple times..."
Url: Log in to your EstimateOne account - you won't be able to access this publicly
Sitemap:
{"_id":"EstimateOne","startUrl":["https://app.estimateone.com/db/"],"selectors":[{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":".next a","type":"SelectorPagination"},{"id":"builder-profile-link","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root","pagination"],"selector":".data-table-link","type":"SelectorLink"},{"id":"builder-name","multiple":false,"parentSelectors":["builder-profile-link"],"regex":"","selector":"h2","type":"SelectorText"},{"id":"builder-address","multiple":false,"parentSelectors":["builder-profile-link"],"regex":"","selector":"div.company_details","type":"SelectorText"},{"id":"builder-number","multiple":false,"parentSelectors":["builder-profile-link"],"regex":"","selector":"div.d-table-row:nth-of-type(1) div.text-right","type":"SelectorText"},{"id":"builder-email","multiple":false,"parentSelectors":["builder-profile-link"],"regex":"","selector":".d-table-cell a","type":"SelectorText"},{"id":"builder-fax","multiple":false,"parentSelectors":["builder-profile-link"],"regex":"","selector":"div.d-table-row:nth-of-type(2) div.text-right","type":"SelectorText"},{"id":"builder-ActiveTenders","multiple":true,"parentSelectors":["builder-profile-link"],"selector":"#DataTables_Table_0 tbody tr","type":"SelectorElement"},{"id":"builder-active-tenders-Category","multiple":false,"parentSelectors":["builder-ActiveTenders"],"regex":"","selector":"td:nth-of-type(4)","type":"SelectorText"},{"id":"builder-active-tenders-ProjectValue","multiple":false,"parentSelectors":["builder-ActiveTenders"],"regex":"","selector":"td:nth-of-type(7)","type":"SelectorText"},{"id":"builder-RecentlyClosedTenders","multiple":true,"parentSelectors":["builder-profile-link"],"selector":"#DataTables_Table_1 tbody tr","type":"SelectorElement"},{"id":"builder-AwardedTenders(Last18Months)","multiple":true,"parentSelectors":["builder-profile-link"],"selector":"#DataTables_Table_2 tbody tr","type":"SelectorElement"},{"id":"builder-RecentlyClosedTenders-Category","multiple":false,"parentSelectors":["builder-RecentlyClosedTenders"],"regex":"","selector":"td:nth-of-type(4)","type":"SelectorText"},{"id":"builder-RecentlyClosedTenders-ProjectValue","multiple":false,"parentSelectors":["builder-RecentlyClosedTenders"],"regex":"","selector":"td:nth-of-type(7)","type":"SelectorText"},{"id":"builder-AwardedTenders(Last18Months)-Category","multiple":false,"parentSelectors":["builder-AwardedTenders(Last18Months)"],"regex":"","selector":"td:nth-of-type(4)","type":"SelectorText"},{"id":"builder-AwardedTenders(Last18Months)-ProjectValue","multiple":false,"parentSelectors":["builder-AwardedTenders(Last18Months)"],"regex":"","selector":"td:nth-of-type(7)","type":"SelectorText"}]}