AJAX pagination but drill down one level

How to?
#1

Hey guys -

I have built several maps successfully but this one isnt' working for me. I need to visit each item's details page to scrape data there but then go the next page using the "next" button that is NOT a link (must use element click selector).

I have tried this a few different ways but can't get it to work.

See below:

{"_id":"pollstarpagetest","startUrl":["https://www.pollstar.com/festival-list"],"selectors":[{"id":"nextpage","type":"SelectorElementClick","selector":"div.ps-action.ps-btn.right","parentSelectors":["_root","nextpage"],"multiple":false,"delay":"2000","clickElementSelector":"div.ps-action.ps-btn.right","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"items","type":"SelectorLink","selector":"div.ps-festival a","parentSelectors":["_root","nextpage"],"multiple":true,"delay":0},{"id":"DetailTabName","type":"SelectorText","selector":"a.artist-name","parentSelectors":["items"],"multiple":false,"regex":"","delay":0},{"id":"pageURL","type":"SelectorText","selector":"div.contact-card__text a.ps-link","parentSelectors":["items"],"multiple":false,"regex":"","delay":0}]}

#2

Hi
this site is not very reliable. I will try to test it later .....

We're sorry, an internal error occurred.

Our supporting staff has been notified of this error and will address the issue shortly.

We apologize for the inconvenience.

Please try clicking your browsers 'back' button or try reloading the home page.

If you continue to receive this message, please try again in a little while.

Thank you for your patience.

#3

Looks like it's back up! They had an issue yesterday i think (bad timing!)

http://www.pollstar.com/festival-list

#4

Hi,

I started yesterday to scrap the first page with an "Element" selector and had to increase the delays parameters for title, location, date because I scraped only 28 items on the first page that contains 30 items.
Perhaps it was due to errors on their plateforme . So today I have not tested if you can leave delay to zero.

On the second step I change the selector : "Element" --> " Element clic" to take into account the pagination.

It seems to scrap correctly all the pages. Here is the sitemap corrected:

{"_id":"test_pollstar","startUrl":["https://www.pollstar.com/festival-list"],"selectors":[{"id":"item","type":"SelectorElementClick","selector":"div.ps-festival","parentSelectors":["_root"],"multiple":true,"delay":"3000","clickElementSelector":"div.ps-action.ps-btn.right","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"link","type":"SelectorLink","selector":"a:nth-of-type(1)","parentSelectors":["item"],"multiple":true,"delay":"0"},{"id":"title","type":"SelectorText","selector":"a.artist-name","parentSelectors":["link"],"multiple":false,"regex":"","delay":"1000"},{"id":"location","type":"SelectorText","selector":"a.ps-spacer","parentSelectors":["link"],"multiple":false,"regex":"","delay":"1000"},{"id":"date-from","type":"SelectorText","selector":"dd#recordFrom","parentSelectors":["link"],"multiple":false,"regex":"","delay":"1000"}]}