Pagination with dynamic popups

james_rockhill4500 · March 8, 2023, 6:39am

Please help. I've spent hours and hours trying to figure out why this won't work.

I'm struggling to figure out how to do pagination with dynamic popups. I have two problems with the following sitemap. There's a total of five pages at this url. Each page loads 10 items. Each item has a button that opens a dynamic popup. The sitemap is supposed to click each of the buttons in turn, grab the data from the popup, close the popup, then open the next popup. After reaching the end of the page, it then is supposed to go to the next page and start over, continuing this until there are no more pages. When I tell it to scrape, it seemingly does this. I can sit here and watch it open all of the popups from each page. However, there are two issues:

First, the first page only opens the first dynamic popup. After opening the first popup on the list of page "1", it doesn't open the next popup, it just skips to page "2". It then starts opening the popups on the new page just like its supposed to. It runs perfectly from there, clicking the link to open each of the 10 popups on page "2" then going to page "3" and opening each of the ten there and so on on each subsequent page until the end. Why does it only open the one popup on page "1" instead of all 10 like it does on the rest of the pages?

Second (and this is the bigger issue), after it runs through the entire scrape, there's no data saved. I don't understand it at all. It seems to work perfectly, but there's just no data when it's done.

Url: https://locator.lcms.org/church?range=0&state=WYOMING&search=WYOMING&s=y

Sitemap:

{"_id":"lcms-wyoming","startUrl":["https://locator.lcms.org/church?range=0&state=WYOMING&search=WYOMING&s=y"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"auto","selector":"a.ui-paginator-page","type":"SelectorPagination"},{"id":"element-card","parentSelectors":["pagination"],"type":"SelectorElement","selector":"app-card td","multiple":true},{"id":"Church-Name","parentSelectors":["element-card"],"type":"SelectorText","selector":"dt","multiple":false,"regex":""},{"id":"profile-link","parentSelectors":["element-card"],"type":"SelectorPopupLink","selector":"div.btn","multiple":false},{"id":"Church-mailining-address","parentSelectors":["profile-link"],"type":"SelectorText","selector":".ui-tabview-panel dt:contains('Mailing Address') + dd","multiple":false,"regex":""},{"id":"church-email","parentSelectors":["profile-link"],"type":"SelectorText","selector":".ui-tabview-panel a.single-contact","multiple":false,"regex":""}]}

james_rockhill4500 · March 8, 2023, 8:23pm

Update: I was able to figure out the pagination issue. I changed the element selector to a parent element, and that seems to have fixed it. The new sitemap is below. It now seemingly runs the scrape flawlessly, but there is no data when it is finished. The preview data buttons return the data just as you would expect. I cannot find the problem with this. Please help!

{"_id":"locator-wyoming","startUrl":["https://locator.lcms.org/church?range=0&city=&state=WYOMING&search=WYOMING&s=y"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"a.ui-paginator-page","type":"SelectorPagination"},{"id":"element-card","multiple":true,"parentSelectors":["pagination"],"selector":"app-card","type":"SelectorElement"},{"id":"Church-Name","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"dt","type":"SelectorText"},{"id":"profile-link","multiple":false,"parentSelectors":["element-card"],"selector":"div.btn","type":"SelectorPopupLink"},{"id":"Church-Mailing Address","multiple":false,"parentSelectors":["profile-link"],"regex":"","selector":".ui-tabview-panel dt:contains('Mailing Address') + dd","type":"SelectorText"},{"id":"Church-Email","multiple":false,"parentSelectors":["profile-link"],"regex":"","selector":".ui-tabview-panel a.single-contact","type":"SelectorText"}]}

ViestursWS · March 9, 2023, 3:43pm

@james_rockhill4500 Hi, the targeted button does not function as a 'pop-up', therefore you should substitute it with an 'Element click' selector instead.

Please, note that the results will start to appear only once the 'Element click' selector has stopped its execution.

Learn more: My scraping job is running, although no results are being returned - Web Scraper Knowledge Base

Sitemap example:

{"_id":"locator-wyoming-2","startUrl":["https://locator.lcms.org/church?range=0&city=&state=WYOMING&search=WYOMING&s=y"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"a.ui-paginator-page","type":"SelectorPagination"},{"clickElementSelector":"app-card div.btn-primary:contains(\"Profile\")","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"element-card","multiple":true,"parentSelectors":["pagination"],"selector":"div.ng-trigger-animation","type":"SelectorElementClick"},{"id":"Church-Name","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"h5","type":"SelectorText"},{"id":"Church-Mailing Address","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel dt:contains('Mailing Address') + dd","type":"SelectorText"},{"id":"Church-Email","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel a.single-contact","type":"SelectorText"},{"clickElementSelector":"div.modal-action","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1800,"discardInitialElements":"discard-when-click-element-exists","id":"close-click","multiple":true,"parentSelectors":["element-card"],"selector":"_parent_","type":"SelectorElementClick"}]}

james_rockhill4500 · March 9, 2023, 7:23pm

@ViestursWS.

This is extremely helpful! I have been playing with the elementclick selector, and, honestly, I've been completely unable to figure out how it works. I feel pretty confident with most of the rest of them. I've not found the support pages for this selector all that helpful. I simply can't wrap my mind around what to put in the "selector" field vs the "click selector" field. Even after looking at this example, I am confused. How did you come up with "parent" in the one and "div.ng-trigger-animation" in the other? I can't even find any CSS that had "ng-trigger-animation" in it if I did understand how exactly this selector works. Can you point me in the direction of some further guidance for this selector, or give me a brief rundown? Thank you so much for your help already, as well. This is great!

ViestursWS · March 10, 2023, 11:21am

@james_rockhill4500 Depending on the sitemap navigation pattern the _parent_ selector is used to target the same parent element as the previous selector. This is all based on the structure of the HTML code.

For example, if the first navigation level 'Element click' selector targets body, the second level navigation selector can use the same parent element without having to copy it multiple times.

The 'div.ng-trigger-animation' corresponds to the detail modal wrapper.

james_rockhill4500 · March 18, 2023, 5:25pm

@ViestursWS Thank you again for your help. I did and do understand parent and child elements. I meant how did you know to use "parent" to target the parent element? I have since found this in the webscraper documentation. Thanks for the information.

I am still having issues with the pagination on this however. I was able to put together a sitemap that grabs all the data I am looking for. However, the pagination still isn't working correctly. Even the example you provided doesn't seem to work. It just keeps looping through page 1.

james_rockhill4500 · March 20, 2023, 10:45pm

I'm still not sure what is going on with the pagination with this. Sometimes it would loop through page one several times. Sometimes it would loop through all the pages. I tried specifying the pagination type instead of using auto, I also tried using the carrot selector for going through the pages. I continued to have the same issue no matter what I tried.

I solved the issue by creating an element click selector that handled pagination. It seems to be working correctly. Here is the sitemap in case anyone else experiences this issue and needs help.

{"_id":"lcms_locator_churches_by_state_manual_pagination","startUrl":["https://locator.lcms.org/church?range=0&state=WYOMING&search=WYOMING&s=y"],"selectors":[{"clickElementSelector":"a.ui-paginator-page","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"pagination","multiple":true,"parentSelectors":["_root","pagination"],"selector":"div.main-panel","type":"SelectorElementClick"},{"clickElementSelector":"app-card div.btn-primary:contains(\"Profile\")","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"element-card","multiple":true,"parentSelectors":["pagination"],"selector":"div.ng-trigger-animation","type":"SelectorElementClick"},{"id":"Name","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"h5","type":"SelectorText"},{"id":"Address","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".address-title","type":"SelectorText"},{"id":"Mailing-Address","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel dt:contains('Mailing Address') + dd","type":"SelectorText"},{"id":"District","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(District)","type":"SelectorText"},{"id":"Cicruit","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(V-Circuit)","type":"SelectorText"},{"id":"Pastor(1)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd","type":"SelectorText"},{"id":"Pastor(2)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(2)","type":"SelectorText"},{"id":"Pastor(3)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(3)","type":"SelectorText"},{"id":"Pastor(4)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(4)","type":"SelectorText"},{"id":"Pastor(5)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(5)","type":"SelectorText"},{"id":"Pastor(6)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(6)","type":"SelectorText"},{"id":"Pastor(7)","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".col-lg-3 dd:nth-of-type(7)","type":"SelectorText"},{"id":"Average-Weekly-Attendance","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(Average Weekly Attendance)","type":"SelectorText"},{"id":"Baptized-Membership","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(Baptized Membership)","type":"SelectorText"},{"id":"Confirmed-Membership","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(Confirmed Membership)","type":"SelectorText"},{"id":"Phone","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"dt:contains('Phone') + .ng-star-inserted a","type":"SelectorText"},{"id":"Email","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"dt:contains('Email') + .ng-star-inserted a","type":"SelectorText"},{"id":"Website","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":"dt:contains('Website') + .ng-star-inserted a","type":"SelectorText"},{"id":"Date Organzied","multiple":false,"parentSelectors":["element-card"],"regex":"","selector":".ui-tabview-panel li:contains(Organized Date)","type":"SelectorText"},{"id":"School(s)","multiple":true,"parentSelectors":["element-card"],"regex":"","selector":"church-schools p.result-title","type":"SelectorText"},{"clickElementSelector":"div.modal-action","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"Close-click","multiple":true,"parentSelectors":["element-card"],"selector":"_parent_","type":"SelectorElementClick"}]}

james_rockhill4500 · March 21, 2023, 1:03am

Of course, as I was leaving for the night, I told this to start scraping, and it is having the same pagination issue as it did with paginator selector. I assume that whatever is going has to do with how the page is rendered, and it doesn't think it is done finding all the profile button elements on that page. If anyone has any suggestions how to fix this, I would appreciate it. Every time I think I have it figured out, I find out that I don't.

ViestursWS · March 23, 2023, 2:11pm

@james_rockhill4500 Hi, it does not appear as if the pagination setup is configured properly.

Here's an updated version:

Scraped data:

james_rockhill4500 · March 25, 2023, 3:02am

@ViestursWS, You're exactly right. I'm unsure why I had it set up the way I did. Anyway, this is perfect! I still wonder why the pagination selector was struggling. Thank you so much for all of your help with this.