Scrape has a mind of it's own! (Pagination issues)

Hey guys - really struggling with this scrape. At first glance it looks an easy straight forward one, but for some reason, the pagination sometimes works, sometimes doesnt. Sometimes rips the first page, sometimes the second, etc etc. Very hit and miss and I cant figure out why.

I am trying to scroll the pages and click their names as a link, then harvest the info inside.

Can anyone help me please? Much appreciated

https://www.loandepot.com/branches/rancho-cucamonga-ca/meet-the-team

Sitemap:
{"_id":"loandepotca1","startUrl":["https://www.loandepot.com/branches/rancho-cucamonga-ca/meet-the-team"],"selectors":[{"id":"agent","type":"SelectorLink","parentSelectors":["_root"],"selector":"h3.lo-card-name a.purple","multiple":true,"delay":0},{"id":"position","type":"SelectorText","parentSelectors":["agent"],"selector":"h3.lo-title","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.col-lg-12 nav","multiple":true,"delay":"4000","clickElementSelector":"a.page-link.page-link-next","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

I fixed this for you. A few things:

1 - Since the page changes but the URL doesn't, that's your clue to use Element Click Selector.

Element Click Selector does two things, it will trigger the page change but it also defines your rows (I.E) defines what you want to scrape. In this case, it.s people - so it helps frame each person as a separate line of data.

You'll notice that ECS has two input elements Selector and Element Click Selector:

Selector = The data you want to scrape. in this case you're going define each employee Card as an element by click on one card and then the next until they all light up and turns red. For me, this ended up being section.lo-team div.col-xs-12

Element Click = the pagination button = a.page-link.page-link-next

Since you've selected each person as it's own element, tick the multiple box.
Since you want to paginate set this to clickMore (continue to click until no new elements are left)
Type = Unique Text
(Save it)

Everything else will go inside (as a child) to this Element Click Selector.
[note, once inside your ECS, there won't be a need to click "multiple" as it's not assumed each action will be replicated across all selected elements defined by the parent..

You'll notice when selecting elements that only the first one is lighting up (that's what you want)

Once inside, I grabbed, Name, Phone, Address and E-mail. Lastly a used a link selector to inside the profile

As a child to the link selector - I then chose cell phone and Bio.

[Done]!

Note: Element click selector will FIRSTLY paginate through all available pages THEN will start scraping from last to first. You won't be able preview any results until AFTER it's run through each page and starts clicking through.

Here is your fixed sitemap

{"_id":"loandepotca1","startUrl":["https://www.loandepot.com/branches/rancho-cucamonga-ca/meet-the-team"],"selectors":[{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"section.lo-team div.col-xs-12","multiple":true,"delay":"4000","clickElementSelector":"a.page-link.page-link-next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"Name","type":"SelectorText","parentSelectors":["pagination"],"selector":"h3.lo-card-name a.purple","multiple":false,"regex":"","delay":0},{"id":"Title","type":"SelectorText","parentSelectors":["pagination"],"selector":"h4.lo-card-title","multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["pagination"],"selector":"strong a.purple","multiple":false,"regex":"","delay":0},{"id":"Email","type":"SelectorText","parentSelectors":["pagination"],"selector":"p.lo-card-contact > a.purple:contains(\"@\")","multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","parentSelectors":["pagination"],"selector":"address","multiple":false,"regex":"","delay":0},{"id":"Link-into-profile","type":"SelectorLink","parentSelectors":["pagination"],"selector":"h3.lo-card-name a.purple","multiple":false,"delay":0},{"id":"NMLS#","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"h4.lo-nmls","multiple":false,"regex":"","delay":0},{"id":"Bio","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"div.container.lo-biography div.row","multiple":false,"regex":"","delay":0},{"id":"Cell Phobe","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"li.mobile-phone","multiple":false,"regex":"","delay":0}]}

After fixing your sitemap, I had a thought. Is it possible to get EVERY employee from loandepot, regardless of location. Yes, yes it is. 900+ results

{"_id":"loandepot-com-full-mapping","startUrl":["https://www.loandepot.com/branches/albuquerque-nm","https://www.loandepot.com/branches/amarillo-tx","https://www.loandepot.com/branches/anacortes-wa","https://www.loandepot.com/branches/ashland-or","https://www.loandepot.com/branches/astoria-ny","https://www.loandepot.com/branches/atlanta-ga","https://www.loandepot.com/branches/austin-tx","https://www.loandepot.com/branches/bakersfield-ca","https://www.loandepot.com/branches/bend-or","https://www.loandepot.com/branches/bethel-ct","https://www.loandepot.com/branches/biloxi-ms","https://www.loandepot.com/branches/birmingham-al","https://www.loandepot.com/branches/boca-raton-fl","https://www.loandepot.com/branches/boston-ma","https://www.loandepot.com/branches/boulder-co","https://www.loandepot.com/branches/branson-mo","https://www.loandepot.com/branches/brentwood-tn","https://www.loandepot.com/branches/bristol-ri","https://www.loandepot.com/branches/north-brooklyn-ny","https://www.loandepot.com/branches/south-brooklyn-ny","https://www.loandepot.com/branches/brownsville-tx","https://www.loandepot.com/branches/canton-mi","https://www.loandepot.com/branches/carlsbad-ca","https://www.loandepot.com/branches/carmel-valley-ca","https://www.loandepot.com/branches/carmel-in","https://www.loandepot.com/branches/cerritos-ca","https://www.loandepot.com/branches/champaign-il","https://www.loandepot.com/branches/charleston-sc","https://www.loandepot.com/branches/charlottesville-va","https://www.loandepot.com/branches/chester-county-pa","https://www.loandepot.com/branches/chicago-il","https://www.loandepot.com/branches/chula-vista-ca","https://www.loandepot.com/branches/clearwater-fl","https://www.loandepot.com/branches/coeur-dalene-id","https://www.loandepot.com/branches/colorado-springs-co","https://www.loandepot.com/branches/columbia-md","https://www.loandepot.com/branches/columbia-sc","https://www.loandepot.com/branches/concord-ca","https://www.loandepot.com/branches/corvallis-or","https://www.loandepot.com/branches/cosat-mesa-ca","https://www.loandepot.com/branches/dallas-tx","https://www.loandepot.com/branches/danvers-ma","https://www.loandepot.com/branches/delafield-wi","https://www.loandepot.com/branches/denver-co","https://www.loandepot.com/branches/dickson-city-pa","https://www.loandepot.com/branches/eagan-mn","https://www.loandepot.com/branches/east-providence-ri","https://www.loandepot.com/branches/eden-prairie-mn","https://www.loandepot.com/branches/edmond-ok","https://www.loandepot.com/branches/el-centro-ca","https://www.loandepot.com/branches/el-paso","https://www.loandepot.com/branches/enid-ok","https://www.loandepot.com/branches/eugene-or","https://www.loandepot.com/branches/exton-pa","https://www.loandepot.com/branches/fairfax-va","https://www.loandepot.com/branches/fairfield-ct","https://www.loandepot.com/branches/fairfield-nj","https://www.loandepot.com/branches/fayetteville-nc","https://www.loandepot.com/branches/findlay-oh","https://www.loandepot.com/branches/foothill-ranch-ca","https://www.loandepot.com/branches/fort-collins-co","https://www.loandepot.com/branches/fort-lee-nj","https://www.loandepot.com/branches/fort-myers-fl","https://www.loandepot.com/branches/fort-worth-tx","https://www.loandepot.com/branches/frederick-md","https://www.loandepot.com/branches/fresno-ca","https://www.loandepot.com/branches/frisco-tx","https://www.loandepot.com/branches/garden-city-ny","https://www.loandepot.com/branches/gilbert-az","https://www.loandepot.com/branches/glastonbury-ct","https://www.loandepot.com/branches/glen-allen-va","https://www.loandepot.com/branches/glencoe-il","https://www.loandepot.com/branches/glenside-pa","https://www.loandepot.com/branches/grants-pass-or","https://www.loandepot.com/branches/greenwich2-ct","https://www.loandepot.com/branches/greenwich-ct","https://www.loandepot.com/branches/greenwood-in","https://www.loandepot.com/branches/hanover-ma","https://www.loandepot.com/branches/harrisburg-pa","https://www.loandepot.com/branches/harwichport-ma","https://www.loandepot.com/branches/henderson-nv","https://www.loandepot.com/branches/hilton-head-sc","https://www.loandepot.com/branches/hoboken-nj","https://www.loandepot.com/branches/houston-tx","https://www.loandepot.com/branches/huntersville-nc","https://www.loandepot.com/branches/huntsville-al","https://www.loandepot.com/branches/hurricane-wv","https://www.loandepot.com/branches/irvine-ca","https://www.loandepot.com/branches/jacksonville-fl","https://www.loandepot.com/branches/jacksonville-south-fl","https://www.loandepot.com/branches/kailua-kona-hi","https://www.loandepot.com/branches/katy-tx","https://www.loandepot.com/branches/marietta-ga","https://www.loandepot.com/branches/kern-ca","https://www.loandepot.com/branches/kirkland-wa","https://www.loandepot.com/branches/klamath-falls-or","https://www.loandepot.com/branches/la-mesa-ca","https://www.loandepot.com/branches/lake-havasu-city-az","https://www.loandepot.com/branches/lake-oswego-or","https://www.loandepot.com/branches/lakeway-tx","https://www.loandepot.com/branches/lancaster-ca","https://www.loandepot.com/branches/lancaster-pa","https://www.loandepot.com/branches/larkspur-ca","https://www.loandepot.com/branches/las-vegas-nv","https://www.loandepot.com/branches/lexington-ky","https://www.loandepot.com/branches/limerick-pa","https://www.loandepot.com/branches/west-lisle-il","https://www.loandepot.com/branches/livermore-ca","https://www.loandepot.com/branches/long-beach-ca","https://www.loandepot.com/branches/lynbrook-ny","https://www.loandepot.com/branches/lynchburg-va","https://www.loandepot.com/branches/maitland-fl","https://www.loandepot.com/branches/manteca-ca","https://www.loandepot.com/branches/marysville-wa","https://www.loandepot.com/branches/maumee-oh","https://www.loandepot.com/branches/medford-or","https://www.loandepot.com/branches/meridian-id","https://www.loandepot.com/branches/middleton-wi","https://www.loandepot.com/branches/annapolis-md","https://www.loandepot.com/branches/minnetonka-mn","https://www.loandepot.com/branches/miramar-fl","https://www.loandepot.com/branches/mission-valley-ca","https://www.loandepot.com/branches/modesto-ca","https://www.loandepot.com/branches/morganfield-ky","https://www.loandepot.com/branches/morristown-nj","https://www.loandepot.com/branches/mountain-home-id","https://www.loandepot.com/branches/mtpleasant-sc","https://www.loandepot.com/branches/muscatine-ia","https://www.loandepot.com/branches/naples-fl","https://www.loandepot.com/branches/new-city-ny","https://www.loandepot.com/branches/new-richmond-wi","https://www.loandepot.com/branches/new-york-ny","https://www.loandepot.com/branches/builder-newport-beach-ca","https://www.loandepot.com/branches/retail-newport-beach-ca","https://www.loandepot.com/branches/northbrook-il","https://www.loandepot.com/branches/oakland-ca","https://www.loandepot.com/branches/rancho-cucamonga-ca","https://www.loandepot.com/branches/lake-nona-fl","https://www.loandepot.com/branches/overland-park-ks","https://www.loandepot.com/branches/palm-coast-fl","https://www.loandepot.com/branches/palm-springs-ca","https://www.loandepot.com/branches/park-ridge-il","https://www.loandepot.com/branches/peoria-az","https://www.loandepot.com/branches/philadelphia-pa","https://www.loandepot.com/branches/phoenix-az","https://www.loandepot.com/branches/pittsburgh-pa","https://www.loandepot.com/branches/plantation-fl","https://www.loandepot.com/branches/plymouth-meeting-pa","https://www.loandepot.com/branches/plymouth-ma","https://www.loandepot.com/branches/port-jefferson-station-ny","https://www.loandepot.com/branches/portsmouth-nh","https://www.loandepot.com/branches/princeton-nj","https://www.loandepot.com/branches/pueblo-west-co","https://www.loandepot.com/branches/pueblo-co","https://www.loandepot.com/branches/puyallup-wa","https://www.loandepot.com/branches/raleigh-nc","https://www.loandepot.com/branches/northern-nv","https://www.loandepot.com/branches/west-riverside-ca","https://www.loandepot.com/branches/east-riverside-ca","https://www.loandepot.com/branches/rock-hill-sc","https://www.loandepot.com/branches/rock-island-il","https://www.loandepot.com/branches/brevard-fl","https://www.loandepot.com/branches/rockville-md","https://www.loandepot.com/branches/rogers-mn","https://www.loandepot.com/branches/rolling-meadows-il","https://www.loandepot.com/branches/roseville-ca","https://www.loandepot.com/branches/rye-ny","https://www.loandepot.com/branches/sacramento-ca","https://www.loandepot.com/branches/san-angelo-tx","https://www.loandepot.com/branches/san-antonio-tx","https://www.loandepot.com/branches/san-bruno-ca","https://www.loandepot.com/branches/san-juan-capistrano-ca","https://www.loandepot.com/branches/san-ramon-ca","https://www.loandepot.com/branches/sandpoint-id","https://www.loandepot.com/branches/sandwich-ma","https://www.loandepot.com/branches/santa-maria-ca","https://www.loandepot.com/branches/santa-rosa-ca","https://www.loandepot.com/branches/schaumburg-il","https://www.loandepot.com/branches/north-scottsdale-az","https://www.loandepot.com/branches/south-scottsdale-az","https://www.loandepot.com/branches/southern-pines-nc","https://www.loandepot.com/branches/spokane-wa","https://www.loandepot.com/branches/st-joseph-mi","https://www.loandepot.com/branches/stamford-ct","https://www.loandepot.com/branches/statesboro-ga","https://www.loandepot.com/branches/stockbridge-ga","https://www.loandepot.com/branches/stockton-ca","https://www.loandepot.com/branches/stoughton-ma","https://www.loandepot.com/branches/suwanee-ga","https://www.loandepot.com/branches/tampa-fl","https://www.loandepot.com/branches/tarrytown-ny","https://www.loandepot.com/branches/temecula-valley-ca","https://www.loandepot.com/branches/temecula-ca","https://www.loandepot.com/branches/tempe-az","https://www.loandepot.com/branches/tigard-or","https://www.loandepot.com/branches/tinley-park-il","https://www.loandepot.com/branches/tracy-ca","https://www.loandepot.com/branches/troy-oh","https://www.loandepot.com/branches/tucson-az","https://www.loandepot.com/branches/valencia-ca","https://www.loandepot.com/branches/valparaiso-in","https://www.loandepot.com/branches/vancouver-wa","https://www.loandepot.com/branches/vero-beach-fl","https://www.loandepot.com/branches/virginia-beach-va","https://www.loandepot.com/branches/waco-tx","https://www.loandepot.com/branches/waco-tx","https://www.loandepot.com/branches/farmingdale-nj","https://www.loandepot.com/branches/walnut-creek-ca","https://www.loandepot.com/branches/walpole-ma","https://www.loandepot.com/branches/wayne-pa","https://www.loandepot.com/branches/west-chester-pa","https://www.loandepot.com/branches/west-hartford-ct","https://www.loandepot.com/branches/west-springfield-ma","https://www.loandepot.com/branches/westborough-ma","https://www.loandepot.com/branches/westford-ma","https://www.loandepot.com/branches/westlake-oh","https://www.loandepot.com/branches/westwood-nj","https://www.loandepot.com/branches/wilmington-oh","https://www.loandepot.com/branches/witon-ct","https://www.loandepot.com/branches/worcester-ma","https://www.loandepot.com/branches/zionsville-in"],"selectors":[{"id":"pagination","type":"SelectorElementClick","parentSelectors":["Meet the Team"],"selector":"section.lo-team div.col-xs-12","multiple":true,"delay":"4000","clickElementSelector":"a.page-link.page-link-next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"Name","type":"SelectorText","parentSelectors":["pagination"],"selector":"h3.lo-card-name a.purple","multiple":false,"regex":"","delay":0},{"id":"Title","type":"SelectorText","parentSelectors":["pagination"],"selector":"h4.lo-card-title","multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["pagination"],"selector":"strong a.purple","multiple":false,"regex":"","delay":0},{"id":"Email","type":"SelectorText","parentSelectors":["pagination"],"selector":"p.lo-card-contact > a.purple:contains(\"@\")","multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","parentSelectors":["pagination"],"selector":"address","multiple":false,"regex":"","delay":0},{"id":"Link-into-profile","type":"SelectorLink","parentSelectors":["pagination"],"selector":"h3.lo-card-name a.purple","multiple":false,"delay":0},{"id":"NMLS#","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"h4.lo-nmls","multiple":false,"regex":"","delay":0},{"id":"Bio","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"div.container.lo-biography div.row","multiple":false,"regex":"","delay":0},{"id":"Cell Phobe","type":"SelectorText","parentSelectors":["Link-into-profile"],"selector":"li.mobile-phone","multiple":false,"regex":"","delay":0},{"id":"Meet the Team","type":"SelectorLink","parentSelectors":["_root"],"selector":"span.inline-left:contains(\"Meet\") a","multiple":false,"delay":0},{"id":"state","type":"SelectorText","parentSelectors":["pagination"],"selector":"address","multiple":false,"regex":"[A-Z][A-Z]","delay":0}]}