How to scrape data that changes

How to?
#1

Hi Guys !

The problem that i have with this website is that some data that i have is on different position. For example one carrier can have email, name and phone number; one can have just email and phone number.

Here is the website: BrokerSnapshot - FM LOGISTICS CORP

Here is the screenshot of what i need to be scraped and to be static. When there is an email to be just the email, not to scrape phone number because it on the same position.

fiver
fiver1629×759 54.5 KB

Here is my sitemap:

{"_id":"novatest","startUrl":["https://brokersnapshot.com/?search=Fm+Logistics"],"selectors":[{"id":"company","parentSelectors":["_root"],"type":"SelectorLink","selector":"td:nth-of-type(5) a","multiple":true,"linkType":"linkFromHref"},{"id":"element","parentSelectors":["company"],"type":"SelectorElement","selector":"table#all-history-table-computer","multiple":true},{"id":"Email","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(1) a[title]","multiple":false,"regex":""},{"id":"phone","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(2) a[title]","multiple":false,"regex":""},{"id":"phone2","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(3) a[title]","multiple":false,"regex":""},{"id":"email2","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(4) a[title]","multiple":false,"regex":""},{"id":"name1","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(5) a","multiple":false,"regex":""},{"id":"name2","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(6) a","multiple":false,"regex":""},{"id":"phonenumber","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(8) a","multiple":false,"regex":""},{"id":"email3","parentSelectors":["element"],"type":"SelectorText","selector":"tr:nth-of-type(9) a","multiple":false,"regex":""}]}```
#2

Hi, it is a bit difficult to inspect the website, since it needs a login, but you can try to make the e-mail selectors more specific, like:

tr:has([title="Email"]):nth-of-type(1) td:has([title="Email"]) a

The same should work with phone etc., you have to find the correct attribute in the HTML

email

#3

Hi JanAp,

Thanks for the reply. One more question how to modify it when there is like 2 or 3 emails? To collect them all.

Here is the link from the following screenshot: BrokerSnapshot - FM LOGISTICS CORP

Like in this screenshot.

screenshot
screenshot1588×469 30.8 KB

#4

Hi JanAp,

I manage to get all the emails but i get the data like this.

data
data842×758 18.9 KB

Is there a way to make them all in different column? Like email 1, email 2 etc etc.

#5

Hi, please try to create a new selector and just change the nth-of-type(1) to nth-of-type(2), and so forth, that should do the trick.

#6

Hi, It's working. Thank you so much for your help. :slight_smile:

#7

Hey sorry to bother you one more question.. Can you help me with the pagination? It just scraping first and second page.

#8

Hi, here is a reference sitemap with pagination:

{"_id":"novatest","startUrl":["https://brokersnapshot.com/?search=Fm+Logistics"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"[class=\"ui paginator right aligned container\"] a:contains(\"»\")","type":"SelectorPagination"},{"id":"company","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":"td:nth-of-type(5) a","type":"SelectorLink"},{"id":"element","multiple":true,"parentSelectors":["company"],"selector":"table#all-history-table-computer","type":"SelectorElement"},{"id":"Email","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(1) a[title]","type":"SelectorText"},{"id":"phone","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(2) a[title]","type":"SelectorText"},{"id":"phone2","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(3) a[title]","type":"SelectorText"},{"id":"email2","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(4) a[title]","type":"SelectorText"},{"id":"name1","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(5) a","type":"SelectorText"},{"id":"name2","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(6) a","type":"SelectorText"},{"id":"phonenumber","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(8) a","type":"SelectorText"},{"id":"email3","multiple":false,"parentSelectors":["element"],"regex":"","selector":"tr:nth-of-type(9) a","type":"SelectorText"}]}
#9

Thank you so much. Really appreciate your help :slight_smile:

#10

Hi JanAp,

Have a question once again.. This website is killing me :).

When i try to do pagination it scrape just 20/30 companies per page, and leaving me a lot of data unscraped.

Here is my sitemap to check if i'm making some mistake.

{"_id":"Brokersnapshot","startUrl":["https://brokersnapshot.com/SearchCompanies/Advanced?state=OH&authority=2&min-units=5&min-inspections=1&limit=100"],"selectors":[{"id":"company","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":"td:nth-of-type(5) a","type":"SelectorLink"},{"id":"email","multiple":true,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(1) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(2) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(3) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(4) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"cellphone","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(1) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(2) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(3) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(4) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"phone1","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(1) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(2) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(3) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(4) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"Contact Name1","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(1) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"Contactname2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(2) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(3) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(4) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"email5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(5) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(6) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(7) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(8) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(9) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(10) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"cellphone5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(5) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(6) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(7) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(8) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(9) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(10) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"phone5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(5) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(6) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(7) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(8) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(9) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(10) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"contactname5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(5) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(6) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(7) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(8) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(9) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(10) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":".paginator a","type":"SelectorPagination"},{"id":"Operating Status","multiple":false,"parentSelectors":["company"],"regex":"","selector":"#OperatingStatusHtml span","type":"SelectorText"},{"id":"Total Number of Trucks","multiple":false,"parentSelectors":["company"],"regex":"","selector":"span#TOT_TRUCKS","type":"SelectorText"}]}```
#11

Hi, I found a slight issue with the pagination. You have to try to target only the 'next page' button when it is available.

image

I have adjusted the pagination selector and also made a slight change in the link selector to target each unique link only once.

{"_id":"Brokersnapshot","startUrl":["https://brokersnapshot.com/SearchCompanies/Advanced?state=OH&authority=2&min-units=5&min-inspections=1&limit=100"],"selectors":[{"id":"company","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":"td:nth-of-type(5) div:nth-of-type(1) a","type":"SelectorLink"},{"id":"email","multiple":true,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(1) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(2) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(3) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(4) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"cellphone","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(1) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(2) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(3) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(4) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"phone1","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(1) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(2) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(3) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(4) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"Contact Name1","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(1) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"Contactname2","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(2) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname3","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(3) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname4","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(4) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"email5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(5) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(6) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(7) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(8) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(9) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"email10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Email\"]):nth-of-type(10) td:has([title=\"Email\"]) a","type":"SelectorText"},{"id":"cellphone5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(5) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(6) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(7) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(8) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(9) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"cellphone10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Cell Phone\"]):nth-of-type(10) td:has([title=\"Cell Phone\"]) a","type":"SelectorText"},{"id":"phone5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(5) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(6) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(7) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(8) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(9) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"phone10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Phone\"]):nth-of-type(10) td:has([title=\"Phone\"]) a","type":"SelectorText"},{"id":"contactname5","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(5) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname6","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(6) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname7","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(7) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname8","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(8) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname9","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(9) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"contactname10","multiple":false,"parentSelectors":["company"],"regex":"","selector":"tr:has([title=\"Contact Name\"]):nth-of-type(10) td:has([title=\"Contact Name\"]) a","type":"SelectorText"},{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":".paginator a:contains(\"»\")","type":"SelectorPagination"},{"id":"Operating Status","multiple":false,"parentSelectors":["company"],"regex":"","selector":"#OperatingStatusHtml span","type":"SelectorText"},{"id":"Total Number of Trucks","multiple":false,"parentSelectors":["company"],"regex":"","selector":"span#TOT_TRUCKS","type":"SelectorText"}]}

Let me know if it helps.

#12

Hi, Thanks for your reply.

It's the same once again. Look at this. Here is the last page of what i need to be scraped: BrokerSnapshot - Search Companies

It has 100 links (companies), this is how much is scraping for me:

image
image985×748 23.9 KB

It's like 34/35 and it goes to the next page. Next page is scraping like 57 records of 100.

#13

Hi, upon closer inspection I found that there are quite many duplicates, which are automatically discarded by the scraper. At the given time there are 2967 unique links. That is why you are seeing less lines in the data output.

#14

Got it,

Thank you so much for your help.