How can I close the popups after scraping them?

How to?
#1

I'm scraping a list of items from a page. When you click on an item a popup opens and I scrape data from the popup. It is a popup in the same url (I think it is built with AngularJS - the popup is sort of an overlay). The problem is that the popup doesn't close after I scrape it, so after a while there are a lot of popups open and the scraper stops working - it becomes too slow. To scrape the items I use element click and then scrape with regular text. How can I close the popups after scraping them?

The website is a web app that needs a code to access so I can't share it...

Sitemap:
{"_id":"",""],"selectors":[{"id":"listings","type":"SelectorElementClick","selector":"div.modal-content","parentSelectors":["_root"],"multiple":true,"delay":"4000","clickElementSelector":"td.without-comments","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"city","type":"SelectorText","selector":"span.smallerText","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"city backup","type":"SelectorText","selector":"div.col-md-2 div.row:nth-of-type(3)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"full address","type":"SelectorText","selector":"h4.modal-title span:nth-of-type(1)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"neighborhood","type":"SelectorText","selector":"div.col-md-2 div.row:nth-of-type(4)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"street backup","type":"SelectorText","selector":"div.panel-body div.col-md-12 div.row:nth-of-type(5)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"rooms","type":"SelectorText","selector":"div.panel-body div.col-md-12 div.row:nth-of-type(6)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"size","type":"SelectorText","selector":"div.row:nth-of-type(7)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"floor","type":"SelectorText","selector":"div.row:nth-of-type(8)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"days on market","type":"SelectorText","selector":"div.col-md-5 div.row","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","selector":"div.col-md-7 div.row","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"details","type":"SelectorText","selector":"div.panel-body div.col-md-12 div.row:nth-of-type(2) div.form-group","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"name","type":"SelectorText","selector":"div.panel-body div.row div.col-md-12 div.row:nth-of-type(1)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"phone 1","type":"SelectorText","selector":"div.panel-body div.row div.col-md-12 div.row:nth-of-type(2)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"phone 2","type":"SelectorText","selector":"div.panel-body div.row div.row:nth-of-type(3)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0},{"id":"email","type":"SelectorText","selector":"div.panel-body div.row div.row:nth-of-type(4)","parentSelectors":["listings"],"multiple":false,"regex":"","delay":0}]}

#2

Can you include the URL?

#3

Hi!

You have to add a selector that closes the pop-up window. If it's a button within a window, you can simply add it to Click Selector that you already use to open it, as second Click selector using comma. OR, you can add another Click selector inside one that clicks button to open pop-up, to close it. Use _parent_ as selector, and close window button as Click.

#4

I can't share the website because it is password protected. Here is a video of the website and the problem I encounter: https://vimeo.com/user86995472/review/278620330/ad77758b9d

iconoclast, Thank you! Indeed I nedd to close the popups after scraping them. I have tried the two ways you offered and I didn't succeed.
This is the name of the button that I need to click on in order to close the popup: "div.modal-header button.close"

About the first method, How do I add another click selector? you can see one of the thing I tried here:

About the second method, is there a selector which is only Click Selector? Right now I am using Element click and I have done what you offered but it didn't work.

Thanks a lot. I highly appreciate it.

#5

Does modal button gets highlighted if you click Element Preview?

Please try this sitemap (explains what I wrote about close button):
{"_id":"modal","startUrl":["https://getbootstrap.com/docs/4.0/components/modal/"],"selectors":[{"id":"click","type":"SelectorElementClick","selector":"div.modal.show div.modal-dialog","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.bd-example:nth-of-type(5) button.btn, div.modal.show span","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

Perhaps you picked a wrong close button, try opening modal window manually, then pick close button for Click selector (and only that button), and try to execute it (hit Preview data).

#6

Thank you again @iconoclast
I followed your advise and built a new scraper that only closes the popup and it worked!
Afterwards I built another scraper that have two separate following selectors. The first open the popup and the second close the popup and it also worked.
Unfortunately it stopped working when I built a third scraper where the second selector that suppose to close was inside of the selector that opens the popup ("open selector" is a parent of "close selector"). Do you have any clue why it happened?
Here is a video of the third scraper that demonstrate the problem:
https://vimeo.com/user86995472/review/278816200/3ad1fc7cc9

#7

Hi Amir!

I forgot to mention that in order to have Element Click to click two buttons you have to tick Multiple inside it.
The sitemap i've shown you above does have this option enabled.

There's one big difference between having few elements inside selector field:

  • If Multiple option is ticked, WebScraper will pick both of the elements one after another;
  • And if it's not ticked, it will pick only first element available - for example, you have two pages on a same website named differently (e.g. div.name and div.surname), and if first is not present at all (data returns null), it will pick only second element available, and vice versa.

Don't forget to add delay to click selector, as if it lacks delay at all, it will click all buttons simultaneously.

image
image.png706×374 13.8 KB

#8

Hi,
@iconoclast, thank you again!
I have implemented your solution and it worked only when the first click selector was for one listing. It was structured exactly has you showed me and it worked. But, when I tried to do it when the first click selector was for multiple listings it failed. Because I need to scrape many listings the first click selector clicks on all of the listings in the list. Each listing at a time and I think the second click selector is activated only after the first click selector finishes (after the first goes over all of the listings in the list).

You can see the two scrapers that I built and how they function in this video:
https://vimeo.com/user86995472/review/278932039/824954f65c

#9

@iconoclast helped me solve this problem with a genius solution!
He wrote a script and used a chrome extension called Tampermonkey to close the popups automatically right after they are opened.
Now the scraper is working great!
Thank you @iconoclast! you are the best!

1 Like
#10

i have same problem, unique popup opened, but i cant get data from popup when oppening