Scraper moves into a new tab and stops working

#1

Hello,
I've tried searching to see if this is happening to anyone else, but I didn't find anything.

I'm scraping details of car ads from a website and recently, the popup window that's doing the scraping will disappear and reappear as a new tab in any open chrome window which causes it to stop running midway through and forces me to restart it.

What causes the scraping window to move to the main chrome window as a new tab?
How can I keep this from happening?

For the record: I've used these sitemaps to scrape many times before without having this issue and it started on Wednesday after successful scrapes on Monday/Tuesday.

Thanks

#2

I confirm this problem too.
It's possible low memory? My system it's more than 90% full
For me when tab move in main window continue scraping but no more data extracted, so I need stop and start again.

#3

Thanks for the reply - I don't think that's the issue.
I have about 50 different sitemaps (different makes and models from the same website) that I've had complete success, from beginning to end, scraping in the past. I usually run anywhere from 2-18 sitemaps to scrape simultaneously. So that's why I don't think its a memory issue.
This problem happens while scraping even just 1 sitemap.

Also to add - when my sitemap goes into the new tab I can sometimes see it still working in that tab although I didn't notice if data is being extracted. But I do have a pagination in the sitemap and I think it completely stops working when it gets to the 'next page' portion of the scraping process.

Thanks,

#4

@jrg718 Hi, are you able to share your sitemap, please?

#5

For the record: As of yesterday, without changing anything to any of the sitemaps, they've started working normally again. I ran about 10 during the day today and 5 completed with no issues, while the other 5 are still running. I do expect it to happen again though as it's always been off and on.

Never shared a sitemap, so let me know if I need to format it in a different way Thanks.
Example Sitemap:
{"_id":"alfa_romero","startUrl":["https://www.cars.com/shopping/results/?dealer_id=&keyword=&list_price_max=&list_price_min=&makes[]=alfa_romeo&maximum_distance=all&mileage_max=&page=1&page_size=100&sort=list_price_desc&stock_type=all&year_max=&year_min=&zip=60605"],"selectors":[{"id":"Pages","parentSelectors":["_root","Pages"],"type":"SelectorLink","selector":"li.sds-pagination__item:nth-of-type(n+2) a","multiple":true,"delay":0},{"id":"Alfa Romero","parentSelectors":["_root","Pages"],"type":"SelectorLink","selector":"a.vehicle-card-link","multiple":true,"delay":0},{"id":"Title","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":".title-section h1","multiple":false,"delay":0,"regex":""},{"id":"Price","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":".gallery-header span.primary-price","multiple":false,"delay":0,"regex":""},{"id":"Engine","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"dt:contains('Engine') + dd","multiple":false,"delay":0,"regex":""},{"id":"Transmission","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"dt:contains('Transmission') + dd","multiple":false,"delay":0,"regex":""},{"id":"Mileage","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"dt:contains('Mileage') + dd","multiple":false,"delay":0,"regex":""},{"id":"Seller Info","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"h3.sds-heading--5","multiple":false,"delay":0,"regex":""},{"id":"Seller Location","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"div.dealer-address","multiple":false,"delay":0,"regex":""},{"id":"Seller Notes","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"div.sellers-notes","multiple":false,"delay":0,"regex":""},{"id":"VIN","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"dt:contains('VIN') + dd","multiple":false,"delay":0,"regex":""},{"id":"New / Used","parentSelectors":["Alfa Romero"],"type":"SelectorText","selector":"p.new-used","multiple":false,"delay":0,"regex":""}]}

#6

If try to refresh data, You can see how it does not continue to extract.

I don't think the problem is the map, the same map (amazon) in firefox doesn't happen, only in chrome, my version of chrome is 96.0.4664.110 and ws 0.6.4
sometimes it happens soon sometimes late others it doesn't.

Maybe it conflicts with another extension, for example I lock part of the page with ublock, kill the cookie with ninja cookie or suspend tabs with Auto Tab Discard.

#7

@jrg718 ​If the scrape has been conducted locally the issue you described possibly occurs due to your location, browser version, OS version, and other factors because I did not experience any issues after performing tests using the extension.

You can always test the sitemap on the Web Scraper Cloud as well - |https://cloud.webscraper.io/login|

#8

The problem persist, chrome is now update to 97.0.4692.71, the first run not moves scraper to new tab but freezee it after a time.
Second test, scraper move into new tab in first page.

#9

If press button "data preview" or "element preview" work correctly.

Uncaught (in promise) Error: Extension context invalidated.
Context
devtools://devtools/bundled/devtools_app.html?remoteBase=https://chrome-devtools-frontend.appspot.com/serve_file/@adefa7837d02a07a604c1e6eff0b3a09422ab88d/&can_dock=true&dockSide=undocked

devtools_panel.js:7210 (anonymous function)

                chrome.runtime.sendMessage(l, t => {

{"error":"{\"message\":\"The message port closed before a response was received.\"}","method":"getElements","request":"{\"method\":\"getElements\",\"params\":[\"tr:contains('Fuente de alimentación') .a-span9 span, tr:contains('Fuente de alimentación') td.a-size-base, .a-list-item > span.a-text-bold:contains('Fuente de alimentación') +\",0]}","stack":"Error\n at o.error (chrome-extension://jnhgnonknehpejjnehehllkliplmbmhn/background_script.js:769:79)\n at chrome-extension://jnhgnonknehpejjnehehllkliplmbmhn/background_script.js:50374:95","url":"https://www.amazon.es/dp/B07MXR611F","domainName":"www.amazon.es","timestamp":1641700798,"level_name":"ERROR","message":"Failed to send message to chrome tab"}
Context
_generated_background_page.html

background_script.js:766 (log)
background_script.js:770 (error)
background_script.js:50374 (anonymous function)

               "ERROR" === t || "WARNING" === t ? (console.error(s), e.env.FAIL_ON_LOG && !this.isMessageExpected(t, i) && e.exit(66)) : console.log(s);

I think the selector is correct:

tr:contains('Fuente de alimentación') .a-span9 span, tr:contains('Fuente de alimentación') td.a-size-base, .a-list-item > span.a-text-bold:contains('Fuente de alimentación') +

The field can be in three different places, in some or in none.
I use the comma for each and the + for optional.
The same for about 30 selectors.

a help, thanks