Scraping a specific page in old websites

How to?
#1

Describe the problem.

I have an issue with a specific page that I can't scrape it. I believe is too old that's why I'm experiencing problems which are:

Get into the main URL,

Url: https://nfe.prefeitura.sp.gov.br/publico/listaPrestadores.aspx

and after that, I need to extract the information below:

E.g.,

CNPJ: 19.910.685/0001-31

RAZÃO SOCIAL: MARMORARIA PEDRAS MATRIZ LTDA ME

ENDEREÇO: AV ENGENHEIRO ARMANDO DE ARRUDA PEREIRA 02912 JABAQUARA - SÃO PAULO/SP - CEP: 04308-001 TELEFONE: 1129258390 SITE: -

and the path to get there is:
bairrotype:jabaquara/click:jabaquara

I need all information that are in cnpj, razão social, endereço, cep, telefone and site.

The website is very old I guess that's the problem.

If anyone helps me out a really appreciate that.

Sitemap:
{id:"sitemap code"}

#2

It does look like older tech (aspx) which is not unusual for .gov websites. I don't think you can get the info in a single pass; when you click on a user, the server generates the url and redirects to a popup.
The popup URL has a predictable pattern, http://nfe.prefeitura.sp.gov.br/gmaps/PrestadorDetalheSite.aspx?inscricao=XXXXXXXX where the XXX is a number which seems to be a page ID for each user. This same number appears in the HTML for each user.

So I think it would be a two-pass operation:

  1. First scrape the page IDs for each user from all the categories on the main site.
  2. Then, generate user URLs with the page IDs and scrape those, which should be quite straightforward.

2019-09-11_081317
2019-09-11_081317.jpg1037×671 256 KB

#3

I tried but I didn't successed I don't know but I keep getting an error or I can scrape just the first page.

#4

Hold on, I just found out about the Link popup selector. Lemme test and I'll post a solution if there's one.

#5

I was using it but didn't work as expected I mean the popup selector.