How to add timestamp to each scraper

How to?
#1

Hello how can I add time stamp to each row of data?

#2

Currently, there does not seem to be a timestamp feature in WS. However on some websites, the current time/date is displayed on every page (usually at the bottom, and based on web server's clock). So you could scrape that instead.

On other sites, the time is not be visible but it may be somewhere else in the page source, especially if the page is dynamically generated.

#4

@cryptoforbreakfast Hello, you can extract the whole script containing the timestamp and afterwards isolate it using regular expressions.

script:contains("currentDateTimeUTC")
regex: currentDateTimeUTC.*

#5

Hi Viesturs,

Thank you for your suggestion!

However I'm not getting it to work yet. When scraping the column remains empty.

Did I set it up wrong?
image

#6

give me your link to check.... seems like it's not too hard to extract current timestamp...

#7

Try:
Type: HTML
Selector: head
Regex: (?<=currentDateTimeUTC":")[^"]+

#8

Hi leemeng

It works! Thank you so much!

Sadly in most of the other sites I'm scraping there is no such time element for me to scrape :slightly_frowning_face: :slightly_frowning_face: :slightly_frowning_face:

Is there really no other general or non-site-specific way somebody can think of? Maybe a Date().getTime() statement or something? Or for example reading AWS/Google server time?

Thanks again!

#9

I believe I found a way to achieve my goal, but it's a bit of a workaround. I'll explain here for anyone looking for the same thing. I'm still open to more elegant solutions though :slight_smile:

First I install this Chrome extension: https://chromewebstore.google.com/detail/minimalist-clock/dhlcfelpfaohophbekdiehhnblhblell

add selector: type Text; div#Ext-134-datetime

So basically the add-in adds a small timebox to each page, and you can just scrape that instead :wink:

If you also want seconds you can also use this extension: https://chromewebstore.google.com/detail/hoverclock/piopcbodmhbaaohefjddlaneoenokmdm

  • Then the selector needs to be: type Grouped; #hoverClock:shadow-root div#dateText, div#clockText
  • But then you get results like this: [{"timestamp":"Thu Jun 20 2024"},{"timestamp":"16:33:45"}]
  • I can work with that in Excel, but if anybody can help me clean this up within webscraper (I'm not very experienced with RegEx), that would be greatly appreciated! I'd really prefer the 2024/06/20 16:33:45 format if that would be possible. If not I'll do it in Excel PowerQuery

If you really only need a minimalist timestamp (no date or seconds): https://chromewebstore.google.com/detail/overlay-clock/oocmeehjidegkkilebmenjdogocfefbd

  • type Text; div#overlay_clock_extension

Good luck, and thanks again!