portugal-running-data
repo with scraper for the portugal running calendar data
| Filename | Source Script | Optional | Description |
|---|---|---|---|
lastmod |
setup-directories |
no | last modification time extracted from the sitemap file |
page.html |
fetch-page |
no | event page from portugalrunning.com |
id |
extract-id |
no | event numeric id from wordpress |
data.json |
fetch-data |
no | json file with some event data |
ics |
fetch-ics |
no | calendar file with location, date and other event information |
location |
fetch-location |
yes | location data for the event |
image |
fetch-image |
yes | cover image for the event |
date |
extract-date |
no | event date extracted from the ics file |
oneline-description |
fetch-oneline-description |
yes | ai generated one line description |
categories |
extract-categories |
no | event categories |
circuits |
extract-circuits |
no | event circuits |
fetch-sitemap
this script fetches the sitemap that contains a list of event page urls and the last modification date
fetch-pages
this script will fetch any missing pages or outdated pages by looking at the lastmod file.
extract-ids
this script will extract the event ids from the page.html file. this id can be used to later fetch other data related to this event.
fetch-ics
this script uses the event id and fetches its ics file.
fetch-data
this script uses the event id to fetch some event data in json format.
fetch-images
some events have a main image in the json data file, this script will fetch that image.
extract-organizer
this script extracts the organizer from the class list in the json data file, if one exists.
extract-categories
this script extracts a list of categories from the class list in the json data file.