Garmelon/PFERD
1
0
Fork 0
mirror of https://github.com/Garmelon/PFERD.git synced 2026-04-12 07:25:04 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add kit-ipd crawler

Browse source
This commit is contained in:
I-Al-Istannen 2021-10-21 12:01:41 +02:00
parent 742632ed8d
commit 6673077397
6 changed files with 196 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
CONFIG.md
View file

@ -136,6 +136,13 @@ crawler simulate a slower, network-based crawler.
requests. (Default: `0.0`)
- `download_speed`: Download speed (in bytes per second) to simulate. (Optional)
### The `kit-ipd` crawler
This crawler crals a KIT ipd page by url. The root page can be crawled from
outside the KIT network so you will be informed about any new/deleted files,
but downloading files requires you to be within. Adding a show delay between
requests is likely a good idea.
### The `kit-ilias-web` crawler
This crawler crawls the KIT ILIAS instance.