PFERD

Garmelon/PFERD

Fork 0

mirror of https://github.com/Garmelon/PFERD.git synced 2026-04-12 15:35:05 +02:00

Commit graph

92886fb8d8 Implement --version flag Joscha 2021-05-19 17:32:23 +02:00
5916626399 Make noqua comment more specific Joscha 2021-05-19 17:16:59 +02:00
a7c025fd86 Implement reusable FileSinkToken for OutputDirectory Joscha 2021-05-19 17:16:23 +02:00
b7a999bc2e Clean up crawler exceptions and (a)noncritical Joscha 2021-05-19 13:25:57 +02:00
3851065500 Fix local crawler's download bars Joscha 2021-05-18 23:23:40 +02:00
4b68fa771f Move logging logic to singleton Joscha 2021-05-18 22:43:46 +02:00
1525aa15a6 Fix link template error and use indeterminate progress bar I-Al-Istannen 2021-05-18 22:40:28 +02:00
db1219d4a9 Create a link file in ILIAS crawler I-Al-Istannen 2021-05-17 21:31:22 +02:00
b8efcc2ca5 Respect filters in ILIAS crawler I-Al-Istannen 2021-05-17 21:30:26 +02:00
0bae009189 Run formatting tools Joscha 2021-05-16 14:32:53 +02:00
3efec53f51 Configure code checking and formatting tools Joscha 2021-05-16 14:31:43 +02:00
8b76ebb3ef Rename IliasCrawler to KitIliasCrawler I-Al-Istannen 2021-05-16 13:28:06 +02:00
467ea3a37e Document ILIAS-Crawler arguments in CONFIG.md I-Al-Istannen 2021-05-16 13:26:58 +02:00
2b6235dc78 Fix pylint warnings (and 2 found bugs) in ILIAS crawler I-Al-Istannen 2021-05-16 13:17:12 +02:00
cd5aa61834 Set max line length for pylint I-Al-Istannen 2021-05-16 13:17:01 +02:00
5ccb17622e Configure pycodestyle to use a max line length of 110 I-Al-Istannen 2021-05-16 13:01:41 +02:00
1c226c31aa Add some repeat annotations to the ILIAS crawler I-Al-Istannen 2021-05-16 13:01:30 +02:00
9ec0d3e16a Implement date-demangling in ILIAS crawler I-Al-Istannen 2021-05-16 11:54:42 +02:00
cf6903d109 Retry crawling on I/O failure I-Al-Istannen 2021-05-15 22:46:26 +02:00
9fd356d290 Ensure tmp files are deleted Joscha 2021-05-15 23:00:40 +02:00
989032fe0c Fix cookies getting deleted Joscha 2021-05-15 22:25:41 +02:00
05573ccc53 Add fancy CLI options Joscha 2021-05-15 21:33:51 +02:00
c454fabc9d Add support for exercises in ILIAS crawler I-Al-Istannen 2021-05-15 21:40:17 +02:00
7d323ec62b Implement video downloads in ilias crawler I-Al-Istannen 2021-05-15 21:29:43 +02:00
c7494e32ce Start implementing crawling in ILIAS crawler I-Al-Istannen 2021-05-15 20:42:18 +02:00
1123c8884d Implement an IliasPage I-Al-Istannen 2021-05-15 18:57:17 +02:00
e1104f888d Add tfa authenticator Joscha 2021-05-15 18:27:16 +02:00
8c32da7f19 Let authenticators provide username and password separately Joscha 2021-05-15 18:24:03 +02:00
d63494908d Properly invalidate exceptions Joscha 2021-05-15 17:37:05 +02:00
b70b62cef5 Make crawler sections start with "crawl:" Joscha 2021-05-15 17:23:33 +02:00
868f486922 Rename local crawler path to target Joscha 2021-05-15 17:12:25 +02:00
b2a2b5999b Implement ILIAS auth and crawl home page I-Al-Istannen 2021-05-15 15:18:51 +02:00
595de88d96 Fix authenticator and crawler names Joscha 2021-05-15 15:18:16 +02:00
a6fdf05ee9 Allow variable whitespace in arrow rules Joscha 2021-05-15 15:13:34 +02:00
f897d7c2e1 Add name variants for all arrows Joscha 2021-05-15 15:06:45 +02:00
b0f731bf84 Make crawlers use transformers Joscha 2021-05-15 14:03:15 +02:00
302b8c0c34 Fix errors loading local crawler config Joscha 2021-05-15 13:32:13 +02:00
acd674f0a0 Change limiter logic Joscha 2021-05-15 13:21:38 +02:00
b0f9e1e8b4 Add vscode directory to gitignore I-Al-Istannen 2021-05-15 11:20:20 +02:00
ed2e19a150 Add reasons for invalid values Joscha 2021-05-15 00:39:55 +02:00
296a169dd3 Make limiter logic more complex Joscha 2021-05-15 00:38:46 +02:00
1591cb9197 Add options to slow down local crawler Joscha 2021-05-14 21:41:24 +02:00
0c9167512c Fix output dir Joscha 2021-05-14 21:28:38 +02:00
a673ab0fae Delete old files Joscha 2021-05-14 00:20:59 +02:00
6e5fdf4e9e Set user agent to "pferd/<version>" Joscha 2021-05-14 00:09:58 +02:00
93a5a94dab Single-source version number Joscha 2021-05-13 23:52:46 +02:00
d565df27b3 Add HttpCrawler Joscha 2021-05-13 22:28:14 +02:00
961f40f9a1 Document simple authenticator Joscha 2021-05-13 19:55:04 +02:00
e3ee4e515d Disable highlighting of primitives Joscha 2021-05-13 19:47:44 +02:00
94d6a01cca Use file mtime in local crawler Joscha 2021-05-13 19:42:40 +02:00
38bb66a776 Update file metadata in more cases Joscha 2021-05-13 19:40:10 +02:00
68781a88ab Fix asynchronous methods being not awaited Joscha 2021-05-13 19:39:49 +02:00
910462bb72 Log stuff happening to files Joscha 2021-05-13 19:37:27 +02:00
6bd6adb977 Fix tmp file names Joscha 2021-05-13 19:36:46 +02:00
0acdee15a0 Let crawlers obtain authenticators Joscha 2021-05-13 18:57:20 +02:00
c3ce6bb31c Fix crawler cleanup not being awaited Joscha 2021-05-11 00:28:45 +02:00
0459ed093e Add simple authenticator Joscha 2021-05-11 00:27:43 +02:00
d5f29f01c5 Use global conductor instance Joscha 2021-05-10 23:50:16 +02:00
595ba8b7ab Remove dummy crawler Joscha 2021-05-10 23:47:46 +02:00
cec0a8e1fc Fix mymy errors Joscha 2021-05-09 01:45:01 +02:00
f9b2fd60e2 Document local crawler and auth Joscha 2021-05-09 01:33:47 +02:00
60cd9873bc Add local file crawler Joscha 2021-05-06 01:02:40 +02:00
273d56c39a Properly load crawler config Joscha 2021-05-05 23:45:10 +02:00
5497dd2827 Add @noncritical and @repeat decorators Joscha 2021-05-05 23:36:54 +02:00
bbfdadc463 Implement output directory Joscha 2021-05-05 18:08:34 +02:00
fde811ae5a Document on_conflict option Joscha 2021-05-05 00:55:55 +02:00
07e831218e Add sync report Joscha 2021-05-02 00:56:10 +02:00
91c33596da Load crawlers from config file Joscha 2021-04-30 16:22:14 +02:00
a8dcf941b9 Document possible redownload settings Joscha 2021-04-30 15:32:56 +02:00
e7a51decb0 Elaborate on transforms and implement changes Joscha 2021-04-29 20:13:46 +02:00
9ec19be113 Document config file format Joscha 2021-04-29 18:55:08 +02:00
f776186480 Use PurePath instead of Path Joscha 2021-04-29 16:52:00 +02:00
0096d83387 Simplify Limiter implementation Joscha 2021-04-29 16:37:42 +02:00
20a24dbcbf Add changelog Joscha 2021-04-29 16:14:50 +02:00
502654d853 Fix mypy errors Joscha 2021-04-29 15:47:52 +02:00
d2103d7c44 Document crawler Joscha 2021-04-29 15:43:20 +02:00
d96a361325 Test and fix exclusive output Joscha 2021-04-29 15:26:10 +02:00
2e85d26b6b Use conductor via context manager Joscha 2021-04-29 14:23:28 +02:00
6431a3fb3d Fix some mypy errors Joscha 2021-04-29 14:23:09 +02:00
ac3bfd7388 Make progress bars easier to use Joscha 2021-04-29 13:53:16 +02:00
3ea86d18a0 Jerry-rig DummyCrawler to run Joscha 2021-04-29 13:45:04 +02:00
bbc792f9fb Implement Crawler and DummyCrawler Joscha 2021-04-29 13:44:29 +02:00
7e127cd5cc Clean up and fix conductor and limiter Joscha 2021-04-29 13:43:50 +02:00
c4fb92c658 Make type hints compatible with Python 3.8 Joscha 2021-04-29 13:11:58 +02:00
8da1ac6cee Extend mypy config Joscha 2021-04-29 11:25:13 +02:00
a18db57e6f Implement terminal conductor Joscha 2021-04-29 11:25:00 +02:00
b915e393dd Implement limiter Joscha 2021-04-29 10:24:28 +02:00
3a74c23d09 Implement transformer Joscha 2021-04-29 09:51:25 +02:00
fbebc46c58 Load and dump config Joscha 2021-04-27 12:41:49 +02:00
024aceb5ee

merge master be7a 2021-04-29 00:30:02 +02:00
11bbfa0c40

fix symbol rename be7a 2021-04-29 00:26:23 +02:00
cccd68e04a Bump version to v2.6.2 v2.6.2 v2 I-Al-Istannen 2021-04-29 00:18:26 +02:00
2bd40a5f30 Fix -p and -u flags I-Al-Istannen 2021-04-29 00:15:12 +02:00
9ad176c3ca

update examples be7a 2021-04-29 00:07:50 +02:00
c800d307db

Add warning about event-loop in ILIASCrawler be7a 2021-04-29 00:02:03 +02:00
3eab236b99

update sync_url.py and fix event_loop housekeeping be7a 2021-04-28 23:58:21 +02:00
2d6be9f5c1

run autopep8 and fix formatting be7a 2021-04-28 23:09:38 +02:00
1d3e169ead

merge upstream be7a 2021-04-28 01:53:42 +02:00
c0ba8b9528

Use shared ILIASCrawler be7a 2021-04-28 01:50:28 +02:00
e3a13143bc

Use shared ILIASCrawler be7a 2021-04-28 01:50:28 +02:00