PFERD

Garmelon/PFERD

Fork 0

mirror of https://github.com/Garmelon/PFERD.git synced 2026-04-13 07:55:05 +02:00

Commit graph

076b8c5a1f Add download strategies to save bandwith I-Al-Istannen 2020-04-23 18:29:20 +02:00
13bc78c889 Display reason for ignoring an element in ilias crawler I-Al-Istannen 2020-04-23 13:54:58 +02:00
dc964a9d98 Remove finished TODOs I-Al-Istannen 2020-04-23 13:30:34 +02:00
c2b14f3db9 ilias crawler: Use direct download link if possible I-Al-Istannen 2020-04-23 13:07:07 +02:00
4b59a7c375 Move around TODOs Joscha 2020-04-23 10:49:01 +00:00
3a57430893 Fix type errors in example_config I-Al-Istannen 2020-04-23 12:35:58 +02:00
bef210ae77 Rename and implement IliasDirectoryFilter I-Al-Istannen 2020-04-23 12:33:38 +02:00
ea005517cf Only remove folders if they exist in tmpdir I-Al-Istannen 2020-04-23 12:09:45 +02:00
3841f27aab Add example config Joscha 2020-04-23 09:49:15 +00:00
df0eb84a44 Fix TmpDir and Location Joscha 2020-04-23 09:48:09 +00:00
2de4255a78 Add Pferd class Joscha 2020-04-23 09:44:13 +00:00
3c808879c9 Add Transforms and Transformables Joscha 2020-04-22 18:25:09 +00:00
a051e3bcca ilias crawler: Add some unhelpful documentation I-Al-Istannen 2020-04-22 17:58:19 +02:00
eb7df036df WIP: ilias crawler: Also crawl assignments I-Al-Istannen 2020-04-22 14:32:20 +02:00
23db59e733 WIP: ilias-crawler: Demangle dates I-Al-Istannen 2020-04-22 12:44:41 +02:00
ac65b06a8e Satisfy pylint a bit I-Al-Istannen 2020-04-22 01:37:34 +02:00
8891041069 WIP: crawler: Add opencast video crawler I-Al-Istannen 2020-04-21 23:01:19 +02:00
70d63e3e90 WIP: Start small ILIAS crawler I-Al-Istannen 2020-04-21 13:32:03 +02:00
b2a7af2e3e Store modification_date in IliasDownloadInfo, remove parameters I-Al-Istannen 2020-04-21 13:31:50 +02:00
23bed48c8c Satisfy autopep8 I-Al-Istannen 2020-04-21 13:30:42 +02:00
0926d33798 Use downloader-specific data classes Joscha 2020-04-20 18:06:21 +00:00
55ba2f4070 Fix pylint in downloaders I-Al-Istannen 2020-04-20 19:48:59 +02:00
d18b48aaf4 Stream in http downloader I-Al-Istannen 2020-04-20 19:45:25 +02:00
4ef0ffe3bf Listen to pylint and mypy Joscha 2020-04-20 17:27:26 +00:00
ce77995c8f Rename http downloader module Joscha 2020-04-20 17:08:51 +00:00
ed9245c14d Remove old organizer I-Al-Istannen 2020-04-20 18:50:23 +02:00
01e6972c96 Add ilias downloader I-Al-Istannen 2020-04-20 18:36:40 +02:00
8181ae5b17 Guard http response in context manager I-Al-Istannen 2020-04-20 18:04:56 +02:00
6407190ae0 Soupify requests responses properly Joscha 2020-04-20 16:38:18 +00:00
87395faac2 Add base for simple HTTP downloader I-Al-Istannen 2020-04-20 17:43:41 +02:00
a9e6e7883d Create temp dir folder in constructor I-Al-Istannen 2020-04-20 17:43:28 +02:00
154d6b29dd Listen to pylint Joscha 2020-04-20 15:15:47 +00:00
62ac569ec4 Revert "Add proposed crawler entry type" I-Al-Istannen 2020-04-20 16:59:08 +02:00
9f1a0a58ab Add proposed crawler entry type I-Al-Istannen 2020-04-20 16:54:32 +02:00
879a2c7c80 Rewrite ILIAS authenticator Joscha 2020-04-20 14:25:55 +00:00
ff06c5215e Fix authenticator Joscha 2020-04-20 14:03:22 +00:00
135a8dce4b Fix resolve_path allowing paths outside its folder I-Al-Istannen 2020-04-20 16:07:14 +02:00
63bbcad918 Add resolve method to tmp_dir I-Al-Istannen 2020-04-20 15:39:52 +02:00
6584d6a905 Elaborate accept_file in new_organizer I-Al-Istannen 2020-04-20 15:39:38 +02:00
5990098ef8 Add UserPassAuthenticator Joscha 2020-04-20 13:26:45 +00:00
f3d3d6bb65 Add some docs to cookie_jar I-Al-Istannen 2020-04-20 14:38:03 +02:00
b2fe7cc064 Add preliminary logging to organizer and tmp_dir I-Al-Istannen 2020-04-20 14:37:44 +02:00
930d821dd7 Add a simple organizer I-Al-Istannen 2020-04-20 14:29:41 +02:00
5c2ff14839 Add "prompt_yes_no" to utils I-Al-Istannen 2020-04-20 14:29:28 +02:00
a3d6dc7873 Clean up temp_folder I-Al-Istannen 2020-04-20 14:28:22 +02:00
53ad1c924b Add cookie jar Joscha 2020-04-20 11:35:26 +00:00
8c431c7d81 Add a simple temporary folder I-Al-Istannen 2020-04-20 12:08:52 +02:00
d5dd5aac06 Fix some mypy errors Joscha 2020-04-20 01:54:47 +00:00
7d48972967 Configure mypy Joscha 2020-04-19 19:50:17 +00:00
25043a4aaa Remove unnecessary files Joscha 2020-04-19 19:49:43 +00:00
7ebeef5873 Clean up gitignore Joscha 2020-04-19 18:47:44 +00:00
9b658776ca

Merge pull request #6 from I-Al-Istannen/master Garmelon 2020-03-01 23:09:32 +00:00
cf3553175f Add OS_Exams synchronizer I-Al-Istannen 2020-02-27 14:51:29 +01:00
bf8b3cf9f7 Hack in support for TI exams I-Al-Istannen 2020-02-24 20:56:43 +01:00
4a5600d5ce

Merge pull request #5 from I-Al-Istannen/master Garmelon 2019-12-12 11:52:05 +00:00
f5bc49160f Lose 50 minutes of my life (and fix the TGI tut) I-Al-Istannen 2019-12-12 12:50:16 +01:00
fa5f82d312

Merge pull request #4 from I-Al-Istannen/master Garmelon 2019-11-18 22:12:32 +00:00
4433696509 [TGI] Add TGi tut I-Al-Istannen 2019-11-14 15:37:57 +01:00
1f5475abc5

Merge pull request #3 from I-Al-Istannen/master Garmelon 2019-10-17 21:10:21 +00:00
1407c6d264 Download all TGI files and not just lectures I-Al-Istannen 2019-10-17 22:14:32 +02:00
e152bfc4a3

Merge pull request #2 from I-Al-Istannen/master Garmelon 2019-10-15 20:01:10 +00:00
1973c931bd Add support for other years in TGI downloader I-Al-Istannen 2019-10-15 15:37:52 +02:00
458cc1c6d6 Add support for TGI website I-Al-Istannen 2019-10-15 15:34:59 +02:00
52852d11a6 Bump version to 1.1.8 v1.1.8 Joscha 2019-09-22 11:56:41 +00:00
f94629a7fa Fix exceptions with weird content types Joscha 2019-09-22 11:55:47 +00:00
c8ee456d33 Bump version to 1.1.7 v1.1.7 Joscha 2019-07-26 08:14:55 +00:00
2752e98621 Fix relative url joining in ti downloader I-Al-Istannen 2019-07-26 10:06:01 +02:00
1572e11da8 Bump version to 1.1.6 v1.1.6 Joscha 2019-07-05 08:49:26 +00:00
ea01dc7cb2 Allow even more types of files Joscha 2019-07-05 08:48:43 +00:00
aba8d46d26 Bump version to 1.1.5 v1.1.5 Joscha 2019-07-04 12:17:33 +00:00
77056e6f8d Allow more types of files Joscha 2019-07-04 12:16:42 +00:00
064f12c14c Ignore mypy files Joscha 2019-07-04 12:16:26 +00:00
2eb834afc3 Bump version to 1.1.4 v1.1.4 Joscha 2019-06-11 12:46:40 +00:00
d468a45662 Allow wolfram files Joscha 2019-06-11 12:42:55 +00:00
50e25346e5 Bump version to 1.1.3 v1.1.3 Joscha 2019-06-07 11:36:41 +00:00
67da4e69fa Add colorful log output I-Al-Istannen 2019-06-07 13:26:23 +02:00
da602366f8 Bump version to 1.1.2 v1.1.2 Joscha 2019-05-17 07:43:32 +00:00
2016f61bf8 Crawl more of the TI page Joscha 2019-05-09 11:04:24 +00:00
59c278da2c Bump version to 1.1.1 v1.1.1 Joscha 2019-05-06 12:07:12 +00:00
c72e92db18 Make Ti downloader authentication more robust Joscha 2019-05-06 12:04:01 +00:00
44b4204517 Add basic Ti downloader Joscha 2019-05-06 11:54:36 +00:00
11b9ff66e4 Bump version to 1.1.0 v1.1.0 Joscha 2019-04-26 09:48:17 +00:00
d730d0064c Conform to other files' __all__ Joscha 2019-04-26 09:44:50 +00:00
ae6cc40fb5 Rename ILIAS crawler to ilias Joscha 2019-04-26 04:24:24 +00:00
0891e7f1bc Fix logging messages not appearing Joscha 2019-04-26 03:58:11 +00:00
571c2a6c98 Clean up README structure Joscha 2019-04-25 20:04:14 +00:00
2d7a2b2d53 Add more usage examples Joscha 2019-04-25 20:01:12 +00:00
8550d1101c Fix backronym in README Joscha 2019-04-25 19:58:30 +00:00
fd71dc6f6e Use python3 everywhere in README Joscha 2019-04-25 19:57:04 +00:00
d48cbebbd4 Bump version to 1.0.0 v1.0.0 Joscha 2019-04-25 19:55:15 +00:00
bc018fffde Update the example config to the new version Joscha 2019-04-25 19:54:15 +00:00
9693e1d968 Make logging easier Joscha 2019-04-25 19:41:26 +00:00
5e9ecd3ce1 Add example setup steps Joscha 2019-04-25 19:30:02 +00:00
f1ba618378 Remove unnecessary files v0.1.0 Joscha 2019-04-25 19:17:50 +00:00
dfddc93039 Move norbert from aiohttp to requests Joscha 2019-04-25 19:15:36 +00:00
f0c42ce8ec Clean up Joscha 2019-04-25 19:02:48 +00:00
82adeb324f Move ffm stuff from aiohttp to requests Joscha 2019-04-25 19:01:53 +00:00
9bae030186 Move ilias stuff from aiohttp to requests Joscha 2019-04-25 18:52:48 +00:00
0e536f476a Reorganize project structure Joscha 2019-04-25 11:48:58 +00:00
45e4500a58 Update example config Joscha 2019-04-25 08:32:25 +00:00