README.md 2.07 KB
Newer Older
Tiago Peixoto's avatar
Tiago Peixoto committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# Netzschleuder

Netzschleuder is a network data catalogue and repository.

The code here can be used to fully replicate the 'official' instance at
<https://networks.skewed.de>, including the upstream data downloading,
parsing and processing.

## Dependencies

The following Python packages are needed to use this code:

* [graph-tool](https://graph-tool.skewed.de)
* [flask](https://flask.palletsprojects.com)
* [pillow](https://pillow.readthedocs.io)
* [google-api-python-client](https://github.com/googleapis/google-api-python-client)
* [google-auth-oauthlib](https://google-auth-oauthlib.readthedocs.io)
* [python-zstandard](https://python-zstandard.readthedocs.io)
* [openpyxl](https://openpyxl.readthedocs.io)
* [pandas](https://pandas.pydata.org)

To render the website you need the following stored inside the
``./static`` directory:

* [ink](http://ink.sapo.pt)
* [mathjax](https://www.mathjax.org)

## Usage

The full repository is populated by running simply: 
```
./do_all.py
```

The above command takes an optional space-separate list of entries as
arguments, in which case only the specified entries are processed.

The processing pipeline consists of the steps ``fetch``, ``parse``,
``convert``, ``analyze``, and ``draw`` taken in sequence. Each of these
steps can also be run in isolation by running the corresponding python
script, e.g. ``fetch.py``.

The processed results will be stored inside a ``cache`` sub-directory. A
local version of the website can be run by evoking ``./net_site.py``.

### Google drive authentication

Some entries require the use of google drive API for fetching. You will
need authenticate with a google account the first time you run the above
scripts, and your credentials will be stored in a local file.

## License

Netzschleuder is free software, you can redistribute it and/or modify it
under the terms of the GNU General Public License, version 3 or
above. See COPYING for details.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.