amprolla

devuan's apt repo merger
git clone https://git.parazyd.org/amprolla
Log | Files | Refs | README | LICENSE

setup.md (5347B)


      1 amprolla setup
      2 ==============
      3 
      4 amprolla should be able to run on any system supporting Python 3, rsync
      5 and GnuPG. However it has only been tested on Devuan and Gentoo.
      6 
      7 
      8 Installation
      9 ------------
     10 
     11 The recommended way to install Python 3 and the needded modules, along
     12 with the extra needed dependencies is using your package manager.
     13 
     14 You will need the following:
     15 
     16 ```
     17 python3, python-requests, gnupg2, rsync
     18 ```
     19 
     20 After installing the required dependencies, clone the amprolla git repo
     21 using git to a place of your preference.
     22 
     23 You will also need to setup a valid gnupg directory structure, along with
     24 a key you shall use for signing the Release files if you require to do so.
     25 
     26 
     27 Configuration
     28 -------------
     29 
     30 To configure amprolla, a default configuration file is provided in
     31 `lib/config.def.py`. Copy the file to `lib/config.py` and edit it to
     32 your needs. The configuration file contains all the information needed
     33 to properly merge the required repositories. The default configuration
     34 is also working as long as you provide a valid gpg fingerprint used to
     35 sign the Release files. If you do not wish to sign Release file, make
     36 sure to disable it in the configuration file.
     37 
     38 The `*dir` variables in the configuration file are the directories
     39 where the files that are being merged are kept and the merges itself
     40 are done. They can be either absolute or relative paths to the root
     41 amprolla directory. The prefered way is to actually have absolute paths
     42 as this will cause less trouble.
     43 
     44 `banpkgs` is a set of package names that amprolla will refuse to merge
     45 if they are found either in the dependencies of a package or if they
     46 are the package itself.
     47 
     48 `repo_order` is a list that holds that is ordered in the priority the
     49 packages are prefered. The preference is ordered first to last.
     50 The dict `repos` itself holds the required information for them.
     51 
     52 ### repos dict structure
     53 
     54 * The variable is a normal dict where the key is a string that should
     55   be contained in `repo_order`. They key's values are another dict with
     56   the following fields:
     57 
     58 	- `name`: the name of the repository used for rewriting the path to
     59 	  the package's upstream (deb file). It is used in the nginx config
     60 	  provided in the `contrib` directory.
     61 
     62 	- `host`: the upstream URL where the repository is being held. Used
     63 	  to know from where to download the necessary Packages/Release files
     64 	  and how to rewrite certain values.
     65 
     66 	- `dists`: the root directory where the suites are held. It is appended
     67 	  to the above URL.
     68 
     69 	- `aliases`: if True, when downloading/rewriting, will look for the
     70 	  suite's alias defined later in the configuration file.
     71 
     72 	- `skipmissing`: a hack helpful to skip missing suites if the repo
     73 	  we are merging does not contain them.
     74 
     75 
     76 `suites` hold our release names and their suites.
     77 
     78 `aliases` hold the suites we want to merge as aliases in case we know a
     79 certain repository doesn't use the same name for it.
     80 
     81 `release_aliases` list our stable and testing branch aliases.
     82 
     83 `categories` hold the package categories we wish to actually merge. apt
     84 repositories usually hold `main`, `contrib`, and `non-free`. With this
     85 we can opt out of any of them.
     86 
     87 `arches` list the actual architectures we want to merge. If we are not
     88 using certain architectures, it is easy to exclude them from merging
     89 this way.
     90 
     91 It is advised to not touch any variables listed below these, as they
     92 are currently setup to provide a correct (valid) apt repository.
     93 
     94 
     95 Running amprolla
     96 ----------------
     97 
     98 After you've setup amprolla, it is needed to perform the initial full
     99 download and full merge. First run `amprolla_init.py`, which is going
    100 to download the necessary directory structures (as defined through the
    101 config file) we will merge afterwards. When the download is done, it is
    102 time to perform the full initial merge of these repositories. This will
    103 provide us with a complete merged repository and we will then be able
    104 to easily perform incremental updates of it.
    105 
    106 After the first merge has been performed, it is advisable to run a
    107 script called `populate_aliases.sh` found in the `contrib` directory.
    108 Make sure you edit it and set a proper path to your merged directory,
    109 and fill out the proper names needed. It will populate the merged
    110 directory with symlinks to certain versions such as `1.0`, `stable`, or
    111 `testing`.
    112 
    113 To merge Contents files, run `amprolla_merge_contents.py`. This module
    114 does not do incremental updates and should not be ran often due to its
    115 heavy IO/RAM requirements.
    116 
    117 Incremental updates are performed through `amprolla_update.py`, however,
    118 for more stable performance and uptime, the incremental updating is
    119 being orchestrated by a shell script called `orchestrate.sh`. This shell
    120 script holds the logic to have near-atomic switching of repositories to
    121 minimize repo downtime during performed merges. Not doing this could
    122 result with users downloading corrupted repository files if they are
    123 requested during an ongoing merge.
    124 
    125 In a screen/tmux session, simply execute the `orchestrate.sh` script
    126 and it will start looping and doing incremental updates every hour.
    127 If you prefer, you can remove this loop and run the shell script through
    128 a cron job based on your needs.
    129 
    130 To actually serve the merged directory over HTTP, a basic nginx
    131 configuration is provided as `contrib/nginx.conf`, and a lighttpd conf
    132 is provided in `contrib/lighttpd.conf`.