Help language development. Donate to The Perl Foundation

Shelve6 cpan:ROBERTLE last updated on 2020-05-18

... A artifact repository for the raku language and friends

The goal is to build a artifact repository service that raku modules, but also
other stuff, can be pushed to and pulled via the usual means (i.e. pushed from
CI or manual workflows, pulled via zef). This would be useful for an organisation 
doing raku development, does not want all their code publicly available, 
yet want to use a regular module-centric, tarball/release-based development flow.

Essentially this is a "content storage" service as described in [S22][1]

## Features

- Upload Raku modules
- Fetch with zef
- Authentication
- Multiple configurable logical repos

## Upcoming and ToDo

- cucumis sextus tests
* adress already in use error handling/reporting
- UI to browse and manage
- API to manage and automate
- Local cache/proxy for other repositories, like CPAN. Could be just a cache,
  or a fetch-ahead full copy. Perhaps both, configurably.
- Rarification/expiry of artifacts in configured repositories
- Web hooks for automation
- Verification and other plugins
- Shared file store, multiple shelve6 instances. Or a database as a store.
- More auth types
- Full-blown monitoring, resilience etc 
- More metadata, like when uploaded and by whom. "on-behalf" in upload script 
  so that a CI or automation job can say on whose command they uploaded

Also grep for the `XXX` fixmes in the code!

## Usage

Shelve6 comes as a web service that you can just start e.g. directly from
the checked-out source repository via `RAKUDOLIB=lib bin/shelve6`, or if it is 
properly installed just via `shelve6`. It reads a config.yaml file, a simple
sample is included, and might look like this:
        port: 8080
        base-url: "http://localhost:8080"
        basedir: store
        - name: p6repo
* `base-url` is where you want the service to be found externally
* `port` is of course the port the service listens on, note that  it currently
  only binds to the first localhost interface, let me know if that gives you grief.
* `basedir` is a directory where shleve6 will store the artifacts
* and the repositories is a list of logical artifact repositories in which you
  can store modules

With the service running, you can use the supplied shelve6-upload script to put
artifacts into shelve6:
    bin/shelve6-upload raku-foo-bar-0.1.tar.gz http://localhost:8080/repos/p6repo

This script is just a thin wrapper around curl, you just need a multipart form
post really.

In order to fetch artifacts, you need to configure your zef to recognise the
repository. In my case I have a `~/.config/zef/config.json`, where in the
`Repository` section I added:
        "short-name" : "shelve6",
        "enabled" : 1,
        "module" : "Zef::Repository::Ecosystems",
        "options" : {
            "name" : "shelve6",
            "auto-update" : 1,
            "mirrors" : [
After that, zef happily pulls from shelve6!

## Authentication and Authorization

If you want to use the repository for private code, it may be a good idea to
enable some security on it. Shelve6 can use credentials in the request and map
them to a set of roles associated with that credential. Currently the only
credential type supported are 'opaque' tokens, these are just striings that are
not looked into (so not JWT or so). These come in a HTTP header like
`Authorization: Bearer supersecret`, where 'supersecret' is the credential. In
the future more credential types can be supported. To configure the mapping of 
credentials to roles, extend the 'server' part of the configuration:
    port: 8080
    base-url: "http://localhost:8080"
            - token: supersecret
              roles: [CI]
              owner: raku-ci-1
            - token: eng8ahquia2kungeitaequie
              roles: [DEV, ADMIN]
              owner: Max Mustermann <[email protected]>
In order to actually require any roles, you need to configure which roles allow
what operation, on the repository:
    - name: p6repo
        upload: [CI, DEV, ADMIN]
        download: [CI, DEV, ADMIN]
        delete: [ADMIN]
Note that the credential is associated with all the roles from the server
config, but any role in the repository section is sufficient for access to be
granted. For example the credential 'eng8ahquia2kungeitaequie' above gives both
the 'DEV' and the 'ADMIN' roles, any of which would be enough to upload and
download artifacts.

The shelve6-upload script supports setting these tokens through a commandline
argument or an environment variable.

In order to enable zef to provide credentials during module fetching, you need
to install the `Zef::Service::AuthenticatedDownload` plugin:
    zef install Zef::Service::AuthenticatedDownload
And configure it. The zef README explains where you can find the zef config, 
where you can add the
plugin and configure it, which is best done before the other web fetchers to
avoid them trying to download and failing due to auth:

            "short-name" : "authenticated-download",
            "module" : "Zef::Service::AuthenticatedDownload",
            "options" : { 
                "configured-hosts" : [
                        "hostname" : "localhost",
                        "auth-type" : "opaque-token",
                        "token" :  "supersecret"
This will make zef use the configured credential for the host in question. If
you do not want to put the credential into the config file, you can also leave
it out and supply it via the ZEF_AUTH_OPAQUE_TOKEN environment variable.

## License

Shelve6 is licensed under the [Artistic License 2.0]( 

## Feedback and Contact

Please let me know what you think: Robert Lemmen <[email protected]>