En:HOWTO: Set Up A Community Repository

From Sabayon Wiki
Revision as of 09:11, 21 February 2012 by Semmelb (Talk | contribs) (Outdated page)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search
i18n: ca de en es fr
Stop.png
This article is outdated and needs a major rework.

Why?

If you want different USE flags on packages or you want that new version you've just found in an overlay or just want to play with something cool then setting up a commnity repo is for you. It allows you to compile portage ebuilds into entropy packages and upload them up to a ftp site.

More information can be found on the En:Community Repositories Page.

Basic Requirements

Here is a list of the requisites you need to satisfy before starting your own repository:

  • Portage or any Portage-compatible Source Package Manager (to compile packages)
  • Entropy >= 0.99 including sys-apps/entropy-server package.
  • a remote FTP where uploading packages and their repository database
  • HTTP or anonymous FTP to allow client downloads
  • A basic Portage terminology knowledge

Setting up your own repository

server.conf

The first stage is setting up your server.conf file located in /etc/entropy.

Let's copy it and start editing

# cp /etc/entropy/server.conf.example /etc/entropy/server.conf

We have only a few parameters to set in this file.

Branches

The first thing to set up is which branch your repository will operate on, this will normally be a number that will relate to the version of sabayon you are building packages on/for. Usually you will only need to specify the primary number, ie 4.1 would be branch "4" and 5.2 would be branch "5".

branches|5

(Please note, one of the branch string listed above must be also set in /etc/entropy/repositories.conf)

Officialserverrepositoryid

The officalserverrepositopryid is the shortened name of your repository, it needs to be a ascii string and can use dots "." and hyphens "-" but not spaces, it is used as the "repository identifier". The Community Repository interface must be aware of what is the default repository, especially when you have more than one.

## this is just an ascii string identifier explained later
officialserverrepositoryid|myrepository

or

officialserverrepositoryid|my-repo.ftw

Repository

This is the main configuration string of for your repository it details:

  • Repository Identifier (see above) - This is the Officialserverrepositoryid that you put above
  • Repository Description - A more verbose description of your Repository.
  • Upload URI(s) - ftp:// paths to your FTP storage in the format "username:[email protected]", there can be multiple URIs to upload to multiple locations, its recommended to use rsync from one primary server as it will be significantly faster.
  • Download URI - The publicly accessable address of your packages, this myst be a HTTP URI pointing to your repos root directory.
  • handlers URL - Handlers are small optional server-side PHP script to ease the work of the interface, the handler parameter must be a HTTP URI pointing to the repository root directory.

This needs to go in a pipe seperated line such as below: See the example below:

## repository|Officialserverrepositoryid|<repository description>|<FTP upload/download URI 1> <FTP upload/download URI 2>|handlers URL (path to repositories root directory)
## we will write
repository|myrepository|My very first repository|ftp://username:[email protected]/myrepositories/%7Chttp://www.sabayonlinux.org/myrepositories

Product

There is another critical parameter listed in /etc/entropy/repositories.conf called "product", at the moment, Sabayon has only one supported product, called "standard". So, be sure to have this set in this file, since we're going to make a repository for Sabayon Linux, product: standard.

=Initialising the Repository

If all is configured properly, you will find that running this command:

# equo community repo status

Will bring the following output

# >>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://myrepository:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: myrepository (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   myrepository
>>  !!! [repo:myrepository] Your default repository is not initialized.
Do you want to initialize your default repository ? [Yes/No]

As you can see Entropy found that your shiny new repository is empty and asked you to initialize it. Do it!

It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.

Anyway, eventually it will finish and you should see somthing along the lines of the below:

>>  !!! Entropy database is probably empty. If you don't agree with what I'm saying, then it's probably corrupted! I won't stop you here btw...
>>  !!! Entropy database: fixing KDE dep strings on etpdb:myrepository. Please wait ...
>>  * Mirrors have not been unlocked. Remember to sync them.

If you do see the above then your repoistory has been setup successfully! In the words of Joost_op "go get yourself a beer".

Compiling and Adding Packages

Overview

The next step (the fun bit) is filling it with some packages.

Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:

# equo database spmsync

Compilation

Now to add a package, its Compile time! The first step is to compile something using any Portage-compabile source package manager (Portage itself, Paludis, Pkgcore - anything that writes /var/db/pkg properly). Let's say we want to compile our own media-sound/amarok perhaps with "postgres" support or with different CFLAGS. Then, we spawn Portage:

# emerge -av media-sound/amarok

Note, if you get Equo called by Portage, you need to select "no". This because of a Portage behavioral change in the latest release (apparently it pulls into its xpak metadata equo's bashrc).

Adding the Package to Your Repository

Single Repository

Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:

# equo community repos update

You will be asked to push the newly compiled packages into your default repository. This is the output you should expect:

 
>>  @@ These are the packages that would be added/updated to the add list:
>>    media-sound/amarok-1.4.9.1
>>   Would you like to package them now ? (inside mynewrepo) [Yes/No] 

Of course yes! We have now pushed media-sound/amarok into our local repository database!

Multiple Repositories

If you have more than one community repository, a quick way to switch the default (working) repository is using ETP_REPO environmental variable:

# ETP_REPO=myotherrepository equo community repos update

You can also export it, and it will persitant for that session.

# export ETP_REPO=myotherrepository 
# equo community repos update

You will see the switch right after having pressed the Enter key:

 
# >>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://mynewrepo:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: myotherrepository (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   mynewrepo
>>   myotherrepository

When uploading you can can add the --syncall argument to sync all the repositories in one go.

Uploading the Packages

We now need to upload the new binary package(s) and the updated repository database.

equo community mirrors sync

This is the output you should expect:

>>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://mynewrepo:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: mynewrepo (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   mynewrepo
>>  * Starting to sync data across mirrors (packages/database) ...
>>  @@ [repo:mynewrepo|sync|branch:3.5] packages sync: sabayonlinux.org
>>  @@ Local statistics:
>>  @@ upload directory:                1 files ready
>>  @@ packages directory:              0 files ready
>>  @@ Remote statistics for: sabayonlinux.org
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/
>>  * [ftp:sabayonlinux.org] switching to: 3.5
>>  @@ remote packages:                 0 files stored
>>  @@ Calculating queues ...
>>  ** Expanding queues:
>>    [branch:3.5|upload] media-sound:amarok-1.4.9.1~0.tbz2 [182.9kB]
>>  @@ Packages to be removed:                  0
>>  @@ Packages to be moved locally:            0
>>  @@ Packages to be uploaded:                 1
>>  @@ Total removal size:                      0.0b
>>  @@ Total upload size:                       182.9kB
>>  @@ Total download size:                     0.0b
Would you like to run the steps above ? [Yes/No] 

Last chance to double check what is getting done. We will choose Yes.

>>  @@ [sabayonlinux.org|upload] connecting to mirror...
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/3.5
>>  @@ [sabayonlinux.org|#1|(1/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2.md5
>>  @@ [sabayonlinux.org|#1|(1/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2.md5: not supported
>>  @@ [sabayonlinux.org|#1|(1/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2.md5
>>  @@ [sabayonlinux.org|#1|(2/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2
>>  @@ [sabayonlinux.org|#1|(2/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2: not supported
>>  @@ [sabayonlinux.org|#1|(2/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2
>>  @@ [branch:5] upload completed successfully: sabayonlinux.org
>> Please insert a commit message:  my first commit
>>  * Remote Entropy Database Repository Status:
>>       Host:  sabayonlinux.org
>>        * Database revision: 0
>>        * Database local revision currently at: 1
>>  @@ [repo:mynewrepo|sabayonlinux.org] mirror successfully locked for download
>>  * [repo:mynewrepo|sabayonlinux.org|upload] preparing to upload database to mirror
>>  * Entropy database is already locked by you :-)
>>  * Mirrors have not been unlocked. Remember to sync them.
>>  * Entropy database is already locked by you :-)
>>  * Mirrors have not been unlocked. Remember to sync them.
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:3] preparing uncompressed database for the upload
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:2] creating compressed database dump + checksum
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>    dump: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2
>>    dump checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2.md5
>>    opener: bz2.BZ2File
>>    Database Export completed.
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>    compressed database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.bz2
>>    compressed checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.md5
>>    opener: bz2.BZ2File
>>  @@ [sabayonlinux.org|upload] connecting to mirror...
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: /
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/database/amd64/
>>  @@ [sabayonlinux.org|#1|(1/14)] uploading: make.conf
>>  @@ [sabayonlinux.org|#1|(1/14)] upload successful: make.conf
>>  @@ [sabayonlinux.org|#1|(2/14)] uploading: packages.db.bz2
>>  @@ [sabayonlinux.org|#1|(2/14)] upload successful: packages.db.bz2
>>  @@ [sabayonlinux.org|#1|(3/14)] uploading: packages.db
>>  @@ [sabayonlinux.org|#1|(3/14)] upload successful: packages.db
>>  @@ [sabayonlinux.org|#1|(4/14)] uploading: packages.db.dump.bz2.md5
>>  @@ [sabayonlinux.org|#1|(4/14)] upload successful: packages.db.dump.bz2.md5
>>  @@ [sabayonlinux.org|#1|(5/14)] uploading: packages.db.revision
>>  @@ [sabayonlinux.org|#1|(5/14)] upload successful: packages.db.revision
>>  @@ [sabayonlinux.org|#1|(6/14)] uploading: packages.rss
>>  @@ [sabayonlinux.org|#1|(6/14)] upload successful: packages.rss
>>  @@ [sabayonlinux.org|#1|(7/14)] uploading: updates.rss
>>  @@ [sabayonlinux.org|#1|(7/14)] upload successful: updates.rss
>>  @@ [sabayonlinux.org|#1|(8/14)] uploading: packages.db.dump.bz2
>>  @@ [sabayonlinux.org|#1|(8/14)] upload successful: packages.db.dump.bz2
>>  @@ [sabayonlinux.org|#1|(9/14)] uploading: package.mask
>>  @@ [sabayonlinux.org|#1|(9/14)] upload successful: package.mask
>>  @@ [sabayonlinux.org|#1|(10/14)] uploading: package.keywords
>>  @@ [sabayonlinux.org|#1|(10/14)] upload successful: package.keywords
>>  @@ [sabayonlinux.org|#1|(11/14)] uploading: packages.db.md5
>>  @@ [sabayonlinux.org|#1|(11/14)] upload successful: packages.db.md5
>>  @@ [sabayonlinux.org|#1|(12/14)] uploading: package.unmask
>>  @@ [sabayonlinux.org|#1|(12/14)] upload successful: package.unmask
>>  @@ [sabayonlinux.org|#1|(13/14)] uploading: profile.link
>>  @@ [sabayonlinux.org|#1|(13/14)] upload successful: profile.link
>>  @@ [sabayonlinux.org|#1|(14/14)] uploading: package.use
>>  @@ [sabayonlinux.org|#1|(14/14)] upload successful: package.use
>>  * [repo:mynewrepo|sabayonlinux.org] mirror successfully unlocked
>>  * [repo:mynewrepo|sync] database sync completed successfully
Should I continue with the tidy procedure ? [Yes/No] 
>>  @@ [repo:mynewrepo|tidy|branches:5] collecting expired packages
>>  @@ [branch:5] collecting expired packages in the selected branches
>>  @@ [branch:5] nothing to remove on this branch 

Done and Done!