Difference between revisions of "En:HOWTO: Set Up A Community Repository"

From Sabayon Wiki
Jump to: navigation, search
(Starter rework of the HOWTO - updated requirements)
m (removed Code tags, they're broken)
(3 intermediate revisions by the same user not shown)
Line 46: Line 46:
 
* Repository Description - A more verbose description of your repository.
 
* Repository Description - A more verbose description of your repository.
 
* Repository URI(s) - URI to your remote server. The format depends on the type of server.
 
* Repository URI(s) - URI to your remote server. The format depends on the type of server.
  In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.
+
 
  In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.
+
In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.
  You can also use file:// URIs for truly local repositories.
+
 
  If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with <p>. If you want the URI to contain only repository information (no package download), you can prefix the URI with <r>.
+
In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.
 +
 +
You can also use file:// URIs for truly local repositories.
 +
 
 +
If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with &lt;p&gt;. If you want the URI to contain only repository information (no package download), you can prefix the URI with &lt;r&gt;.
  
 
See the example below:
 
See the example below:
{{Code|Snippet from /etc/entropy/server.conf|<pre class="clear">repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/</pre>}}
+
{{Console|<pre class="clear">$ vim /etc/entropy/server.conf
 +
repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/</pre>}}
  
 
You can enter more than one repository, just append more repository = ... lines.
 
You can enter more than one repository, just append more repository = ... lines.
Line 66: Line 71:
 
* Repository Description - A more verbose description of your Repository.
 
* Repository Description - A more verbose description of your Repository.
 
* Package URI(s) - URI to your remote server. The format depends on the type of server.
 
* Package URI(s) - URI to your remote server. The format depends on the type of server.
  In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.
+
In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.
  In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.
+
 
  You can also use file:// URIs for truly local repositories.
+
In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.
 +
 
 +
You can also use file:// URIs for truly local repositories.
 
* Repository URI - Also an URI to your remote server in the same format as above with an optional suffix #bz2 for bzip2 compression, #gz for gzip compression.
 
* Repository URI - Also an URI to your remote server in the same format as above with an optional suffix #bz2 for bzip2 compression, #gz for gzip compression.
  
Line 75: Line 82:
  
 
See the example below:
 
See the example below:
{{Code|Snippet from /etc/entropy/repository.conf.d/entropy_astralstorm.conf|<pre class="clear">
+
{{Console|<pre class="clear">$ vim /etc/entropy/repository.conf.d/entropy_astralstorm.conf
 
repository = astralstorm|AstralStorm's Sabayon Overlay|https://hackerspace.pl/~astralstorm/sabayon|https://hackerspace.pl/~astralstorm/sabayon#bz2
 
repository = astralstorm|AstralStorm's Sabayon Overlay|https://hackerspace.pl/~astralstorm/sabayon|https://hackerspace.pl/~astralstorm/sabayon#bz2
 
repository = astralstorm|AstralStorm's Sabayon Overlay|file:///home/astralstorm/sabayon|
 
repository = astralstorm|AstralStorm's Sabayon Overlay|file:///home/astralstorm/sabayon|

Revision as of 10:20, 14 September 2013

i18n: ca de en es fr
Stop.png
This article was outdated and is undergoing a major rework.

Why?

If you want different USE flags on packages or you want that new version you've just found in an overlay or just want to play with something cool then setting up a commnity repo is for you. It allows you to compile portage ebuilds into entropy packages and upload them up to a ftp, ssh or even http or https WebDAV site.

More information can be found on the En:Community Repositories Page.

Basic Requirements

Here is a list of the requisites you need to satisfy before starting your own repository:

  • Portage or any Portage-compatible Source Package Manager (to compile packages)
  • sys-apps/entropy-server package. The tutorial was tested using Entropy version 175.
  • Remote server to upload packages - can use FTP, SSH or WebDAV. Note that entropy-server does not support client SSL certificates for HTTPS.
  • HTTP, HTTPS or anonymous FTP to allow client downloads
  • A basic Portage terminology knowledge.
  • Optionally: some knowledge about Layman overlay manager.

Setting up your own repository

In order to set up your repository you need to create and modify a few configuration files first, then use eit to push and commit the packages.

/etc/entropy/server.conf

The first stage is setting up your server.conf file located in /etc/entropy.

Copy it and begin to edit with your editor of choice..

# cp /etc/entropy/server.conf.example /etc/entropy/server.conf

There are only a few parameters truly necessary to set in this file.

Community mode

This option means that your repository is an overlay over one of the complete repositories, e.g. sabayonlinux.org or sabayon-weekly. Set this to enabled unless you want to upload all packages your newly built ones depend on, even if they're in the main repositories.

Only set this to disabled if you're trying to fork Sabayon, which nobody really wants.

Default repository

The option default-repository contains the name of the default repository you will be adding packages to with En:Eit repository manager. The name can contains ASCII characters, dot (.), hyphen (-) and underscore (_) only.

Repository

This is the main configuration string of for your repository. It consists of the following fields:

  • Repository Identifier (see above) - This is the repository name in the same form as in the default-repository field.
  • Repository Description - A more verbose description of your repository.
  • Repository URI(s) - URI to your remote server. The format depends on the type of server.

In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.

In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.

You can also use file:// URIs for truly local repositories.

If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with <p>. If you want the URI to contain only repository information (no package download), you can prefix the URI with <r>.

See the example below:

$ vim /etc/entropy/server.conf
repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/

You can enter more than one repository, just append more repository = ... lines.

/etc/entropy/repositories.conf.d/

You can put a file similar to /etc/entropy/repositories.conf with your chosen name and .conf suffix. Files starting with _ are considered inactive repositories. These files are only used for downloading, syncing, upgrading and installing packages and not for pushing packages.

Repository

This is the main configuration string of for your repository. It consists of a slightly different set of fields than in /etc/entropy/server.conf.

  • Repository Identifier (see above) - This is the repository name in the same form as in the default-repository field.
  • Repository Description - A more verbose description of your Repository.
  • Package URI(s) - URI to your remote server. The format depends on the type of server.

In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.

In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.

You can also use file:// URIs for truly local repositories.

  • Repository URI - Also an URI to your remote server in the same format as above with an optional suffix #bz2 for bzip2 compression, #gz for gzip compression.

Subsequent lines with the same Repository Identifier are treated as package mirrors. It is recommended to create one file per repository instead of adding multiple repositories with different Identifiers in the same file.

See the example below:

$ vim /etc/entropy/repository.conf.d/entropy_astralstorm.conf
repository = astralstorm|AstralStorm's Sabayon Overlay|https://hackerspace.pl/~astralstorm/sabayon|https://hackerspace.pl/~astralstorm/sabayon#bz2
repository = astralstorm|AstralStorm's Sabayon Overlay|file:///home/astralstorm/sabayon|

=Initialising the Repository

Stop.png
This rest of the article is outdated and is yet to undergo undergoing a major rework.

If all is configured properly, you will find that running this command:

# equo status

Will bring the following output

# >>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://myrepository:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: myrepository (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   myrepository
>>  !!! [repo:myrepository] Your default repository is not initialized.
Do you want to initialize your default repository ? [Yes/No]

As you can see Entropy found that your shiny new repository is empty and asked you to initialize it. Do it!

It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.

Anyway, eventually it will finish and you should see somthing along the lines of the below:

>>  !!! Entropy database is probably empty. If you don't agree with what I'm saying, then it's probably corrupted! I won't stop you here btw...
>>  !!! Entropy database: fixing KDE dep strings on etpdb:myrepository. Please wait ...
>>  * Mirrors have not been unlocked. Remember to sync them.

If you do see the above then your repoistory has been setup successfully! In the words of Joost_op "go get yourself a beer".

Compiling and Adding Packages

Overview

The next step (the fun bit) is filling it with some packages.

Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:

# equo database spmsync

Compilation

Now to add a package, its Compile time! The first step is to compile something using any Portage-compabile source package manager (Portage itself, Paludis, Pkgcore - anything that writes /var/db/pkg properly). Let's say we want to compile our own media-sound/amarok perhaps with "postgres" support or with different CFLAGS. Then, we spawn Portage:

# emerge -av media-sound/amarok

Note, if you get Equo called by Portage, you need to select "no". This because of a Portage behavioral change in the latest release (apparently it pulls into its xpak metadata equo's bashrc).

Adding the Package to Your Repository

Single Repository

Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:

# equo community repos update

You will be asked to push the newly compiled packages into your default repository. This is the output you should expect:

 
>>  @@ These are the packages that would be added/updated to the add list:
>>    media-sound/amarok-1.4.9.1
>>   Would you like to package them now ? (inside mynewrepo) [Yes/No] 

Of course yes! We have now pushed media-sound/amarok into our local repository database!

Multiple Repositories

If you have more than one community repository, a quick way to switch the default (working) repository is using ETP_REPO environmental variable:

# ETP_REPO=myotherrepository equo community repos update

You can also export it, and it will persitant for that session.

# export ETP_REPO=myotherrepository 
# equo community repos update

You will see the switch right after having pressed the Enter key:

 
# >>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://mynewrepo:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: myotherrepository (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   mynewrepo
>>   myotherrepository

When uploading you can can add the --syncall argument to sync all the repositories in one go.

Uploading the Packages

We now need to upload the new binary package(s) and the updated repository database.

equo community mirrors sync

This is the output you should expect:

>>  @@ Entropy Server Mirrors Interface loaded:
>>   mirror: ftp://mynewrepo:[email protected]/
>>  @@ Entropy Server Interface Instance on repository: mynewrepo (type: community repository)
>>  @@ Currently configured repositories:
>>   __system__
>>   mynewrepo
>>  * Starting to sync data across mirrors (packages/database) ...
>>  @@ [repo:mynewrepo|sync|branch:3.5] packages sync: sabayonlinux.org
>>  @@ Local statistics:
>>  @@ upload directory:                1 files ready
>>  @@ packages directory:              0 files ready
>>  @@ Remote statistics for: sabayonlinux.org
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/
>>  * [ftp:sabayonlinux.org] switching to: 3.5
>>  @@ remote packages:                 0 files stored
>>  @@ Calculating queues ...
>>  ** Expanding queues:
>>    [branch:3.5|upload] media-sound:amarok-1.4.9.1~0.tbz2 [182.9kB]
>>  @@ Packages to be removed:                  0
>>  @@ Packages to be moved locally:            0
>>  @@ Packages to be uploaded:                 1
>>  @@ Total removal size:                      0.0b
>>  @@ Total upload size:                       182.9kB
>>  @@ Total download size:                     0.0b
Would you like to run the steps above ? [Yes/No] 

Last chance to double check what is getting done. We will choose Yes.

>>  @@ [sabayonlinux.org|upload] connecting to mirror...
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/3.5
>>  @@ [sabayonlinux.org|#1|(1/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2.md5
>>  @@ [sabayonlinux.org|#1|(1/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2.md5: not supported
>>  @@ [sabayonlinux.org|#1|(1/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2.md5
>>  @@ [sabayonlinux.org|#1|(2/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2
>>  @@ [sabayonlinux.org|#1|(2/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2: not supported
>>  @@ [sabayonlinux.org|#1|(2/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2
>>  @@ [branch:5] upload completed successfully: sabayonlinux.org
>> Please insert a commit message:  my first commit
>>  * Remote Entropy Database Repository Status:
>>       Host:  sabayonlinux.org
>>        * Database revision: 0
>>        * Database local revision currently at: 1
>>  @@ [repo:mynewrepo|sabayonlinux.org] mirror successfully locked for download
>>  * [repo:mynewrepo|sabayonlinux.org|upload] preparing to upload database to mirror
>>  * Entropy database is already locked by you :-)
>>  * Mirrors have not been unlocked. Remember to sync them.
>>  * Entropy database is already locked by you :-)
>>  * Mirrors have not been unlocked. Remember to sync them.
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:3] preparing uncompressed database for the upload
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:2] creating compressed database dump + checksum
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>    dump: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2
>>    dump checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2.md5
>>    opener: bz2.BZ2File
>>    Database Export completed.
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
>>    compressed database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.bz2
>>    compressed checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.md5
>>    opener: bz2.BZ2File
>>  @@ [sabayonlinux.org|upload] connecting to mirror...
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
>>  * [ftp:sabayonlinux.org] switching to: /
>>  * [ftp:sabayonlinux.org] switching to: /
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/database/amd64/
>>  @@ [sabayonlinux.org|#1|(1/14)] uploading: make.conf
>>  @@ [sabayonlinux.org|#1|(1/14)] upload successful: make.conf
>>  @@ [sabayonlinux.org|#1|(2/14)] uploading: packages.db.bz2
>>  @@ [sabayonlinux.org|#1|(2/14)] upload successful: packages.db.bz2
>>  @@ [sabayonlinux.org|#1|(3/14)] uploading: packages.db
>>  @@ [sabayonlinux.org|#1|(3/14)] upload successful: packages.db
>>  @@ [sabayonlinux.org|#1|(4/14)] uploading: packages.db.dump.bz2.md5
>>  @@ [sabayonlinux.org|#1|(4/14)] upload successful: packages.db.dump.bz2.md5
>>  @@ [sabayonlinux.org|#1|(5/14)] uploading: packages.db.revision
>>  @@ [sabayonlinux.org|#1|(5/14)] upload successful: packages.db.revision
>>  @@ [sabayonlinux.org|#1|(6/14)] uploading: packages.rss
>>  @@ [sabayonlinux.org|#1|(6/14)] upload successful: packages.rss
>>  @@ [sabayonlinux.org|#1|(7/14)] uploading: updates.rss
>>  @@ [sabayonlinux.org|#1|(7/14)] upload successful: updates.rss
>>  @@ [sabayonlinux.org|#1|(8/14)] uploading: packages.db.dump.bz2
>>  @@ [sabayonlinux.org|#1|(8/14)] upload successful: packages.db.dump.bz2
>>  @@ [sabayonlinux.org|#1|(9/14)] uploading: package.mask
>>  @@ [sabayonlinux.org|#1|(9/14)] upload successful: package.mask
>>  @@ [sabayonlinux.org|#1|(10/14)] uploading: package.keywords
>>  @@ [sabayonlinux.org|#1|(10/14)] upload successful: package.keywords
>>  @@ [sabayonlinux.org|#1|(11/14)] uploading: packages.db.md5
>>  @@ [sabayonlinux.org|#1|(11/14)] upload successful: packages.db.md5
>>  @@ [sabayonlinux.org|#1|(12/14)] uploading: package.unmask
>>  @@ [sabayonlinux.org|#1|(12/14)] upload successful: package.unmask
>>  @@ [sabayonlinux.org|#1|(13/14)] uploading: profile.link
>>  @@ [sabayonlinux.org|#1|(13/14)] upload successful: profile.link
>>  @@ [sabayonlinux.org|#1|(14/14)] uploading: package.use
>>  @@ [sabayonlinux.org|#1|(14/14)] upload successful: package.use
>>  * [repo:mynewrepo|sabayonlinux.org] mirror successfully unlocked
>>  * [repo:mynewrepo|sync] database sync completed successfully
Should I continue with the tidy procedure ? [Yes/No] 
>>  @@ [repo:mynewrepo|tidy|branches:5] collecting expired packages
>>  @@ [branch:5] collecting expired packages in the selected branches
>>  @@ [branch:5] nothing to remove on this branch 

Done and Done!