Difference between revisions of "En:HOWTO: Set Up A Community Repository"

From Sabayon Wiki
Jump to: navigation, search
(Repository: Removed the box)
(Single Repository)
(6 intermediate revisions by 3 users not shown)
Line 19: Line 19:
  
 
==Setting up your own repository==
 
==Setting up your own repository==
In order to set up your repository you need to create and modify a few configuration files first, then use eit to push and commit the packages.
+
In order to set up your repository you need to create and modify a configuration file first, then use eit to push and commit the packages.
  
 
===/etc/entropy/server.conf===
 
===/etc/entropy/server.conf===
Line 35: Line 35:
  
 
Only set this to disabled if you're trying to fork Sabayon, which nobody really wants.
 
Only set this to disabled if you're trying to fork Sabayon, which nobody really wants.
 +
 +
{{Console|<pre class="clear">community-mode = enable</pre>}}
  
 
====Default repository====
 
====Default repository====
 
The option default-repository contains the name of the default repository you will be adding packages to with [[En:Eit]] repository manager.
 
The option default-repository contains the name of the default repository you will be adding packages to with [[En:Eit]] repository manager.
 
The name can contains ASCII characters, dot (.), hyphen (-) and underscore (_) only.
 
The name can contains ASCII characters, dot (.), hyphen (-) and underscore (_) only.
 +
 +
This is an example for the Nirvana Community Repository
 +
 +
{{Console|<pre class="clear">default-repository = nirvana</pre>}}
  
 
====Repository====
 
====Repository====
Line 53: Line 59:
 
You can also use file:// URIs for truly local repositories.
 
You can also use file:// URIs for truly local repositories.
  
If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with <p>. If you want the URI to contain only repository information (no package download), you can prefix the URI with <r>.
+
If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with &lt;p&gt;. If you want the URI to contain only repository information (no package download), you can prefix the URI with &lt;r&gt;.
  
See the example below:
+
See the examples below:
{{Code|Snippet from /etc/entropy/server.conf|<pre class="clear">repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/</pre>}}
+
{{Console|<pre class="clear">repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/</pre>}}
 +
{{Console|<pre class="clear">repository|nirvana|Nirvana Sabayon Linux Community Repository|ftp://danysk:[email protected]/public/nirvana/</pre>}}
  
 
You can enter more than one repository, just append more repository = ... lines.
 
You can enter more than one repository, just append more repository = ... lines.
Line 62: Line 69:
 
===/etc/entropy/repositories.conf.d/===
 
===/etc/entropy/repositories.conf.d/===
 
You can put a file similar to /etc/entropy/repositories.conf with your chosen name and .conf suffix. Files starting with _ are considered inactive repositories.
 
You can put a file similar to /etc/entropy/repositories.conf with your chosen name and .conf suffix. Files starting with _ are considered inactive repositories.
These files are only used for downloading, syncing, upgrading and installing packages and not for pushing packages.
+
These files are only used for downloading, syncing, upgrading and installing packages and not for pushing packages. You will share this file to give other users access to your repository.
  
====Repository====
+
Example from Nirvana:
This is the main configuration string of for your repository. It consists of a slightly different set of fields than in /etc/entropy/server.conf.
+
{{Console|<pre class="clear">[nirvana]
 
+
desc = Nirvana Sabayon Linux Community Repository
* Repository Identifier (see above) - This is the repository name in the same form as in the default-repository field.
+
repo = http://www.ols18.com/danysk/nirvana#bz2
* Repository Description - A more verbose description of your Repository.
+
enabled = true
* Package URI(s) - URI to your remote server. The format depends on the type of server.
+
pkg = http://www.ols18.com/danysk/nirvana
In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.
+
 
+
In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.
+
 
+
You can also use file:// URIs for truly local repositories.
+
* Repository URI - Also an URI to your remote server in the same format as above with an optional suffix #bz2 for bzip2 compression, #gz for gzip compression.
+
 
+
Subsequent lines with the same Repository Identifier are treated as package mirrors.
+
It is recommended to create one file per repository instead of adding multiple repositories with different Identifiers in the same file.
+
 
+
See the example below:
+
{{Code|Snippet from /etc/entropy/repository.conf.d/entropy_astralstorm.conf|<pre class="clear">
+
repository = astralstorm|AstralStorm's Sabayon Overlay|https://hackerspace.pl/~astralstorm/sabayon|https://hackerspace.pl/~astralstorm/sabayon#bz2
+
repository = astralstorm|AstralStorm's Sabayon Overlay|file:///home/astralstorm/sabayon|
+
 
</pre>}}
 
</pre>}}
  
===Initialising the Repository==
+
==Initialising the Repository==
{{Warning| '''This rest of the article is outdated and is yet to undergo undergoing a major rework.'''}}
+
You can start with:
 
+
If all is configured properly, you will find that running this command:
+
 
+
{{Console| <pre class="clear"># equo status</pre>}}
+
 
+
Will bring the following output
+
  
{{Console| <pre class="clear">
+
{{Console| <pre class="clear"># eit init reponame</pre>}}
# >>  @@ Entropy Server Mirrors Interface loaded:
+
>>  mirror: ftp://myrepository:[email protected]/
+
>>  @@ Entropy Server Interface Instance on repository: myrepository (type: community repository)
+
>>  @@ Currently configured repositories:
+
>>  __system__
+
>>  myrepository
+
>>  !!! [repo:myrepository] Your default repository is not initialized.
+
Do you want to initialize your default repository ? [Yes/No]
+
</pre>}}  
+
  
As you can see Entropy found that your shiny new repository is empty and asked you to initialize it. Do it!  
+
Entropy will find that your shiny new repository is empty and asked you to initialize it. Do it!  
  
 
It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.
 
It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.
 
Anyway, eventually it will finish and you should see somthing along the lines of the below:
 
 
{{Console| <pre class="clear">
 
>>  !!! Entropy database is probably empty. If you don't agree with what I'm saying, then it's probably corrupted! I won't stop you here btw...
 
>>  !!! Entropy database: fixing KDE dep strings on etpdb:myrepository. Please wait ...
 
>>  * Mirrors have not been unlocked. Remember to sync them.
 
</pre>}}
 
 
If you do see the above then your repoistory has been setup successfully! In the words of Joost_op "go get yourself a beer".
 
  
 
==Compiling and Adding Packages==
 
==Compiling and Adding Packages==
Line 125: Line 93:
  
 
Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:
 
Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:
{{Console| <pre class="clear"># equo database spmsync</pre>}}
+
{{Console| <pre class="clear"># equo rescue spmsync</pre>}}
  
 
===Compilation===
 
===Compilation===
Line 138: Line 106:
 
Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:
 
Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:
  
{{Console| <pre class="clear"># equo community repos update</pre>}}
+
{{Console| <pre class="clear"># eit add "package-name"</pre>}}
  
You will be asked to push the newly compiled packages into your default repository.  
+
To specify one or more packages.
This is the output you should expect:
+
  
{{Console| <pre class="clear">  
+
{{Console| <pre class="clear"># eit commit</pre>}}
>>  @@ These are the packages that would be added/updated to the add list:
+
>>    media-sound/amarok-1.4.9.1
+
>>  Would you like to package them now ? (inside mynewrepo) [Yes/No]
+
</pre>}}
+
  
Of course yes! We have now pushed media-sound/amarok into our local repository database!
+
To add every packages compiled locally in your remote repository.
 
+
====Multiple Repositories====
+
If you have more than one community repository, a quick way to switch the default (working) repository is using ETP_REPO environmental variable:
+
 
+
{{Console| <pre class="clear"># ETP_REPO=myotherrepository equo community repos update</pre>}}
+
 
+
You can also export it, and it will persitant for that session.
+
 
+
{{Console| <pre class="clear"># export ETP_REPO=myotherrepository
+
# equo community repos update</pre>}}
+
 
+
You will see the switch right after having pressed the Enter key:
+
 
+
{{Console| <pre class="clear">
+
# >>  @@ Entropy Server Mirrors Interface loaded:
+
>>  mirror: ftp://mynewrepo:[email protected]/
+
>>  @@ Entropy Server Interface Instance on repository: myotherrepository (type: community repository)
+
>>  @@ Currently configured repositories:
+
>>  __system__
+
>>  mynewrepo
+
>>  myotherrepository
+
</pre>}}
+
 
+
When uploading you can can add the --syncall argument to sync all the repositories in one go.
+
 
+
==Uploading the Packages==
+
We now need to upload the new binary package(s) and the updated repository database.
+
 
+
{{Console| <pre class="clear">
+
equo community mirrors sync
+
</pre>}}
+
 
+
This is the output you should expect:
+
 
+
{{Console| <pre class="clear">
+
>>  @@ Entropy Server Mirrors Interface loaded:
+
>>  mirror: ftp://mynewrepo:[email protected]/
+
>>  @@ Entropy Server Interface Instance on repository: mynewrepo (type: community repository)
+
>>  @@ Currently configured repositories:
+
>>  __system__
+
>>  mynewrepo
+
>>  * Starting to sync data across mirrors (packages/database) ...
+
>>  @@ [repo:mynewrepo|sync|branch:3.5] packages sync: sabayonlinux.org
+
>>  @@ Local statistics:
+
>>  @@ upload directory:                1 files ready
+
>>  @@ packages directory:              0 files ready
+
>>  @@ Remote statistics for: sabayonlinux.org
+
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
+
>>  * [ftp:sabayonlinux.org] switching to: /
+
>>  * [ftp:sabayonlinux.org] switching to: /
+
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/
+
>>  * [ftp:sabayonlinux.org] switching to: 3.5
+
>>  @@ remote packages:                0 files stored
+
>>  @@ Calculating queues ...
+
>>  ** Expanding queues:
+
>>    [branch:3.5|upload] media-sound:amarok-1.4.9.1~0.tbz2 [182.9kB]
+
>>  @@ Packages to be removed:                  0
+
>>  @@ Packages to be moved locally:            0
+
>>  @@ Packages to be uploaded:                1
+
>>  @@ Total removal size:                      0.0b
+
>>  @@ Total upload size:                      182.9kB
+
>>  @@ Total download size:                    0.0b
+
Would you like to run the steps above ? [Yes/No]
+
</pre>}}
+
  
Last chance to double check what is getting done. We will choose Yes.
+
The package will be created and stored locally. Once done, it is time to push it remotely!
  
{{Console| <pre class="clear">
+
{{Console| <pre class="clear"># eit push --quick</pre>}}
>>  @@ [sabayonlinux.org|upload] connecting to mirror...
+
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
+
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/packages/amd64/3.5
+
>>  @@ [sabayonlinux.org|#1|(1/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2.md5
+
>>  @@ [sabayonlinux.org|#1|(1/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2.md5: not supported
+
>>  @@ [sabayonlinux.org|#1|(1/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2.md5
+
>>  @@ [sabayonlinux.org|#1|(2/2)] uploading: media-sound:amarok-1.4.9.1~0.tbz2
+
>>  @@ [sabayonlinux.org|#1|(2/2)] digest verification: media-sound:amarok-1.4.9.1~0.tbz2: not supported
+
>>  @@ [sabayonlinux.org|#1|(2/2)] upload successful: media-sound:amarok-1.4.9.1~0.tbz2
+
>>  @@ [branch:5] upload completed successfully: sabayonlinux.org
+
>> Please insert a commit message:  my first commit
+
>>  * Remote Entropy Database Repository Status:
+
>>      Host:  sabayonlinux.org
+
>>        * Database revision: 0
+
>>        * Database local revision currently at: 1
+
>>  @@ [repo:mynewrepo|sabayonlinux.org] mirror successfully locked for download
+
>>  * [repo:mynewrepo|sabayonlinux.org|upload] preparing to upload database to mirror
+
>>  * Entropy database is already locked by you :-)
+
>>  * Mirrors have not been unlocked. Remember to sync them.
+
>>  * Entropy database is already locked by you :-)
+
>>  * Mirrors have not been unlocked. Remember to sync them.
+
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:3] preparing uncompressed database for the upload
+
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
+
>>  * [repo:mynewrepo|sabayonlinux.org|EAPI:2] creating compressed database dump + checksum
+
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
+
>>    dump: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2
+
>>    dump checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.dump.bz2.md5
+
>>    opener: bz2.BZ2File
+
>>    Database Export completed.
+
>>    database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db
+
>>    compressed database path: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.bz2
+
>>    compressed checksum: /var/lib/entropy/server/mynewrepo/database/amd64/packages.db.md5
+
>>    opener: bz2.BZ2File
+
>>  @@ [sabayonlinux.org|upload] connecting to mirror...
+
>>  * [ftp:sabayonlinux.org] connecting with user: mynewrepo
+
>>  * [ftp:sabayonlinux.org] switching to: /
+
>>  * [ftp:sabayonlinux.org] switching to: /
+
>>  @@ [sabayonlinux.org|upload] changing directory to standard/mynewrepo/database/amd64/...
+
>>  * [ftp:sabayonlinux.org] switching to: standard/mynewrepo/database/amd64/
+
>>  @@ [sabayonlinux.org|#1|(1/14)] uploading: make.conf
+
>>  @@ [sabayonlinux.org|#1|(1/14)] upload successful: make.conf
+
>>  @@ [sabayonlinux.org|#1|(2/14)] uploading: packages.db.bz2
+
>>  @@ [sabayonlinux.org|#1|(2/14)] upload successful: packages.db.bz2
+
>>  @@ [sabayonlinux.org|#1|(3/14)] uploading: packages.db
+
>>  @@ [sabayonlinux.org|#1|(3/14)] upload successful: packages.db
+
>>  @@ [sabayonlinux.org|#1|(4/14)] uploading: packages.db.dump.bz2.md5
+
>>  @@ [sabayonlinux.org|#1|(4/14)] upload successful: packages.db.dump.bz2.md5
+
>>  @@ [sabayonlinux.org|#1|(5/14)] uploading: packages.db.revision
+
>>  @@ [sabayonlinux.org|#1|(5/14)] upload successful: packages.db.revision
+
>>  @@ [sabayonlinux.org|#1|(6/14)] uploading: packages.rss
+
>>  @@ [sabayonlinux.org|#1|(6/14)] upload successful: packages.rss
+
>>  @@ [sabayonlinux.org|#1|(7/14)] uploading: updates.rss
+
>>  @@ [sabayonlinux.org|#1|(7/14)] upload successful: updates.rss
+
>>  @@ [sabayonlinux.org|#1|(8/14)] uploading: packages.db.dump.bz2
+
>>  @@ [sabayonlinux.org|#1|(8/14)] upload successful: packages.db.dump.bz2
+
>>  @@ [sabayonlinux.org|#1|(9/14)] uploading: package.mask
+
>>  @@ [sabayonlinux.org|#1|(9/14)] upload successful: package.mask
+
>>  @@ [sabayonlinux.org|#1|(10/14)] uploading: package.keywords
+
>>  @@ [sabayonlinux.org|#1|(10/14)] upload successful: package.keywords
+
>>  @@ [sabayonlinux.org|#1|(11/14)] uploading: packages.db.md5
+
>>  @@ [sabayonlinux.org|#1|(11/14)] upload successful: packages.db.md5
+
>>  @@ [sabayonlinux.org|#1|(12/14)] uploading: package.unmask
+
>>  @@ [sabayonlinux.org|#1|(12/14)] upload successful: package.unmask
+
>>  @@ [sabayonlinux.org|#1|(13/14)] uploading: profile.link
+
>>  @@ [sabayonlinux.org|#1|(13/14)] upload successful: profile.link
+
>>  @@ [sabayonlinux.org|#1|(14/14)] uploading: package.use
+
>>  @@ [sabayonlinux.org|#1|(14/14)] upload successful: package.use
+
>>  * [repo:mynewrepo|sabayonlinux.org] mirror successfully unlocked
+
>>  * [repo:mynewrepo|sync] database sync completed successfully
+
Should I continue with the tidy procedure ? [Yes/No]
+
>>  @@ [repo:mynewrepo|tidy|branches:5] collecting expired packages
+
>>  @@ [branch:5] collecting expired packages in the selected branches
+
>>  @@ [branch:5] nothing to remove on this branch </pre>}}
+
  
Done and Done!
+
We have now pushed media-sound/amarok into our repository database!
  
 
[[Category:Entropy|En:HOWTO: Set Up A Community Repository]]
 
[[Category:Entropy|En:HOWTO: Set Up A Community Repository]]
 
[[Category:HOWTO|En:HOWTO: Set Up A Community Repository]]
 
[[Category:HOWTO|En:HOWTO: Set Up A Community Repository]]

Revision as of 16:04, 6 April 2014

i18n: ca de en es fr
Stop.png
This article was outdated and is undergoing a major rework.

Why?

If you want different USE flags on packages or you want that new version you've just found in an overlay or just want to play with something cool then setting up a commnity repo is for you. It allows you to compile portage ebuilds into entropy packages and upload them up to a ftp, ssh or even http or https WebDAV site.

More information can be found on the En:Community Repositories Page.

Basic Requirements

Here is a list of the requisites you need to satisfy before starting your own repository:

  • Portage or any Portage-compatible Source Package Manager (to compile packages)
  • sys-apps/entropy-server package. The tutorial was tested using Entropy version 175.
  • Remote server to upload packages - can use FTP, SSH or WebDAV. Note that entropy-server does not support client SSL certificates for HTTPS.
  • HTTP, HTTPS or anonymous FTP to allow client downloads
  • A basic Portage terminology knowledge.
  • Optionally: some knowledge about Layman overlay manager.

Setting up your own repository

In order to set up your repository you need to create and modify a configuration file first, then use eit to push and commit the packages.

/etc/entropy/server.conf

The first stage is setting up your server.conf file located in /etc/entropy.

Copy it and begin to edit with your editor of choice..

# cp /etc/entropy/server.conf.example /etc/entropy/server.conf

There are only a few parameters truly necessary to set in this file.

Community mode

This option means that your repository is an overlay over one of the complete repositories, e.g. sabayonlinux.org or sabayon-weekly. Set this to enabled unless you want to upload all packages your newly built ones depend on, even if they're in the main repositories.

Only set this to disabled if you're trying to fork Sabayon, which nobody really wants.

community-mode = enable

Default repository

The option default-repository contains the name of the default repository you will be adding packages to with En:Eit repository manager. The name can contains ASCII characters, dot (.), hyphen (-) and underscore (_) only.

This is an example for the Nirvana Community Repository

default-repository = nirvana

Repository

This is the main configuration string of for your repository. It consists of the following fields:

  • Repository Identifier (see above) - This is the repository name in the same form as in the default-repository field.
  • Repository Description - A more verbose description of your repository.
  • Repository URI(s) - URI to your remote server. The format depends on the type of server.

In case of FTP and HTTP(S) WebDAV, it's simple: ftp://user:[email protected]/some/folders.

In case of SSH, it is slightly different: ssh:[email protected]:some/folders. It is recommended to set up ssh-agent, as otherwise entropy-server will be asking many times about the password.

You can also use file:// URIs for truly local repositories.

If you need the repository URI to be package only (no way to run entropy sync), you can prefix the URI with <p>. If you want the URI to contain only repository information (no package download), you can prefix the URI with <r>.

See the examples below:

repository = astralstorm|AstralStorm's Sabayon General Repository|file:///home/astralstorm/sabayon/ ssh:[email protected]:~/public_html/sabayon/
repository|nirvana|Nirvana Sabayon Linux Community Repository|ftp://danysk:[email protected]/public/nirvana/

You can enter more than one repository, just append more repository = ... lines.

/etc/entropy/repositories.conf.d/

You can put a file similar to /etc/entropy/repositories.conf with your chosen name and .conf suffix. Files starting with _ are considered inactive repositories. These files are only used for downloading, syncing, upgrading and installing packages and not for pushing packages. You will share this file to give other users access to your repository.

Example from Nirvana:

[nirvana]
desc = Nirvana Sabayon Linux Community Repository
repo = http://www.ols18.com/danysk/nirvana#bz2
enabled = true
pkg = http://www.ols18.com/danysk/nirvana

Initialising the Repository

You can start with:

# eit init reponame

Entropy will find that your shiny new repository is empty and asked you to initialize it. Do it!

It will then connect to your FTP, setup the directory and database structure. Initally, the process could take a while, especially the FTP directory listing, we've noticed that if it's empty, Python ftplib takes a while.

Compiling and Adding Packages

Overview

The next step (the fun bit) is filling it with some packages.

Before starting, we must be sure that our Entropy installed packages database is in sync with Portage. When you install a package using Entropy, the Portage database gets updated too. The command to merge your Portage-updated packages in Entropy is this:

# equo rescue spmsync

Compilation

Now to add a package, its Compile time! The first step is to compile something using any Portage-compabile source package manager (Portage itself, Paludis, Pkgcore - anything that writes /var/db/pkg properly). Let's say we want to compile our own media-sound/amarok perhaps with "postgres" support or with different CFLAGS. Then, we spawn Portage:

# emerge -av media-sound/amarok

Note, if you get Equo called by Portage, you need to select "no". This because of a Portage behavioral change in the latest release (apparently it pulls into its xpak metadata equo's bashrc).

Adding the Package to Your Repository

Single Repository

Once you have compiled and installed the application using portage its time to tell Entropy to push this package into our repository. You can do it calling:

# eit add "package-name"

To specify one or more packages.

# eit commit

To add every packages compiled locally in your remote repository.

The package will be created and stored locally. Once done, it is time to push it remotely!

# eit push --quick

We have now pushed media-sound/amarok into our repository database!