5.4. The Distribution Files
Prev Chapter 5. Configuring the Makefile Next

5.4. The Distribution Files

The second part of the Makefile describes the files that must be downloaded to build the port, and where they can be downloaded.

5.4.1. DISTNAME

DISTNAME is the name of the port as called by the authors of the software. DISTNAME defaults to ${PORTNAME}-${DISTVERSIONPREFIX}${DISTVERSION}${DISTVERSIONSUFFIX}, and if not set, DISTVERSION defaults to ${PORTVERSION} so override DISTNAME only if necessary. DISTNAME is only used in two places. First, the distribution file list (DISTFILES) defaults to ${DISTNAME}${EXTRACT_SUFX}. Second, the distribution file is expected to extract into a subdirectory named WRKSRC, which defaults to work/${DISTNAME}.

Some vendor's distribution names which do not fit into the ${PORTNAME}-${PORTVERSION}-scheme can be handled automatically by setting DISTVERSIONPREFIX, DISTVERSION, and DISTVERSIONSUFFIX. PORTVERSION will be derived from DISTVERSION automatically.

Important:

Only one of PORTVERSION and DISTVERSION can be set at a time. If DISTVERSION does not derive a correct PORTVERSION, do not use DISTVERSION.

If the upstream version scheme can be derived into a ports-compatible version scheme, set some variable to the upstream version, do not use DISTVERSION as the variable name. Set PORTVERSION to the computed version based on the variable you created, and set DISTNAME accordingly.

If the upstream version scheme cannot easily be coerced into a ports-compatible value, set PORTVERSION to a sensible value, and set DISTNAME with PORTNAME with the verbatim upstream version.

Example 5.6. Deriving PORTVERSION Manually

BIND9 uses a version scheme that is not compatible with the ports versions (it has - in its versions) and cannot be derived using DISTVERSION because after the 9.9.9 release, it will release a “patchlevels” in the form of 9.9.9-P1. DISTVERSION would translate that into 9.9.9.p1, which, in the ports versioning scheme means 9.9.9 pre-release 1, which is before 9.9.9 and not after. So PORTVERSION is manually derived from an ISCVERSION variable to output 9.9.9p1.

The order into which the ports framework, and pkg, will sort versions is checked using the -t argument of pkg-version(8):

% pkg version -t 9.9.9 9.9.9.p1
> 1
% pkg version -t 9.9.9 9.9.9p1
< 2

1

The > sign means that the first argument passed to -t is greater than the second argument. 9.9.9 is after 9.9.9.p1.

2

The < sign means that the first argument passed to -t is less than the second argument. 9.9.9 is before 9.9.9p1.

In the port Makefile, for example dns/bind99, it is achieved by:

PORTNAME=	bind
PORTVERSION=	${ISCVERSION:S/-P/P/:S/b/.b/:S/a/.a/:S/rc/.rc/} 1
CATEGORIES=	dns net
MASTER_SITES=	ISC/bind9/${ISCVERSION} 2
PKGNAMESUFFIX=	99
DISTNAME=	${PORTNAME}-${ISCVERSION} 3

MAINTAINER=	mat@FreeBSD.org
COMMENT=	BIND DNS suite with updated DNSSEC and DNS64

LICENSE=	ISCL

# ISC releases things like 9.8.0-P1 or 9.8.1rc1, which our versioning does not like
ISCVERSION=	9.9.9-P6 4

4

Define upstream version in ISCVERSION, with a comment saying why it is needed.

1

Use ISCVERSION to get a ports-compatible PORTVERSION.

2

Use ISCVERSION directly to get the correct URL for fetching the distribution file.

3

Use ISCVERSION directly to name the distribution file.


Example 5.7. Derive DISTNAME from PORTVERSION

From time to time, the distribution file name has little or no relation to the version of the software.

In comms/kermit, only the last element of the version is present in the distribution file:

PORTNAME=	kermit
PORTVERSION=	9.0.304
CATEGORIES=	comms ftp net
MASTER_SITES=	ftp://ftp.kermitproject.org/kermit/test/tar/
DISTNAME=	cku${PORTVERSION:E}-dev20 1

1

The :E make(1) modifier returns the suffix of the variable, in this case, 304. The distribution file is correctly generated as cku304-dev20.tar.gz.


Example 5.8. Exotic Case 1

Sometimes, there is no relation between the software name, its version, and the distribution file it is distributed in.

From audio/libworkman:

PORTNAME=       libworkman
PORTVERSION=    1.4
CATEGORIES=     audio
MASTER_SITES=   LOCAL/jim
DISTNAME=       ${PORTNAME}-1999-06-20

Example 5.9. Exotic Case 2

In comms/librs232, the distribution file is not versioned, so using DIST_SUBDIR is needed:

PORTNAME=       librs232
PORTVERSION=    20160710
CATEGORIES=     comms
MASTER_SITES=   http://www.teuniz.net/RS-232/
DISTNAME=       RS-232
DIST_SUBDIR=	${PORTNAME}-${PORTVERSION}

Note:

PKGNAMEPREFIX and PKGNAMESUFFIX do not affect DISTNAME. Also note that if WRKSRC is equal to ${WRKDIR}/${DISTNAME} while the original source archive is named something other than ${PORTNAME}-${PORTVERSION}${EXTRACT_SUFX}, leave DISTNAME alone— defining only DISTFILES is easier than both DISTNAME and WRKSRC (and possibly EXTRACT_SUFX).

5.4.2. MASTER_SITES

Record the directory part of the FTP/HTTP-URL pointing at the original tarball in MASTER_SITES. Do not forget the trailing slash (/)!

The make macros will try to use this specification for grabbing the distribution file with FETCH if they cannot find it already on the system.

It is recommended that multiple sites are included on this list, preferably from different continents. This will safeguard against wide-area network problems.

Important:

MASTER_SITES must not be blank. It must point to the actual site hosting the distribution files. It cannot point to web archives, or the FreeBSD distribution files cache sites. The only exception to this rule is ports that do not have any distribution files. For example, meta-ports do not have any distribution files, so MASTER_SITES does not need to be set.

5.4.2.1. Using MASTER_SITE_* Variables

Shortcut abbreviations are available for popular archives like SourceForge (SOURCEFORGE), GNU (GNU), or Perl CPAN (PERL_CPAN). MASTER_SITES can use them directly:

MASTER_SITES=	GNU/make

The older expanded format still works, but all ports have been converted to the compact format. The expanded format looks like this:

MASTER_SITES=		${MASTER_SITE_GNU}
MASTER_SITE_SUBDIR=	make

These values and variables are defined in Mk/bsd.sites.mk. New entries are added often, so make sure to check the latest version of this file before submitting a port.

Tip:

For any MASTER_SITE_FOO variable, the shorthand FOO can be used. For example, use:

MASTER_SITES=	FOO

If MASTER_SITE_SUBDIR is needed, use this:

MASTER_SITES=	FOO/bar

Note:

Some MASTER_SITE_* names are quite long, and for ease of use, shortcuts have been defined:

Table 5.3. Shortcuts for MASTER_SITE_* Macros
MacroShortcut
PERL_CPANCPAN
GITHUBGH
GITHUB_CLOUDGHC
LIBREOFFICE_DEVLODEV
NETLIBNL
RUBYGEMSRG
SOURCEFORGESF

5.4.2.2. Magic MASTER_SITES Macros

Several “magic” macros exist for popular sites with a predictable directory structure. For these, just use the abbreviation and the system will choose a subdirectory automatically. For a port named Stardict, of version 1.2.3, and hosted on SourceForge, adding this line:

MASTER_SITES=	SF

infers a subdirectory named /project/stardict/stardict/1.2.3. If the inferred directory is incorrect, it can be overridden:

MASTER_SITES=	SF/stardict/WyabdcRealPeopleTTS/${PORTVERSION}

This can also be written as

MASTER_SITES=	SF
MASTER_SITE_SUBDIR=	stardict/WyabdcRealPeopleTTS/${PORTVERSION}
Table 5.4. Magic MASTER_SITES Macros
MacroAssumed subdirectory
APACHE_COMMONS_BINARIES${PORTNAME:S,commons-,,}
APACHE_COMMONS_SOURCE${PORTNAME:S,commons-,,}
APACHE_JAKARTA${PORTNAME:S,-,/,}/source
BERLIOS${PORTNAME:tl}.berlios
CHEESESHOPsource/${DISTNAME:C/(.).*/\1/}/${DISTNAME:C/(.*)-[0-9].*/\1/}
CPAN${PORTNAME:C/-.*//}
DEBIANpool/main/${PORTNAME:C/^((lib)?.).*$/\1/}/${PORTNAME}
FARSIGHT${PORTNAME}
FESTIVAL${PORTREVISION}
GCCreleases/${DISTNAME}
GENTOOdistfiles
GIMP${PORTNAME}/${PORTVERSION:R}/
GH${GH_ACCOUNT}/${GH_PROJECT}/tar.gz/${GH_TAGNAME}?dummy=/
GHC${GH_ACCOUNT}/${GH_PROJECT}/
GNOMEsources/${PORTNAME}/${PORTVERSION:C/^([0-9]+\.[0-9]+).*/\1/}
GNU${PORTNAME}
GNUPG${PORTNAME}
GNU_ALPHA${PORTNAME}
HORDE${PORTNAME}
LODEV${PORTNAME}
MATE${PORTVERSION:C/^([0-9]+\.[0-9]+).*/\1/}
MOZDEV${PORTNAME:tl}
NL${PORTNAME}
QTarchive/qt/${PORTVERSION:R}
SAMBA${PORTNAME}
SAVANNAH${PORTNAME:tl}
SF${PORTNAME:tl}/${PORTNAME:tl}/${PORTVERSION}

5.4.3. USE_GITHUB

If the distribution file comes from a specific commit or tag on GitHub for which there is no officially released file, there is an easy way to set the right DISTNAME and MASTER_SITES automatically. These variables are available:

Table 5.5. USE_GITHUB Description
VariableDescriptionDefault
GH_ACCOUNTAccount name of the GitHub user hosting the project${PORTNAME}
GH_PROJECTName of the project on GitHub${PORTNAME}
GH_TAGNAMEName of the tag to download (2.0.1, hash, ...) Using the name of a branch here is incorrect. It is also possible to use the hash of a commit id to do a snapshot.${DISTVERSIONPREFIX}${DISTVERSION}${DISTVERSIONSUFFIX}
GH_SUBDIRWhen the software needs an additional distribution file to be extracted within ${WRKSRC}, this variable can be used. See the examples in Section 5.4.3.1, “Fetching Multiple Files from GitHub” for more information.(none)
GH_TUPLEGH_TUPLE allows putting GH_ACCOUNT, GH_PROJECT, GH_TAGNAME, and GH_SUBDIR into a single variable. The format is account:project:tagname:group/subdir. The /subdir part is optional. It is helpful when there is more than one GitHub project from which to fetch. 

Important:

Do not use GH_TUPLE for the default distribution file, as it has no default.

Example 5.10. Simple Use of USE_GITHUB

While trying to make a port for version 1.2.7 of pkg from the FreeBSD user on github, at https://github.com/freebsd/pkg, The Makefile would end up looking like this (slightly stripped for the example):

PORTNAME=	pkg
DISTVERSION=	1.2.7

USE_GITHUB=	yes
GH_ACCOUNT=	freebsd

It will automatically have MASTER_SITES set to GH GHC and WRKSRC to ${WRKDIR}/pkg-1.2.7.


Example 5.11. More Complete Use of USE_GITHUB

While trying to make a port for the bleeding edge version of pkg from the FreeBSD user on github, at https://github.com/freebsd/pkg, the Makefile ends up looking like this (slightly stripped for the example):

PORTNAME=	pkg-devel
DISTVERSION=	1.3.0.a.20140411

USE_GITHUB=	yes
GH_ACCOUNT=	freebsd
GH_PROJECT=	pkg
GH_TAGNAME=	6dbb17b

It will automatically have MASTER_SITES set to GH GHC and WRKSRC to ${WRKDIR}/pkg-6dbb17b.

Tip:

20140411 is the date of the commit referenced in GH_TAGNAME, not the date the Makefile is edited, or the date the commit is made.


Example 5.12. Use of USE_GITHUB with DISTVERSIONPREFIX

From time to time, GH_TAGNAME is a slight variation from DISTVERSION. For example, if the version is 1.0.2, the tag is v1.0.2. In those cases, it is possible to use DISTVERSIONPREFIX or DISTVERSIONSUFFIX:

PORTNAME=	foo
DISTVERSIONPREFIX=	v
DISTVERSION=	1.0.2

USE_GITHUB=	yes

It will automatically set GH_TAGNAME to v1.0.2, while WRKSRC will be kept to ${WRKDIR}/foo-1.0.2.


Example 5.13. Using USE_GITHUB When Upstream Does Not Use Versions

If there never was a version upstream, do not invent one like 0.1 or 1.0. Create the port with a DISTVERSION of gYYYYMMDD, where g is for Git, and YYYYMMDD represents the date the commit referenced in GH_TAGNAME.

PORTNAME=	bar
DISTVERSION=	g20140411

USE_GITHUB=	yes
GH_TAGNAME=	c472d66b

This creates a versioning scheme that increases over time, and that is still before version 0 (see Example 5.1, “Using pkg-version(8) to Compare Versions.” for details on pkg-version(8)):

% pkg version -t g20140411 0
<

Which means using PORTEPOCH will not be needed in case upstream decides to cut versions in the future.


Example 5.14. Using USE_GITHUB to Access a Commit Between Two Versions

If the current version of the software uses a Git tag, and the port needs to be updated to a newer, intermediate version, without a tag, use git-describe(1) to find out the version to use:

% git describe --tags f0038b1
v0.7.3-14-gf0038b1

v0.7.3-14-gf0038b1 can be split into three parts:

v0.7.3

This is the last Git tag that appears in the commit history before the requested commit.

-14

This means that the requested commit, f0038b1, is the 14th commit after the v0.7.3 tag.

-gf0038b1

The -g means “Git”, and the f0038b1 is the commit hash that this reference points to.

PORTNAME=	bar
DISTVERSIONPREFIX=  v
DISTVERSION=	0.7.3-14
DISTVERSIONSUFFIX=  -gf0038b1

USE_GITHUB=	yes

This creates a versioning scheme that increases over time (well, over commits), and does not conflict with the creation of a 0.7.4 version. (See Example 5.1, “Using pkg-version(8) to Compare Versions.” for details on pkg-version(8)):

% pkg version -t 0.7.3 0.7.3.14
<
% pkg version -t 0.7.3.14 0.7.4
<

Note:

If the requested commit is the same as a tag, a shorter description is shown by default. The longer version is equivalent:

% git describe --tags c66c71d
v0.7.3
% git describe --tags --long c66c71d
v0.7.3-0-gc66c71d

5.4.3.1. Fetching Multiple Files from GitHub

The USE_GITHUB framework also supports fetching multiple distribution files from different places in GitHub. It works in a way very similar to Section 5.4.9, “Multiple Distribution or Patches Files from Multiple Locations”.

Multiple values are added to GH_ACCOUNT, GH_PROJECT, and GH_TAGNAME. Each different value is assigned a group. The main value can either have no group, or the :DEFAULT group. A value can be omitted if it is the same as the default as listed in Table 5.5, “USE_GITHUB Description”.

GH_TUPLE can also be used when there are a lot of distribution files. It helps keep the account, project, tagname, and group information at the same place.

For each group, a ${WRKSRC_group} helper variable is created, containing the directory into which the file has been extracted. The ${WRKSRC_group} variables can be used to move directories around during post-extract, or add to CONFIGURE_ARGS, or whatever is needed so that the software builds correctly.

Caution:

The :group part must be used for only one distribution file. It is used as a unique key and using it more than once will overwrite the previous values.

Note:

As this is only syntactic sugar above DISTFILES and MASTER_SITES, the group names must adhere to the restrictions on group names outlined in Section 5.4.9, “Multiple Distribution or Patches Files from Multiple Locations”

When fetching multiple files from GitHub, sometimes the default distribution file is not fetched from GitHub. To disable fetching the default distribution, set:

USE_GITHUB=	nodefault

Important:

When using USE_GITHUB=nodefault, the Makefile must set DISTFILES in its top block. The definition should be:

DISTFILES=    ${DISTNAME}${EXTRACT_SUFX}
Example 5.15. Use of USE_GITHUB with Multiple Distribution Files

From time to time, there is a need to fetch more than one distribution file. For example, when the upstream git repository uses submodules. This can be done easily using groups in the GH_* variables:

PORTNAME=	foo
DISTVERSION=	1.0.2

USE_GITHUB=	yes
GH_ACCOUNT=	bar:icons,contrib
GH_PROJECT=	foo-icons:icons foo-contrib:contrib
GH_TAGNAME=	1.0:icons fa579bc:contrib
GH_SUBDIR=	ext/icons:icons

CONFIGURE_ARGS=	--with-contrib=${WRKSRC_contrib}

This will fetch three distribution files from github. The default one comes from foo/foo and is version 1.0.2. The second one, with the icons group, comes from bar/foo-icons and is in version 1.0. The third one comes from bar/foo-contrib and uses the Git commit fa579bc. The distribution files are named foo-foo-1.0.2_GH0.tar.gz, bar-foo-icons-1.0_GH0.tar.gz, and bar-foo-contrib-fa579bc_GH0.tar.gz.

All the distribution files are extracted in ${WRKDIR} in their respective subdirectories. The default file is still extracted in ${WRKSRC}, in this case, ${WRKDIR}/foo-1.0.2. Each additional distribution file is extracted in ${WRKSRC_group}. Here, for the icons group, it is called ${WRKSRC_icons} and it contains ${WRKDIR}/foo-icons-1.0. The file with the contrib group is called ${WRKSRC_contrib} and contains ${WRKDIR}/foo-contrib-fa579bc.

The software's build system expects to find the icons in a ext/icons subdirectory in its sources, so GH_SUBDIR is used. GH_SUBDIR makes sure that ext exists, but that ext/icons does not already exist. Then it does this:

post-extract:
      @${MV} ${WRKSRC_icons} ${WRKSRC}/ext/icons

Example 5.16. Use of USE_GITHUB with Multiple Distribution Files Using GH_TUPLE

This is functionally equivalent to Example 5.15, “Use of USE_GITHUB with Multiple Distribution Files”, but using GH_TUPLE:

PORTNAME=	foo
DISTVERSION=	1.0.2

USE_GITHUB=	yes
GH_TUPLE=	bar:foo-icons:1.0:icons/ext/icons \
		bar:foo-contrib:fa579bc:contrib

CONFIGURE_ARGS=	--with-contrib=${WRKSRC_contrib}

Grouping was used in the previous example with bar:icons,contrib. Some redundant information is present with GH_TUPLE because grouping is not possible.


Example 5.17. How to Use USE_GITHUB with Git Submodules?

Ports with GitHub as an upstream repository sometimes use submodules. See git-submodule(1) for more information.

The problem with submodules is that each is a separate repository. As such, they each must be fetched separately.

Using finance/moneymanagerex as an example, its GitHub repository is https://github.com/moneymanagerex/moneymanagerex. It has a .gitmodules file at the root. This file describes all the submodules used in this repository, and lists additional repositories needed. This file will tell what additional repositories are needed:

[submodule "lib/wxsqlite3"]
	path = lib/wxsqlite3
	url = https://github.com/utelle/wxsqlite3.git
[submodule "3rd/mongoose"]
	path = 3rd/mongoose
	url = https://github.com/cesanta/mongoose.git
[submodule "3rd/LuaGlue"]
	path = 3rd/LuaGlue
	url = https://github.com/moneymanagerex/LuaGlue.git
[submodule "3rd/cgitemplate"]
	path = 3rd/cgitemplate
	url = https://github.com/moneymanagerex/html-template.git
[...]

The only information missing from that file is the commit hash or tag to use as a version. This information is found after cloning the repository:

% git clone --recurse-submodules https://github.com/moneymanagerex/moneymanagerex.git
Cloning into 'moneymanagerex'...
remote: Counting objects: 32387, done.
[...]
Submodule '3rd/LuaGlue' (https://github.com/moneymanagerex/LuaGlue.git) registered for path '3rd/LuaGlue'
Submodule '3rd/cgitemplate' (https://github.com/moneymanagerex/html-template.git) registered for path '3rd/cgitemplate'
Submodule '3rd/mongoose' (https://github.com/cesanta/mongoose.git) registered for path '3rd/mongoose'
Submodule 'lib/wxsqlite3' (https://github.com/utelle/wxsqlite3.git) registered for path 'lib/wxsqlite3'
[...]
Cloning into '/home/mat/work/freebsd/ports/finance/moneymanagerex/moneymanagerex/3rd/LuaGlue'...
Cloning into '/home/mat/work/freebsd/ports/finance/moneymanagerex/moneymanagerex/3rd/cgitemplate'...
Cloning into '/home/mat/work/freebsd/ports/finance/moneymanagerex/moneymanagerex/3rd/mongoose'...
Cloning into '/home/mat/work/freebsd/ports/finance/moneymanagerex/moneymanagerex/lib/wxsqlite3'...
[...]
Submodule path '3rd/LuaGlue': checked out 'c51d11a247ee4d1e9817dfa2a8da8d9e2f97ae3b'
Submodule path '3rd/cgitemplate': checked out 'cd434eeeb35904ebcd3d718ba29c281a649b192c'
Submodule path '3rd/mongoose': checked out '2140e5992ab9a3a9a34ce9a281abf57f00f95cda'
Submodule path 'lib/wxsqlite3': checked out 'fb66eb230d8aed21dec273b38c7c054dcb7d6b51'
[...]
% cd moneymanagerex
% git submodule status
 c51d11a247ee4d1e9817dfa2a8da8d9e2f97ae3b 3rd/LuaGlue (heads/master)
 cd434eeeb35904ebcd3d718ba29c281a649b192c 3rd/cgitemplate (cd434ee)
 2140e5992ab9a3a9a34ce9a281abf57f00f95cda 3rd/mongoose (6.2-138-g2140e59)
 fb66eb230d8aed21dec273b38c7c054dcb7d6b51 lib/wxsqlite3 (v3.4.0)
[...]

It can also be found on GitHub. Each subdirectory that is a submodule is shown as directory @ hash, for example, mongoose @ 2140e59.

Note:

While getting the information from GitHub seems more straightforward, the information found using git submodule status will provide more meaningful information. For example, here, lib/wxsqlite3's commit hash fb66eb2 correspond to v3.4.0. Both can be used interchangeably, but when a tag is available, use it.

Now that all the required information has been gathered, the Makefile can be written (only GitHub-related lines are shown):

PORTNAME=	moneymanagerex
DISTVERSIONPREFIX=	v
DISTVERSION=	1.3.0

USE_GITHUB=	yes
GH_TUPLE=	utelle:wxsqlite3:v3.4.0:wxsqlite3/lib/wxsqlite3 \
		moneymanagerex:LuaGlue:c51d11a:lua_glue/3rd/LuaGlue \
		moneymanagerex:html-template:cd434ee:html_template/3rd/cgitemplate \
		cesanta:mongoose:2140e59:mongoose/3rd/mongoose \
		[...]

5.4.4. USE_GITLAB

Similar to GitHub, if the distribution file comes from gitlab.com or is hosting the GitLab software, these variables are available for use and might need to be set.

Table 5.6. USE_GITLAB Description
VariableDescriptionDefault
GL_SITESite name hosting the GitLab projecthttps://gitlab.com
GL_ACCOUNTAccount name of the GitLab user hosting the project${PORTNAME}
GL_PROJECTName of the project on GitLab${PORTNAME}
GL_COMMITThe commit hash to download. Must be the full 160 bit, 40 character hex sha1 hash. This is a required variable for GitLab.(none)
GL_SUBDIRWhen the software needs an additional distribution file to be extracted within ${WRKSRC}, this variable can be used. See the examples in Section 5.4.4.1, “Fetching Multiple Files from GitLab” for more information.(none)
GL_TUPLEGL_TUPLE allows putting GL_SITE, GL_ACCOUNT, GL_PROJECT, GL_COMMIT, and GL_SUBDIR into a single variable. The format is site:account:project:commit:group/subdir. The site: and /subdir part is optional. It is helpful when there are more than one GitLab project from which to fetch. 

Example 5.18. Simple Use of USE_GITLAB

While trying to make a port for version 1.14 of libsignon-glib from the accounts-sso user on gitlab.com, at https://gitlab.com/accounts-sso/libsignon-glib, The Makefile would end up looking like this for fetching the distribution files:

PORTNAME=	libsignon-glib
DISTVERSION=	1.14

USE_GITLAB=	yes
GL_ACCOUNT=	accounts-sso
GL_COMMIT=	e90302e342bfd27bc8c9132ab9d0ea3d8723fd03

It will automatically have MASTER_SITES set to gitlab.com and WRKSRC to ${WRKDIR}/libsignon-glib-e90302e342bfd27bc8c9132ab9d0ea3d8723fd03-e90302e342bfd27bc8c9132ab9d0ea3d8723fd03.


Example 5.19. More Complete Use of USE_GITLAB

A more complete use of the above if port had no versioning and foobar from the foo user on project bar on a self hosted GitLab site https://gitlab.example.com, the Makefile ends up looking like this for fetching distribution files:

PORTNAME=	foobar
DISTVERSION=	g20170906

USE_GITLAB=	yes
GL_SITE=	https://gitlab.example.com
GL_ACCOUNT=	foo
GL_PROJECT=	bar
GL_COMMIT=	9c1669ce60c3f4f5eb43df874d7314483fb3f8a6

It will have MASTER_SITES set to "https://gitlab.example.com" and WRKSRC to ${WRKDIR}/bar-9c1669ce60c3f4f5eb43df874d7314483fb3f8a6-9c1669ce60c3f4f5eb43df874d7314483fb3f8a6.

Tip:

20170906 is the date of the commit referenced in GL_COMMIT, not the date the Makefile is edited, or the date the commit to the FreeBSD ports tree is made.

Note:

GL_SITE's protocol, port and webroot can all be modified in the same variable.


5.4.4.1. Fetching Multiple Files from GitLab

The USE_GITLAB framework also supports fetching multiple distribution files from different places from GitLab and GitLab hosted sites. It works in a way very similar to Section 5.4.9, “Multiple Distribution or Patches Files from Multiple Locations” and Section 5.4.4.1, “Fetching Multiple Files from GitLab”.

Multiple values are added to GL_SITE, GL_ACCOUNT, GL_PROJECT and GL_COMMIT. Each different value is assigned a group. Table 5.6, “USE_GITLAB Description”.

GL_TUPLE can also be used when there are a lot of distribution files. It helps keep the site, account, project, commit, and group information at the same place.

For each group, a ${WRKSRC_group} helper variable is created, containing the directory into which the file has been extracted. The ${WRKSRC_group} variables can be used to move directories around during post-extract, or add to CONFIGURE_ARGS, or whatever is needed so that the software builds correctly.

Caution:

The :group part must be used for only one distribution file. It is used as a unique key and using it more than once will overwrite the previous values.

Note:

As this is only syntactic sugar above DISTFILES and MASTER_SITES, the group names must adhere to the restrictions on group names outlined in Section 5.4.9, “Multiple Distribution or Patches Files from Multiple Locations”

When fetching multiple files using GitLab, sometimes the default distribution file is not fetched from a GitLab site. To disable fetching the default distribution, set:

USE_GITLAB=	nodefault

Important:

When using USE_GITLAB=nodefault, the Makefile must set DISTFILES in its top block. The definition should be:

DISTFILES=    ${DISTNAME}${EXTRACT_SUFX}
Example 5.20. Use of USE_GITLAB with Multiple Distribution Files

From time to time, there is a need to fetch more than one distribution file. For example, when the upstream git repository uses submodules. This can be done easily using groups in the GL_* variables:

PORTNAME=	foo
DISTVERSION=	1.0.2

USE_GITLAB=	yes
GL_SITE=	https://gitlab.example.com:9434/gitlab:icons
GL_ACCOUNT=	bar:icons,contrib
GL_PROJECT=	foo-icons:icons foo-contrib:contrib
GL_COMMIT=	c189207a55da45305c884fe2b50e086fcad4724b ae7368cab1ca7ca754b38d49da064df87968ffe4:icons 9e4dd76ad9b38f33fdb417a4c01935958d5acd2a:contrib
GL_SUBDIR=	ext/icons:icons

CONFIGURE_ARGS= --with-contrib=${WRKSRC_contrib}

This will fetch two distribution files from gitlab.com and one from gitlab.example.com hosting GitLab. The default one comes from https://gitlab.com/foo/foo and commit is c189207a55da45305c884fe2b50e086fcad4724b. The second one, with the icons group, comes from https://gitlab.example.com:9434/gitlab/bar/foo-icons and commit is ae7368cab1ca7ca754b38d49da064df87968ffe4. The third one comes from https://gitlab.com/bar/foo-contrib and is commit 9e4dd76ad9b38f33fdb417a4c01935958d5acd2a. The distribution files are named foo-foo-c189207a55da45305c884fe2b50e086fcad4724b_GL0.tar.gz, bar-foo-icons-ae7368cab1ca7ca754b38d49da064df87968ffe4_GL0.tar.gz, and bar-foo-contrib-9e4dd76ad9b38f33fdb417a4c01935958d5acd2a_GL0.tar.gz.

All the distribution files are extracted in ${WRKDIR} in their respective subdirectories. The default file is still extracted in ${WRKSRC}, in this case, ${WRKDIR}/foo-c189207a55da45305c884fe2b50e086fcad4724b-c189207a55da45305c884fe2b50e086fcad4724b. Each additional distribution file is extracted in ${WRKSRC_group}. Here, for the icons group, it is called ${WRKSRC_icons} and it contains ${WRKDIR}/foo-icons-ae7368cab1ca7ca754b38d49da064df87968ffe4-ae7368cab1ca7ca754b38d49da064df87968ffe4. The file with the contrib group is called ${WRKSRC_contrib} and contains ${WRKDIR}/foo-contrib-9e4dd76ad9b38f33fdb417a4c01935958d5acd2a-9e4dd76ad9b38f33fdb417a4c01935958d5acd2a.

The software's build system expects to find the icons in a ext/icons subdirectory in its sources, so GL_SUBDIR is used. GL_SUBDIR makes sure that ext exists, but that ext/icons does not already exist. Then it does this:

post-extract:
        @${MV} ${WRKSRC_icons} ${WRKSRC}/ext/icons

Example 5.21. Use of USE_GITLAB with Multiple Distribution Files Using GL_TUPLE

This is functionally equivalent to Example 5.20, “Use of USE_GITLAB with Multiple Distribution Files”, but using GL_TUPLE:

PORTNAME=	foo
DISTVERSION=	1.0.2

USE_GITLAB=	yes
GL_COMMIT=	c189207a55da45305c884fe2b50e086fcad4724b
GL_TUPLE=	https://gitlab.example.com:9434/gitlab:bar:foo-icons:ae7368cab1ca7ca754b38d49da064df87968ffe4:icons/ext/icons \
		bar:foo-contrib:9e4dd76ad9b38f33fdb417a4c01935958d5acd2a:contrib

CONFIGURE_ARGS= --with-contrib=${WRKSRC_contrib}

Grouping was used in the previous example with bar:icons,contrib. Some redundant information is present with GL_TUPLE because grouping is not possible.


5.4.5. EXTRACT_SUFX

If there is one distribution file, and it uses an odd suffix to indicate the compression mechanism, set EXTRACT_SUFX.

For example, if the distribution file was named foo.tar.gzip instead of the more normal foo.tar.gz, write:

DISTNAME=	foo
EXTRACT_SUFX=	.tar.gzip

The USES=tar[:xxx], USES=lha or USES=zip automatically set EXTRACT_SUFX to the most common archives extensions as necessary, see Chapter 17, Using USES Macros for more details. If neither of these are set then EXTRACT_SUFX defaults to .tar.gz.

Note:

As EXTRACT_SUFX is only used in DISTFILES, only set one of them..

5.4.6. DISTFILES

Sometimes the names of the files to be downloaded have no resemblance to the name of the port. For example, it might be called source.tar.gz or similar. In other cases the application's source code might be in several different archives, all of which must be downloaded.

If this is the case, set DISTFILES to be a space separated list of all the files that must be downloaded.

DISTFILES=	source1.tar.gz source2.tar.gz

If not explicitly set, DISTFILES defaults to ${DISTNAME}${EXTRACT_SUFX}.

5.4.7. EXTRACT_ONLY

If only some of the DISTFILES must be extracted—for example, one of them is the source code, while another is an uncompressed document—list the filenames that must be extracted in EXTRACT_ONLY.

DISTFILES=	source.tar.gz manual.html
EXTRACT_ONLY=	source.tar.gz

When none of the DISTFILES need to be uncompressed, set EXTRACT_ONLY to the empty string.

EXTRACT_ONLY=

5.4.8. PATCHFILES

If the port requires some additional patches that are available by FTP or HTTP, set PATCHFILES to the names of the files and PATCH_SITES to the URL of the directory that contains them (the format is the same as MASTER_SITES).

If the patch is not relative to the top of the source tree (that is, WRKSRC) because it contains some extra pathnames, set PATCH_DIST_STRIP accordingly. For instance, if all the pathnames in the patch have an extra foozolix-1.0/ in front of the filenames, then set PATCH_DIST_STRIP=-p1.

Do not worry if the patches are compressed; they will be decompressed automatically if the filenames end with .Z, .gz, .bz2 or .xz.

If the patch is distributed with some other files, such as documentation, in a compressed tarball, using PATCHFILES is not possible. If that is the case, add the name and the location of the patch tarball to DISTFILES and MASTER_SITES. Then, use EXTRA_PATCHES to point to those files and bsd.port.mk will automatically apply them. In particular, do not copy patch files into ${PATCHDIR}. That directory may not be writable.

Tip:

If there are multiple patches and they need mixed values for the strip parameter, it can be added alongside the patch name in PATCHFILES, e.g:

PATCHFILES=	patch1 patch2:-p1

This does not conflict with the master site grouping feature, adding a group also works:

PATCHFILES=	patch2:-p1:source2

Note:

The tarball will have been extracted alongside the regular source by then, so there is no need to explicitly extract it if it is a regular compressed tarball. Take extra care not to overwrite something that already exists in that directory if extracting it manually. Also, do not forget to add a command to remove the copied patch in the pre-clean target.

5.4.9. Multiple Distribution or Patches Files from Multiple Locations

(Consider this to be a somewhat “advanced topic”; those new to this document may wish to skip this section at first).

This section has information on the fetching mechanism known as both MASTER_SITES:n and MASTER_SITES_NN. We will refer to this mechanism as MASTER_SITES:n.

A little background first. OpenBSD has a neat feature inside DISTFILES and PATCHFILES which allows files and patches to be postfixed with :n identifiers. Here, n can be any word containing [0-9a-zA-Z_] and denote a group designation. For example:

DISTFILES=	alpha:0 beta:1

In OpenBSD, distribution file alpha will be associated with variable MASTER_SITES0 instead of our common MASTER_SITES and beta with MASTER_SITES1.

This is a very interesting feature which can decrease that endless search for the correct download site.

Just picture 2 files in DISTFILES and 20 sites in MASTER_SITES, the sites slow as hell where beta is carried by all sites in MASTER_SITES, and alpha can only be found in the 20th site. It would be such a waste to check all of them if the maintainer knew this beforehand, would it not? Not a good start for that lovely weekend!

Now that you have the idea, just imagine more DISTFILES and more MASTER_SITES. Surely our “distfiles survey meister” would appreciate the relief to network strain that this would bring.

In the next sections, information will follow on the FreeBSD implementation of this idea. We improved a bit on OpenBSD's concept.

Important:

The group names cannot have dashes in them (-), in fact, they cannot have any characters out of the [a-zA-Z0-9_] range. This is because, while make(1) is ok with variable names containing dashes, sh(1) is not.

5.4.9.1. Simplified Information

This section explains how to quickly prepare fine grained fetching of multiple distribution files and patches from different sites and subdirectories. We describe here a case of simplified MASTER_SITES:n usage. This will be sufficient for most scenarios. More detailed information are available in Section 5.4.9.2, “Detailed Information”.

Some applications consist of multiple distribution files that must be downloaded from a number of different sites. For example, Ghostscript consists of the core of the program, and then a large number of driver files that are used depending on the user's printer. Some of these driver files are supplied with the core, but many others must be downloaded from a variety of different sites.

To support this, each entry in DISTFILES may be followed by a colon and a “group name”. Each site listed in MASTER_SITES is then followed by a colon, and the group that indicates which distribution files are downloaded from this site.

For example, consider an application with the source split in two parts, source1.tar.gz and source2.tar.gz, which must be downloaded from two different sites. The port's Makefile would include lines like Example 5.22, “Simplified Use of MASTER_SITES:n with One File Per Site”.

Example 5.22. Simplified Use of MASTER_SITES:n with One File Per Site
MASTER_SITES=	ftp://ftp1.example.com/:source1 \
		http://www.example.com/:source2
DISTFILES=	source1.tar.gz:source1 \
		source2.tar.gz:source2

Multiple distribution files can have the same group. Continuing the previous example, suppose that there was a third distfile, source3.tar.gz, that is downloaded from ftp.example2.com. The Makefile would then be written like Example 5.23, “Simplified Use of MASTER_SITES:n with More Than One File Per Site”.

Example 5.23. Simplified Use of MASTER_SITES:n with More Than One File Per Site
MASTER_SITES=	ftp://ftp.example.com/:source1 \
		http://www.example.com/:source2
DISTFILES=	source1.tar.gz:source1 \
		source2.tar.gz:source2 \
		source3.tar.gz:source2

5.4.9.2. Detailed Information

Okay, so the previous example did not reflect the new port's needs? In this section we will explain in detail how the fine grained fetching mechanism MASTER_SITES:n works and how it can be used.

  1. Elements can be postfixed with :n where n is [^:,]+, that is, n could conceptually be any alphanumeric string but we will limit it to [a-zA-Z_][0-9a-zA-Z_]+ for now.

    Moreover, string matching is case sensitive; that is, n is different from N.

    However, these words cannot be used for postfixing purposes since they yield special meaning: default, all and ALL (they are used internally in item ii). Furthermore, DEFAULT is a special purpose word (check item 3).

  2. Elements postfixed with :n belong to the group n, :m belong to group m and so forth.

  3. Elements without a postfix are groupless, they all belong to the special group DEFAULT. Any elements postfixed with DEFAULT, is just being redundant unless an element belongs to both DEFAULT and other groups at the same time (check item 5).

    These examples are equivalent but the first one is preferred:

    MASTER_SITES=	alpha
    MASTER_SITES=	alpha:DEFAULT

  4. Groups are not exclusive, an element may belong to several different groups at the same time and a group can either have either several different elements or none at all.

  5. When an element belongs to several groups at the same time, use the comma operator (,).

    Instead of repeating it several times, each time with a different postfix, we can list several groups at once in a single postfix. For instance, :m,n,o marks an element that belongs to group m, n and o.

    All these examples are equivalent but the last one is preferred:

    MASTER_SITES=	alpha alpha:SOME_SITE
    MASTER_SITES=	alpha:DEFAULT alpha:SOME_SITE
    MASTER_SITES=	alpha:SOME_SITE,DEFAULT
    MASTER_SITES=	alpha:DEFAULT,SOME_SITE

  6. All sites within a given group are sorted according to MASTER_SORT_AWK. All groups within MASTER_SITES and PATCH_SITES are sorted as well.

  7. Group semantics can be used in any of the variables MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR, PATCH_SITE_SUBDIR, DISTFILES, and PATCHFILES according to this syntax:

    1. All MASTER_SITES, PATCH_SITES, MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements must be terminated with the forward slash / character. If any elements belong to any groups, the group postfix :n must come right after the terminator /. The MASTER_SITES:n mechanism relies on the existence of the terminator / to avoid confusing elements where a :n is a valid part of the element with occurrences where :n denotes group n. For compatibility purposes, since the / terminator was not required before in both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR elements, if the postfix immediate preceding character is not a / then :n will be considered a valid part of the element instead of a group postfix even if an element is postfixed with :n. See both Example 5.24, “Detailed Use of MASTER_SITES:n in MASTER_SITE_SUBDIR” and Example 5.25, “Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories”.

      Example 5.24. Detailed Use of MASTER_SITES:n in MASTER_SITE_SUBDIR
      MASTER_SITE_SUBDIR=	old:n new/:NEW

      • Directories within group DEFAULT -> old:n

      • Directories within group NEW -> new


      Example 5.25. Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories
      MASTER_SITES=	http://site1/%SUBDIR%/ http://site2/:DEFAULT \
		http://site3/:group3 http://site4/:group4 \
		http://site5/:group5 http://site6/:group6 \
		http://site7/:DEFAULT,group6 \
		http://site8/%SUBDIR%/:group6,group7 \
		http://site9/:group8
DISTFILES=	file1 file2:DEFAULT file3:group3 \
		file4:group4,group5,group6 file5:grouping \
		file6:group7
MASTER_SITE_SUBDIR=	directory-trial:1 directory-n/:groupn \
		directory-one/:group6,DEFAULT \
		directory

      The previous example results in this fine grained fetching. Sites are listed in the exact order they will be used.

      • file1 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site1/directory-trial:1/

        • http://site1/directory-one/

        • http://site1/directory/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file2 will be fetched exactly as file1 since they both belong to the same group

        • MASTER_SITE_OVERRIDE

        • http://site1/directory-trial:1/

        • http://site1/directory-one/

        • http://site1/directory/

        • http://site2/

        • http://site7/

        • MASTER_SITE_BACKUP

      • file3 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site3/

        • MASTER_SITE_BACKUP

      • file4 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site4/

        • http://site5/

        • http://site6/

        • http://site7/

        • http://site8/directory-one/

        • MASTER_SITE_BACKUP

      • file5 will be fetched from

        • MASTER_SITE_OVERRIDE

        • MASTER_SITE_BACKUP

      • file6 will be fetched from

        • MASTER_SITE_OVERRIDE

        • http://site8/

        • MASTER_SITE_BACKUP


  8. How do I group one of the special macros from bsd.sites.mk, for example, SourceForge (SF)?

    This has been simplified as much as possible. See Example 5.26, “Detailed Use of MASTER_SITES:n with SourceForge (SF)”.

    Example 5.26. Detailed Use of MASTER_SITES:n with SourceForge (SF)
    MASTER_SITES=	http://site1/ SF/something/1.0:sourceforge,TEST
DISTFILES=	something.tar.gz:sourceforge

    something.tar.gz will be fetched from all sites within SourceForge.


  9. How do I use this with PATCH*?

    All examples were done with MASTER* but they work exactly the same for PATCH* ones as can be seen in Example 5.27, “Simplified Use of MASTER_SITES:n with PATCH_SITES”.

    Example 5.27. Simplified Use of MASTER_SITES:n with PATCH_SITES
    PATCH_SITES=	http://site1/ http://site2/:test
PATCHFILES=	patch1:test

5.4.9.3. What Does Change for Ports? What Does Not?

  1. All current ports remain the same. The MASTER_SITES:n feature code is only activated if there are elements postfixed with :n like elements according to the aforementioned syntax rules, especially as shown in item 7.

  2. The port targets remain the same: checksum, makesum, patch, configure, build, etc. With the obvious exceptions of do-fetch, fetch-list, master-sites and patch-sites.

    • do-fetch: deploys the new grouping postfixed DISTFILES and PATCHFILES with their matching group elements within both MASTER_SITES and PATCH_SITES which use matching group elements within both MASTER_SITE_SUBDIR and PATCH_SITE_SUBDIR. Check Example 5.25, “Detailed Use of MASTER_SITES:n with Comma Operator, Multiple Files, Multiple Sites and Multiple Subdirectories”.

    • fetch-list: works like old fetch-list with the exception that it groups just like do-fetch.

    • master-sites and patch-sites: (incompatible with older versions) only return the elements of group DEFAULT; in fact, they execute targets master-sites-default and patch-sites-default respectively.

      Furthermore, using target either master-sites-all or patch-sites-all is preferred to directly checking either MASTER_SITES or PATCH_SITES. Also, directly checking is not guaranteed to work in any future versions. Check item B for more information on these new port targets.

  3. New port targets

    1. There are master-sites-n and patch-sites-n targets which will list the elements of the respective group n within MASTER_SITES and PATCH_SITES respectively. For instance, both master-sites-DEFAULT and patch-sites-DEFAULT will return the elements of group DEFAULT, master-sites-test and patch-sites-test of group test, and thereon.

    2. There are new targets master-sites-all and patch-sites-all which do the work of the old master-sites and patch-sites ones. They return the elements of all groups as if they all belonged to the same group with the caveat that it lists as many MASTER_SITE_BACKUP and MASTER_SITE_OVERRIDE as there are groups defined within either DISTFILES or PATCHFILES; respectively for master-sites-all and patch-sites-all.

5.4.10. DIST_SUBDIR

Do not let the port clutter /usr/ports/distfiles. If the port requires a lot of files to be fetched, or contains a file that has a name that might conflict with other ports (for example, Makefile), set DIST_SUBDIR to the name of the port (${PORTNAME} or ${PKGNAMEPREFIX}${PORTNAME} are fine). This will change DISTDIR from the default /usr/ports/distfiles to /usr/ports/distfiles/${DIST_SUBDIR}, and in effect puts everything that is required for the port into that subdirectory.

It will also look at the subdirectory with the same name on the backup master site at http://distcache.FreeBSD.org (Setting DISTDIR explicitly in Makefile will not accomplish this, so please use DIST_SUBDIR.)

Note:

This does not affect MASTER_SITES defined in the Makefile.

Prev Up Next
5.3. Categorization Home 5.5. MAINTAINER

All FreeBSD documents are available for download at https://download.freebsd.org/ftp/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.