Description of GPT XML files

Grid Packaging Tool XML DTD Description

Overview

The Grid Packaging Tool (GPT) uses XML to describe all the information that is needed for an installation. These XML files hold all of the information that the GPT will need in order to create a Grid instance on a particular system. The intent of this XML is to construct a framework and a set of Grid packages that can be used to create tailored Grid to meet individual needs. This will allow for the distribution and release of individual Grid components, rather then a single monolithic release. This concept, will allow organizations to construct both source and binary distributions of packages they are interested in rather then being forced to build and configure components that are of no interest.

By using the GPT, it will be possible to release updates to an individual component without necessarily releasing updates to all other components. Additionally, an individual component can be built and tested without configuring and building unrelated components. Thus the packaging framework substantially increases the efficiency of the development and release process.

In order to accomplish this plan, information needs to be available that describes all the pieces that are needed for a release. The current GPT makes use of files to organize and hold this data. The content of these files is a series of XML elements and attributes that describe the various parts of an installation. For an overview of these parts and a discussion of the GPT concepts please refer to the Packaging RFC document that is included with this release. The lay of XML file is defined by a DTD document. The remainder of this discussion will address the elements and attributes of the XML file as defined by the DTD.

Definitions

Elements

Description

BinaryBuild

This value indicates if the items in the binary package are compile or link time dependent.

BinaryDependencies

This element will be composed of either a BinaryBuild element or a BinaryRuntime element.

BinaryRuntime

This element indicates the binary package item and the type of dependency it constitutes. The element will also include one or more dependency elements that show items on which this element is dependent.

BuildDependency

Each package shall store only it's direct dependencies. For example, if the source code in package foo had an include statement which referenced a header file in package fum then package foo has a direct dependency to package fum. On the other hand if the header file in fum includes headers from other packages those dependency belong to package fum not package foo. A packaging system will have to examine all of the dependent packages in order to obtain the entire dependency tree. Care should be taken to avoid circular dependencies.

A circular dependency is defined by the situation where dependencies between two or more packages can only be accommodated if all of the packages are installed simultaneously. For example if packages foo and fum depend on each other then a PM install cannot install foo before fum. Nor can it install fum before foo. This situation shall be resolved by splitting up foo and fum in such a way that the dependency tree becomes a directed acyclic graph.

BuildDependencies

This element identifies an item on which the build is dependent, the type of dependency is also indicated. The element will also contain a list of items that it is dependent on.

BuildEnvironment

Element has CFLAGS line containing defines needed to use the header files. LIBS line containing external libraries need to link with the libraries in this package. And various other build flags.

BuildInstructions

This element will contain instructions on how each of the items in the package should be built. Steps taken in the build process and also what Flavors should be applied.

BuildStep

This element indicates the steps that need to be taken in the build process.

BuildType

Indicates an item and at what point in the build process it is used.

Cflags

List of flags that will be used by the compiler to build the package.

CompatibilityIdentifier

Element holds the basic units needed for compatibility checking. The major and minor release numbers along with the age. We have adopted a variation of libtool's compatibility numbering scheme. In libtool's scheme, each compatibility number consists of three fields, major release number, minor release number, and a "compatibility range" number. The major release number is bumped for any interface change, the minor release number gets bumped for bugfixes, etc in a given interface, and the third number represents the range for which the first number is backwardly compatible. For example, 5.4.3 is backwardly compatible with 2.x.x and up (through 5), since 5-3=2. We will refer to this release numbering scheme as "aging".

CompatibilityRequirement

Defines the release numbering information that will be used to determine an items compatibility with other software components. This information can either be an element of type Simple or Range.

ContactInfo

Information on the package builder: name and email address.

DataPackage

This package contains data files which cannot be modified by users. It will always be generated from a source package and will share the source package's name and compatibility. If the package shall also have a flavor as part of its identity if any data files are configured for flavored. Otherwise it will be noflavored.

Data packages have run-time dependencies if data files include files from other data packages.

Dependency

Description

Element will hold a text description of the package.

DevelopmentPackage

This package contains flavored header files, static libraries, and libtool library files. It will always be generated from a source package and will share the source package's name and compatibility. The package shall always have a flavor as part of its identity.

Development packages can have run-time dependencies if their libraries call executables and scripts in other program packages. They could also have runtime dependencies on data files and documents.

Even though development packages are not installed for run-time they can still have run-time dependencies with other pgm, data, and doc packages if the libraries access files or programs in these packages. The run-time dependencies of a static library will have to be absorbed by a pgm_static package if it contains an executable that was linked with the library.

A development package can have a compile dependency to another dev package if it contains a header file that includes headers from the other package.

A development package can also have linking dependencies if its libraries use symbols from libraries contained in other packages. These dependencies are contained here so that the dependency tree for an executable (from a pgm_static) can be recursively extracted when the executable is built.

DocPackage

This package contains documents. It will always be generated from a source package and will share the source package's name and compatibility. It will always be noflavored.

Document packages have run-time dependencies if document files include files from other doc packages.

FeatureSetIdentity

Element allows for mapping to another file. Indicates the feature set and the sets compatibility index.

Flavor

Flavors used to create items of package.

FlavorChoice

A list of options used during the building of the package contents.

FlavorConfig

List of what Flavors must be used to build package and a list of Flavors that must not be used in the build.

FlavorOption

Holds the information on the Flavors that will be used to build item.

GPTPackageMetadata

The Grid software has some requirements that dictate what the strategy for splitting up the software will be. This element encapsulates these requirements and provides the high level definition of the data structure. The data structure can be decomposed using the following ideas:

1. The Grid software has a complex network of dependencies between its various components. Many of these dependencies are circular unless The Grid software is split into atomic sized packages (ie. 1 package = 1 library). This network of dependencies also dictate that the software has to be treated as a distribution of packages rather than a loose collection of packages.

2. The development portion of the software is built with a number of compiler flavors. A compiler flavor is defined as the set of build environment variables (compiler choice, linker choice, compiler/linker flags etc.) which every binary that will be linked together has to use. Flavors cannot be mixed which means that for any given source dependency tree there is an equivalent binary dependency tree for every flavor.

3. A significant portion of the Grid software is not compiled at all. Instead it consist of scripts and data files which are not flavor specific.

4. The installed files that are the product of The Grid software source code can be used in a number of different ways as discussed in the Overview. None of these uses require the exact same set of installed files. For example, consider a source code package that installs the program foo, the static and shared versions of the library libfoo, and the header foo.h. Program foo is needed at runtime. The shared library libfoo.so is need at runtime if other programs which link to it are installed. Both static and shared libraries libfoo.a and libfoo.so as well as the header file are needed when the package foo is used in a development tree.

5. Several Grid software components use files that have to be modified after installation. These files have to be treated special so that the Grid software user does not lose the modifications when packages are re-installed or updated. (These files are by definition runtime configuration files, as mentioned above.)

6. Because of the complex dependencies anticipated between Grid software packages a flexible compatibility checking scheme is needed. This scheme needs to allow both the package maintainer and the package users to individually express compatibility among the package releases. For example, the maintainer for package foo is releasing an update. This maintainer needs to be able to express that the update can be safely used in place of the previous release. Conversely, the maintainer needs to be able to express that the update is incompatible with previous release of foo. In the same way, the users of package foo need to be able to express which release of foo their package depends on.

The idea behind package types is that smaller is better because it provides flexibility. For example, dependency checking between packages becomes simple if the contents of a particular package serve one consistent purpose. If dependency checking is simple, then it can be automated through package manager convenience tools. From this, users can manage a large number of packages in groups rather than individually.

The source code contained in a source code package shall always be released together. There is no provision in our current packaging framework for different binary package types generated from the same source package to have different release numbers. In other words all of the source code in a source code package uses the same name and release number. Thus, putting the source to a library, and to programs using that library into the same source package implies that you will never want to release a new release of the library without releasing a new release of the programs, and vice versa.

One source code package shall generate several binary packages. For example, if the package generates binaries than each build flavor these binaries were built with would need to be in a seperate binary package. These packages will use the name and compatibility of the source package and add their own unique extension. This allows dependency checking to be tailored to a particular user requirement (ie. run-time vs development tree) which will simplify the checking.

All of the installed files of a given source code package build shall be contained in several different binary package types depending on how the files are used. No file shall belong to more than one binary package. Consider the example discussed in requirement 4 in the previous section.

Here is how the installed files would be divided into seperate binary packages:

1. Dynamically linked program foo. This is used if the runtime environment supports shared libraries.

2. Statically linked program foo. This is used if the runtime environment does not support shared libraries.

3. Shared library libfoo.so. This is used when other dynamically linked programs are linked with this library.

4. Static library libfoo.a and foo.h. Used in a development tree.

Any binary package that contains compiled code or files configured by flavor shall be tagged with a flavor name. Any binary package that does not contain these types of files will be tagged as a "noflavor" package.

InvalidFlavors

A list of Flavors that can not be used when building this package.

PackageDocs

Lists information about any documentation that is available on the package.  Also provides link information on where to find documents.

PackageInfo

Information about the Package.  Includes Contact information of Package author and link information to documentation on package.

PackageLibs

Contains the libraries provided by this package.

PackageReleaseInfo

Release information on Package.  Includes the CompatibilityIdentifier for the package and the PacakgeStability.

PackageStability

Indicates the distribution level of the items found within the package.

PGMPackage

This package contains dynamically linked executables and scripts. It will always be generated from a source package and will share the source package's name and compatibility. If the package contains executables it shall also have a flavor as part of its identity. If the package contains only scripts then it can be designated as "noflavor".

Program packages can have run-time dependencies if their executables and scripts call executables and scripts in other program packages. They could also have runtime dependencies on data files and documents.

A program package can also have runtime linking dependencies if its executables are linked with libraries from rtl packages. For example, say that an executable links with libfoo in package fum. If the executable is linked to the shared library libfoo.so then the linking dependency translates to the fum_rtl package which will have to be installed before the program package.

PGMStaticPackage

This package contains statically linked executables. It will always be generated from a source package and will share the source package's name and compatibility. The package shall also have a flavor as part of its identity.

Static program packages can have run-time dependencies if their executables call executables and scripts in other program packages. They could also have runtime dependencies on data files and documents. In addition these packages absorb the runtime dependencies of the static libraries they are linked with. For example, consider a program foo that statically links with a library libfee.a that has a system call to still another program fum. The library libfee has a runtime dependency to the program fum. The program foo will have to absorb this dependency so that program fum is installed before program foo is installed.

A program package can also have regeneration dependencies if its executables are linked with libraries from other packages. For example, say that an executable links with libfoo in package fum. If the library was linked statically to libfoo.a then the dependency is translated to the fum_dev package. In this case the program package will have to be regenerated any time fum_dev is updated. A build number will be updated to reflect the regeneration.

None of the executables in a program package shall ever be built with a mixture of static and shared package libraries because this complicates the compatibility checks needed at runtime to make sure that all of the libraries are compatible.

PostInstallMessage

Message to be used at completion of installation process.

PostInstallProgram

Any program that should be run at completion of installation.

Range

Element allows for mapping to another file.

To provide flexibility in specifying releases as a part of a package dependency the following shall be done. A packager will be able to specify the release of a dependency as either one release number or a range. If one release number is specified, then the framework will use the compatibility range mentioned previously to determine whether a dependency is met or not. If a range is specified then the packaging framework will look for releases only within that range. In the source package a packager will be able to specify a list of ranges and simple releases. This list shall be in order of preference. We will refer to this list as a "release specification".

As an example consider the installed package foo which has a release number of 5.3. As was mentioned in the previous section, this specifies a compatibility range of 2 to 5. Now we want to install package fum which depends on foo. The following table shows how the release numbering works:

Release specification for Dependency foo to fum	Dependency is met?
1	No
1 to 4	No
4 to 4	No
3	Yes
3 to 6	Yes

Binary packages will only have the first release specification that was met when the source package was built. So for specifications listed in the example, binary package foo would have the release specification of 2.3.

RTLPackage

This package contains libraries used at run-time by programs and scripts. It will always be generated from a source package and will share the source package's name and compatibility. If the package contains binaries it shall also have a flavor as part of its identity. Otherwise it is a noflavor.

Runtime packages can have run-time dependencies if their libraries call executables and scripts in other program packages. They could also have runtime dependencies on data files and documents.

Runtime packages have linking dependencies which are needed at runtime. For example when a program using shared library foo starts execution, it needs to load libfoo.so as well as all of the shared libraries that libfoo depends on for symbols.

RuntimeType

Indicates the type of runtime item and any dependencies the item might have turning its build.

SetupPackage

This package will be used to create items needed for the setup of other packages. It will list all sources items needed and how to build these items. Also, post install instructions will be given. These instructions will begin installing other items.

These are Grid or host specific. They deal with the localization of the software. Currently this can be localized to either an organization (Grid) or a host (specific machine).

SetupType

Indicate the type if item that is being setup. Also, all items that this element is dependent on.

Simple

Element allows for mapping to another file.

Release specification for Dependency foo to fum	Dependency is met?
1	No
1 to 4	No
4 to 4	No
3	Yes
3 to 6	Yes

SourceDependencies

A listing of dependency items for package. Each SourceDependencies element can be of a single type. If several different types of dependencies are needed several instances of this element will have to be created.

SourcePackage

This package consists of source code, scripts, and documents which are configured and built to produce binary packages. One source package will produce one or more binary packages each of which is a different package type. Source packages have two sources of dependencies to consider. The first source are the compile and link dependencies that are present when the source code is being built. The second source is the run-time dependencies that need to be stored in the binary packages when they are generated.

Source packages are different from all of the other package types, in that they are not managed by the package manager. Source packages are not installed into the installation tree, so they do not need to include data for the purpose of their own uninstallation. Rather, the data included in a source package is necessary for ensuring that the compile and link dependencies are satisfied when building the binary packages, and for generating the data necessary for each binary package being produced. The data can also be used by a convenience tool which builds/installs/ generates binary packages from multiple source package ordered by their dependencies.

SystemIncludes

A list of header files that need to come from outside of the package but are needed for the build. Including headers in this list may cause a recompile of the source file associated with the header.

SystemLibs

Contains external libraries need to link with the libraries in this package.

VersionLabel

Element allows for mapping to another file. A single label by which the package release will be distinguished.

WithFlavors

Indicates if flavors should be used when building package.

Structure

Schema gpt_dtd.dtd

<?xml version="1.0" encoding="UTF-8"?>

<!ATTLIST GPTPackageMetadata

Name CDATA #REQUIRED

FormatVersion CDATA #REQUIRED

<!ELEMENT PackageInfo (Description, ContactInfo*, PackageDocs*)>

<!ELEMENT ContactInfo EMPTY>

<!ATTLIST ContactInfo

ContactName CDATA #REQUIRED

ContactEmail CDATA #REQUIRED

<!ELEMENT PackageDocs EMPTY>

<!ATTLIST PackageDocs

PackageDocsDesc CDATA #REQUIRED

PackageDocsURL CDATA #REQUIRED

<!ELEMENT Description (#PCDATA)>

<!ELEMENT PackageReleaseInfo (PackageStability, CompatibilityIdentifier)>

<!ELEMENT PackageStability EMPTY>

<!ATTLIST PackageStability

Release (experimental | alpha | beta | production) #REQUIRED

<!ELEMENT SetupPackage (SourceDependencies*, BuildInstructions?, BuildEnvironment?, PostInstallMessage, PostInstallProgram)>

<!ELEMENT SourcePackage (WithFlavors, SourceDependencies*, BuildInstructions?, BuildEnvironment?)>

<!ELEMENT PGMPackage (Flavor, BinaryDependencies*, PostInstallMessage, PostInstallProgram)>

<!ELEMENT PGMStaticPackage (Flavor, BinaryDependencies*, PostInstallMessage, PostInstallProgram)>

<!ELEMENT DevelopmentPackage (Flavor, BinaryDependencies*, BuildEnvironment?)>

<!ELEMENT RTLPackage (Flavor, BinaryDependencies*)>

<!ELEMENT DataPackage (BinaryDependencies*)>

<!ELEMENT DocPackage (BinaryDependencies*)>

<!ELEMENT TestPackage (WithFlavors, Flavor, SourceDependencies*, BinaryDependencies*, BuildInstructions?, BuildEnvironment?, PostInstallMessage, PostInstallProgram)>

<!ELEMENT Flavor (#PCDATA)>

<!ELEMENT WithFlavors EMPTY>

<!ATTLIST WithFlavors

Build (yes | no) #REQUIRED

<!ELEMENT Dependency (CompatibilityRequirement)>

<!ATTLIST Dependency

Name CDATA #REQUIRED

Package (data | dev | doc | hdr | pgm | rtl) #REQUIRED

AllowSubstitution (No | Yes) #REQUIRED

<!ELEMENT BuildDependency (CompatibilityRequirement)>

<!ATTLIST BuildDependency

Name CDATA #REQUIRED

AllowSubstitution (No | Yes) #REQUIRED

<!ELEMENT SourceDependencies (RuntimeType | SetupType | BuildType)>

<!ELEMENT RuntimeType (Dependency+, FlavorChoice?)>

<!ATTLIST RuntimeType

Type (data_runtime | doc_runtime | pgm_runtime | lib_runtime) #REQUIRED

<!ELEMENT SetupType (Dependency+)>

<!ATTLIST SetupType

Type (data_setup | doc_setup | pgm_setup | lib_setup) #REQUIRED

<!ELEMENT BuildType (BuildDependency+)>

<!ATTLIST BuildType

Type (compile | pgm_link | lib_link) #REQUIRED

<!ELEMENT BinaryDependencies (BinaryBuild | BinaryRuntime)>

<!ELEMENT BinaryBuild (BuildDependency+)>

<!ATTLIST BinaryBuild

Type (Compile | Build_Link) #REQUIRED

<!ELEMENT BinaryRuntime (Dependency+, FlavorChoice?)>

<!ATTLIST BinaryRuntime

Type (Regeneration | Runtime_Link | Runtime | Setup) #REQUIRED

<!ELEMENT BuildEnvironment (Cflags?, PackageLibs?, SystemLibs?, SystemIncludes?)>

<!ELEMENT Cflags (#PCDATA)>

<!ELEMENT SystemIncludes (#PCDATA)>

<!ELEMENT PackageLibs (#PCDATA)>

<!ELEMENT SystemLibs (#PCDATA)>

<!ELEMENT BuildInstructions (BuildStep*, FlavorChoice*)>

<!ELEMENT BuildStep (#PCDATA)>

<!ELEMENT PostInstallMessage (#PCDATA)>

<!ELEMENT PostInstallProgram (#PCDATA)>

<!ELEMENT FlavorChoice (FlavorConfig+)>

<!ELEMENT FlavorConfig (FlavorOption+, InvalidFlavors*)>

<!ELEMENT FlavorOption EMPTY>

<!ATTLIST FlavorOption

FlavorSwitch NMTOKEN #REQUIRED

FlavorLabel CDATA #REQUIRED

<!ELEMENT InvalidFlavors EMPTY>

<!ATTLIST InvalidFlavors

FlavorLabel CDATA #REQUIRED

<!ELEMENT CompatibilityRequirement (Simple*, Range*)>

<!ELEMENT Simple EMPTY>

<!ATTLIST Simple

Major CDATA #REQUIRED

<!ELEMENT Range EMPTY>

<!ATTLIST Range

LowerMajor CDATA #REQUIRED

LowerMinor CDATA #REQUIRED

UpperMajor CDATA #REQUIRED

UpperMinor CDATA #REQUIRED

<!ELEMENT VersionLabel (#PCDATA)>

<!ELEMENT FeatureSetIdentity (CompatibilityIdentifier)>

<!ATTLIST FeatureSetIdentity

FeatureSubName CDATA #REQUIRED

<!ELEMENT CompatibilityIdentifier EMPTY>

<!ATTLIST CompatibilityIdentifier

Major CDATA #REQUIRED

Minor CDATA #REQUIRED

Age CDATA #REQUIRED