• About
• News
• License
• Download
• Installation
• Documentation
• Reference
• Support
• Security
• Privacy
• Troubleshooting
• Bugs: 1 2
• Sources
• Publications
• Presentations
• MyProxyLogon
• OAuth
• NCSA Server
• TeraGrid
• Protocol
• Anniversary

MyProxy > Docs > Admin Guide > High Availability

MyProxy servers can be configured to provide high availability in the presence of network failures, power outages, or other faults that make a subset of the MyProxy servers unreachable by MyProxy clients. To achieve high availability, redundant MyProxy servers should be deployed to minimize shared points of failure. Geographically distributed MyProxy servers can be deployed to provide service in the case of wide area network outages and partitions. Redundant MyProxy servers can also be deployed in a single machine room (ideally with separate power supplies and network links) to provide recovery from server hardware failures.

A MyProxy server can be configured to act as a certificate authority, a credential repository, or both. The following sections describe how to configure redundant MyProxy servers for these configuration options. It is assumed that you are already familiar with the MyProxy server installation process.

Contents

• Client Configuration
• Replicating a MyProxy CA
• Replicating a MyProxy Repository

Client Configuration

MyProxy clients locate the myproxy-server based on the hostname(s) given as input (on the command line, from the MYPROXY_SERVER environment variable, etc.). MyProxy clients will attempt to connect to each myproxy-server hostname (in order), trying each IP address (in order) returned by DNS for that hostname, until a connection is successfully established or all possible addresses have been attempted. The myproxy-server hostnames can be specified in order according to network distance (i.e., try the "nearest" server first) or according to a primary/backup scheme, as determined by the myproxy-server administrator.

For example:

In the above example, we pass the -v flag for verbose output showing that the myproxy-logon command attempts to connect to the IP addresses associated with the three configured MYPROXY_SERVER hostnames, and the third attempt is successful.

Here is a similar example using the Java CoG (with debugging output enabled):

Replicating a MyProxy CA

For the most part, replicating a MyProxy CA simply requires installing MyProxy servers on multiple machines and configuring them the same way, i.e., with the same certificate_mapfile and PAM setup. However, there are two options for managing the private keys.

One option is to create multiple CAs, each with their own private keys. The CAs can have names like "CN=My Organization CA #1" and "CN=My Organization CA #2" or "CN=My Organization CA East" and "CN=My Organization CA West" or whatever scheme makes the most sense in your situation. In this case you must distribute the public keys (usually as part of self-signed certificates) for all of the CAs to your relying parties, which adds some complexity over having a single CA, but not too much (i.e., you can distribute them in an archive format such as a zip or tar file). The benefit of this approach is the possibility of removing one of the CAs from the trust list if it is compromised, without affecting the other CAs. To this end, it can be useful to make the CAs subordinate to a root CA that can revoke them if needed.

The other option is to use the same private key for all the MyProxy CA replicas. In this case, when setting up the replicas, copy the CA private key and and certificate to the replicas very carefully, using an encrypted channel, such as scp or sftp. Then, edit the certificate_serialfile so that each CA instance issues certificates with different serial numbers. (Note that this file contains the serial number in hexadecimal.) For example, one CA can be initialized with "0", another with "100000", another with "200000", and so forth, so they can each issue over a million certificates without conflicting serial numbers. Or, set certificate_serial_skip in myproxy-server.config (supported by myproxy-server v4.7 and later) to stagger the serial numbers across the replicas. The benefit of this approach is that you can have many replicas with needing only to distribute a single CA public key (certificate). However, many copies of the CA private key on the replicas increases the possibility of compromise, so it should be used with caution.

Replicating a MyProxy Repository

Replicating a MyProxy repository requires maintaining consistent repository state across the replicas, so a credential stored on one replica can be retrieved from another replica. We can envision multiple approaches to repository replication:

• Shared back-end filesystem: To fully replicate MyProxy repository functionality across multiple myproxy-server instances, you can host the MyProxy repository on a shared network filesystem mounted across the instances. Because the myproxy-server keeps all state in the filesystem, any repository updates in one instance will be immediately available (via the network filesystem) to the other instances. This approach is most suitable for multiple myproxy-server instances deployed in the same local area, and it depends on the availability and performance of the network file system.
• Primary-Backup replication: Primary-backup replication implements a looser coupling between myproxy-server replicas, making it an attractive option for wide-area replication. The myproxy-replicate tool supports repository replication to secondary servers via the MyProxy protocol. Primary-backup replication could alternatively be implemented using rsync over ssh or another external mechanism. This method is described in more detail below.

Primary-Backup Replication

The primary-backup scheme uses passive replication. Repository updates (credential store, delete, and change password) are performed on the primary server only, while read-only operations (authenticate, retrieve credential) can also be performed on secondary servers, which hold a backup copy of the repository. Manual reconfiguration is required to set a new primary server.

First, all MyProxy servers should be installed and configured according to the standard server installation process. For consistency, the servers should be configured identically, except as described below.

Configuring the Primary Server

Configuring myproxy-replicate on the primary myproxy-server is simply a matter of modifying myproxy-server.config:

#
# Secondary server list
#
# When the myproxy-replicate program is run, this list of servers is
# used to indicate where the repository information is to be sent.
# The list is comprised of hostnames and optional port numbers. The
# hostname may be in the form:
# name
# FQDN
# tcp/ip address
#
# The host name and port number must be separated by a ':'.
#
# If multiple secondary servers are given, each secondary must be separated with
# a ';'.
#
# Example:
# grids1
# grids1.ncsa.uiuc.edu
# grids1:9000
# grids1;grids2.ncsa.uiuc.edu:9000;141.142.96.41
#
slave_servers <server1>:<port1>;<server2>;...

Configuring Secondary Servers

Setting up a machine to run as a secondary MyProxy server is simple and straight forward. As with the primary, MyProxy software should be installed and configured as described in the installation instructions. Once this is done, changes need to be made to the myproxy-server.config.

The secondary server should only accept credentials from the primary server, so a user will never be able to store directly to a secondary and cause an inconsistency in the repository. User interaction with secondary servers is limited to myproxy-logon and myproxy-retrieve. All other commands must be performed via the primary. In normal operation, all commands should be sent to the primary, and users should not need to know about servers running as secondary machines. Secondary servers should only be accessed by clients when the primary is unreachable. In order to limit credential storage on the secondary to only the primary server, the value of accepted_credentials must be set to the DN of the primary. All other myproxy-server.config values should be set as they are on the primary. It is simplest and safest to copy the myproxy-server.config file from the primary to the secondary and change the value of accepted_credentials. The following shows a simple secondary configuration. Only the primary MyProxy server on myproxy.ncsa.uiuc.edu is allowed to modify the credentials in the repository of the secondary.

accepted_credentials "/C=US/O=NCSA/CN=myproxy.ncsa.uiuc.edu"
authorized_retrievers "*"
default_retrievers "*"
authorized_renewers "*"
default_renewers "none"

The secondary server is now ready to be run. The server should be run as a system service and instructions for doing this can be found in the server installation guide. Once the secondary servers are started, it is safe to start the myproxy-replicate utility.

Start Replication

The MyProxy server which has been configured as the primary is also responsible for replication of its repository. Replication is accomplished via the myproxy-replicate utility. This utility will read the repository on the local machine as specified by its arguments. Each stored credential found in the repository will be sent to any configured secondary servers according to the slave_servers tag found in myproxy-server.config. It is recommended that this utility be called via cron or some other automatic mechanism. For example, the following script can be used to call myproxy-replicate:

#!/bin/sh

# The MyProxy replicate cron script calls the replicate utility and logs output.

# Set GLOBUS_LOCATION as appropriate for your installation.
GLOBUS_LOCATION="/usr/grid"
export GLOBUS_LOCATION

. ${GLOBUS_LOCATION}/libexec/globus-script-initializer

# The -r and -c options need to be changed to correct location for the server.
# [-storage|-r]=<path to repository> Directory of the MyProxy repository.
# [-config|-c]=<path to config file> Directory of the MyProxy Server
${GLOBUS_LOCATION}/sbin/myproxy-replicate -r /var/myproxy \
  -c $GLOBUS_LOCATION/etc/myproxy-server.config 2>&1 \
  | logger -t myproxy-replicate.cron -p cron.info

exit 0

This would then be scheduled with cron by updating the /etc/crontab file with something like the following.

SHELL=/bin/sh
0,10,20,30,40,50 * * * * /usr/local/globus/sbin/myproxy-replicate.cron >/dev/null 2>&1

This line causes the replicate script to be run every 10 minutes. The time interval is at the discretion of the MyProxy administrator but the granularity should be small enough to avoid large deltas between repositories.

Before running the replicate utility verify the secondary servers are running and configured to accept credentials from the primary.

Last modified 12/08/09.
©2000-2019 Board of Trustees of the University of Illinois.