CredentialManager Troubleshooting

When troubleshooting a MyProxy problem, it is important to consult the CredentialManager-server logs. If you don't have access to the CredentialManager-server logs, please contact your CredentialManager-server administrator for help. When you run the client program, if the error message does not give you enough information, please use -debug option in your command to print out more detailed error message.

The exceptions thrown by CredentialService could be caused by unauthorized operation, unqualified password, expired credential, unexisted service instance ... In most cases, the detailed error messages using -debug option can give you enough information. Here, we focus on documenting several problems that are caused either by the underlying Globus or the configuration of CredentialManager, and the error messages usually cannot give you enough information.

  1. Expired service instance cannnot be destroyed
  2. Exception thrown in predestroy()
  3. Cluttered IndexService
  4. IndexService crash

  1. Expired service instance cannnot be destroyed

    This problem is caused by `removeSubscription' method when the container sweeps out the expired service instance. The container calls predestroy() to clean up the expired service instance. `removeSubscription' method will be called inside predestroy() to remove subscription to the IndexService and predestroy() returns right after `removeSubscription' returns. With the return of predestroy(), that expired service instance does not exist anymore.
    The problem is that `removeSubscription' returns before the actual removal action is done. Furthermore, the actual removal action performed later will fail because the predestroy() has already returned and the service instance disappears! As a result, the outdated service data in IndexService never get removed and some problems will occur. For instance, a user stores a credential named "test". Sometime later, he stores another credential using the same name. It should be no problem if the first one already expires. However, since the service data of the expired service instance does not get removed, two service instances with the same credential name will be found and an exception will be thrown. To solve the problem for now, we add 100 ms delay after `removeSubscription' to allow the IndexService to finish removing the subscription before predestroy() returns. It is not guaranteed that 100 ms delay is enough under all circumstances.
    Note: if we use the other methods to invoke the preDestroy, such as the client issues destroy command, there is no such problem happen.

    indexService.removeSubscription(subscriptionID);
    // allow the indexService to remove the subscription successfully.
    Thread.sleep (100);
    

  2. Exception thrown in predestroy()

    This problem is a bug in Globus and please refer here for more details. In CredentialManager, the most common case when an exception is thrown in predestroy() is unauthorized destroy. For instance, if a user is not the owner of a credential and tries to destroy it with no password specified, the exception "Unauthorized destroy" will be thrown in predestroy(). Once that happens, the user has to restart globus-start-container to make the following commands work correctly.

  3. Cluttered IndexService

    When a service instance is destroyed by the user, the subscription to the IndexService is removed. However, the IndexService still contains that service instance's aggregated service data. To solve this problem, we use a flag as part of the service data to indicate whether or not that service has been destroyed. However, it might be possible that the IndexService get increasingly cluttered up with obsolete service data. One way of removing the obsolete service data is to restart the IndexService. We will try to solve this problem in the next release.

  4. IndexService crash

    The CredentialManager service instances polls the IndexService and subscribes to it once per hour by default. The problem is that if the IndexService crashes and restarts, the IndexService will not contain any service data until the next time the service instances ask him to subscribe their service data. However, if the subscription interval is set too low, it may be too expensive to subscribe again and again.