Lessons from the Field: A directory transition from DSEE to OUD 11gR2PS1

By Sylvain Duloutre on Feb 18, 2014

I was recently involved in a LDAP directory services transition project, from DSEE to OUD 11gR2PS1, for a large manufacturing enterprise. Directory service is medium-sized with a few of million LDAP entries, and is accessed by a wide range of services and applications, ranging from Corporate Directory to Identity Store for Identity Management and user management for intranet and extranet portals.

Here is an overview of the steps we followed and the issues we addressed during this project to successfully transition the infrastructure to OUD.

1. Transition Assessment

Putting in place a sound methodology and design is a key success factor for a directory migration, regardless the final migration options selected.
This assessment was conducted over a period of 3 weeks and included the following:

– Identification and formalization of the main drivers and requirements for transition
– Inventory of the current directory infrastructure, including identification of the application portfolio accessing the directory
– Identification of transition options compatible with requirements
– Estimation of transition effort
– Identification of training requirement for the staff and skills required during the transition, especially people with business knowledge of the data stored in the directory.

For this project, the 4 main drivers to migrate to OUD were:

– native support of EUS (ability to store Oracle DB accounts used by Enterprise User Security) as deployment of EUS was a Corporate decision
– smooth integration and official support of OUD with the Oracle Identity stack, especially OIM and OAM
– superior scalability, able to deliver SLA required by new services to be rolled out in the coming years
– support for global account lockout

About 10,000 applications word-wide access the directory. Among them, 10 were identified as critical for the transition, mostly provisioning applications.


Figure 1: Existing DSEE topology with 4 masters, 6 read-only replicas in each DC + 2 read-only replicas in remote branches

OUD provides several options to transition from DSEE. For this project, the DSEE and OUD topologies have to cohabit for 6 months in production as transition was planned incrementally on a geo basis. Furthermore, client applications heavily relies on password policy, so strong data consistency is required across the 2 topologies during transition. Based on that, tightly-coupled cohabitation via the replication gateway was selected. In addition to that, this strategy provides smooth and incremental transition without interruption of service.

Transition analysis and design was conducted over a period of 1 week. The goal of this critical phase was to adapt schema, configuration and data to OUD, define and automate procedures to deploy an OUD directory server able to deliver a service equivalent to existing DSEE servers in staging area. Business knowledge of directory data was really important during this phase. Transition of the replication topology was also addressed during this phase.

4 OUD servers have been deployed as a directory backbone. Actual roll-out of other OUD servers (20+ servers all over the world) will be performed incrementally over the next months.

2. Transition Analysis and Design

Transition analysis and design heavily relied on the transitioning tools provided with OUD. Transitioning to OUD implies adapting ODSEE configuration (and sometimes the data) to the OUD format.
The OUD delivery provides tools to automatically adapt ODSEE configuration and data. The few configuration elements that can’t be adapted automatically are identified by the OUD diagnostic tools and require manual adaptation.

The work was broken down into the following steps:

  1. – Diagnose the DSEE deployment
  2. – Migrate the schema, configuration and data to a reference OUD instance
  3. – Validate configuration and settings
  4. – Deploy additional OUD instances in a replicated topology
  5. – Upgrade 2 DSEE masters in place to ODSEE 11gR1 PS2
  6. – Deploy Replication Gateway to make the link between the 2 topologies
  7. – Use T2P procedure (Test to Production) to roll out server configuration  in production

2.1 Diagnosing the DSEE deployment

OUD ships with the ds2oud tool able to diagnose DSEE configuration, schema and existing user data. You can use this tool to identify areas that are likely to require special attention during the transition.
Many of the differences spotted by this tool can be automatically migrated, especially those related to schema and server configuration. Others issues, related to advanced server configuration and user data requires manual intervention as they often require business knowledge or architectural decisions.

In order to run this tool, you must have administrative access to one running DSEE instance. For sake of security, one additional DSEE master server was deployed specifically for that purpose so that production systems are not impacted at all by this diagnostic phase.

2.2 Migrating the schema, configuration and data to a reference OUD instance
LDAP Schema
LDAP schemas were compared first. Custom schema extensions were properly imported to the OUD side. However, we faced a couple of issues with some experimental schemas, e.g. RFC 2307: OUD ships with the latest versions of the RFCs, but in this case, DSEE was using an older and incompatible version of the schema. A huge amount of existing LDAP entries were relying on the older schema, so we decided to use this old schema on OUD too.

Server Configuration
We also used ds2oud to migrate the server configuration. The -F option is used to produce a batch file containing a list of configuration changes to be applied to the OUD directory server. This batch file was reused to setup subsequent OUD servers during server roll-out.

Global parameters, database suffixes, indexes, global password policy were migrated automatically.

Note: Support of global password policy and account lockout during cohabitation of DSEE and OUD via the replication gateway requires that DSEE use the ‘DS6’ password policy mode. For this deployment, the DSEE topology was already using this mode, so no additional action was required on that side.

User Data
For this project, most of the transition effort was related to user data migration: LDAP was deployed more than 15 years ago. LDAP entries were provisioned over time by a huge variety of applications. The number of provisioning applications is now quite limited. However, it appeared that about 5% of the LDAP entries did not strictly conform to the LDAP schema and/or to the LDAP standard.  By default, OUD strictly enforce the LDAP standard and the LDAP schema, including attribute value syntax check. DSEE does not check attribute value syntax.

Based on that, it was decided to use that transition project to conduct a detailed data assessment and sanitization. This assessment was completed over the 1 week period as originally planned. However, it required involvement of additional stakeholders to figure out whether entries that did not match the LDAP schema were correct or incorrect from a business and application perspective.

Here are a few road blocks hit and addressed:

– Most LDAP entries contained more than one structural LDAP object class
This non-standard setting is accepted by ODSEE but by default, it is rejected by OUD. That check was relaxed on OUD because applications creating these entries could not be modified.
– Some entries contained attribute type extensions (e.g criteria:criteria;x-custom-extension) that violate LDAP standards because they contain underscores.
That check was relaxed on OUD as well
– Some DNs and telephoneNumber attributes contain incorrect values
These values were cleaned up on DSEE.

Directory Metadata
The transition strategy chosen is based on the replication protocol. It provides strong data consistency and all data including directory metadata are replicated back and forth between DSEE and OUD.
For that project, directory metadata, mainly access controls, could be replication w/o any problem, However, we had to adapt directory metadata related to account-based resource limits settings:

Some DSEE entries contain the following resource limit attributes, namely nsSizeLimit,nsTimeLimit, nsLookThroughLimit, nsIdleTimeout. Corresponding attributes on OUD are ds-rlim-size-limit, ds-rlim-time-limit, ds-rlim-lookthrough-limit,ds-rlim-idle-time-limit. In order to replicate the functionality correctly, the OUD schema was modified so that each DSEE attribute name related to resource limits is declared as an alias name for each corresponding OUD attribute.

2.3 Validating OUD Configuration and Settings

A few OUD servers were deployed in staging area and configured as defined above. Then traffic from key customer applications identified during the transition assessment was redirected to the OUD infrastructure.

This phase is very important to validate the changes above and identify behavioral differences between OUD and DSEE that are not always detected automatically. We detected a few problems that required additional configuration changes on the OUD side. These changes were added to the list of configuration to be applied to OUD servers.

Here are a few road blocks hit and how we addressed them:

– Few applications were performing unindexed operations. By default, OUD reject such searches.
For technical reasons, it was not acceptable to create the appropriate indexes to suppress unindexed searches: For this project, we recommend to disable privileges leading to aci behavioral differences between OUD and ODSEE.
– Attributes present in the rootDSE and in the schema are flagged as operational, so they are not returned to client applications unless they are explicitly specified in the search attribute list. On ODSEE, these attributes are systematically returned.
Client applications relied on the DSEE behavior; we had to modify the OUD configuration so that rootDSE entries are returned like user attributes
– By default, in OUD, unauthenticated users are not granted access to cn=schema nor the rootDSE.
Appropriate access controls was added to OUD

2.4 Deploy additional instances in a replicated topology

In such medium/large replication topology, it is advisable to separate the directory server and replication server instances into separate JVMs, and to limit the number of replication servers:

– 4 instances having both Directory server and Replication Server roles are deployed, 2 in each data center.
– The number of directory server instances serving search operations could be reduced to 4 due to superior OUD performances.
– 2 replication groups are defined so that DSs in one data center preferably connect to a RS within the same data center.
– 2 directory servers deployed in remote branches are configured as read-only replicas to conform to corporate rules. These 2 servers can connect to replication server from either data center to receive updates.

Note: Unlike DSEE topology, every directory server running in the main data centers are read-write master. The corresponding servers in DSEE handled a limited write traffic that was redirected to DSEE masters via referrals. The new OUD topology eliminates the need for referrals.


Figure 2: New OUD topology with 4 RS+DS, 4 DS in each DC + 2 read-only DS in remote branches

3 Deploying the OUD topology

The main outcome of the transition analysis and design phase is a collection of commands to be applied to set up an OUD directory server instance.
Additional OUD directory server instances were setup then configured. The Test to Production feature provided by OUD is used to clone configurations to pre-production environment.

Data are exported from DSEE (with the –opends flag) to preserve replication metadata, so that replication can be established between the 2 environments. Data are imported in a single OUD directory server, then replication was enabled between servers and database files are copied to the other servers. In the customer environment, this initialization strategy was preferred over an over-the-network full initialization.

The minimum version required for tightly coupled coexistence is ODSEE 11g Release 1 (11.1.1) for the ODSEE master that communicates directly with the replication gateway. However, the rest of the ODSEE topology does not need to be uniformly based on this version and remain in 6.x, so we upgraded 2 DSEE masters to the latest ODSEE 11gR1 PS2 ( Instances were automatically upgraded in place without having to copy, export or import anything. An alternate solution would have been to deploy 2 new ODSEE 11g instances as replication gateway companion.

At that stage, 2 replication gateways are deployed as described in the OUD administration guide. This is the recommended setting to avoid single point of failure.

Backup strategy was adapted to reflect the new hybrid topology: In a replicated environment involving ODSEE and OUD, you must perform regular backups on the ODSEE side and on the OUD side. A backup must always be restored in the topology it is associated with.


Figure 3: Cohabitation DSEE/OUD

The current plan is to keep the DSEE-OUD cohabitation for 6 months as applications are progressively redirected to OUD.

4 Conclusion

Tightly coupled coexistence of OUD with ODSEE is achieved by deploying OUD and ODSEE in a replicated topology using the “Replication Gateway”. The replication gateway provides out-of-the-box live transition without service interruption.
This enables you to run OUD and DSEE in parallel in a mixed environment so that you can transition to OUD over time, validate your upgrade strategy application by application, and most importantly, without downtime.

Migration tools shipped with OUD addresses most of the transitioning issues. However, data cleaning and/or manual adaptation is sometimes required during this process, so some time should be allocated to address that during the transition analysis and design.

Transition from DSEE to OUD: Top 5 tips

The ds2oud tool can be used to migrate DSEE configuration to OUD. However, a few additional OUD configuration changes might be required on a case by case basis to provide seamless transition for applications.

Here are the top 5 differences spotted during real transition projects and how to address them:

#1 Syntax checking

DSEE does not check attribute value syntax. OUD does, so attribute values must conform to the attribute syntax defined in the schema. For instance, an attribute with Boolean syntax can hold TRUE or FALSE values only. Ideally, data should be fixed by the customer. However, this is not always possible and takes time. Furthermore, somne client application may rely on the incorrect data.

To disable attribute value syntac checking on OUD, the invalid-attribute-syntax-behavior property in the global configuration  can be changed to ‘warn’ or accept

#2 Structural objectclasses

Every user entry must have exactly one STRUCTURAL object-class to conform to Directory Standards. If a ODSEE entry has 0 or more than one structural object-class, the entry would be rejected during an import. ODSEE does not differentiate between the two object-class types, so this kind of schema inconsistency is commonly found in real deployments. It is recommended that you fix such user entries on the ODSEE side before transitioning to OUD.

Alternatively, you can disable this schema checking  as described in https://blogs.oracle.com/sduloutr/entry/cohabitation_odsee_oud_schema_checking

# Schema and root DSE access

The root DSE entry (empty DN) and the schema entry (cn=schema) contains several operational attributes. DSEE systematically returns these attributes even when the client application does not list them explilcitely in the search attribute list. This does not conform to the LDAP standard. By default OUD does not return them. However, it is possible to configure OUD to behave like DSEE using the procedure described in https://blogs.oracle.com/sduloutr/entry/oracle_unified_directory_root_dse

#4 Unindexed searches

By default, OUD does not allow unindexed searches as they may impact overall directory services performances. DSEE does.
It is recommended to limit the number of unindexed searches by creating additional indexes. However, unindex searches are valid patterns in some specific situations.
It is possible to grant unindexed search privilege on a per user account basis as described in https://blogs.oracle.com/sduloutr/entry/cohabitation_migration_odsee_oud_privileges

#5 Anonymous access

By default, DSEE accepts requests with DN and no passsword. Such requests are processed as anonymous.
By default, OUD rejects such requests. This behaviour can be changed by setting the property bind-with-dn-requires-password to false in the global OUD configuration

Don’t forget to have a look at the additional OUD KM notes available on OTN . They can be accessed as described in https://blogs.oracle.com/sduloutr/entry/how_to_subscribe_my_oracle

New Oracle White Paper about Directory Services Integration with Database Enterprise User Security

I’ve written a new Oracle White Paper about Directory Services Integration with
Database Enterprise User Security based on 2 recent posts,https://blogs.oracle.com/sduloutr/entry/oud_eus_take_2_db and https://blogs.oracle.com/sduloutr/entry/oud_eus_take_1_db

The official document is available at http://www.oracle.com/technetwork/database/security/dirsrv-eus-integration-133371.pdf

Migrating SSL Certificates to OUD

By default, self-signed certificates are automatically asssigned to OUD instances.

In some cases, you might want to reuse a DSEE server certificate for the new OUD instance, so that the migration is transparent for SSL clients. Note that this might require installation of the OUD instance on the same box as the DSEE depending on SSL certificate options used.

If you want to have your OUD instance reuse the SSL servert certificate,  perform the following steps

1. export the DSEE server certificate to a PKCS12 file (e.g dsee.p12) as described in the ODSEE admin guide
The exact procedure may depend on the DSEE release. On DSEE 6.x, DSEE 7.x and ODSEE, run the command below:

    dsadm export-cert -o dsee.p12  <instance_path> defaultCert

Note: By default, the alias of the DSEE server cert is defaultCert. Use the appropriate alias in case you choosed to use another value.

2. copy the PKCS12 file to <OUD_INSTANCE>/config

3. create a pin file containing the pkcs12 file password e.g. dsee.p12.pin in the <OUD_INSTANCE>/config directory

At that stage, the DSEE server certificate can be imported in the OUD instance in 2 different ways:
– either configure a PKCS12 OUD keystore pointing to the file exported from DSEE
– import the DSEE certificate to the default JKS OUD keystore

To configure a OUD PKCS12 keystore, perform the following steps:

4.1 Configure the PKCS12 keystore

dsconfig set-key-manager-provider-prop \
–provider-name PKCS12 \
–set key-store-file:config/dsee.p12 \
–set key-store-pin-file:config/dsee.p12.pin \
–set enabled:true \

4.2 Configure the LDAPS connection handler to use the pkcs#12 keystore

dsconfig set-connection-handler-prop \
–handler-name LDAPS\ Connection\ Handler \
–set key-manager-provider:PKCS12 \

To import the DSEE certificate key pair to the existing OUD JKS keystore, perform the following steps:

5.1 Locate the JAVA_HOME of the jvm used by OUD

The version of the JVM used is displayed at startup in the OUD error log

5.2 Run the following command to import the DSEE certificate

JAVA_HOME/bin/keytool -v -importkeystore -srckeystore <Path to PKCS12 cert file exported from DSEE>  -srcstoretype PKCS12 -destkeystore <OUD_INSTANCE_DIR>/OUD/config/keystore  -deststoretype JKS

When prompted, specify the JKS pin (available in <OUD_INSTANCE_DIR>/OUD/config/keystore.pin  and the PKCS12 pin you used to export the DSEE server cert

5.3 Check import

To list the content of the OUD JKS keystore, use the following:

    JAVA_HOME/bin/keytool -list -keystore <OUD_INSTANCE_DIR>/OUD/config/keystore

Enter keystore password:

Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries

defaultcert, Aug 29, 2013, PrivateKeyEntry,
Certificate fingerprint (MD5): 10:63:DC:B5:6B:C8:F3:A0:6B:A7:23:9E:0B:EA:9C:30

server-cert, Aug 29, 2013, PrivateKeyEntry,
Certificate fingerprint (MD5): BE:C9:F3:8A:49:98:96:15:EF:AC:B4:08:6F:76:FB:05

By default, the DSEE server cert alias is defaultcert.
By default, the OUD server cert alias is server-cert.
By default, OUD let java  automatically choose the best server-cert amongst those present in the keystore. If you want to force the use of  one certificate, do the following:

dsconfig set-connection-handler-prop \
–handler-name LDAPS\ Connection\ Handler \
–set ssl-cert-nickname:defaultcert \

Oracle Virtual Desktop Infrastructure and Unified Directory

Oracle Virtual Desktop Infrastructure offers a complete solution for managing and providing access to virtualized desktop environments hosted in the datacenter.  Oracle Virtual Desktop Instrastructure enables organizations to simplify administration, reduce operating costs, increase the utilization of existing IT assets, and boost security by moving from a tradtional desktop environment to a virtual desktop architecture.

Typically, you configure Oracle VDI to use the information held in a corporate user directory, like Oracle Unified Directory Server.

You can use the OUD setup or the ODSM to create a suffix holding users, eg,  ou=People,dc=oscr,dc=uk,dc=oracle,dc=com using existing schema.
Then create a few user entries with the fields User Name, First Name, Last Name, User ID and User Password.  So for my account it is

User Name : Sylvain Duloutre
First Name : Sylvain
Last Name : Duloutre
User ID : sduloutr
User Password : ****

To install Virtual Desktop Infrastructure, follow the install guide, then connect to the VDI Web UI using your preferred browser. Here is a screenshot showing the setup of the VDI server :


Next are 2 screenshots showing the LDAP settings and how they map to VDI:


As you can see there isn’t actually a lot of configuration to do.  You  can now login to VDI from a Sunray or from the Oracle Virtual Desktop Client using the login name and password stored in OUD.

Thanks to Rob for VDI snapshots and testing.

OUD Database Indexes & Performance Tuning

Oracle Unified Directory (OUD) database indexes enable (search) directory requests  to be processed efficiently.

Indexes are files that contain lists of values, where each value is associated with a list of entry identifiers to suffixes in the directory server database. When the directory server processes a search request, it searches the database using the list of entry identifiers in the indexes, thus speeding up the search. If indexes did not exist, the directory server would have to look up each entry in the database, which dramatically degrades performance.

First of all, unindexed searches are rejected by default unless the (authenticated)  user has the privilege to perform them because unindexed searches  may negatively impact overall directory performances. In some circumstances, unindexed searches are a valid approach, so it is possible to explicitely  assign the unindexed-search privilege to a user or a group of user. A user attempting to perform an unindexed search gets the message “you do not have sufficient privileges to perform an unindexed search”.

Note that unindexed searches always complete sucessfully when the amount of entries in the database does not exceed the value if index-entry-limit (4000 by default), which is unlikely to be seen in real deployments.

The verify-index command can be used to check the consistency between the index and entry data within the directory server database. This command also provides information about the number of index keys that have reached the index entry limit (See below).

The rebuild-index command can be used to rebuild an index. It is useful in the following cases:

  • When the index-entry-limit property of an index changes
  • When a new index is created

The index-entry-limit tuning parameter (known as AllIDs threshold in Oracle Directory Server Entreprise Edition) controls the maximum numbers of entries kept in an index record as scaning the whole database may become a more efficient option when the number of entries associated with a given index value becomes huge.

Tuning indexes upfront may not be an obvious task, that’s the reason why OUD provides you with  both  static and  dynamic index analyser tools:

The static analyser is delivered as part of the dbtest tool.  The dbtest list-index-status -n <dbName> -b <dbSuffix> command let you figure out how many entries are associated with each index value, and tune index-entry-limit when needed..

The dynamic analyser can be used to monitor live search filters received by OUD and corresponding  index usage: Live index analysis must be first enabled at the database level. At that point, subsequent searches hitting OUD are analysed and the corresponding statistics are kept in memory.  Statistics can be retrieved from the in-memory suffix cn=monitor as part of the database environmenent entry.

To enable live index data collection, run  the following command:

$ dsconfig set-workflow-element-prop –element-name userRoot  –set index-filter-analyzer-enabled:true  –set max-entries:100 -h localhost -p 4444 -D cn=directory\ manager -w <password>

The set-max-entries parameter controls how many statistical records are kept in memory.

To retrieve the results, run the following search:

$ ldapsearch -p 1389 -D cn=directory\ manager -w <password>  -b “cn=<dbName> Database Environment,cn=monitor”  “(objectclass=*)”  filter-usefilter-use: (description=*) hits:2 maxmatches:-1 message:presence index type is disabled for the description attribute

Important note:  It is not recommanded to enable statistic collection in a production system for a long time as it consumes ressources.

It is also possible to use the operational attribute debugsearchindex to figure out which entries are targetted by each component of a LDAP search filter. This is a lightweight yet convenient way to debug index-related issues.

$ ./ldapsearch -p 11389 -b “o=example” -D “cn=directory manager” -w *** “(&(uid=user.9*)(objectclass=*))”debugsearchindex
dn: cn=debugsearch
debugsearchindex: filter=(&(objectClass=*)[NOT-INDEXED](uid=user.9*)[COUNT:111]) [COUNT:111] scope=wholeSubtree[LIMIT-EXCEEDED:4101] final=[COUNT:111] 

Migration Stategy to Oracle Unified Directory

Developing a good strategy is a key element of a migration from third-party directories to OUD.
For sake of simplification, migration can be broken down in 5 steps as described below:

User Data Migration

Most companies defined some custom LDAP attributes and object classes. They use them  in conjunction with standard LDAP schema. LDAP provides a standard way to define schema extensions, so migration of user data is in general quite straight-forward:  Custom schema extensions need to be added to the OUD configuration, user data are exported from the existing directory to the standard LDIF format then re-imported into OUD.

By default, OUD schema checking is strict, some user entries may be rejected when they do not strictly adhere with the LDAP schema. In such case, either fix the data, fix the schema or relax the corresponding schema option in OUD configuration.

Migration of passwords may cause problems if they are encrypted with non-standard algorithms. I plan to cover that in a separate post soon.

Directory Metadata Migration

Most directories, including OUD, store meta data along with the User Data. This may  include access control information (aci), collective attributes, ldap sub entries etc. Each directory vendor uses its own model, so this aspect of the migration requires attention  and must be carefully planned.

Directory Configuration Migration

Each directory has its own configuration model, so the configuration must be ported to OUD. It includes the LDAP ports the directory listen on, the LDAP naming contexts exposed, database indexes, replication settings, security settings, performance setting, etc. This can be done using OUD graphical interface (ODSM) or using command line dsconfig. This is in general quite simple to migrate the directory configuration to OUD. Special care is needed to manage migration of SSL server certificates if certificate renewal is not an option.

Dealing with hard-wired dependencies in client applications

Some LDAP client application have hard-wired dependencies on a directory vendor and/or version. For instance, an application would query the directory service version string and would take some decision based on that. Some applications may also create/update directory-specific metadata. It is quite difficult to identify such issues upfront, but it is usually good policy to classify client applications based on their LDAP traffic patterns: traffic of provisioning applications should be review first, as the probabilities to have dependencies on vendor-specific interface is higher than for application doing simple authentication.

Oracle Virtual Directory (OVD), part of the  Oracle Directory Services Plus can be used to emulate directory-specific features.

Switching from existing directory to OUD

From an operational perspective, it is key to define how the actual switch to OUD will occur: Some customers would favor export and import w/o maintaining the 2 environments in sync. This seems very simple, but this methodology cannot ensure an highly-available deployment with up-to-date entries on both sides. When this is not acceptable, synchronization tools like DIP (Directory Integration Platform) which is a part of Oracle Directory Services Plus can be used to synchronize user data.

Additional options exist to migrate from Oracle Directory Enterprise Edition (DSEE) to OUD as described here.