Recently, I’ve been facing an issue more and more frequently where a Repository cannot register itself to a local/remote Docbroker. The root cause of the issue is simply that the hostname used for the projection to the Docbroker is too long which prevents the Repository to properly register to it. I’ve seen it recently because when working with Kubernetes, you can pretty quickly end-up with very very long hostnames… The issue isn’t linked to K8s in itself, it just helps to reproduce the issue. Alternatively, you can do the same thing by just tweeking the /etc/hosts file to lenghten a name that can then be used for the projection.

Alright so let’s demonstrate this issue. The simplest way to reproduce the issue is to have a FQDN that is strictly longer than 59 characters and a short hostname that is shorter:

[dmadmin@documentum-server-0 ~]$ hostname | wc -m
20
[dmadmin@documentum-server-0 ~]$ hostname
documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ hostname -f | wc -m
63
[dmadmin@documentum-server-0 ~]$ hostname -f
documentum-server-0.documentum-server.dbi-ns.svc.cluster.local
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep "^host=" $DOCUMENTUM/dba/dm_launch_Docbroker
host=documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
Successful reply from docbroker at host (documentum-server-0) on port(1490) running software version (16.4.0110.0167 Linux64).
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname -f` -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
Successful reply from docbroker at host (documentum-server-0) on port(1490) running software version (16.4.0110.0167 Linux64).
[dmadmin@documentum-server-0 ~]$

As you can see above, everything seems fine. Let’s see what is the current configuration and if it works properly:

[dmadmin@documentum-server-0 ~]$ grep "^host=" $DOCUMENTUM/dba/dm_launch_Docbroker
host=documentum-server-0
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_stop_Docbroker
./dmshutdown 16.4.0000.0248  Linux64 Copyright (c) 2018. OpenText Corporation.
Shutdown request was processed by Docbroker on host documentum-server-0 (INET_ADDR: 01 2d3 01010164 documentum-server-0 1.1.1.100)
Reply status indicates a success: OK
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_launch_Docbroker
starting connection broker on current host: [documentum-server-0]
with connection broker log: [$DOCUMENTUM/dba/log/docbroker.documentum-server-0.1489.log]
connection broker pid: 19073
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ cat $DOCUMENTUM/dba/log/docbroker.documentum-server-0.1489.log
OpenText Documentum Connection Broker (version 16.4.0110.0167  Linux64)
Copyright (c) 2018. OpenText Corporation
2019-11-16T11:30:44.737512 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 19073"
2019-11-16T11:30:44.747745 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1489)."
2019-11-16T11:30:44.747797 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1489, host: documentum-server-0 (1.1.1.100, e01234ac))"
2019-11-16T11:30:44.747812 [DM_DOCBROKER_I_REGISTERED_PORT]info:  "The Docbroker registered using port (1490)."
2019-11-16T11:30:44.747828 [DM_DOCBROKER_I_LISTENING]info:  "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1490, host: documentum-server-0 (1.1.1.100, e01234ac))"
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET\]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_start_gr_repo
starting Documentum server for repository: [gr_repo]
with server log: [$DOCUMENTUM/dba/log/gr_repo.log]
server pid: 19245
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:31:48.749813      19245[19245]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:31:49.008865      19318[19318]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0) with port (1490).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : documentum-server-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 documentum-server-0 1.1.1.100
Docbroker version         : 16.4.0110.0167  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234567
Docbase description : dbi dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0110.0167  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@documentum-server-0 ~]$

So with the above short name on both the Docbroker start script and the Repository projection (server.ini), you can see that it’s all working. Now let’s replicate the issue:

[dmadmin@documentum-server-0 ~]$ $DOCUMENTUM/dba/dm_shutdown_gr_repo
Stopping Documentum server for repository: [gr_repo]

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase gr_repo.gr_repo
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68780012505 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> shutdown,c,T,T
...
OK
API> exit
Bye
Waiting for 90 seconds for server pid, 19245, to disappear.

Sat Nov 16 11:35:35 UTC 2019: Waiting for shutdown of repository: [gr_repo]
Sat Nov 16 11:35:35 UTC 2019: checking for pid: 19245

Sat Nov 16 11:35:45 UTC 2019: Waiting for shutdown of repository: [gr_repo]
Sat Nov 16 11:35:45 UTC 2019: checking for pid: 19245

repository: [gr_repo] has been shutdown
checking that all children (19251 19253 19254 19275 19297 19319 19362 19387 19570) have shutdown
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET\]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ sed -i "s,^host.*,host = `hostname -f`," $DOCUMENTUM/dba/config/gr_repo/server.ini
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep -A3 "TARGET\]" $DOCUMENTUM/dba/config/gr_repo/server.ini
[DOCBROKER_PROJECTION_TARGET]
host = documentum-server-0.documentum-server.dbi-ns.svc.cluster.local
port = 1489
proximity = 1
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ grep broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:38:34.593108      19715[19715]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:38:34.655402      19787[19787]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo) with port (1489).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$

The message seems correct, right? You might be fooled by the message but if you look carefully at it, you can actually see that the host in the parenthesis is cut. The value is “documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo” instead of “documentum-server-0.documentum-server.dbi-ns.svc.cluster.local“. That’s 59 characters exactly and nothing more. Since our FQDN is longer, then the Repository cut the hostname and keep only the first 59 characters. If you look at the log file directly of if you include the error messages as well in the grep (with the -i option for example for the uppercase BROKER errors), then you can see the failure:

[dmadmin@documentum-server-0 ~]$ grep -i broker $DOCUMENTUM/dba/log/gr_repo.log
2019-11-16T11:38:34.593108      19715[19715]    0000000000000000        [DM_SERVER_I_START]info:  "Sending Initial Docbroker check-point "
2019-11-16T11:38:34.655141      19787[19787]    0112d68780000003        [DM_DOCBROKER_E_NETWORK_ERROR]error:  "An error occured performing a network operation: ((112) Error opening connection).  Network specific error: ( errno: 0, message: Success)."
2019-11-16T11:38:34.655364      19787[19787]    0112d68780000003        [DM_DOCBROKER_E_CONNECT_FAILED_EX]error:  "Unable to connect to DocBroker. Clients please check your dfc.properties file for a correct host.  Server please check your server.ini or target attributes in server config.  Network address: (INET_ADDR: family: 2, port: 1489, host: documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo (0.0.0.0, 00000000))."
2019-11-16T11:38:34.655402      19787[19787]    0112d68780000003        [DM_DOCBROKER_I_PROJECTING]info:  "Sending information to Docbroker located on host (documentum-server-0.documentum-server.dbi-ns.svc.cluster.lo) with port (1489).  Information: (Config(gr_repo), Proximity(1), Status(Open), Dormancy Status(Active))."
[dmadmin@documentum-server-0 ~]$
[dmadmin@documentum-server-0 ~]$ dmqdocbroker -t `hostname` -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0110.0058
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            :
Docbroker port            :
Docbroker network address :
Docbroker version         :
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
[dmadmin@documentum-server-0 ~]$

So as you can see above, just putting a hostname too long in the server.ini (or inside the Repository) will prevent it to project itself to the Docbroker and therefore you will never be able to work or even stop the Repository (kill is the only option). Therefore, take care that the hostname you are using for the projection isn’t too long.

Cet article Documentum – Beware of long hostnames for the Docbroker est apparu en premier sur Blog dbi services.

In a previous blog, I talked about the D2 utility that can be used to automate the registration, approval and global management of the DFC Client IDs and Privileged Clients. The instructions I gave were the optimal (“normal”) ones… However, in reality, this utility or at least the shell script that is supposed to facilitate its execution is kinda bad. Therefore, if you really want to use it, then you first need to fix it.

I saw these issues so far on the shell script (‘D2PrivilegedClientUtil.sh‘):

• D2 16.4.0:
  • No shebang
  • First line is just ‘:’
  • log4j_configuration variable is set to ‘/Documentum/config/log4j.properties’
• D2 16.5.0:
  • Same as D2 16.4.0 (the 3 issues)
  • The file is in DOS format
• D2 16.5.1:
  • Same as D2 16.5.0 (the 4 issues)

If you are trying to execute the D2 16.5.1 utility with its default content, you will therefore have the following:

[weblogic@weblogic-server-0 d2privilegedclient]$ chmod 700 D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${gr} -u ${user} -p ${pw} | grep -v "|-"
./D2PrivilegedClientUtil.sh: line 1: $':\r': command not found
./D2PrivilegedClientUtil.sh: line 2: $'\r': command not found
./D2PrivilegedClientUtil.sh: line 4: $'\r': command not found
./D2PrivilegedClientUtil.sh: line 7: $'\r': command not found
Running D2PrivilegedClientUtil
./D2PrivilegedClientUtil.sh: line 9: $'\r': command not found
./D2PrivilegedClientUtil.sh: line 13: $'\r': command not found
./D2PrivilegedClientUtil.sh: line 20: $'\r': command not found
./D2PrivilegedClientUtil.sh: line 35: syntax error: unexpected end of file
[weblogic@weblogic-server-0 d2privilegedclient]$

Therefore, the first issue you can see is the DOS format. Let’s fix this first:

[weblogic@weblogic-server-0 d2privilegedclient]$ cp D2PrivilegedClientUtil.sh D2PrivilegedClientUtil.sh.orig
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ awk '{ sub("\r$", ""); print }' D2PrivilegedClientUtil.sh > temp.sh
[weblogic@weblogic-server-0 d2privilegedclient]$ mv temp.sh D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$ chmod 700 D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$

Once the DOS format issue is fixed, let’s try again to execute the shell script:

[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${gr} -u ${user} -p ${pw} | grep -v "|-"
Running D2PrivilegedClientUtil

log4j:ERROR Could not read configuration file from URL [file:/Documentum/config/log4j.properties].
java.io.FileNotFoundException: /Documentum/config/log4j.properties (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at java.io.FileInputStream.<init>(FileInputStream.java:93)
        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:524)
        at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:415)
        at com.documentum.fc.common.impl.logging.LoggingConfigurator.configureLog4j(LoggingConfigurator.java:141)
        at com.documentum.fc.common.impl.logging.LoggingConfigurator.performInitialConfiguration(LoggingConfigurator.java:33)
        at com.documentum.fc.common.DfLogger.<clinit>(DfLogger.java:624)
        at com.documentum.fc.common.impl.logging.LoggingConfigurator.onPreferencesInitialized(LoggingConfigurator.java:178)
        at com.documentum.fc.common.DfPreferences.initialize(DfPreferences.java:71)
        at com.documentum.fc.common.DfPreferences.getInstance(DfPreferences.java:43)
        at com.documentum.fc.client.DfSimpleDbor.getDefaultDbor(DfSimpleDbor.java:78)
        at com.documentum.fc.client.DfSimpleDbor.<init>(DfSimpleDbor.java:66)
        at com.documentum.fc.client.DfClient$ClientImpl.<init>(DfClient.java:350)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:766)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.emc.xcp.rest.core.dfc.dao.impl.PrivilegeClientDaoImpl.fetchAndGrantPermissions(PrivilegeClientDaoImpl.java:40)
        at com.opentext.d2.util.D2PrivilegedClientUtil.main(D2PrivilegedClientUtil.java:90)
log4j:ERROR Ignoring configuration file [file:/Documentum/config/log4j.properties].
0 [main] WARN com.documentum.fc.common.impl.logging.LoggingConfigurator  - Using default log4j configuration
552 [main] WARN com.documentum.fc.client.impl.docbroker.ServerMapBuilder  - [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker "documentum-server-0:1489" failed
DfIOException:: THREAD: main; MSG: [DM_SESSION_E_RPC_ERROR]error:  "Server communication failure"; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.DfIOException.newCommunicationFailureException(DfIOException.java:16)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:216)
        at com.documentum.fc.client.impl.connection.docbroker.netwise.NetwiseDocbrokerRpcClient.requestObject(NetwiseDocbrokerRpcClient.java:58)
        at com.documentum.fc.client.impl.connection.docbroker.DocbrokerConnection.requestObject(DocbrokerConnection.java:80)
        at com.documentum.fc.client.impl.docbroker.MapBuilder.issueRequest(MapBuilder.java:34)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java:169)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java:60)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:192)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:122)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.connect(ModuleManager.java:397)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.init(ModuleManager.java:352)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.getInstance(ModuleManager.java:43)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.<init>(DfcIdentityPublisher.java:44)
        at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.emc.xcp.rest.core.dfc.dao.impl.PrivilegeClientDaoImpl.fetchAndGrantPermissions(PrivilegeClientDaoImpl.java:40)
        at com.opentext.d2.util.D2PrivilegedClientUtil.main(D2PrivilegedClientUtil.java:90)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at com.documentum.fc.impl.util.io.MessageChannel.writeSocket(MessageChannel.java:261)
        at com.documentum.fc.impl.util.io.MessageChannel.write(MessageChannel.java:219)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:211)
        ... 33 more
564 [main] WARN com.documentum.fc.client.impl.bof.classmgmt.ModuleManager  - [DFC_BOF_COULDNT_CONNECT_TO_REGISTRY] Unable to connect to module registry, docbase name gr_repo username dm_bof_registry.
DfDocbrokerException:: THREAD: main; MSG: [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker "documentum-server-0:1489" failed; ERRORCODE: ff; NEXT: null
        at com.documentum.fc.client.DfDocbrokerException.newRequestFailedException(DfDocbrokerException.java:12)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java:72)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:192)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:122)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.connect(ModuleManager.java:397)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.init(ModuleManager.java:352)
        at com.documentum.fc.client.impl.bof.classmgmt.ModuleManager.getInstance(ModuleManager.java:43)
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.<init>(DfcIdentityPublisher.java:44)
        at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.emc.xcp.rest.core.dfc.dao.impl.PrivilegeClientDaoImpl.fetchAndGrantPermissions(PrivilegeClientDaoImpl.java:40)
        at com.opentext.d2.util.D2PrivilegedClientUtil.main(D2PrivilegedClientUtil.java:90)
Caused by: DfIOException:: THREAD: main; MSG: [DM_SESSION_E_RPC_ERROR]error:  "Server communication failure"; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.DfIOException.newCommunicationFailureException(DfIOException.java:16)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:216)
        at com.documentum.fc.client.impl.connection.docbroker.netwise.NetwiseDocbrokerRpcClient.requestObject(NetwiseDocbrokerRpcClient.java:58)
        at com.documentum.fc.client.impl.connection.docbroker.DocbrokerConnection.requestObject(DocbrokerConnection.java:80)
        at com.documentum.fc.client.impl.docbroker.MapBuilder.issueRequest(MapBuilder.java:34)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java:169)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java:60)
        ... 28 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at com.documentum.fc.impl.util.io.MessageChannel.writeSocket(MessageChannel.java:261)
        at com.documentum.fc.impl.util.io.MessageChannel.write(MessageChannel.java:219)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:211)
        ... 33 more
577 [main] WARN com.documentum.fc.client.security.internal.RegistrationMgr  - [DFC_SECURITY_GR_PUBLICATION_FAILED] Publication of DFC instance with global registry failed
DfException:: THREAD: main; MSG: [DFC_BOF_GLOBAL_REGISTRY_UNAVAILABLE] Unable to connect to the global registry no session; ERRORCODE: ff; NEXT: null
        at com.documentum.fc.client.security.impl.DfcIdentityPublisher.<init>(DfcIdentityPublisher.java:55)
        at com.documentum.fc.client.security.internal.RegistrationMgr.register(RegistrationMgr.java:34)
        at com.documentum.fc.impl.RuntimeContext.<clinit>(RuntimeContext.java:195)
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:772)
        at com.documentum.com.DfClientX.getLocalClient(DfClientX.java:43)
        at com.emc.xcp.rest.core.dfc.dao.impl.PrivilegeClientDaoImpl.fetchAndGrantPermissions(PrivilegeClientDaoImpl.java:40)
        at com.opentext.d2.util.D2PrivilegedClientUtil.main(D2PrivilegedClientUtil.java:90)
2019-09-07 11:12:03,915 UTC [ERROR] (main) - c.e.x.r.c.d.dao.impl.PrivilegeClientDaoImpl   : [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker "documentum-server-0:1489" failed
com.documentum.fc.client.DfDocbrokerException: [DFC_DOCBROKER_REQUEST_FAILED] Request to Docbroker "documentum-server-0:1489" failed
        at com.documentum.fc.client.DfDocbrokerException.newRequestFailedException(DfDocbrokerException.java:12)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java:72)
        at com.documentum.fc.client.impl.docbroker.DocbrokerClient.getServerMap(DocbrokerClient.java:152)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoices(ServerChoiceManager.java:159)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.updateServerChoicesIfNecessary(ServerChoiceManager.java:148)
        at com.documentum.fc.client.impl.connection.docbase.ServerChoiceManager.getServerChoices(ServerChoiceManager.java:47)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getServerChoices(DocbaseConnection.java:273)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.establishNewRpcClient(DocbaseConnection.java:227)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.open(DocbaseConnection.java:126)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:100)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.<init>(DocbaseConnection.java:60)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionFactory.newDocbaseConnection(DocbaseConnectionFactory.java:26)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.createNewConnection(DocbaseConnectionManager.java:192)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:122)
        at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
        at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
        at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
        at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
        at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
        at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
        at com.emc.xcp.rest.core.dfc.dao.impl.PrivilegeClientDaoImpl.fetchAndGrantPermissions(PrivilegeClientDaoImpl.java:47)
        at com.opentext.d2.util.D2PrivilegedClientUtil.main(D2PrivilegedClientUtil.java:90)
Caused by: com.documentum.fc.client.DfIOException: [DM_SESSION_E_RPC_ERROR]error:  "Server communication failure"
        at com.documentum.fc.client.DfIOException.newCommunicationFailureException(DfIOException.java:16)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:216)
        at com.documentum.fc.client.impl.connection.docbroker.netwise.NetwiseDocbrokerRpcClient.requestObject(NetwiseDocbrokerRpcClient.java:58)
        at com.documentum.fc.client.impl.connection.docbroker.DocbrokerConnection.requestObject(DocbrokerConnection.java:80)
        at com.documentum.fc.client.impl.docbroker.MapBuilder.issueRequest(MapBuilder.java:34)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getDataFromDocbroker(ServerMapBuilder.java:169)
        at com.documentum.fc.client.impl.docbroker.ServerMapBuilder.getMap(ServerMapBuilder.java:60)
        ... 20 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at com.documentum.fc.impl.util.io.MessageChannel.writeSocket(MessageChannel.java:261)
        at com.documentum.fc.impl.util.io.MessageChannel.write(MessageChannel.java:219)
        at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.sendMessage(AbstractNetwiseRpcClient.java:211)
        ... 25 common frames omitted
null
[weblogic@weblogic-server-0 d2privilegedclient]$

Above, we can see two issues:

• The wrong path for the log4j.properties file
• An SSL Handshake error, which cause the connection to the docbroker to fail

The first issue is simple to fix. It’s technically not mandatory to have a log4j.properties file, the utility will still run properly but it will create two files in the local folder: “log4j.log” and “trace.log”. It will also print an exception, the one shown above. Since D2 doesn’t come with a log4j.properties file, it’s a little bit strange to use that for a newly developed utility but it’s like that. Anyway, either remove the files after the execution and ignore the errors or use a custom log4j.properties file. We are usually adding a custom log4j.properties file for the DFC logs in D2. Therefore, in my case, I will just use it.

The second issue is actually a bug. When I first executed it, I saw this message and based on the stack trace, I highly suspected that the issue was linked to the DFC connection mode. Since we are always using secure communications by default for our Applications, I switched it to native and tried again: it worked. Therefore, I opened a SR with the OpenText Support (#4190973) which resulted in two defects. First, DTWO-46920 to fix the different issues in the shell script linked to linux (the utility was obviously developed for Windows and never tested on linux). Second, DTWO-46921 to document the bug that prevent the D2 utility to work over secure channels.

Since none of these issues are blocking, I applied the workaround for each of these (+ the missing shebang) to execute the D2 utility a third time:

[weblogic@weblogic-server-0 d2privilegedclient]$ sed -i 's,^:$,#!/bin/bash,' D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ sed -i 's,^log4j_configuration=.*,log4j_configuration=${webinfdir}/classes/log4j.properties,' D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ grep secure_connect_default $APP_DATA/D2/dfc.properties
dfc.session.secure_connect_default=secure
[weblogic@weblogic-server-0 d2privilegedclient]$ sed -i 's,^dfc.session.secure_connect_default=.*,dfc.session.secure_connect_default=native,' $APP_DATA/D2/dfc.properties
[weblogic@weblogic-server-0 d2privilegedclient]$ grep secure_connect_default $APP_DATA/D2/dfc.properties
dfc.session.secure_connect_default=native
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${gr} -u ${user} -p ${pw} | grep -v "|-"
Running D2PrivilegedClientUtil

2019-09-07 11:17:27,359 UTC [INFO ] (main) - c.e.x.r.c.d.dao.impl.PrivilegeClientDaoImpl   : Checking dm_client_rights for dfc: dfc_Lp76piwfz4tpAupiCktfKFfndOAa
D2_WS1_weblogic-server-0_fndOAa
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${repo} -u ${user} -p ${pw} | grep -v "|-"
Running D2PrivilegedClientUtil

2019-09-07 11:18:05,735 UTC [INFO ] (main) - c.emc.xcp.rest.core.dfc.utils.IpAndRcHelper   : filling in {0} a new record with this persistent certificate:
{1}
2019-09-07 11:18:06,082 UTC [INFO ] (main) - c.e.x.r.c.d.dao.impl.PrivilegeClientDaoImpl   : Checking dm_client_rights for dfc: dfc_Lp76piwfz4tpAupiCktfKFfndOAa
D2_WS1_weblogic-server-0_fndOAa
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ grep secure_connect_default $APP_DATA/D2/dfc.properties
dfc.session.secure_connect_default=native
[weblogic@weblogic-server-0 d2privilegedclient]$ sed -i 's,^dfc.session.secure_connect_default=.*,dfc.session.secure_connect_default=secure,' $APP_DATA/D2/dfc.properties
[weblogic@weblogic-server-0 d2privilegedclient]$ grep secure_connect_default $APP_DATA/D2/dfc.properties
dfc.session.secure_connect_default=secure
[weblogic@weblogic-server-0 d2privilegedclient]$

After applying all these fixes, the D2 utility is finally working and it does register and approve the DFC Client ID correctly. I’m curious to see when exactly they are going to fix all that and provide an utility that can work over secure communications.

Cet article Documentum – Fixing issues with the D2 Privileged Client utility est apparu en premier sur Blog dbi services.

In the past several months, I have been working extensively with several other colleagues on a migration of Documentum from VMs to K8s pods. We have been doing/using silent installations for several years already but there were still several manual tasks for the customization of the repository, deployment of some applications, aso… In this blog, I will talk about what can be done to automate the registration and approval of DFC Clients s o it can be done without any manual intervention.

DFC Clients registration & approval is useful in several areas. Some example of where you would need it:

• If you want to use D2 4.7 or above for example, then you will need to register and approve the D2 DFC Client IDs
• If you want to use High Availability on the Content Server, then you will also need to register and approve the DFC Client IDs of the CS&JMS dfc.keystore files. In addition to that, for the HA part, you will also need to add the two options ‘Enables trusted login‘ and ‘Enables trusted server privilege‘
• You could also want to have the trusted options for a custom application so that your code doesn’t require authentication and it could just run on the target repository: obviously you would need to be careful with such things but it is nonetheless possible

To configure a DFC Client, you can simply use Documentum Administrator (Administration > Client Rights Management > Privileged Clients). However, this is always a pain in the *** when you install a new application to retrieve the Client ID from the dfc.keystore, then login to DA, approve the ID, add the options that are needed, doing it for all repositories needed, aso…

So, how is a DFC Client ID managed inside the repository? Well, when a new dfc.keystore is created, it should register itself, on the first use, with the Global Registry repository defined in the associated dfc.properties file. What this does is that it adds a new row/entry inside the ‘dm_client_registration‘ table of the GR repository only. In DA, when you open the “Manage Clients” page (File > Manage Clients) and add a DFC Client ID to the list of Privileged Clients (right side), it will create a new row/entry in the ‘dm_client_rights‘ table as well as in the ‘dm_public_key_certificate‘ table (if it’s not already present). You can do that for the GR repository obviously but you can also do it for a normal repository. In both cases, the list of DFC Clients that you see on the “Manage Clients” page of DA is actually the list of the entries from the ‘dm_client_registration‘ of the GR repository.

Permissions related to a specific DFC Client ID are then managed directly in the ‘dm_client_rights’:

• Approving a DFC Client ID will set the ‘allow_all_priv_modules‘ to true
• Adding the ‘Enables trusted login‘ will set the ‘principal_auth_priv‘ to true
• Adding the ‘Enables trusted server privilege‘ will set the ‘server_trust_priv‘ to true

So, it seems to be simple to automate, right? Well, not really… The thing is that when you are trying to automate things, you need to be sure that the behavior is consistent. There is also the issue that what DA is doing on the “Manage Clients” page (add entry to the Privileged Clients list and therefore creation of rows/entries in the ‘dm_client_rights‘ + ‘dm_public_key_certificate‘) isn’t public. We tried to ask OpenText about what is DA actually doing in the background but the feedback we got is that they cannot tell us.

So, there is no way then? Is this blog useless? Well, of course not but there are some restrictions. The D2 Team actually developed a utility that is available starting with D2 16.4 and that can be used to do exactly that. The problem here is that this utility is linked to D2. It actually doesn’t use any D2 specific classes but only Documentum OOTB ones. However, since it was developed by the D2 Team, it means that you need to have a D2 license to be able to use it. I asked already in a SR (#4299215) for OpenText to consider migrating this utility on the CS side because this is needed as well for High Availability and other purposes. There is no reasons for this to be a D2 only tool, let’s see what they will decide in the future.

The D2 utility can only be used to register the DFC Client ID and add it to the list of the Privileged Clients but it can do it properly for both GR and normal repositories so that’s all we needed. This is how it works:

[weblogic@weblogic-server-0 ~]$ cd $APPLICATIONS
[weblogic@weblogic-server-0 applications]$ 
[weblogic@weblogic-server-0 applications]$ ls -l | grep D2
drwxr-x---.  9 weblogic weblogic      8192 Sep  3 16:05 D2-Config
drwxr-x---. 15 weblogic weblogic      8192 Sep  3 16:10 D2
[weblogic@weblogic-server-0 applications]$ 
[weblogic@weblogic-server-0 applications]$ cd D2/utils/d2privilegedclient/
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ ls -l
total 40
-rw-r-----. 1 weblogic weblogic 18470 May 20 17:23 D2PrivilegedClientUtil-16.5.1.jar
-rw-r-----. 1 weblogic weblogic  1214 May 20 17:23 D2PrivilegedClientUtil.cmd
-rw-r-----. 1 weblogic weblogic  1141 May 20 17:23 D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ grep "webinfdir" D2PrivilegedClientUtil.sh
webinfdir=../../WEB-INF
    classpath="${utiljar}:${webinfdir}/classes/:${webinfdir}/lib/*"
    classpath="${utiljar};${webinfdir}/classes/;${webinfdir}/lib/*"
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ cat ../../WEB-INF/classes/dfc.properties
#include $APP_DATA/D2/dfc.properties
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ grep -E "keystore|repo" $APP_DATA/D2/dfc.properties
dfc.globalregistry.repository=gr_repo
dfc.security.keystore.file=$APP_DATA/D2/dfc.keystore
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ strings $APP_DATA/D2/dfc.keystore | grep dfc_ | gawk '{print substr($1, 1, length($1) - 1); exit}'
dfc_Lp76piwfz4tpAupiCktfKFfndOAa
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ gr="gr_repo"
[weblogic@weblogic-server-0 d2privilegedclient]$ repo="REPO1"
[weblogic@weblogic-server-0 d2privilegedclient]$ user="dmadmin"
[weblogic@weblogic-server-0 d2privilegedclient]$ pw="P4ssw0rd"
[weblogic@weblogic-server-0 d2privilegedclient]$ 
[weblogic@weblogic-server-0 d2privilegedclient]$ chmod 700 D2PrivilegedClientUtil.sh
[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${gr} -u ${user} -p ${pw} | grep -v "|-"
Running D2PrivilegedClientUtil

2019-09-07 11:17:27,359 UTC [INFO ] (main) - c.e.x.r.c.d.dao.impl.PrivilegeClientDaoImpl   : Checking dm_client_rights for dfc: dfc_Lp76piwfz4tpAupiCktfKFfndOAa
D2_WS1_weblogic-server-0_fndOAa
[weblogic@weblogic-server-0 d2privilegedclient]$
[weblogic@weblogic-server-0 d2privilegedclient]$ ./D2PrivilegedClientUtil.sh -d ${repo} -u ${user} -p ${pw} | grep -v "|-"
Running D2PrivilegedClientUtil

2019-09-07 11:18:05,735 UTC [INFO ] (main) - c.emc.xcp.rest.core.dfc.utils.IpAndRcHelper   : filling in {0} a new record with this persistent certificate:
{1}
2019-09-07 11:18:06,082 UTC [INFO ] (main) - c.e.x.r.c.d.dao.impl.PrivilegeClientDaoImpl   : Checking dm_client_rights for dfc: dfc_Lp76piwfz4tpAupiCktfKFfndOAa
D2_WS1_weblogic-server-0_fndOAa
[weblogic@weblogic-server-0 d2privilegedclient]$

As you can see above, the shell script is setting up the classpath so that it fetches the internal dfc.properties of D2. In my case, this dfc.properties loads an external one. The D2 utility will use the dfc.properties file to fetch information regarding the location of the dfc.keystore as well as information regarding the GR repository.

There are several things wrong with the D2 utility. If you are trying to execute it as shown above without fixing these issues first, then it won’t be working properly. To fix the D2 utility, you can take a look at this blog.

Once the D2 utility has been executed for both the Global Registry repository and all normal repositories that will use D2, then you are done. You don’t need to do anything else for D2.

For the HA configuration of a CS for example, then in addition, you will also need to add the two options ‘Enables trusted login‘ and ‘Enables trusted server privilege‘, as said previously. To do that, I already gave all the needed information since I mentioned the link between what can be done in DA and the effect it has on the repository. Using a CS dfc.keystore for this example:

[dmadmin@content-server-0 ~]$ gr="gr_repo"
[dmadmin@content-server-0 ~]$ repo="REPO1"
[dmadmin@content-server-0 ~]$ 
[dmadmin@content-server-0 ~]$ client_id=`strings $DOCUMENTUM_SHARED/config/dfc.keystore | grep dfc_ | gawk '{print substr($1, 1, length($1) - 1); exit}'`
[dmadmin@content-server-0 ~]$ 
[dmadmin@content-server-0 ~]$ iapi ${gr} -U${USER} -Pxxx <<EOF
> retrieve,c,dm_client_rights where client_id='${client_id}'
> set,c,l,allow_all_roles
> T
> set,c,l,allow_all_priv_modules
> T
> set,c,l,principal_auth_priv
> T
> set,c,l,server_trust_priv
> T
> save,c,l
> EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase gr_repo
[DM_SESSION_I_SESSION_START]info:  "Session 010f12345001014e started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> ...
080f1234500005a1
API> SET> ...
OK
API> SET> ...
OK
API> SET> ...
OK
API> SET> ...
OK
API> ...
OK
API> Bye
[dmadmin@content-server-0 ~]$ 
[dmadmin@content-server-0 ~]$ 
[dmadmin@content-server-0 ~]$ iapi ${repo} -U${USER} -Pxxx <<EOF
> retrieve,c,dm_client_rights where client_id='${client_id}'
> set,c,l,allow_all_roles
> T
> set,c,l,allow_all_priv_modules
> T
> set,c,l,principal_auth_priv
> T
> set,c,l,server_trust_priv
> T
> save,c,l
> EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f2345600069bf started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> ...
080f2345600005be
API> SET> ...
OK
API> SET> ...
OK
API> SET> ...
OK
API> SET> ...
OK
API> ...
OK
API> Bye
[dmadmin@content-server-0 ~]$

With that, you will be able to automate the registration, approval and permissions of all your DFC Client IDs, assuming you have a valid D2 license. Hopefully OpenText will reconsider and make it a OOTB utility.

Cet article Documentum – Automatic/Silent registration & approval of DFC Clients est apparu en premier sur Blog dbi services.

If you have been working with Documentum, then you probably already created/configured an LDAP/LDAPs Server Config Object (or several) so that your users can be globally managed in your organization. There are several compatible LDAP Servers so I will just take one (Sun One/Netscpae/iPlanet Directory Server). To create this LDAP/LDAPs Server Config Object, you probably used Documentum Administrator because it’s simple and quick to setup, however that’s not enough for automation. In this blog, I will show and explain the steps needed to configure the same but without any need for DA.

The problem with DA is that it usually does some magic and you cannot always do exactly the same without it. Here, this also applies but to a smaller extent since it is only the SSL part (LDAPs) that needs specific steps. For this, there is a KB created by EMC some years ago (migrated to OpenText): KB6321243.

Before starting, let’s setup some parameters that will be used in this blog:

[dmadmin@content-server-0 ~]$ repo="repo01"
[dmadmin@content-server-0 ~]$ dm_location_name="ldap_chain"
[dmadmin@content-server-0 ~]$ file_path="$DOCUMENTUM/dba/secure/ldapdb/${dm_location_name}.pem"
[dmadmin@content-server-0 ~]$ ldap_server_name="Internal_LDAP"
[dmadmin@content-server-0 ~]$ ldap_host="ldap.domain.com"
[dmadmin@content-server-0 ~]$ ldap_ssl=1 #0 for LDAP, 1 for LDAPs
[dmadmin@content-server-0 ~]$ ldap_port=636
[dmadmin@content-server-0 ~]$ location=`if ((${ldap_ssl} == 1)); then echo ${dm_location_name}; else echo "ldapcertdb_loc"; fi`
[dmadmin@content-server-0 ~]$ ldap_principal="ou=APP,ou=applications,ou=intranet,dc=dbi services,dc=com"
[dmadmin@content-server-0 ~]$ ldap_pwd="T3stP4ssw0rd"
[dmadmin@content-server-0 ~]$ ldap_user_filter="objectclass=person"
[dmadmin@content-server-0 ~]$ ldap_user_class="person"
[dmadmin@content-server-0 ~]$ ldap_group_filter="objectclass=groupofuniquenames"
[dmadmin@content-server-0 ~]$ ldap_group_class="groupofuniquenames"

1. Preparation steps for LDAPs

The steps in this section are only needed in case you need to configure SSL communications between your LDAP Server and Documentum. It can be done upfront without any issues. So let’s start with setting up the environment. Without the use of DA, the only way you have to import/trust SSL Certificate for the LDAPs connection is by adding an environment variable named “DM_LDAP_CERT_FILE” and setting it to “1”. This will allow Documentum to use certificate files for the trust chain instead of doing what DA is doing (the magic part) that we cannot replicate.

It is a little bit out of scope for this blog but a second variable is often needed “DM_LDAP_IGNORE_HOSTNAME_CHECK” which drives the validation of the hostname. Setting this to “1” will disable the hostname check and therefore allow you to use an LDAP Server that is behind a Proxy or a Load Balancer. This would also be needed with a LDAP (non-secure).

[dmadmin@content-server-0 ~]$ grep DM_LDAP ~/.bash_profile
[dmadmin@content-server-0 ~]$ echo $DM_LDAP_CERT_FILE -- $DM_LDAP_IGNORE_HOSTNAME_CHECK
--
[dmadmin@content-server-0 ~]$
[dmadmin@content-server-0 ~]$ echo "export DM_LDAP_CERT_FILE=1" >> ~/.bash_profile
[dmadmin@content-server-0 ~]$ echo "export DM_LDAP_IGNORE_HOSTNAME_CHECK=1" >> ~/.bash_profile
[dmadmin@content-server-0 ~]$
[dmadmin@content-server-0 ~]$ grep DM_LDAP ~/.bash_profile
export DM_LDAP_CERT_FILE=1
export DM_LDAP_IGNORE_HOSTNAME_CHECK=1
[dmadmin@content-server-0 ~]$
[dmadmin@content-server-0 ~]$ source ~/.bash_profile
[dmadmin@content-server-0 ~]$ echo $DM_LDAP_CERT_FILE -- $DM_LDAP_IGNORE_HOSTNAME_CHECK
1 -- 1
[dmadmin@content-server-0 ~]$

For the variables to take effect, you will need to restart the Repositories. I usually set everything up (LDAPs specific pieces + LDAP steps) and only then restart the repositories so it’s done once at the very end of the setup.

The next step is then to create/prepare the Trust Chain. In DA, you can import the Trust Chain one certificate at a time, the Root first and then the Intermediate one. While using “DM_LDAP_CERT_FILE=1” (so without DA), you can unfortunately use only one file per LDAP and therefore this file will need to contain the full Trust Chain. To do that, simply put in a file the content of both Root and Intermediate SSL Certificates one after the other. So in the end, you file should contain something like that:

[dmadmin@content-server-0 ~]$ vi ${dm_location_name}.pem
[dmadmin@content-server-0 ~]$ cat ${dm_location_name}.pem
-----BEGIN CERTIFICATE-----
<<<content_of_root_ca>>>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<<<content_of_intermediate_ca>>>
-----END CERTIFICATE-----
[dmadmin@content-server-0 ~]$
[dmadmin@content-server-0 ~]$ mv ${dm_location_name}.pem ${file_path}
[dmadmin@content-server-0 ~]$

Once you have the file, you can put it wherever you want with the name that you want but it absolutely needs to be a “.pem” extension. You can check this blog, which explains what happens if this isn’t the case and what needs to be done to fix it. As you can see above, I choose to put the file where DA is putting them as well.

The last step for this SSL specific part is then to create a “dm_location” Object that will reference the file that has been created so that the LDAP Server Config Object can use it and trust the target LDAP Server. Contrary to the LDAP Certificate Database Management in DA, which is global to all Repositories (so it needs to be done only one), here you will need to create the “dm_location” Object in all the Repositories that are going to use the LDAP Server. This can be done very easily via iAPI:

[dmadmin@content-server-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
create,c,dm_location
set,c,l,object_name
${dm_location_name}
set,c,l,path_type
file
set,c,l,file_system_path
${file_path}
save,c,l
exit
EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase repo01
[DM_SESSION_I_SESSION_START]info:  "Session 0112d6878000111b started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> ...
3a12d68780002522
API> SET> ...
OK
API> SET> ...
OK
API> SET> ...
OK
API> ...
OK
API> Bye
[dmadmin@content-server-0 ~]$

The name of the “dm_location” Object doesn’t have to be the same as the name of the Trust Chain file. I’m just using the same here so it’s simpler to see the relation between both. These are the only steps that are specific to SSL communications between your LDAP Server and Documentum.

2. Global steps for LDAP

This section applies to all cases. Whether you are trying to setup an LDAP or LDAPs Server, then you will need to create the “dm_ldap_config” Object and everything else described below. As mentioned previously, I’m using one type of LDAP Server for this example (value of “dm_ldap_config.a_application_type“). If you aren’t very familiar with the settings inside the Repository, then the simplest thing to do to find out which parameters you would need (and the associated values) is simply to create one LDAP Config Object using DA. Once done, just dump it and you can re-use that same configuration in the future.

So let’s start with creating a sample LDAP Server Config Object:

[dmadmin@content-server-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
create,c,dm_ldap_config
set,c,l,object_name
${ldap_server_name}
set,c,l,map_attr[0]
user_name
set,c,l,map_attr[1]
user_login_name
set,c,l,map_attr[2]
user_address
set,c,l,map_attr[3]
group_name
set,c,l,map_attr[4]
client_capability
set,c,l,map_attr[5]
user_xprivileges
set,c,l,map_attr[6]
default_folder
set,c,l,map_attr[7]
workflow_disabled
set,c,l,map_val[0]
uniqueDisplayName
set,c,l,map_val[1]
uid
set,c,l,map_val[2]
mail
set,c,l,map_val[3]
cn
set,c,l,map_val[4]
2
set,c,l,map_val[5]
32
set,c,l,map_val[6]
/Home/${uniqueDisplayName}
set,c,l,map_val[7]
false
set,c,l,map_attr_type[0]
dm_user
set,c,l,map_attr_type[1]
dm_user
set,c,l,map_attr_type[2]
dm_user
set,c,l,map_attr_type[3]
dm_group
set,c,l,map_attr_type[4]
dm_user
set,c,l,map_attr_type[5]
dm_user
set,c,l,map_attr_type[6]
dm_user
set,c,l,map_attr_type[7]
dm_user
set,c,l,map_val_type[0]
A
set,c,l,map_val_type[1]
A
set,c,l,map_val_type[2]
A
set,c,l,map_val_type[3]
A
set,c,l,map_val_type[4]
V
set,c,l,map_val_type[5]
V
set,c,l,map_val_type[6]
E
set,c,l,map_val_type[7]
V
set,c,l,ldap_host
${ldap_host}
set,c,l,port_number
${ldap_port}
set,c,l,person_obj_class
${ldap_user_class}
set,c,l,group_obj_class
${ldap_group_class}
set,c,l,per_search_base
${ldap_principal}
set,c,l,grp_search_base
${ldap_principal}
set,c,l,per_search_filter
${ldap_user_filter}
set,c,l,grp_search_filter
${ldap_group_filter}
set,c,l,bind_dn
${ldap_principal}
set,c,l,user_subtype
dm_user
set,c,l,deactivate_user_option
T
set,c,l,import_mode
groups
set,c,l,bind_type
bind_by_dn
set,c,l,ssl_mode
${ldap_ssl}
set,c,l,ssl_port
${ldap_port}
set,c,l,certdb_location
${location}
set,c,l,map_rejection[0]
2
set,c,l,map_rejection[1]
2
set,c,l,map_rejection[2]
0
set,c,l,map_rejection[3]
2
set,c,l,map_rejection[4]
0
set,c,l,map_rejection[5]
0
set,c,l,map_rejection[6]
2
set,c,l,map_rejection[7]
0
set,c,l,retry_count
3
set,c,l,retry_interval
3
set,c,l,failover_use_interval
300
set,c,l,r_is_public
F
set,c,l,a_application_type
netscape
set,c,l,a_full_text
T
save,c,l
exit
EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase repo01
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68780001123 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> ...
0812d6878000252c
API> SET> ...
OK
...
...
...
[dmadmin@content-server-0 ~]$

Once the LDAP Server Config Object has been created, you can register it in the “dm_server_config” Objects. In our silent scripts, we have using the r_object_id of the object just created so that we are sure it is the correct value but below, for simplification, I’m using a select to retrieve the r_object_id based on the LDAP Object Name (so make sure it’s unique if you use the below):

[dmadmin@content-server-0 ~]$ iapi ${repo} -U${USER} -Pxxx << EOF
?,c,update dm_server_config object set ldap_config_id=(select r_object_id from dm_ldap_config where object_name='${ldap_server_name}')
exit
EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase repo01
[DM_SESSION_I_SESSION_START]info:  "Session 0112d6878000112f started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> objects_updated
---------------
              1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "1 objects were affected by your UPDATE statement."

API> Bye
[dmadmin@content-server-0 ~]$

Then, it is time to encrypt the password of the LDAP Account used that is used for the “bind_dn” (${ldap_principal} above):

[dmadmin@content-server-0 ~]$ crypto_docbase=`grep ^dfc.crypto $DOCUMENTUM_SHARED/config/dfc.properties | tail -1 | sed 's,.*=[[:space:]]*,,'`
[dmadmin@content-server-0 ~]$ 
[dmadmin@content-server-0 ~]$ iapi ${crypto_docbase} -U${USER} -Pxxx << EOF
encrypttext,c,${ldap_pwd}
exit
EOF


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase gr_repo
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68880001135 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> ...
DM_ENCR_TEXT_V2=AAAAEHQfx8vF52wIC1Lg8KoxAflW/I7ZnbHwEDJCciKx/thqFZxAvIFNtpsBl6JSGmI4XKYCCuUl/NMY7BTsCa2GeIdUebL2LYfA/nJivzuikqOt::gr_repo
API> Bye
[dmadmin@content-server-0 ~]$

Finally, the only thing left is to create the file “$DOCUMENTUM/dba/config/${repo}/ldap_${dm_ldap_config_id}.cnt” and put in it the content of the encrypted password (the whole line “DM_ENCR_TEXT_V2=…::gr_repo“). As mentioned previously, after a small restart of the Repository, you should then be able to run your dm_LDAPSynchronization job. You might want to configure the job with some specific properties but that’s up to you.

With all the commands above, you have already a very good basis to automate the creation/setup of your LDAP/LDAPs Server without issue. In our automation, instead of printing the result of the iAPI commands to the console, we are usually putting that in a log file. With that, we can automatically retrieve the result of the previous commands and continue the execution based on the outcome so there is no need for any human interaction. In the scope of this blog, it was much more human friendly to display it directly.

Maybe one final note: the above steps are for a Primary Content Server. In case you are trying to do the same thing on a Remote Content Server (RCS/CFS), then some steps aren’t needed. For example, you will need to put the Trust Chain to the correct location but you won’t need to create the “dm_location” or “dm_ldap_config” Objects since they are inside the Repository and therefore already present.

Cet article Documentum – Automatic/Silent creation of LDAP/LDAPs Server Config Objects est apparu en premier sur Blog dbi services.

In a previous blog, I talked about the automation of the LDAP/LDAPs creation. However, the first time that I applied these steps, I actually faced an issue and I couldn’t really get my head around it, at first. This will be a rather short post but I still wanted to share my thoughts because it might avoid you some headache. The issue is only linked to the SSL part of the setup so there is no problem with the basis non-secure LDAP communications.

So, after applying all the steps, everything went fine and I therefore tried to run the dm_LDAPSynchronization job to validate the setup. Doing so, the generated log file wasn’t so great:

[dmadmin@content-server-0 ~]$ cat $DOCUMENTUM/dba/log/repo01/sysadmin/LDAPSynchronizationDoc.txt
LDAPSynchronization Report For DocBase repo01 As Of 2019/09/22 12:45:57

2019-09-22 12:45:56:124 UTC [default task-79]: LDAP Synchronization Started @ Sun Sep 22 12:45:56 UTC 2019
2019-09-22 12:45:56:124 UTC [default task-79]:
2019-09-22 12:45:56:124 UTC [default task-79]: $JMS_HOME/server/DctmServer_MethodServer/deployments/ServerApps.ear/lib/dmldap.jar
2019-09-22 12:45:56:125 UTC [default task-79]: ---------------------------------------------------------------------------------
2019-09-22 12:45:56:125 UTC [default task-79]: Product-Name : Content Server-LDAPSync
2019-09-22 12:45:56:125 UTC [default task-79]: Product-Version : 16.4.0110.0167
2019-09-22 12:45:56:125 UTC [default task-79]: Implementation-Version : 16.4.0110.0167
2019-09-22 12:45:56:125 UTC [default task-79]: ---------------------------------------------------------------------------------
2019-09-22 12:45:56:125 UTC [default task-79]:
2019-09-22 12:45:56:126 UTC [default task-79]: Preparing LDAP Synchronization...
2019-09-22 12:45:57:101 UTC [default task-79]: INFO: Job Status: [LDAP Synchronization Started @ Sun Sep 22 12:45:56 UTC 2019]
2019-09-22 12:45:57:120 UTC [default task-79]: INFO: Job Status updated
2019-09-22 12:45:58:415 UTC [default task-79]: INFO: List of Ldap Configs chosen for Synchronization
2019-09-22 12:45:58:415 UTC [default task-79]: INFO:    >>>0812d6878000252c - Internal_LDAP<<<
2019-09-22 12:45:58:415 UTC [default task-79]: INFO:
2019-09-22 12:45:58:418 UTC [default task-79]:
2019-09-22 12:45:58:418 UTC [default task-79]: ==================================================================================
2019-09-22 12:45:58:420 UTC [default task-79]: Starting Sychronization for ldap config object >>>Internal_LDAP<<< ...
2019-09-22 12:45:58:425 UTC [default task-79]: Unexpected Error. Caused by: [DM_LDAP_SYNC_E_EXCEPTION_ERROR]error:  "Ldap Config Property "certdb_location" has invalid value "ldap_chain"."
2019-09-22 12:45:58:426 UTC [default task-79]: ERROR: DmLdapException:: THREAD: default task-79; MSG: [DM_LDAP_SYNC_E_EXCEPTION_ERROR]error:  "Ldap Config Property "certdb_location" has invalid value "ldap_chain"."; ERRORCODE: 100; NEXT: null
        at com.documentum.ldap.internal.sync.SynchronizationContextBuilder.getCertDbLocation(SynchronizationContextBuilder.java:859)
        at com.documentum.ldap.internal.sync.SynchronizationContextBuilder.setCertDbLocation(SynchronizationContextBuilder.java:225)
        at com.documentum.ldap.internal.sync.SynchronizationContextBuilder.buildSynchronizationContext(SynchronizationContextBuilder.java:49)
        at com.documentum.ldap.LDAPSync.prepareSync(LDAPSync.java:438)
        at com.documentum.ldap.LDAPSync.processJob(LDAPSync.java:238)
        at com.documentum.ldap.LDAPSync.execute(LDAPSync.java:80)
        at com.documentum.mthdservlet.DfMethodRunner.runIt(Unknown Source)
        at com.documentum.mthdservlet.AMethodRunner.runAndReturnStatus(Unknown Source)
        at com.documentum.mthdservlet.DoMethod.invokeMethod(Unknown Source)
        at com.documentum.mthdservlet.DoMethod.doPost(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
        at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
        at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
        at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
        at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
        at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
        at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)
        at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)
        at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
        at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)
        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)
        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)
        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

2019-09-22 12:45:58:426 UTC [default task-79]: WARNING:   **** Skipping Ldap Config Object - Internal_LDAP ****
2019-09-22 12:45:58:775 UTC [default task-79]: Synchronization of ldap config object >>>Internal_LDAP<<< is finished
2019-09-22 12:45:58:775 UTC [default task-79]: ==================================================================================
2019-09-22 12:45:58:775 UTC [default task-79]:
2019-09-22 12:45:58:786 UTC [default task-79]: INFO: Job Status: [dm_LDAPSynchronization Tool had ERRORS at 2019/09/22 12:45:58. Total duration was 2 seconds.View the job's report for details.]
2019-09-22 12:45:58:800 UTC [default task-79]: INFO: Job Status updated
2019-09-22 12:45:58:800 UTC [default task-79]: LDAP Synchronization Ended @ Sun Sep 22 12:45:58 UTC 2019
2019-09-22 12:45:58:800 UTC [default task-79]: Session s2 released successfully
Report End  2019/09/22 12:45:58
[dmadmin@content-server-0 ~]$

After a bunch of checks inside the repository, everything seemed fine. All the objects had the correct content, the correct references, aso… However there was one thing that wasn’t exactly as per the KB6321243 and that was the extension of the Trust Chain file. If you look at the basis of SSL Certificate encodings, then there are two main possibilities: DER (binary = not readable) or PEM (ASCII = readable). In addition to that, you can also have files with CRT or CER extensions but they are always either DER or PEM encoded. The KB asks you to have a PEM encoded SSL Certificate so this file can technically have either a “.pem” or “.cer” or “.crt” extension, that’s almost synonymous. Therefore, here I was, thinking that I could keep my “.crt” extension for the PEM encoded SSL Trust Chain.

To validate that this was the issue, I switched my file to the “.pem” extension and updated the “dm_location” Object:

[dmadmin@content-server-0 ~]$ cd $DOCUMENTUM/dba/secure/ldapdb
[dmadmin@content-server-0 ldapdb]$ mv ldap_chain.crt ldap_chain.pem
[dmadmin@content-server-0 ldapdb]$ 
[dmadmin@content-server-0 ldapdb]$ iapi repo01 -U${USER} -Pxxx


        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0110.0058

Connecting to Server using docbase repo01
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68780001402 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0110.0167  Linux64.Oracle
Session id is s0
API> retrieve,c,dm_location where object_name='ldap_chain'
...
3a12d68780002522
API> get,c,l,file_system_path
...
$DOCUMENTUM/dba/secure/ldapdb/ldap_chain.cer
API> set,c,l,file_system_path
SET> $DOCUMENTUM/dba/secure/ldapdb/ldap_chain.pem
...
OK
API> get,c,l,file_system_path
...
$DOCUMENTUM/dba/secure/ldapdb/ldap_chain.pem
API> save,c,l
...
OK
API> ?,c,UPDATE dm_job OBJECTS set run_now=true, set a_next_invocation=DATE(NOW) WHERE object_name='dm_LDAPSynchronization'
objects_updated
---------------
              1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "1 objects were affected by your UPDATE statement."

API> exit
Bye
[dmadmin@content-server-0 ldapdb]$

With the above, I just changed the extension of the file on the file system and its reference in the “dm_location” Object. The last iAPI command triggered the dm_LDAPSynchronization job to run. Checking the new log file, the issue was indeed solved which confirmed that despite the fact that the Trust Chain was a PEM encoded certificate, it wasn’t enough. There is actually a hardcoded value/check inside Documentum, which forces you to use the “.pem” extension and nothing else:

[dmadmin@content-server-0 ldapdb]$ cat $DOCUMENTUM/dba/log/repo01/sysadmin/LDAPSynchronizationDoc.txt
LDAPSynchronization Report For DocBase repo01 As Of 2019/09/22 13:19:33

2019-09-22 13:19:30:360 UTC [default task-87]: LDAP Synchronization Started @ Sun Sep 22 13:19:30 UTC 2019
2019-09-22 13:19:30:360 UTC [default task-87]:
2019-09-22 13:19:30:361 UTC [default task-87]: $JMS_HOME/server/DctmServer_MethodServer/deployments/ServerApps.ear/lib/dmldap.jar
2019-09-22 13:19:30:367 UTC [default task-87]: ---------------------------------------------------------------------------------
2019-09-22 13:19:30:367 UTC [default task-87]: Product-Name : Content Server-LDAPSync
2019-09-22 13:19:30:367 UTC [default task-87]: Product-Version : 16.4.0110.0167
2019-09-22 13:19:30:367 UTC [default task-87]: Implementation-Version : 16.4.0110.0167
2019-09-22 13:19:30:367 UTC [default task-87]: ---------------------------------------------------------------------------------
2019-09-22 13:19:30:367 UTC [default task-87]:
2019-09-22 13:19:30:370 UTC [default task-87]: Preparing LDAP Synchronization...
2019-09-22 13:19:32:425 UTC [default task-87]: INFO: Job Status: [LDAP Synchronization Started @ Sun Sep 22 13:19:30 UTC 2019]
2019-09-22 13:19:32:453 UTC [default task-87]: INFO: Job Status updated
2019-09-22 13:19:34:292 UTC [default task-87]: INFO: List of Ldap Configs chosen for Synchronization
2019-09-22 13:19:34:292 UTC [default task-87]: INFO:    >>>0812d6878000252c - Internal_LDAP<<<
2019-09-22 13:19:34:292 UTC [default task-87]: INFO:
2019-09-22 13:19:34:294 UTC [default task-87]:
2019-09-22 13:19:34:294 UTC [default task-87]: ==================================================================================
2019-09-22 13:19:34:297 UTC [default task-87]: Starting Sychronization for ldap config object >>>Internal_LDAP<<< ...
2019-09-22 13:19:35:512 UTC [default task-87]: INFO: Directory Type: Sun ONE Directory Server ...
2019-09-22 13:19:35:517 UTC [default task-87]: INFO: Ldap Connection: SSL connection
2019-09-22 13:19:35:517 UTC [default task-87]: INFO: ldap://ldap.domain.com:636
2019-09-22 13:19:35:517 UTC [default task-87]: INFO: {java.naming.provider.url=ldap://ldap.domain.com:636, java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory}
2019-09-22 13:19:35:597 UTC [Thread-91752]: INFO: DM_LDAP_IGNORE_HOSTNAME_CHECK environment variable is enabled.
2019-09-22 13:19:35:598 UTC [Thread-91752]: INFO: Skipping hostname check
2019-09-22 13:19:35:598 UTC [Thread-91752]: INFO: DctmTrustMangaer.checkServerTrusted(): Successfully validated the certificate chain sent from server.
2019-09-22 13:19:35:635 UTC [default task-87]: INFO: DM_LDAP_IGNORE_HOSTNAME_CHECK environment variable is enabled.
2019-09-22 13:19:35:635 UTC [default task-87]: INFO: Skipping hostname check
2019-09-22 13:19:35:635 UTC [default task-87]: INFO: DctmTrustMangaer.checkServerTrusted(): Successfully validated the certificate chain sent from server.
2019-09-22 13:19:35:663 UTC [default task-87]: INFO: LDAP Search Retry: is_child_context = true
2019-09-22 13:19:35:663 UTC [default task-87]: INFO: LDAP Search Retry: Retry count = 1
2019-09-22 13:19:35:665 UTC [default task-87]: Starting the group synchronization...
...
2019-09-22 13:19:35:683 UTC [default task-87]: Group synchronization finished.
2019-09-22 13:19:35:683 UTC [default task-87]:
2019-09-22 13:19:35:683 UTC [default task-87]: INFO: Updating Last Run Time: [20190922131935Z]
2019-09-22 13:19:35:683 UTC [default task-87]: INFO: Updating Last Change No: [null]
2019-09-22 13:19:35:749 UTC [default task-87]: INFO: Ldap Config Object >>>>Internal_LDAP<<<< updated
2019-09-22 13:19:35:751 UTC [default task-87]: Disconnected from LDAP Server successfully.
2019-09-22 13:19:36:250 UTC [default task-87]: Synchronization of ldap config object >>>Internal_LDAP<<< is finished
2019-09-22 13:19:36:250 UTC [default task-87]: ==================================================================================
2019-09-22 13:19:36:250 UTC [default task-87]:
2019-09-22 13:19:36:265 UTC [default task-87]: INFO: Job Status: [dm_LDAPSynchronization Tool Completed with WARNINGS at 2019/09/22 13:19:36. Total duration was 6 seconds.]
2019-09-22 13:19:36:278 UTC [default task-87]: INFO: Job Status updated
2019-09-22 13:19:36:278 UTC [default task-87]: LDAP Synchronization Ended @ Sun Sep 22 13:19:36 UTC 2019
2019-09-22 13:19:36:278 UTC [default task-87]: Session s2 released successfully
Report End  2019/09/22 13:19:36
[dmadmin@content-server-0 ldapdb]$

A pretty annoying design but there is nothing you can do about it. Fortunately, it’s not hard to fix the issue once you know what’s the problem!

Cet article Documentum – LDAP Config Object “certdb_location” has invalid value est apparu en premier sur Blog dbi services.

As you know, there is a lot of vendors for Enterprise Content Management systems (ECM) and they all have different interfaces. Content Management Interoperability Services (CMIS) is a standard for improving interoperability between Enterprise Content Management systems. The vendor-neutral OASIS Web services interface specification published two versions of CMIS v1.0 and v1.1, then the Technical Committee was closed on 09 May 2017 and is no longer active but the standard is still used by almost all CMS products.

What this means?
In fact, you can write applications targeting Alfresco, Documentum, SharePoint and other ECM systems using CMIS without writing one line of a vendor-specific code, so one source code to interoperate with them all… Compare it with JDBC to connect to any database that provides a JDBC driver
This kind of standardization must survive and be constantly up to date, because it enable applications to target different ECM repositories uniformly with a common interface and I think that is really cool.

CMIS Basics

CMIS Repository

At the root of the CMIS model and services is a repository, which is an instance of the content management system and its store of metadata, content, and indexes.

The repository is the end point to which all requests are directed. In the RESTful model, it is the root path of the resources being addressed in CMIS, then it is capable of describing itself and its capabilities.

CMIS Query

A CMIS Query is based upon SQL-92. The query is read-only and presents no data manipulation capabilities!

The syntax consists of the following clauses:

• SELECT with a target list.
• FROM with the object types being queried.
• JOIN to perform a join between object types.
• WHERE with the predicate.
• IN and ANY to query multi-value properties.
• CONTAINS to specify a full-text qualification.
• IN_FOLDER and IN_TREE to search within a folder hierarchy.
• ORDERBY to sort the results.

The CMIS query maps the object type into a relational structure where object type approximates a table, the object approximates a row, and the property approximates a column that can be multi-valued.

CMIS Services

CMIS provides services that you can access using SOAP or AtomPub, and include the following:

• Repository services let you discover available repositories, get the capabilities of these repositories, and provide basic Data Dictionary information of what types are available in the repository.
• Navigation services let you navigate the repository by accessing the folder tree and traversing the folder/child hierarchy.
• Object services provide the basics (Create, Read, Update, Delete) and Control services on any object, including document, folder, policy, and relationship objects. For document objects, this includes setting and getting of properties, policies, and content streams.
• Object services retrieve objects by path or object ID. Applications may also discover what actions users are allowed to perform.
• Multi-filing services let you establish the hierarchy by adding or removing an object to or from a folder.
• Discovery services provide Query and Change services, and a means of paging the results of the query.
• Change services let you discover what content has changed since the last time checked, as specified by a special token. You can use Change services for external search indexing and replication services.
• Versioning services control concurrent operation of the Object services by providing Check In and Check Out services. Version services also provide version histories for objects that are versioned.
• Relationship services let you create, manage, and access relationships or associations between objects as well as allow an application to traverse those associations.
• Policy services apply policies on document objects. Policies are free-form objects and can be used by implementations for security, record, or control policies.
• ACL services let you create, manage, and access Access Control Lists to control who can perform certain operations on an object.

CMIS Bindings

Web Services (SOAP)
This binding is based on the SOAP protocol All services and operations defined in the CMIS domain model specification are present in the Web Services binding. For example, in Alfresco you can get a summary of the CMIS services from the following URL:

http://<hostname>:<port>/alfresco/cmis

AtomPUB (REST)
This RESTful binding is based on the Atom Publishing Protocol. Clients communicate with the repository by requesting the service document, which is obtained through a well-known URI. For example, in Alfresco the service document is at:

http://<hostname>:<port>/alfresco/api/-default-/public/cmis/versions/1.1/atom

Personnaly, I used AtomPUB which works very well, and I think is being the most performant and the most popular. But if you’ve ever looked at the XML that comes back from a CMIS AtomPub call you know how verbose it can be!

CMIS v1.0 vs v1.1

CMIS 1.1 has some exciting new features comparing to v1.0. I will not list all v1.1 news, but you can find below the two important news (for me):

Browser Binding

The Browser Binding is based on JSON, so the payloads that go between client and server are smaller, making it the fastest of the three bindings.
The original purpose of the Browser Binding was to make it easy for those building “single page” web apps, but I think apps of all types will move to the Browser Binding because it is easier to work with.

Type Mutability

This allows CMIS developers to create and update the repository’s content model with code. Sometimes you will need content types to store objects with metadata specific to your solution.
Before CMIS 1.1, you have to ship repository-specific content models and configuration instructions with your app.
With CMIS 1.1, developers can simply write install and config logic as part of the solution that will interrogate the repository to determine if any changes need to be made to the content model to support the solution.

So, for the moment we have CMIS 1.0 and 1.1, which is cool but as you know things are moving fast and the CMIS need to follow… Do you think that we will see a CMIS 2.0 soon?

Cet article CMIS – What importance does it have in ECM World? est apparu en premier sur Blog dbi services.

In a previous blog I talked about CMIS in general and showed you what importance it does have in ECM World. In fact, I already experienced CMIS in a project with Documentum and Alfresco involved! The goal was to make the same operations on both ECM from a single interface, so CMIS was the solution. In this blog, I will talk about CMIS with Documentum.

Preparation

Is CMIS deployed on Content Server by default?

Old versions (6.7 and older) has the CMIS on the Content Server, which is not the case of later versions. So, having CS 7.3 I had to follow the below procedure to deploy and configure CMIS.

Application archive

To deploy CMIS, you will need to have one of the following files:

• emc-cmis.war: For Apache Tomcat, JBoss
• emc-cmis-weblogic.ear: For WebLogic

I must confess that I didn’t find these archives on OpenText website, so I had to ask the OpenText support to share the WebLogic one.

JVM Configuration

You have to provide adequate heap space and PermGen space for the CMIS application, for my test case I used the following JVM settings:

• -Xms512m
• -Xms512m
• -XX:MaxPermSize=128m

The JVM settings are very important for a CMIS smooth running, like any other application.

Using urandom generators on Linux systems

In fact, there are some issues with implementation of peudo-random number generators on Linux. For more efficient randomization, Linux systems should use urandom generators that are faster but less secure!
To change the source of secure random numbers from random to urandom, set the system property as follows:

-Djava.security.egd=file:///dev/urandom

For my test case, I set this option in WebLogic start script.

DFC Configuration file

Like any other client (DA, D2, D2-Config), you have to configure the DFC in the dfc.properties file, it provides property settings for the Documentum Foundation Classes. This file is located in APP-INF/classes if you are deploying an EAR file, or in WEB-INF/classes if you are deploying a WAR file.

You can find below the most important properties to adapt:

• dfc.docbroker.host: The fully qualified hostname for the Docbroker.
• dfc.docbroker.port: 1489 by default, specify it if you want to use another port.
• dfc.globalregistry.repository: The name of the Global Registry.
• dfc.globalregistry.username: dm_bof_registry by default.
• dfc.globalregistry.password: An encrypted password for the global registry user.

Deployment

On WebLogic

Having the emc-cmis-weblogic.ear configured as explained before, you have only one thing to do before you deploy the CMIS application:

Authentication configuration
In fact, for WebLogic Server, client requests that use HTTP BASIC authentication must pass WebLogic Server authentication, even if access control is not enabled on the target resource. The setting of the Security Configuration enforce-valid-basic-auth-credentials determines this behavior. It specifies whether or not the system should allow requests with invalid HTTP BASIC authentication credentials to access unsecured resources.
So, you have tow solutions:

• Disable the WebLogic HTTP Basic authentication:
  I used it for my test case to write this Blog, but is NOT recommended! You have to be careful with this setting as this affects the complete domain. If you explicitly set the enforce-valid-basic-auth-credentials flag to false, WebLogic Server does not perform authentication for HTTP BASIC authentication client requests which is dangerous from a security view.
• Create a user (WebLogic or LDAP) and use it to connect:
  This solution is highly recommended, don’t hesitate to ask if you need more information regarding how to do it. I will be happy to share with you what we did in our project.

As said, for my test case I disabled the WebLogic HTTP Basic authentication, so I edited the following WebLogic config.xml file directly:
$WLS_HOME/domains/$DOMAIN_NAME/config/config.xml

In the section, if the enforce-valid-basic-auth-credentials is already defined then change its value to false. Otherwise, add the following line at the end of the section:

<enforce-valid-basic-auth-credentials>false</enforce-valid-basic-auth-credentials>

Once done restart the WebLogic Server, then deploy the CMIS application using the WebLogic Console.

Post Deployment

On successful deployment you should be able to access the CMIS using below URL:

http://<host>:<port>/<contextPath>

The application context path will vary depending on your deployment.

RESTful AtomPub service
The service defining the RESTful AtomPub can be obteined from this URL:

http://<host>:<port>/<contextPath>/resources

Or you can click on the button “RESTful AtomPub Binding”

Web service
You can view WSDL for the repository service via the URL:

http://<host>:<port>/<contextPath>/services/RepositoryService?wsdl

Or you can click on the button “Web Services Binding”

Welcome to the CMIS world with Documentum, test it you will see what can it be useful if you want to have common interface. Any questions? As usual, feel free to ask.

Cet article CMIS – Documentum – How’s it going on? est apparu en premier sur Blog dbi services.

In the scope of an upgrade project, with some colleagues, we have been deploying some D2-REST applications on Kubernetes pods using WebLogic Server. At the beginning, we started using D2-REST 16.4 and that was working properly (once the issue described here is fixed (and some others linked to FIPS 140-2, aso…)). After that, we tried to switch to higher versions (16.5.0 Pxx, 16.5.1 P00 or P04) but it stopped working with some error. We were able to replicate the issue with WebLogic Server 12.2.1.3 and 12.2.1.4 so it’s not just specific to one small use case but it seems more global to the D2-REST 16.5.x versions on WebLogic. It might impact other Application Servers as well, that would need some testing.

Upon accessing the D2-REST URL (E.g.: https://lb_url/D2-REST), the service seemed to be working but while going further on the product information page for example (E.g.: https://lb_url/D2-REST/product-info), then the following error was always displayed:

<error xmlns="http://identifiers.emc.com/vocab/documentum" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <status>500</status>
  <code>E_INTERNAL_SERVER_ERROR</code>
  <message>An internal server error occurs.</message>
  <details>
    org.owasp.esapi.reference.DefaultSecurityConfiguration cannot be cast to com.emc.d2.web.security.D2SecurityConfiguration
  </details>
  <id>51872a76-g47f-4d6e-9d47-e9fa5d8c1291</id>
</error>

The error generated on the D2-REST logs at that time was:

java.lang.ClassCastException: org.owasp.esapi.reference.DefaultSecurityConfiguration cannot be cast to com.emc.d2.web.security.D2SecurityConfiguration
	at com.emc.d2.web.security.D2HttpUtilities.getHeader(D2HttpUtilities.java:40)
	at com.emc.documentum.d2.rest.filter.AppInfoFilter.getRemoteAddr(AppInfoFilter.java:82)
	at com.emc.documentum.d2.rest.filter.AppInfoFilter.doFilter(AppInfoFilter.java:36)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.emc.documentum.rest.security.filter.RepositoryNamingFilter.doFilter(RepositoryNamingFilter.java:40)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.emc.documentum.rest.filter.RestCorsFilter.doFilterInternal(RestCorsFilter.java:47)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.emc.documentum.rest.filter.CompressionFilter.doFilter(CompressionFilter.java:73)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.emc.documentum.rest.log.MessageLoggingFilter.doFilter(MessageLoggingFilter.java:69)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.emc.documentum.rest.security.filter.ExceptionHandlerFilter.doFilterInternal(ExceptionHandlerFilter.java:31)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3797)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3763)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
	at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
	at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
	at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
	at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2451)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2299)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2277)
	at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1720)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1680)

Since we couldn’t find anything obvious, we opened an OpenText Support case (#4322241). There is a KB (KB14050670) around Internal Server Error but it didn’t help us in this case. After some research on OpenText side, it seems that this is a known issue and there is a solution for it but it is not documented at the moment: that’s the whole purpose of this blog. The thing is that the solution is going to be in the next version of the D2FS REST Services Development Guide and therefore if you are looking into the OpenText Support Site, you won’t find anything related to this error yet. Don’t ask me why it will be in the Development guide, maybe they didn’t find another suitable location.

So the solution is very simple, you just have to add a small piece into the D2-REST web.xml file:

[weblogic@wsd2rest-0 ~]$ cd $APPLICATIONS
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ jar -xvf D2-REST.war WEB-INF/web.xml
 inflated: WEB-INF/web.xml
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ cat WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
         http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0"
         metadata-complete="true">

  <display-name>D2-REST</display-name>
  <description>D2-REST</description>
  <error-page>
    <error-code>404</error-code>
    <location>/errors/redirect/404</location>
  </error-page>
  <error-page>
    <error-code>500</error-code>
    <location>/errors/redirect/500</location>
  </error-page>
</web-app>
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ sed -i 's,</web-app>,  <listener>\n&,' WEB-INF/web.xml
[weblogic@wsd2rest-0 dbi]$ sed -i 's,</web-app>,    <listener-class>com.emc.d2.rest.context.WebAppContextListener</listener-class>\n&,' WEB-INF/web.xml
[weblogic@wsd2rest-0 dbi]$ sed -i 's,</web-app>,  </listener>\n&,' WEB-INF/web.xml
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ cat WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
         http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0"
         metadata-complete="true">

  <display-name>D2-REST</display-name>
  <description>D2-REST</description>
  <error-page>
    <error-code>404</error-code>
    <location>/errors/redirect/404</location>
  </error-page>
  <error-page>
    <error-code>500</error-code>
    <location>/errors/redirect/500</location>
  </error-page>
  <listener>
    <listener-class>com.emc.d2.rest.context.WebAppContextListener</listener-class>
  </listener>
</web-app>
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ jar -uvf D2-REST.war WEB-INF/web.xml
adding: WEB-INF/web.xml(in = 753) (out= 326)(deflated 56%)
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ rm -rf WEB-INF/
[weblogic@wsd2rest-0 dbi]$

As you can see above, it’s all about adding a new listener into the web.xml file for the “WebAppContextListener“. This class – based on its name – has absolutely nothing to do with the error shown above and yet, adding this listener will solve the cast issue. So just redeploy/update your Application in WebLogic and that’s it, the issue should be gone.

Cet article Documentum – Cast trouble with D2-REST 16.5.x on WebLogic est apparu en premier sur Blog dbi services.

In the scope of an upgrade project, with some colleagues, we have been deploying some D2-REST applications on Kubernetes pods using WebLogic Server. As explained in a previous blog, we first tried to upgrade our D2-REST 4.x into 16.4 but faced a small error. I don’t know if you already used/deployed D2-REST but it seems to me that the deployment is always kinda chaotic. Sometimes you will need to apply some steps and then for the next version it’s not needed anymore but later it will be needed again, aso… So in the end, we always try to deploy the OOTB with some small improvements and whenever we face an error, we try to fix it for this specific version and this version only. Never assume that a fix for a version is good for all versions.

Below, I will be using some scripts and properties files that are present in this “dbi_resources” folder: it’s some utilities and stuff that we are using for automation and to simplify our lives. So we tried to deploy the D2-REST 16.4 on our WebLogic Servers 12.2.1.x:

[weblogic@wsd2rest-0 ~]$ cd $APPLICATIONS
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ wlst_cmd="$ORACLE_HOME/oracle_common/common/bin/wlst.sh"
[weblogic@wsd2rest-0 dbi]$ wlst_script="${dbi_resources}/manageApplication.wls"
[weblogic@wsd2rest-0 dbi]$ domain_prop="${dbi_resources}/domain.properties"
[weblogic@wsd2rest-0 dbi]$ deploy_prop="${dbi_resources}/D2-REST.deploy"
[weblogic@wsd2rest-0 dbi]$ undeploy_prop="${dbi_resources}/D2-REST.undeploy"
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ ${wlst_cmd} ${wlst_script} ${domain_prop} ${deploy_prop}

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

>>> Loaded the properties file: ${dbi_resources}/domain.properties
>>> Loaded the properties file: ${dbi_resources}/D2-REST.deploy
>>> Connected to the AdminServer.
>>> Edit Session started.

<Dec 11, 2019 4:49:19 PM UTC> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, D2-REST [archive: $APPLICATIONS/D2-REST.war], to msD2-REST-02 msD2-REST-01 .>
ERROR... check error messages for cause.
Error occurred while performing activate : Error while Activating changes. : java.lang.NoSuchMethodError: org.apache.log4j.Logger.setResourceBundle(Ljava/util/ResourceBundle;)V
Use dumpStack() to view the full stacktrace :
Problem invoking WLST - Traceback (innermost last):
  File "${dbi_resources}/manageApplication.wls", line 77, in ?
  File "<iostream>", line 569, in stopEdit
  File "<iostream>", line 553, in raiseWLSTException
WLSTException: Error occurred while performing stopEdit : Cannot call stopEdit without an edit session in progress

[weblogic@wsd2rest-0 dbi]$

At this point, the application has been deployed but it cannot be started properly. It will therefore be stuck in “New” status on the WebLogic side. On the D2-REST log file, the error message looks like this:

2019-12-11 16:49:57,112 UTC [ERROR] ([ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)') - o.springframework.web.context.ContextLoader   : Context initialization failed
java.lang.NoSuchMethodError: org.apache.log4j.Logger.setResourceBundle(Ljava/util/ResourceBundle;)V
        at com.documentum.fc.common.DfLogger.<clinit>(DfLogger.java:622) ~[dfc-16.4.jar:na]
        at com.documentum.fc.common.impl.logging.LoggingConfigurator.onPreferencesInitialized(LoggingConfigurator.java:178) ~[dfc-16.4.jar:na]
        at com.documentum.fc.common.DfPreferences.initialize(DfPreferences.java:71) ~[dfc-16.4.jar:na]
        at com.documentum.fc.common.DfPreferences.getInstance(DfPreferences.java:43) ~[dfc-16.4.jar:na]
        at com.documentum.fc.client.DfSimpleDbor.getDefaultDbor(DfSimpleDbor.java:78) ~[dfc-16.4.jar:na]
        at com.documentum.fc.client.DfSimpleDbor.<init>(DfSimpleDbor.java:66) ~[dfc-16.4.jar:na]
        at com.documentum.fc.client.DfClient$ClientImpl.<init>(DfClient.java:350) ~[dfc-16.4.jar:na]
        at com.documentum.fc.client.DfClient.<clinit>(DfClient.java:766) ~[dfc-16.4.jar:na]
        at com.emc.documentum.rest.context.WebAppContextInitializer.getDfcVersion(WebAppContextInitializer.java:104) ~[_wl_cls_gen.jar:na]
        at com.emc.documentum.rest.context.WebAppContextInitializer.collectInfo(WebAppContextInitializer.java:81) ~[_wl_cls_gen.jar:na]
        at com.emc.documentum.rest.context.WebAppContextInitializer.preloadAppEnvironment(WebAppContextInitializer.java:67) ~[_wl_cls_gen.jar:na]
        at com.emc.documentum.rest.context.WebAppContextInitializer.initialize(WebAppContextInitializer.java:38) ~[_wl_cls_gen.jar:na]
        at com.emc.documentum.rest.context.WebAppContextInitializer.initialize(WebAppContextInitializer.java:31) ~[_wl_cls_gen.jar:na]
        at org.springframework.web.context.ContextLoader.customizeContext(ContextLoader.java:482) ~[spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:442) ~[spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325) ~[spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) [spring-web-4.3.10.RELEASE.jar:4.3.10.RELEASE]
        at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:705) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:328) [com.oracle.weblogic.security.subject.jar:12.2.1.3]
        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197) [com.oracle.weblogic.security.subject.jar:12.2.1.3]
        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.EventsManager.executeContextListener(EventsManager.java:251) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:204) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:192) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1921) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3106) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1843) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:884) [com.oracle.weblogic.servlet.jar:12.2.1.3]
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:360) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:356) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:138) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:233) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:228) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:78) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:52) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:752) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:45) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:262) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:52) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165) [com.oracle.weblogic.application.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:90) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:631) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:171) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:121) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:348) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:907) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1468) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:459) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:181) [com.oracle.weblogic.deploy.jar:12.2.1.3]
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:217) [com.oracle.weblogic.deploy.service.jar:12.2.1.3]
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:14) [com.oracle.weblogic.deploy.service.jar:12.2.1.3]
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:69) [com.oracle.weblogic.deploy.service.jar:12.2.1.3]
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:678) [com.bea.core.weblogic.workmanager.jar:12.2.1.3]
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352) [com.bea.core.utils.full.jar:12.2.1.3]
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337) [com.bea.core.utils.full.jar:12.2.1.3]
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57) [com.oracle.weblogic.work.jar:12.2.1.3]
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41) [com.bea.core.weblogic.workmanager.jar:12.2.1.3]
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652) [com.bea.core.weblogic.workmanager.jar:12.2.1.3]
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420) [com.bea.core.weblogic.workmanager.jar:12.2.1.3]
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:360) [com.bea.core.weblogic.workmanager.jar:12.2.1.3]

The solution is quite simple, it’s just a conflict for the log4j jars that comes with the OOTB war file provided by OpenText… So you just need to undeploy the application, remove the conflict and redeploy it afterwards. If you are facing the error above, then it’s linked to the “log4j-over-slf4j” jar file and you can solve it like that:

[weblogic@wsd2rest-0 dbi]$ ${wlst_cmd} ${wlst_script} ${domain_prop} ${undeploy_prop}

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

>>> Loaded the properties file: ${dbi_resources}/domain.properties
>>> Loaded the properties file: ${dbi_resources}/D2-REST.undeploy
>>> Connected to the AdminServer.
>>> Edit Session started.

<Dec 11, 2019 4:54:46 PM UTC> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating undeploy operation for application, D2-REST [archive: null], to msD2-REST-01 msD2-REST-02 .>

Current Status of your Deployment:
Deployment command type: undeploy
Deployment State : completed
Deployment Message : no message
None

>>> Execution completed.
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ jar -tvf D2-REST.war | grep "WEB-INF/lib/log4j"
481535 Tue Jan 05 05:02:00 UTC 2016 WEB-INF/lib/log4j-1.2.16.jar
 12359 Mon Dec 12 03:29:02 UTC 2016 WEB-INF/lib/log4j-over-slf4j-1.6.1.jar
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ zip -d D2-REST.war WEB-INF/lib/log4j-over-slf4j*
deleting: WEB-INF/lib/log4j-over-slf4j-1.6.1.jar
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$
[weblogic@wsd2rest-0 dbi]$ ${wlst_cmd} ${wlst_script} ${domain_prop} ${deploy_prop}

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

>>> Loaded the properties file: ${dbi_resources}/domain.properties
>>> Loaded the properties file: ${dbi_resources}/D2-REST.deploy
>>> Connected to the AdminServer.
>>> Edit Session started.

<Dec 11, 2019 4:56:05 PM UTC> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, D2-REST [archive: $APPLICATIONS/D2-REST.war], to msD2-REST-02 msD2-REST-01 .>

Current Status of your Deployment:
Deployment command type: deploy
Deployment State : completed
Deployment Message : no message
None

>>> Execution completed.
[weblogic@wsd2rest-0 dbi]$

As you can see above, the D2-REST 16.4 is now successfully deployed. You can access it and work with it without any issues.

[weblogic@wsd2rest-0 dbi]$ curl -s -k https://lb_url/D2-REST/product-info | python -mjson.tool
{
    "links": [
        {
            "href": "https://lb_url/D2-REST/product-info",
            "rel": "self"
        }
    ],
    "name": "documentum-rest-services-product-info",
    "properties": {
        "build_number": "0511",
        "major": "16.4",
        "minor": "0000",
        "product": "Documentum D2 REST Services",
        "product_version": "16.4.0000.0511",
        "revision_number": "NA"
    }
}
[weblogic@wsd2rest-0 dbi]$

Cet article Documentum – NoSuchMethodError on setResourceBundle with D2-REST 16.4 on WebLogic est apparu en premier sur Blog dbi services.

You have finally completed that migration and need to restart all the Documentum processes. So, you shut down the docbroker and move on to the repositories but then you receive an error message about them not being reachable any more. Or conversely, you want to start all the Documentum processes and you start first the repositories and later the docbrokers. Next, you want to connect to one repository and you receive the same error message. Of course, you finally remember, since the docbroker is a requirement for the repositories, it must be started first and shut down last but it is too late now. How to get out if this annoying situation ? You could just (re)start the docbroker and wait for the next repostories’ checkpoint, at most 5 minutes by default. If this is not acceptable, at first sight, there is no other solution than to “kill -9” the repositories’ processes, start the docbroker and only next the repositories. Let’s see if we can find a better way. Spoiler alert: to stop this insufferable suspens, I must say up front that there is no other way, sorry, but there are a few ways to alleviate this inconvenience.

A quick clarification

Let’s first clarify a point of terminology here: there is a difference between docbases/repositories and content servers. A docbase encompasses the actual content and their persistent data and technical information whereas the content server is the set of running processes that give access to and manage one docbase. It is very similar to Oracle’s databases and instances, where one database can be served by several instances, providing parallelism and high availability. A docbase can be served by more than one content server, generally spread over different machines, with its own set of dm_start_docbase and dm_shutdown_docbase scripts and server.ini. A docbase knows how many content servers use it because they each have their own dm_server_config object. If there is just one content server, both docbase and content server can be used interchangeably but when there are several content servers for the same docbase, when one says “stopping the docbase” it really means “stopping one particular content server”, and this is the meaning that will be used in the rest of the article. If the docbase has more than one content servers, just extend the presented manipulations to each of them.

Connecting to the repositories without a docbroker

If one could connect to a repository without a running docbroker, the situation that triggered this article, things would be much easier. In the ancient, simpler times, the dmcl.ini parameters below could help working around an unavailable docbroker:

[DOCBROKER_DEBUG]
docbase_id = <id of docbase as specified in its server.ini file>
host =  <host's name the docbase server is running on>
port = <docbase's port as specified in /etc/services>
service = <docbase's service name as specified in /etc/services>

and they used to work.
After the switch to the dfc.properties file, those parameters were renamed as follows:

dfc.docbroker.debug.docbase_id=<id of docbase as specified in its server.ini file>
dfc.docbroker.debug.host=<host's name the docbase server is running on>
dfc.docbroker.debug.port=<docbase's port as specified in /etc/services>
dfc.docbroker.debug.service=<docbase's service name as specified in /etc/services>

Unfortunately, they don’t work any more. Actually, although they are still documented in the dfcfull.properties, they have not been implemented and will never be according to OTX. Moreover, they will be removed in the future. Too bad, that would have been such a cheap way to extricate oneself from an uncomfortable situation.

Preventing the situation

The best solution is obviously to prevent it to happen. This can be easily realized by using a central script for stopping and starting the Documentum stack. And, while we are at it, inquiring its status.
Documentum already provides such a script, e.g. see here Linux scripts for automatic startup and shutdown of Documentum Content Server. Here is another more sophisticated implementation:

#!/bin/bash
#
# See Usage() function below for explanations; 
# cec - dbi-services - April 2019
#

general_status=0

Usage() {
   cat <<EoU
Usage:
    start-stop.sh [(help) | start | stop | status] [(all) | docbases | docbrokers | docbase={,} | docbroker={,} | method_server]
 E.g.:
    display this help screen:
       start-stop.sh
    start all:
       start-stop.sh start [all]
    stop all:
       start-stop.sh stop [all]
    status all:
       start-stop.sh status [all]
    start docbroker01:
       start-stop.sh start docbroker=docbroker01
    start docbases global_registry and dmtest01:
       start-stop.sh docbase=global_registry,dmtest01
    start all the docbases:
       start-stop.sh docbases
    start all the docbrokers:
       start-stop.sh docbrokers
EoU
}

start_docbroker() {
   docbroker=$1
   echo "starting up docbroker $docbroker ..."
   ./dm_launch_${docbroker}
}

start_all_docbrokers() {
   echo "starting the docbrokers ..."
   DOCBROKERS=`ls -1 dm_launch_* 2>/dev/null | cut -f3 -d_`
   nb_items=0
   for docbroker in $DOCBROKERS; do
      start_docbroker $docbroker
      (( nb_items++ ))
   done
   echo "$nb_items docbrokers started"

}

start_docbase() {
   docbase=$1
   echo "starting $docbase"
   ./dm_start_${docbase}
}

start_all_docbases() {
   echo "starting the repositories ..."
   DOCBASES=`ls -1 config 2>/dev/null `
   nb_items=0
   for docbase in $DOCBASES; do
      start_docbase $docbase
      (( nb_items++ ))
   done
   echo "$nb_items repositories started"
}

start_method_server() {
   echo "starting the method server ..."
   cd ${DOCUMENTUM}/${JBOSS}/server
   nohup ${DOCUMENTUM}/${JBOSS}/server/startMethodServer.sh 2>&1 > /tmp/nohup.log &
   echo "method server started"
}

start_all() {
   echo "starting all the documentum processes ..."
   start_all_docbrokers
   start_all_docbases
   start_method_server
}

status_docbroker() {
   docbroker_name=$1
   docbroker_host=$(grep "^host=" /app/dctm/dba/dm_launch_${docbroker_name} | cut -d= -f2)
   docbroker_port=$(grep "dmdocbroker -port " /app/dctm/dba/dm_launch_${docbroker_name} | cut -d\  -f3)
   dmqdocbroker -t $docbroker_host -p $docbroker_port -c ping 2> /dev/null 1> /dev/null
   local_status=$?
   if [ $local_status -eq 0 ]; then
      echo "$(date +"%Y/%m/%d %H:%M:%S"): successfully pinged docbroker $docbroker_name listening on port $docbroker_port on host $docbroker_host"
   else
      echo "$(date +"%Y/%m/%d %H:%M:%S"): docbroker $docbroker_name listening on port $docbroker_port on host $docbroker_host is unhealthy"
      general_status=1
   fi
   echo "status for docbroker $docbroker_name:$docbroker_port: $local_status, i.e. $(if [[ $local_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

status_all_docbrokers() {
   DOCBROKERS=`ls -1 dm_launch_* 2>/dev/null | cut -f3 -d_`
   DOCBROKERS_PORTS=`grep -h "./dmdocbroker" dm_launch_* | cut -f3 -d\ `
   for f in `ls -1 dm_launch_* 2>/dev/null `; do
      docbroker_name=`echo $f | cut -f3 -d_`
      docbroker_port=`grep "./dmdocbroker" $f | cut -f3 -d\ `
      status_docbroker $docbroker_name $docbroker_port
   done
   echo "general status for all docbrokers: $general_status, i.e. $(if [[ $general_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

status_docbase() {
   docbase=$1
   timeout --preserve-status 30s idql $docbase -Udmadmin -Pxx 2> /dev/null 1> /dev/null <<eoq
     quit
eoq
   local_status=$?
   if [[ $local_status -eq 0 ]]; then
      echo "$(date +"%Y/%m/%d %H:%M:%S"): successful connection to repository $docbase"
   else
      echo "$(date +"%Y/%m/%d %H:%M:%S"): repository $docbase is unhealthy"
      general_status=1
   fi
   echo "status for docbase $docbase: $local_status, i.e. $(if [[ $local_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

status_all_docbases() {
   DOCBASES=`ls -1 config 2>/dev/null `
   for docbase in $DOCBASES; do
      status_docbase $docbase
   done
   echo "general status for all docbases: $general_status, i.e. $(if [[ $general_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

status_method_server() {
   # check the method server;
   curl --silent --fail -k http://${HOSTNAME}:9080/DmMethods/servlet/DoMethod 2>&1 > /dev/null
   local_status=$?
   if [ $local_status -eq 0 ]; then
      echo "$(date +"%Y/%m/%d %H:%M:%S"): method server successfully contacted"
   else
      echo "$(date +"%Y/%m/%d %H:%M:%S"): method server is unhealthy"
      general_status=1
   fi
   echo "status for method_server: $local_status, i.e. $(if [[ $local_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

status_all() {
   status_all_docbrokers
   status_all_docbases
   status_method_server
   echo "General status: $general_status, i.e. $(if [[ $general_status -eq 0 ]]; then echo OK; else echo NOK;fi)"
}

stop_docbase() {
   echo "stopping $docbase"
   docbase=$1
   ./dm_shutdown_${docbase}
   echo "docbase $docbase stopped"
}

stop_all_docbases() {
   echo "stopping the repositories ..."
   DOCBASES=`ls -1 config 2>/dev/null `
   nb_items=0
   for docbase in $DOCBASES; do
      stop_docbase $docbase
      (( nb_items++ ))
   done
   echo "$nb_items repositories stopped"
}

stop_docbroker() {
   echo "stopping docbroker $docbroker ..."
   docbroker=$1
   ./dm_stop_${docbroker}
   echo "docbroker $docbroker stopped"
}

stop_all_docbrokers() {
   echo "stopping the docbrokers ..."
   DOCBROKERS=`ls -1 dm_stop_* 2>/dev/null | cut -f3 -d_`
   nb_items=0
   for docbroker in $DOCBROKERS; do
      stop_docbroker $docbroker
      (( nb_items++ ))
   done
   echo "$nb_items docbrokers stopped"
}

stop_method_server() {
   echo "stopping the method server ..."
   ${DOCUMENTUM}/${JBOSS}/server/stopMethodServer.sh
   echo "method server stopped"
}

stop_all() {
   echo "stopping all the documentum processes ..."
   stop_all_docbases
   stop_method_server
   stop_all_docbrokers
   echo "all documentum processes stopped"
   ps -ajxf | egrep '(PPID|doc|java)' | grep -v grep | sort -n -k2,2
}

# -----------
# main;
# -----------
   [[ -f ${DM_HOME}/bin/dm_set_server_env.sh ]] && . ${DM_HOME}/bin/dm_set_server_env.sh
   cd ${DOCUMENTUM}/dba
   if [[ $# -eq 0 ]]; then
      Usage
      exit 0
   else
      while [[ $# -ge 1 ]]; do
         case $1 in
	    help)
	       Usage
	       exit 0
	    ;;
            start|stop|status)
	       cmd=$1
	       shift
	       if [[ -z $1 || $1 = "all" ]]; then
	          ${cmd}_all
	       elif [[ $1 = "docbases" ]]; then
	          ${cmd}_all_docbases
	       elif [[ $1 = "docbrokers" ]]; then
	          ${cmd}_all_docbrokers
	       elif [[ ${1%%=*} = "docbase" ]]; then
	          docbases=`echo ${1##*=} | gawk '{gsub(/,/, " "); print}'`
                  for docbase in $docbases; do
	             ${cmd}_docbase $docbase
	          done
	       elif [[ ${1%%=*} = "docbroker" ]]; then
	          docbrokers=`echo ${1##*=} | gawk '{gsub(/,/, " "); print}'`
                  for docbroker in $docbrokers; do
	             ${cmd}_docbroker $docbroker
	          done
	       elif [[ $1 = "method_server" ]]; then
                  ${cmd}_method_server
               fi
               exit $general_status
            ;;
            *)
               echo "syntax error"
	       Usage
	       exit 1
	    ;;
         esac
         shift
      done
   fi

See lines 11 to 29 for its usage.
Note on line 110 the timeout command when attempting to connect to a docbase to check its status; see the article Adding a timeout in monitoring probes for an explanation.
We couldn’t help but adding the option to address each component individually, or a few of them, in addition to all of them at once. So, the script lets us stop, start and inquire the status of one particular docbroker or docbase or method server, or a list of docbrokers or a list of docbases, or everything at once.
After a maintenance task, to stop all the Documentum processes, the command below could be used:

$ start-stop.sh stop all

Similarly, to start everything:

$ start-stop.sh start all

Thus, the proper order is guaranteed to be used and human error is prevented. By standardizing on such script and using it as shown, the aforementioned problem won’t occur anymore.

That is fine but if we didn’t use the script and find ourselves in the situation where no docbroker is running and we must shut down the repositories, is there a way to do it easily and cleanly ? Well, easily, certainly, but cleanly, no. Please, continue reading on Part II.

Cet article A Ruthless Repository Shutdown Utility, Part I est apparu en premier sur Blog dbi services.

Stopping the unreachable repositories

Suppose that the docbroker has been stopped prematurely and that we want to shut down the repositories but the out-of-the-box dm_shutdown_repository is not effective. Why is it so by the way ? If we look closely inside the shutdown script, we quickly notice the reason:

#!/bin/sh
################## DOCUMENTUM SERVER SHUTDOWN FILE ######################
#
# 1994-2018 OpenText Corporation. All rights reserved
# Version 16.4 of the Documentum Server.
#
# A generated server shutdown script for a repository.
# This file was generated on Fri Aug 30 12:15:10 CEST 2019 by user dmadmin.
#
check_connect_status() {
status=$1
if [ ! $status = 0 ] ; then
  cat <<-END
  ***** $0: ERROR
  ***** Unable to complete shutdown - unable to connect
  ***** to the server to issue $2 request.
END
  exit 1
fi
}
...
# Stop the server
echo Stopping Documentum server for repository: [dmtestgr02]
echo ''
DM_DMADMIN_USER=dmadmin
#
# Get the pid for the root process
#
DM_PID=`./iapi dmtestgr02 -U$DM_DMADMIN_USER -P -e << EOF  | grep 'root_pid' | sed -e 's/ .*[: A-Za-z]//'
apply,s0,NULL,LIST_SESSIONS
next,s0,q0
dump,s0,q0
exit
EOF`
status=$?
check_connect_status $status LIST_SESSIONS
...
            kill -9 $child_pid
...
  kill -9 $DM_PID
...
         kill -9 $child_pid
...

On line 29, the shutdown script first attempts to connect to the repository in order to retrieve the root pid of the server processes. On line 36, this attempt’s result is checked by the function check_connect_status defined earlier in the script at line 10. If something went wrong during the connection, iapi’s return status will be != 0 and check_connect_status will simply exit the script on line 18. So, if a repository has gone berserk, or no free sessions are available, or the docbroker is unreachable, the script will not be able to stop it. That logic is quite restrictive and we must fall back to killing the repository’s processes ourselves anyway.
Strangely enough, the script is not scared of killing processes, it does this from several places, but it rather looks like it is a bit shy in identifying the right ones and therefore relies on the server itself or, ultimately, on the user, for help in this area.
Admittedly, it is not always easy to pinpoint the right processes from the list returned by the command ps, especially if the repository is running in HA on the same machine, or if several repositories share the same machine, so extra care must be used in order not to kill the wrong ones. The dm_shutdown_docbase avoids this difficulty altogether by asking the content server (aka CS) its root pid and that is why it aborts if it cannot contact it.
Historically, the “kill” command could only “kill -9” (SIGKILL, forceful, coercive kill) but nowadays it has been generalized to send signals and could just as well have been forked to “signal” or “send”. So, can a signal be sent to the main executable ${DM_HOME}/bin/documentum to ask it to cleanly shut down the repository ? We wish but this has not been implemented. Signals such as SIGQUIT, SIGTRAP, SIGINT and SIGABRT are trapped indeed but will only kill the server after printing to the server’s log the last executed SQL or the call stack trace, e.g. after a SIGINT was sent:

2019-10-11T13:14:14.045467 24429[24429] 0100c35080004101 Error: dm_bear_trap: Unexpected exception, (SIGINT: interrupt: (2) at (Connection Failure)), during new session creation in module dmapply.cxx after line 542. Process exiting.
Last SQL statement executed by DB was:
 
 
Last SQL statement executed by DB was:
Last SQL statement executed by DB was:
 
 
 
 
Last SQL statement executed by DB was:
 
 
(23962) Outer Exception handler caught exception: SIGINT: interrupt: (2) at (RPC MAIN)

Thus, a corruption is theoretically possible while using any of those signals, just as it is when a SIGKILL signal is issued.
According to OTX Support, a trap handler that shuts down cleanly the repository has not been implemented because it needs a session to invoke the shutdown server method. OK, and what if a hidden session were opened at startup time and kept around just for such administrative cases ? How about a handler to immediately force a projection to the available docbrokers instead of waiting for the next checkpoint cycle ? As you see, there are ways to make the shutdown more resilient but my overall feeling is there is a lack of willingness to improve the content server.
Therefore, if waiting about 5 minutes for the repository to project to a docbroker is not acceptable, there is no other alternative than kill -9 the repository’s processes, start the docbroker(s) and then the repository. Other signals can work, but not always, and are not any safer.
In order to use that command, one needs to know the content server’s root pid and since the CS does not accept any connection at this point, one must get it from another source. Once the root pid is available, it can be given to the kill command with a slight subtlety: in order to include its children processes, the root pid must be negated, e.g.:

# use the standalone /bin/kill command;
$ /bin/kill --signal SIGKILL -12345
# or use bash's kill builtin:
$ command kill -s SIGKILL -12345

This will transitively kill the process with pid 12345 and all the others in same group, which are the ones it started itself, directly or indirectly.
If a numeric signal is preferred, the equivalent command is:

$ /bin/kill -9 -12345

I leave it to you to decide which one is more readable.
So now, we need to identify the repository’s root process. Once found, we can send its negated value the SIGKILL signal, which will propagate to all the child processes. Let’s see now how to identify this root process.

Identifying the content server’s root process

Ordinarily, the LIST_SESSIONS server method returns a collection containing the root_pid attribute among other valuable information, e.g.:

API> apply,c,NULL,LIST_SESSIONS
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES
 
  root_start                      : 12/11/2019 22:53:19
  root_pid                        : 25329
  shared_mem_id                   : 2588691
  semaphore_id                    : 0
  session                      [0]: 0100c3508000a11c
                               [1]: 0100c3508000a102
                               [2]: 0100c3508000a101
  db_session_id                [0]: 272
                               [1]: 37
                               [2]: 33
  typelockdb_session_id        [0]: -1
                               [1]: -1
                               [2]: -1
  tempdb_session_ids           [0]: -1
                               [1]: 45
                               [2]: 36
  pid                          [0]: 17686
                               [1]: 26512
                               [2]: 26465
  user_name                    [0]: dmadmin
                               [1]: dmadmin
                               [2]: dmadmin
  user_authentication          [0]: Trusted Client
                               [1]: Password
                               [2]: Trusted Client
  client_host                  [0]: docker
                               [1]: 172.19.0.3
                               [2]: docker
  client_lib_ver               [0]: 16.4.0070.0035
                               [1]: 16.4.0070.0035
                               [2]: 16.4.0070.0035
...

But in our case, the CS is not reachable so it cannot be queried.
An easy alternative is to simply look into the CS’s log:

dmadmin@docker:/app/dctm$ less /app/dctm/dba/log/dmtest.log
 
    OpenText Documentum Content Server (version 16.4.0080.0129  Linux64.Oracle)
    Copyright (c) 2018. OpenText Corporation
    All rights reserved.
 
2019-12-11T22:53:19.757264      25329[25329]    0000000000000000        [DM_SERVER_I_START_SERVER]info:  "Docbase dmtest attempting to open"
 
2019-12-11T22:53:19.757358      25329[25329]    0000000000000000        [DM_SERVER_I_START_KEY_STORAGE_MODE]info:  "Docbase dmtest is using database for cryptographic key storage"
...

The number 25329 is the root_pid. It can be extracted from the log file as shown below:

$ grep "\[DM_SERVER_I_START_SERVER\]info" /app/dctm/dba/log/dmtest.log | gawk '{if (match($2, /\[[0-9]+\]/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}'
25329
# or compacter:
gawk '{if (match($0, /\[([0-9]+)\].+\[DM_SERVER_I_START_SERVER\]info/, root_pid)) {print root_pid[1]; exit}}' /app/dctm/dba/log/dmtest.log
25329


The extracted root_pid can be confirmed by the ps command with options ajxf showing a nice tree-like view of the running processes. E.g.:

dmadmin@docker:/app/dctm$ ps_gpid 25329
 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
    1 25329 25329 25329 ?           -1 Ss    1001   0:01 ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 25370 25329 25329 ?           -1 S     1001   0:00  \_ /app/dctm/product/16.4/bin/mthdsvr master 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 25329 dmtest /app/dctm/dba/log
25370 25371 25329 25329 ?           -1 Sl    1001   0:05  |   \_ /app/dctm/product/16.4/bin/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 0 dmtest /app/dctm/dba/log
25370 25430 25329 25329 ?           -1 Sl    1001   0:05  |   \_ /app/dctm/product/16.4/bin/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 1 dmtest /app/dctm/dba/log
25370 25451 25329 25329 ?           -1 Sl    1001   0:05  |   \_ /app/dctm/product/16.4/bin/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 2 dmtest /app/dctm/dba/log
25370 25464 25329 25329 ?           -1 Sl    1001   0:05  |   \_ /app/dctm/product/16.4/bin/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 3 dmtest /app/dctm/dba/log
25370 25482 25329 25329 ?           -1 Sl    1001   0:05  |   \_ /app/dctm/product/16.4/bin/mthdsvr worker 0xe901fc83, 0x7f084db15000, 0x223000 50000  5 4 dmtest /app/dctm/dba/log
25329 25431 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 25432 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 25453 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 25465 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 25489 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
25329 26439 25329 25329 ?           -1 Sl    1001   0:11  \_ ./dm_agent_exec -docbase_name dmtest.dmtest -docbase_owner dmadmin -sleep_duration 0
25329 26465 25329 25329 ?           -1 S     1001   0:00  \_ ./documentum -docbase_name dmtest -security acl -init_file /app/dctm/dba/config/dmtest/server.ini
    1 10112 25329 25329 ?           -1 Rl    1001   0:03 ./dm_agent_exec -docbase_name dmtest.dmtest -docbase_owner dmadmin -trace_level 0 -job_id 0800c3508000218b -log_directory /app/dctm/dba/log -docbase_id 50000

On line 3, the CS for docbase dmtest was started with pid 25329 and same value for its pgid. This process started then a few child processes all with the pgid 25329.
ps_pgid on line 1 is a bash function defined in ~/.bashrc as follows:

# returns the lines from ps -ajxf with given gpid;
# the ps command's header line is printed only if at least 1 entry is found;
function ps_pgid {
   pgid=$1
   ps -ajxf | gawk -v pgid=$pgid 'BEGIN {getline; header = $0; h_not_printed = 1} {if ($3 == pgid) {if (h_not_printed) {print header; h_not_printed = 0}; print}}'
}

The command does not show the method server nor the docbroker as they were started separately from the CS.
Thus, if we execute the command below:

$ kill --signal SIGKILL -25329

the CS will be killed along with all its child processes, which is exactly what we want.

Putting both commands together, we get:

kill --signal SIGKILL -$(grep "\[DM_SERVER_I_START_SERVER\]info" /app/dctm/dba/log/dmtest.log | gawk '{if (match($2, /\[[0-9]+\]/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}')

It may be worth defining a bash function for it too:

function kill_cs {
   repo=$1
   kill --signal SIGKILL -$(grep "\[DM_SERVER_I_START_SERVER\]info" /app/dctm/dba/log/${repo}.log | gawk '{if (match($2, /\[[0-9]+\]/)) {print substr($2, RSTART + 1, RLENGTH - 2); exit}}')
}
 
# source it:
. ~/.bashrc
 
# call it:
kill_cs dmtest

where test is the content server to kill.
The naive way to search the running content server via the command “ps -ef | grep docbase_name” can be too ambiguous in case of multiple content servers for the same repository (e.g. in a high-availability installation) or when docbase_name is the stem of a family of docbases (e.g. dmtest_1, dmtest_2, …, dmtest_10, etc…). Besides, even if no ambiguity were possible, it would return too many processes to be killed individually. xargs could do it at once, sure, but why risk killing the wrong ones ? The above ps_pgid function is directly looking for the given group id which is the root_pid of the content server of interest taken straight out of its log file, no ambiguity here.

Hardening start-stop.sh

This ruthless kill functionality could be added to the start-stop script listed above, either as a command-line option to the stop parameter (say, like -k as in the dm_shutdown_repository script) or as a full parameter on a par with the stop | start | status ones, i.e.:

start-stop.sh stop | start | status | kill ...

or, simply by deciding that a stop should always succeed and forcing a kill if needed. In such variant, the stop_docbase() function becomes:

stop_docbase() {
   echo "stopping $docbase"
   docbase=$1
   ./dm_shutdown_${docbase}
   if [[ $? -eq 1 ]]; then
      echo "killing docbase $docbase"
      kill_cs $docbase
   fi
   echo "docbase $docbase stopped"
}

Conclusion

If the content server were open source we wouldn’t have this article’s title. Instead, it would be “Forcing impromptu projections to docbrokers through signal handling in content server: an implementation” or “Shutting down a content server by sending a signal: a proposal”. We could send this request to the maintainers and probably receive a positive answer. Or we could implement the changes ourselves and submit them as a RFC. This model does not work so much in closed, commercial source which evolves following its own marketing agenda. Nonetheless, this situation gives us the opportunity to rant about it and find work-arounds. Imagine a world where all software were flawless, would it be as fun ?

Cet article A Ruthless Repository Shutdown Utility, Part II est apparu en premier sur Blog dbi services.

How can you connect an external DFC Client to docbrokers and Repositories hosted on Kubernetes Pods? That seems to be a very simple question yet it might prove difficult… Let’s talk about this challenge in this blog and possible solutions/workarounds.

As you all know, Kubernetes is using containers so just like for a basic Docker container, you won’t be able to access it from the outside by default. On Docker, you will need to expose some ports and then you can interact with whatever is running on that port. For Kubernetes, it’s the same principle but it obviously add other layers in addition which makes it even more complicated. Therefore, if you want to be able to connect to a docbroker inside a K8s Pod from the outside of K8s, then you will need to do a few things:

• at the container level, to open the ports 1489/1490 (default ones, you can change them obviously)
• a K8s Service to expose these ports inside K8s
• an Nginx Ingres Controller for which the TCP ports 1489/1490 have been configured for external accesses (or other ports if these are already used for another namespace for example)
• a “Load Balancer” K8s Service (still at the Nginx Ingres Controller level) which exposes these ports using an external IP

Once you have that, you should be able to communicate with a docbroker that is inside a K8s pod. If you want to have a chance to talk to a Repository, then you will also need to do the same thing but for the Repository ports. When you install a repo, you will specify in the /etc/services the ports it should use (just like for the docbroker).

For this example, let’s start simple with the same ports internally and externally:

• DFC Client host: vm
• K8s pod short name (hostname): cs-0
• K8s pod full name (headless service / full hostname): cs-0.cs.dbi-ns01.svc.cluster.local
• K8s pod IP: 1.1.1.100
• K8s pod docbroker port: 1489/1490
• K8s pod Repositories port: gr_repo=49400/49401    //    REPO1=49402/49403
• K8s external hostname/lb: k8s-cs-dbi-ns01.domain.com
• K8s external IP: 2.2.2.200
• K8s external docbroker port: 1489/1490
• K8s external Repositories port: gr_repo=49400/49401    //    REPO1=49402/49403

Considering the above setup (both the docbroker and Repositories ports configured on K8s), you can already talk to the docbroker properly:

[dmadmin@vm ~]$ grep "dfc.docbroker" dfc.properties
dfc.docbroker.host[0]=k8s-cs-dbi-ns01.domain.com
dfc.docbroker.port[0]=1489
[dmadmin@vm ~]$
[dmadmin@vm ~]$ nc -v k8s-cs-dbi-ns01.domain.com 1489
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 2.2.2.200:1489.
^C
[dmadmin@vm ~]$ 
[dmadmin@vm ~]$ nc -v k8s-cs-dbi-ns01.domain.com 49402
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 2.2.2.200:49402.
^C
[dmadmin@vm ~]$ 
[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c ping
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1489
Successful reply from docbroker at host (cs-0) on port(1490) running software version (16.4.0170.0234  Linux64).
[dmadmin@vm ~]$ 
[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : cs-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100
Docbroker version         : 16.4.0170.0234  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234567
Docbase description : dbi-ns01 dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
Docbase name        : REPO1
Docbase id          : 1234568
Docbase description : dbi-ns01 dev k8s repo1
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       :
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@vm ~]$

So as you can see above, the docbroker does respond properly with the list of Repositories that it is aware of (Repo name, ID, hostname, …) and for that purpose, there is no need for the Repositories’ ports to be opened, only the docbroker is enough. However, as soon as you want to go further and start talking to the Repositories, you will obviously need to open these additional ports as well. Above, I used 49402/49403 for the REPO1 Repository (both internal and external). When trying to login to a target Repository, it will fail:

[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$

Why is that? Well the reason is that to connect to a docbroker, a DFC Client will use the value from the well-known “dfc.properties” file. By reading it, it will know where the docbroker can be found: in our case, it’s “k8s-cs-dbi-ns01.domain.com:1489“. When that is done, the docbroker replies with the list of Repositories known and it will also reply with the “host” that should be used to communicate with the Repositories. That’s because the Repositories might not be on the same host as the docbroker and therefore it needs to provides the information to the DFC Client. However, that “host” is actually an IP! When a Repository register itself with a docbroker, the docbroker records the source IP of the request and it will then forward this IP to the DFC Client that wants to talk to this Repository.

The problem here is that the Repositories are installed on K8s Pods and therefore the IP that the docbroker knows is actually the IP of the K8s Pod… Which is, therefore, not reachable from outside of K8s!

1. IP Forwarding, a solution?

If you want to validate a setup or do some testing, it’s pretty simple on Linux, you can quickly setup an IP Forwarding between the IP of the K8s Pod (which points to nothing) and the IP of the K8s LB Service that you configured previously for the docbroker and Repositories ports. Here is an example:

[dmadmin@vm ~]$ nslookup k8s-cs-dbi-ns01.domain.com
Server: 1.1.1.10
Address: 1.1.1.10#53

k8s-cs-dbi-ns01.domain.com     canonical name = k8s-cluster-worker2.domain.com.
Name:   k8s-cluster-worker2.domain.com
Address: 2.2.2.200
[dmadmin@vm ~]$
[dmadmin@vm ~]$ external_ip=2.2.2.200
[dmadmin@vm ~]$ ping -c 1 ${external_ip}
PING 2.2.2.200 (2.2.2.200) 56(84) bytes of data.
64 bytes from 2.2.2.200: icmp_seq=1 ttl=63 time=0.980 ms

--- 2.2.2.200 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.980/0.980/0.980/0.000 ms
[dmadmin@vm ~]$
[dmadmin@vm ~]$ internal_ip=1.1.1.100
[dmadmin@vm ~]$ ping -c 1 ${internal_ip}
PING 1.1.1.100 (1.1.1.100) 56(84) bytes of data.

--- 1.1.1.100 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$
[dmadmin@vm ~]$
[dmadmin@vm ~]$ sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
[dmadmin@vm ~]$ 
[dmadmin@vm ~]$ sudo iptables -t nat -A OUTPUT -d ${internal_ip} -j DNAT --to-destination ${external_ip}
[dmadmin@vm ~]$ 
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 0112d6888000152a started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$

As you can see above, as soon as you configure an IP Forwarding from the Pod IP to the K8s LB IP, then the Repository connection is successful. Here, I just executed a “quit” command to close the iAPI session but it shows that the session creation is working so you are sure that the end-to-end communication is fine.

Obviously, that is just for testing… Indeed, your Pod IP is going to change in the future (after each restart of the pod for example) which means that the IP Forwarding will be broken at that time. This also requires being setup on the client directly because the DFC Client will try to communicate with a specific IP… But this IP most probably doesn’t point to anything and therefore the only way to make it happen correctly is either setting that up on the client or on the network layer, which is super annoying and isn’t really reliable anyway so this isn’t a solution.

2. Docbroker Translation, a solution?

Several years ago, a feature has been introduced in the docbroker that was initially planned to handle blocking rules on a FireWall: IP and Port Translation. I believe it was introduced for Documentum 6.5 but I might be wrong, it was a long time ago… Since the issue here for K8s is pretty similar to what would happen with a FireWall blocking the IP, we can actually use this feature to help us. Contrary to the IP Forwarding, which is done on the client side, the Translation is done on the server side which is therefore global for all clients. This has an obvious advantage that you can just do it once for all clients (or rather you will need to re-do this configuration at each start of your K8s Pod since the IP will be changed). However, it also has a drawback which is that there is no exception: all communications will be translated, even K8s internal communications… So this might be a problem. There is a KB to describe how it works (KB7701941) and you can also look at the documentation as well. However, the documentation might not be really correct. Indeed, if you look at the CS 7.1 documentation, you will find this definition:

[TRANSLATION]
port=inside_firewall_port=outside_firewall_port
{,inside_firewall_port=outside_firewall_port}
host=inside_firewall_IP=outside_firewall_IP
{,inside_firewall_IP=outside_firewall_IP}

If you look at the CS 16.4 documentation, you will find this definition:

[TRANSLATION]
port=inside_firewall_port=outside_firewall_port
{,inside_firewall_port=outside_firewall_port}
host=outside_firewall_IP=inside_firewall_IP
{,outside_firewall_IP=inside_firewall_IP}

Finally, if you look at the CS 16.7 documentation, you will find yet another definition:

[TRANSLATION]port=["]outside_firewall_port=inside_firewall_port
{,outside_firewall_port=inside_firewall_port}["]
host=["]outside_firewall_ip=inside_firewall_ip
{,outside_firewall_ip=inside_firewall_ip}["]

Three documentations on the same feature, three different definitions :D. In addition to that, there is an example in the documentation which is also wrong, on the three documentations. The real definition is the last one, after fixing the formatting errors that is… So in short, this is what you can do with the docbroker translation:

[TRANSLATION]
port=["]ext_port_1=int_port_1{,ext_port_2=int_port_2}{,ext_port_3=int_port_3}{,...}["]
host=["]ext_ip_1=int_ip_1{,ext_ip_2=int_ip_2}{,ext_ip_3=int_ip_3}{,...}["]

From what I could see, the double quotes aren’t mandatory but you can use them if you want to…

Let’s test all that after removing the IP Forwarding, obviously:

[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : cs-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100
Docbroker version         : 16.4.0170.0234  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234567
Docbase description : dbi-ns01 dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
Docbase name        : REPO1
Docbase id          : 1234568
Docbase description : dbi-ns01 dev k8s repo1
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       :
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO1
^C[dmadmin@vm ~]$
[dmadmin@vm ~]$

On the docbroker side (k8s), let’s configure the translation properly and restart for the new configuration to be applied:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ external_ip=2.2.2.200
[dmadmin@cs-0 dba]$ external_port=1489
[dmadmin@cs-0 dba]$ internal_ip=1.1.1.100
[dmadmin@cs-0 dba]$ internal_port=1489
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ echo "[TRANSLATION]" >> Docbroker.ini
[dmadmin@cs-0 dba]$ echo "port=${external_port}=${internal_port}" >> Docbroker.ini
[dmadmin@cs-0 dba]$ echo "host=${external_ip}=${internal_ip}" >> Docbroker.ini
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[TRANSLATION]
port=1489=1489
host=2.2.2.200=1.1.1.100
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ ./dm_stop_Docbroker; sleep 1; ./dm_launch_Docbroker
./dmshutdown 16.4.0000.0248  Linux64 Copyright (c) 2018. OpenText Corporation.
Shutdown request was processed by Docbroker on host cs-0 (INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100)
Reply status indicates a success: OK
starting connection broker on current host: [cs-0.cs.dbi-ns01.svc.cluster.local]
with connection broker log: [$DOCUMENTUM/dba/log/docbroker.cs-0.cs.dbi-ns01.svc.cluster.local.1489.log]
connection broker pid: 18219
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ head -7 log/docbroker.cs-0.cs.dbi-ns01.svc.cluster.local.1489.log
OpenText Documentum Connection Broker (version 16.4.0170.0234  Linux64)
Copyright (c) 2018. OpenText Corporation
HOST TRANSLATION TABLE:
    [1] From(1.1.1.100), to(2.2.2.200)
PORT TRANSLATION TABLE:
    [1] From(1489), to(1489)
2019-12-15T10:25:22.307379 [DM_DOCBROKER_I_START]info:  "Docbroker has started.  Process id: 18219"
[dmadmin@cs-0 dba]$

Once that is done, back on the DFC Client side, trying to connect to the Repository:

[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns01.domain.com -p 1489 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1489
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : cs-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 01 2d3 01010164 cs-0 1.1.1.100
Docbroker version         : 16.4.0170.0234  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234567
Docbase description : dbi-ns01 dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
Docbase name        : REPO1
Docbase id          : 1234568
Docbase description : dbi-ns01 dev k8s repo1
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       :
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@vm ~]$

As you can see above, the dmqdocbroker will still print the Internal IP (1.1.1.100), that’s fine/normal. However the Repository connection should now work:

[dmadmin@vm ~]$ echo "quit" | iapi REPO1 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 0112d6888000175b started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$

So as you can see above, using the docbroker translation mechanisms is indeed a solution to be able to connect to a Repository that is inside a K8s pod. There are drawbacks as mentioned above but at least, that’s a valid workaround.

3. Using different ports externally

Above, I have always been using the same ports internally and externally. However, in a real case, you will probably have, in the end, hundreds or even thousands of CS pods. So how do you manage that? Well you saw above that the docbroker translation can be used to translate an external port into an internal port but it’s not just for the docbroker port! You can actually use that for the Repository ports as well.

Let’s say for this example that I have a second namespace (dbi-ns02) with the following:

• DFC Client Host: vm
• K8s pod short name (hostname): cs-0
• K8s pod full name (headless service / full hostname): cs-0.cs.dbi-ns02.svc.cluster.local
• K8s pod IP: 1.1.1.200
• K8s pod docbroker port: 1489/1490
• K8s pod Repositories port: gr_repo=49400/49401    //    REPO2=49402/49403
• K8s external hostname/lb: k8s-cs-dbi-ns02.domain.com
• K8s external IP: 2.2.2.200
• K8s external docbroker port: 1491/1492
• K8s external Repositories port: gr_repo=49404/49405    //    REPO2=49406/49407

The external IP is still the same because it’s the same K8s Cluster but the external ports are now different. The internal IP is also different because it’s another namespace. So with the default docbroker configuration (no translation), then we have the same issue, obviously, where the iAPI session will hang and never respond because of the IP that doesn’t exist.

So if we try to setup the basic docbroker translation just like what we did above, then on the K8s pod, we will have the following:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ ifconfig | grep inet | grep -v 127.0.0.1
        inet 1.1.1.200  netmask 255.255.255.255  broadcast 0.0.0.0
[dmadmin@cs-0 dba]$
[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[TRANSLATION]
port=1491=1489
host=2.2.2.200=1.1.1.200
[dmadmin@cs-0 dba]$

With this configuration, if you are trying to connect from an external DFC Client, then it will be able to talk to the docbroker (assuming you have all the K8s stuff in place for redirecting the ports properly) but won’t be able to talk to the Repository:

[dmadmin@vm ~]$ dmqdocbroker -t k8s-cs-dbi-ns02.domain.com -p 1491 -c getdocbasemap
dmqdocbroker: A DocBroker Query Tool
dmqdocbroker: Documentum Client Library Version: 16.4.0000.0185
Using specified port: 1491
**************************************************
**     D O C B R O K E R    I N F O             **
**************************************************
Docbroker host            : cs-0
Docbroker port            : 1490
Docbroker network address : INET_ADDR: 02 3d4 02020286 cs-0 1.1.1.200
Docbroker version         : 16.4.0170.0234  Linux64
**************************************************
**     D O C B A S E   I N F O                  **
**************************************************
--------------------------------------------
Docbase name        : gr_repo
Docbase id          : 1234569
Docbase description : dbi-ns02 dev k8s gr
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       : Global Registry
Docbase Dormancy Status     :
--------------------------------------------
Docbase name        : REPO2
Docbase id          : 1234570
Docbase description : dbi-ns02 dev k8s repo2
Govern docbase      :
Federation name     :
Server version      : 16.4.0170.0234  Linux64.Oracle
Docbase Roles       :
Docbase Dormancy Status     :
--------------------------------------------
[dmadmin@vm ~]$
[dmadmin@vm ~]$ echo "quit" | iapi REPO2 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO2
:Wrong docbase id: (1234570) expecting: (1234568)

Could not connect
[dmadmin@vm ~]$

The reason for that is that I have been talking to the docbroker on the external port 1491, which is therefore the docbroker 1489 of the second namespace (“dbi-ns02“). This docbroker replied to the DFC Client that the Repository is using the port 49402/49403, which is true but only internally… Therefore, my DFC Client has been trying to connect to the Repository REPO2 (from the second namespace) using the port which is actually the one used by the REPO1 (from the first namespace) and therefore there is a mismatch in the Repository ID.

For that purpose, you can update the docbroker translation to include the Repositories ports as well:

[dmadmin@cs-0 dba]$ cat Docbroker.ini
[DOCBROKER_CONFIGURATION]
secure_connect_mode=dual
[TRANSLATION]
port=1491=1489,49404=49400,49405=49401,49406=49402,49407=49403
host=2.2.2.200=1.1.1.200
[dmadmin@cs-0 dba]$

With this new docbroker translation configuration, the external DFC Client should be able to communicate properly with the repository:

[dmadmin@vm ~]$ echo "quit" | iapi REPO2 -Udmadmin -P${dm_pw}

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0000.0185

Connecting to Server using docbase REPO2
[DM_SESSION_I_SESSION_START]info:  "Session 0112d68a80001403 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@vm ~]$

Alternatively to all that, you might want to take a look at Traefik or Istio which might also help you to configure the correct communications from the outside of K8s to the inside. I had a case opened with the OpenText Support so that they could correct the documentation for all versions.

Cet article Documentum – Connection to docbrokers and Repositories inside K8s from an external DFC Client est apparu en premier sur Blog dbi services.

Recently, I was doing some sanity checks on a Documentum Server and I saw a Java exception stack while logging in using iAPI/iDQL to a Repository. It was reproducible for all Repositories. I’ve never seen something like that before (or at least I don’t remember it) so I was a little bit surprised. Whenever there are errors upon login, it is usually Documentum error messages that are printed and there is no exception stack. Since it took me some efforts finding the root cause, I thought about sharing it.

The exception stack displayed was the following one:

[dmadmin@cs-0 ~]$ echo "quit" | iapi gr_repo -Udmadmin -Pxxx

        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2016
        All rights reserved.
        Client Library Release 7.3.0040.0025

Connecting to Server using docbase gr_repo
DfException:: THREAD: main; MSG: [DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c9214 started for user dmadmin."; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbase.DocbaseExceptionMapper.newException(DocbaseExceptionMapper.java:57)
        at com.documentum.fc.client.impl.connection.docbase.MessageEntry.getException(MessageEntry.java:39)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseMessageManager.getExceptionForAllMessages(DocbaseMessageManager.java:176)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getExceptionForAllMessages(DocbaseConnection.java:1518)
        at com.documentum.fc.client.impl.session.Session.getExceptionsForAllMessages(Session.java:1603)
        at com.documentum.fc.client.impl.session.SessionHandle.getExceptionsForAllMessages(SessionHandle.java:1301)
        at com.documentum.dmcl.impl.ApiContext.addMessages(ApiContext.java:423)
        at com.documentum.dmcl.impl.ApiContext.collectExceptionsForReporting(ApiContext.java:370)
        at com.documentum.dmcl.impl.GetMessageHandler.get(GetMessageHandler.java:23)
        at com.documentum.dmcl.impl.DmclApi.get(DmclApi.java:49)
        at com.documentum.dmcl.impl.DmclApiNativeAdapter.get(DmclApiNativeAdapter.java:145)
        at com.documentum.dmcl.impl.DmclApiNativeAdapter.get(DmclApiNativeAdapter.java:130)


Connected to Documentum Server running Release 7.3.0050.0039  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@cs-0 ~]$

The login was successful but still, a strange exception stack appeared. The first thing I did was checking the Repository log file but there was nothing out of the ordinary inside it except for one thing:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba/log
[dmadmin@cs-0 log]$
[dmadmin@cs-0 log]$ grep -A3 "Agent Exec" gr_repo.log
Wed Sep 11 10:38:29 2019 [INFORMATION] [AGENTEXEC 1477] Detected during program initialization: Agent Exec connected to server gr_repo:  DfException:: THREAD: main; MSG: [DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c8904 started for user dmadmin."; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbase.DocbaseExceptionMapper.newException(DocbaseExceptionMapper.java:57)
        at com.documentum.fc.client.impl.connection.docbase.MessageEntry.getException(MessageEntry.java:39)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseMessageManager.getExceptionForAllMessages(DocbaseMessageManager.java:176)
        at com.documentu
[dmadmin@cs-0 log]$

While starting, the Agent Exec was therefore facing the same behavior with the exact same stack (which is cut at the 4th line but it’s the same stack until then so it’s safe to assume it’s the same). Therefore, to dig deeper and to find when the issue started exactly, I checked the logs from the agentexec/jobs since this will be kept until cleanup from the log purge and since it does login to the Repository:

[dmadmin@cs-0 log]$ cd $DOCUMENTUM/dba/log/gr_repo/agentexec
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ # Check the last file
[dmadmin@cs-0 agentexec]$ cat $(ls -tr job_* | tail -1)
Wed Sep 11 18:00:21 2019 [INFORMATION] [LAUNCHER 3184] Detected while preparing job dm_ConsistencyChecker for execution: Agent Exec connected to server gr_repo:  DfException:: THREAD: main; MSG: [DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c8974 started for user dmadmin."; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbase.DocbaseExceptionMapper.newException(DocbaseExceptionMapper.java:57)
        at com.documentum.fc.client.impl.connection.docbase.MessageEntry.getException(MessageEntry.java:39)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseMessageManager.getExceptionForAllMessages(DocbaseMessageManager.java:176)
        at com.documentu
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ # Finding the first file with the error
[dmadmin@cs-0 agentexec]$ for f in $(ls -tr); do r=$(grep "_I_SESSION_START.*ERRORCODE" "${f}"); if [[ "${r}" != "" ]]; then echo "${r}"; break; fi; done
Tue Sep 10 18:00:06 2019 [INFORMATION] [LAUNCHER 31113] Detected while preparing job dm_ConsistencyChecker for execution: Agent Exec connected to server gr_repo:  DfException:: THREAD: main; MSG: [DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c8827 started for user dmadmin."; ERRORCODE: 100; NEXT: null
[dmadmin@cs-0 agentexec]$

In all the job’s sessions files, there were the same stack (or rather a piece of the stack). At first, I didn’t understand where this was coming from, all I know was that it was linked somehow to the login inside the Repository and that it appeared for the first time on the date returned by my last command above. It was not really an error message since it wasn’t showing any “_E_” messages but it was still printing an exception.

Knowing when it appeared the first time, I looked at all the files that have been modified on that day and among log files, which are expected and can be ignored, there were the dfc.properties file. This provided me the reason for this message: it was actually due to enabling the diagnostic mode on the dfc.properties of the Documentum Server. To be exact, it was due to the “dfc.diagnostics.exception.include_stack=true” entry:

[dmadmin@cs-0 agentexec]$ tail -5 $DOCUMENTUM_SHARED/config/dfc.properties
dfc.session.secure_connect_default=secure
dfc.time_zone=UTC
dfc.diagnostics.resources.enable=true
dfc.diagnostics.exception.include_stack=true
dfc.tracing.print_exception_stack=true
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ echo "quit" | iapi gr_repo -Udmadmin -Pxxx

        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2016
        All rights reserved.
        Client Library Release 7.3.0040.0025

Connecting to Server using docbase gr_repo
DfException:: THREAD: main; MSG: [DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c9235 started for user dmadmin."; ERRORCODE: 100; NEXT: null
        at com.documentum.fc.client.impl.docbase.DocbaseExceptionMapper.newException(DocbaseExceptionMapper.java:57)
        at com.documentum.fc.client.impl.connection.docbase.MessageEntry.getException(MessageEntry.java:39)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseMessageManager.getExceptionForAllMessages(DocbaseMessageManager.java:176)
        at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.getExceptionForAllMessages(DocbaseConnection.java:1518)
        at com.documentum.fc.client.impl.session.Session.getExceptionsForAllMessages(Session.java:1603)
        at com.documentum.fc.client.impl.session.SessionHandle.getExceptionsForAllMessages(SessionHandle.java:1301)
        at com.documentum.dmcl.impl.ApiContext.addMessages(ApiContext.java:423)
        at com.documentum.dmcl.impl.ApiContext.collectExceptionsForReporting(ApiContext.java:370)
        at com.documentum.dmcl.impl.GetMessageHandler.get(GetMessageHandler.java:23)
        at com.documentum.dmcl.impl.DmclApi.get(DmclApi.java:49)
        at com.documentum.dmcl.impl.DmclApiNativeAdapter.get(DmclApiNativeAdapter.java:145)
        at com.documentum.dmcl.impl.DmclApiNativeAdapter.get(DmclApiNativeAdapter.java:130)


Connected to Documentum Server running Release 7.3.0050.0039  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ sed -i sed 's,^dfc.diagnostics.exception.include_stack,#&,' $DOCUMENTUM_SHARED/config/dfc.properties
[dmadmin@cs-0 agentexec]$
[dmadmin@cs-0 agentexec]$ echo "quit" | iapi gr_repo -Udmadmin -Pxxx

        EMC Documentum iapi - Interactive API interface
        (c) Copyright EMC Corp., 1992 - 2016
        All rights reserved.
        Client Library Release 7.3.0040.0025

Connecting to Server using docbase gr_repo
[DM_SESSION_I_SESSION_START]info:  "Session 0112d687800c9237 started for user dmadmin."

Connected to Documentum Server running Release 7.3.0050.0039  Linux64.Oracle
Session id is s0
API> Bye
[dmadmin@cs-0 agentexec]$

As you can see above, commenting the line “dfc.diagnostics.exception.include_stack=true” (meaning setting it to false, the default value) caused the exception stack to disappear. Since I was curious about this stack and wanted confirmation that this is “expected”, I opened a case with the OpenText Support (#4331438) and they confirmed me after a few days that it wasn’t considered an “ERROR“, it was more of an “INFO” message. It’s a strange way to display informative messages but hey, who am I to judge!

Cet article Documentum – Java exception stack on iAPI/iDQL login est apparu en premier sur Blog dbi services.

Containers running under docker can have their stdout observed from the outside through the “docker logs”command; here is an excerpt of its usage:

docker logs --help
Usage:	docker logs [OPTIONS] CONTAINER

Fetch the logs of a container

Options:
      --details        Show extra details provided to logs
  -f, --follow         Follow log output
      --since string   Show logs since timestamp (e.g. 2013-01-02T13:23:37) or relative (e.g. 42m
                       for 42 minutes)
      --tail string    Number of lines to show from the end of the logs (default "all")
  -t, --timestamps     Show timestamps
      --until string   Show logs before a timestamp (e.g. 2013-01-02T13:23:37) or relative (e.g.
                       42m for 42 minutes)

e.g.:

Example of output:

docker logs --follow --timestamps container05bis
...
2019-07-10T03:50:38.624862914Z ==> /app/dctm/dba/log/docbroker.container05bis.1489.log <==
2019-07-10T03:50:38.624888183Z OpenText Documentum Connection Broker (version 16.4.0000.0248 Linux64)
2019-07-10T03:50:38.624893936Z Copyright (c) 2018. OpenText Corporation
2019-07-10T03:50:38.624898034Z 2019-07-10T05:50:38.519721 [DM_DOCBROKER_I_START]info: "Docbroker has started. Process id: 35"
2019-07-10T03:50:38.624902047Z 2019-07-10T05:50:38.521502 [DM_DOCBROKER_I_REGISTERED_PORT]info: "The Docbroker registered using port (1489)."
2019-07-10T03:50:38.624906087Z 2019-07-10T05:50:38.521544 [DM_DOCBROKER_I_LISTENING]info: "The Docbroker is listening on network address: (INET_ADDR: family: 2, port: 1489, host: container05bis (192.168.33.7, 0721a8c0))"
2019-07-10T03:50:38.624911984Z
2019-07-10T03:50:38.624915369Z ==> /app/dctm/dba/log/dmtest05bis.log <==
2019-07-10T03:50:38.625040316Z
2019-07-10T03:50:38.625050474Z ==> /app/dctm/dba/log/dmtest05bis/agentexec/agentexec.log <==
2019-07-10T03:50:38.625055299Z Wed Jul 10 03:33:48 2019 [INFORMATION] [LAUNCHER 4251] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625059065Z Wed Jul 10 03:34:18 2019 [INFORMATION] [LAUNCHER 4442] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625071866Z Wed Jul 10 03:34:48 2019 [INFORMATION] [LAUNCHER 4504] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625075268Z Wed Jul 10 03:36:18 2019 [INFORMATION] [LAUNCHER 4891] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625078688Z Wed Jul 10 03:36:49 2019 [INFORMATION] [LAUNCHER 4971] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625082182Z Wed Jul 10 03:48:18 2019 [INFORMATION] [LAUNCHER 6916] Detected during program initialization: Version: 16.4.0000.0248 Linux64
2019-07-10T03:50:38.625096886Z
2019-07-10T03:50:38.625101275Z ==> /app/dctm/wildfly9.0.1/server/DctmServer_MethodServer/logs/ServerApps.log <==
2019-07-10T03:50:38.625105098Z at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:511)
2019-07-10T03:50:38.625108575Z at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101)
2019-07-10T03:50:38.625112342Z at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)
2019-07-10T03:50:38.625116110Z at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019-07-10T03:50:38.625120084Z at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019-07-10T03:50:38.625123672Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-07-10T03:50:38.625127341Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-07-10T03:50:38.625131122Z at java.lang.Thread.run(Thread.java:748)
2019-07-10T03:50:38.625134828Z at org.jboss.threads.JBossThread.run(JBossThread.java:320)
2019-07-10T03:50:38.625139133Z 05:34:58,050 INFO [ServerService Thread Pool -- 96] com.documentum.cs.otds.DfSessionHandler - DFC Client Successfully initialized
2019-07-10T03:50:38.625143291Z
2019-07-10T03:50:38.625146939Z ==> /app/dctm/wildfly9.0.1/server/DctmServer_MethodServer/logs/ServerApps_trace.log <==
2019-07-10T03:50:38.625150991Z
2019-07-10T03:50:38.625154528Z ==> /app/dctm/wildfly9.0.1/server/DctmServer_MethodServer/logs/dmotdsrest.log <==
2019-07-10T03:50:38.625159563Z 03:33:16,505 [ServerService Thread Pool -- 66] DFC Client Successfully initialized
2019-07-10T03:50:38.625163445Z 05:34:58,050 [ServerService Thread Pool -- 96] DFC Client Successfully initialized
2019-07-10T03:50:38.625955045Z Setting up watches. Beware: since -r was given, this may take a while!
2019-07-10T03:50:38.627044196Z Watches established.
2019-07-10T03:50:38.934648542Z
2019-07-10T03:50:38.934668467Z ==> /app/dctm/dba/log/dmtest05bis.log <==
2019-07-10T03:50:38.934673076Z Wed Jul 10 05:50:38 2019[DM_STARTUP_W_DOCBASE_OWNER_NOT_FOUND] *** warning *** : The database user, dmtest05bisc as specified by your server.ini is not a valid user as determined using the system password check api. This will likely severly impair the operation of your docbase.
2019-07-10T03:50:39.001793811Z
2019-07-10T03:50:39.001816266Z
2019-07-10T03:50:39.001821414Z OpenText Documentum Content Server (version 16.4.0000.0248 Linux64.Oracle)
2019-07-10T03:50:39.001825146Z Copyright (c) 2018. OpenText Corporation
2019-07-10T03:50:39.001828983Z All rights reserved.
2019-07-10T03:50:39.001832816Z
2019-07-10T03:50:39.005318448Z 2019-07-10T05:50:39.005068 193[193] 0000000000000000 [DM_SERVER_I_START_SERVER]info: "Docbase dmtest05bis attempting to open"
2019-07-10T03:50:39.005337158Z
2019-07-10T03:50:39.005342472Z 2019-07-10T05:50:39.005172 193[193] 0000000000000000 [DM_SERVER_I_START_KEY_STORAGE_MODE]info: "Docbase dmtest05bis is using database for cryptographic key storage"
...

If the information is plethoric, the above command can be narrowed to a time window by adding −−since and −−until restrictions, e.g.:

docker logs --timestamps --follow --since 5m <container>

Thus, everything sent to the container’s stdout can be aggregated and viewed very simply from one place, yielding a cheap console log. In particular, Documentum containers could expose their well-known log files to the outside world, e.g. the docbroker log, the content server(s) log(s) and the method server logs. To this effect, it would be enough to just start a “tail -F ” on those files from within the entrypoint as illustrated below:

tail -F ${DOCUMENTUM}/dba/log/docbroker.log ${DOCUMENTUM}/dba/log/dmtest.log ${DOCUMENTUM}/wildfly9.0.1/server/DctmServer_MethodServer/logs/ServerApps.log ...

The -F option guarantees that the logs continue being followed even after a possible rotation.
Admittedly, this output can be a bit hard to read because the logs are interlaced, i.e. lines or block of lines from different logs are displayed sorted by the time they were produced and not by their origin. Actually, this is a benefit because it makes it easier to find correlations between distinct, apparently unrelated events.
Viewing a particular log is still possible from without the container, e.g.:

docker exec <container_name> /bin/bash -c "tail -f \${DOCUMENTUM}/dba/log/dmtest05bis.log"

provided the tail command exists in the container, which is not obvious as there is a definitive will to make images as stripped down as possible.
As those files are statically known (i.e. at build time), such command could be defined as early as in the buildfile and invoked in its entrypoint script.
Unfortunately, the content server logs are not very verbose and the most useful messages are directed to session or ad hoc logs. The session logs are dynamically created for each new session with the session id as their name, which makes it unpredictable. Since those names are only known at run-time, the above buildfile’s “tail -F” command cannot include them and consequently they are not displayed by the “docker logs” command. The same applies to on-demand trace files with variable names, e.g. with a timestamp suffix.
So, is there a way to follow those dynamic session logs (or any dynamically named files at that) anyway ? An obvious way is to use a file or directory watcher to be notified of any created or modified file. If a watcher process running inside the container could wait for such conditions and signal any occurence thereof, a listener process, also running inside the container, could receive the notification and dynamically fork a tail -F command to follow that file. Externally, “docker logs” would continue displaying whatever is sent to the container’s stdout, including the newly discovered files.
Under Linux, we can use inotifywait as the file watcher. Let’s see how to set it up.

Installing inotifywait

Under a Debian Linux derivative such as Ubuntu, inotifywait can be easily installed from the inotify-tools package through the usual command:

sudo apt-get install inotify-tools

Under a Red Hat Linux derivative such as Centos, a two-step method is to first grab the rpm package from its on-line repository and then install it; the latest release as of this writing is the 3.14-9:

  curl https://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/i/inotify-tools-3.14-9.el7.x86_64.rpm -o inotify-tools-3.14-9.el7.x86_64.rpm
  sudo rpm -Uvh inotify-tools-3.14-9.el7.x86_64.rpm 

In order to verify that the installation was successful, just try to launch “inotifywait”:

$ inotifywait
No files specified to watch!

Good, the command exists; let’s now test it.

Testing inotifywait

The command inotifywait has the following invocation syntax:

inotifywait [-hcmrq] [-e  ] [-t  ] [--format  ] [--timefmt  ]  [ ... ]

The man page explains very well each of the parameters so, please, refer there for details.
As the whole point is to detect new or changed files whose name we don’t know at image build time, but whose location is known (e.g. a path such as ${DOCUMENTUM}/dba/log), we will be watching directories. In such a case, one parameter will be a list of directories to recursively watch for any new or modified files.
That command was designed to output to stdout any event whose occurence it was configured to wait for.
The default output format is the following:

watched_filename EVENT_NAMES event_filename

one line per file.
EVENT_NAMES is the event that occurred, in our case mainly one of CREATE or MODIFY as requested through the -e command-line parameter.
As we watch directories, watched_filename is the name of the watched directory where the event occured and event_filename, the created/modified file.
Here is an example of use as a test with custom formatting for a better presentation:

$ inotifywait --quiet --monitor --event create,modify,attrib --recursive --timefmt "%Y/%m/%d-%H:%M:%S" --format "%T %e %w%f" /tmp &

The command will recursively (option −−recursive) watch the /tmp directory for any created or modified (option −−event …) file or a change of file attributes such as the timestamp or permissions. It will run in the background (option −−monitor) and issue events to stdout prefixed with a properly formatted time stamp (options −−timefmt and −−format).
With inotifywait running in the background, create now a dummy file in /tmp:

$ touch /tmp/xx
# the following lines get displayed:
2019/12/31-17:43:45 CREATE /tmp/xx
2019/12/31-17:43:45 ATTRIB /tmp/xx


Actually, the /tmp directory is a busy directory and a lot of file activity occurs in there very quickly:

2019/12/31-17:43:52 CREATE /tmp/hsperfdata_dmadmin/471
2019/12/31-17:43:52 MODIFY /tmp/hsperfdata_dmadmin/471
2019/12/31-17:43:52 MODIFY /tmp/hsperfdata_dmadmin/471
2019/12/31-17:44:06 MODIFY /tmp/nohup.log
2019/12/31-17:44:06 MODIFY /tmp/nohup.log
2019/12/31-17:44:06 MODIFY /tmp/nohup.log
2019/12/31-17:45:06 MODIFY /tmp/nohup.log
2019/12/31-17:45:06 MODIFY /tmp/nohup.log
2019/12/31-17:45:06 MODIFY /tmp/nohup.log
2019/12/31-17:45:22 CREATE /tmp/hsperfdata_dmadmin/510
2019/12/31-17:45:22 MODIFY /tmp/hsperfdata_dmadmin/510
2019/12/31-17:45:22 MODIFY /tmp/hsperfdata_dmadmin/510
2019/12/31-17:45:50 CREATE /tmp/runc-process785366568
2019/12/31-17:45:50 MODIFY /tmp/runc-process785366568
...


Let’s see if the directory is really watched recursively:

$ mkdir -p /tmp/dir1/dir2
> 2019/12/31-17:49:51 CREATE,ISDIR /tmp/dir1
> 2019/12/31-17:49:51 CREATE,ISDIR /tmp/dir1/dir2


We notice that the directory creation is trapped too.

$ touch /tmp/dir1/dir2/xx
> 2019/12/31-17:49:59 CREATE /tmp/dir1/dir2/xx
> 2019/12/31-17:49:59 ATTRIB /tmp/dir1/dir2/xx


It works as advertised. Moreover, two events are raised here, CREATE for the file creation as it didn’t previously exist, and ATTRIB for the change of timestamp. Let’s verify this:

$ touch /tmp/dir1/dir2/xx
> 2019/12/31-17:50:01 ATTRIB /tmp/dir1/dir2/xx
$ touch /tmp/dir1/dir2/xx
> 2019/12/31-17:50:03 ATTRIB /tmp/dir1/dir2/xx

Indeed.
Let’s see if a change of attribute is also noticed:

$ chmod -r /tmp/dir1/dir2/xx
2019/12/31-18:03:50 ATTRIB /tmp/dir1/dir2/xx
$ chmod +r /tmp/dir1/dir2/xx
2019/12/31-18:03:55 ATTRIB /tmp/dir1/dir2/xx

It is, fine.

Using inotify in a [containerized] Documentum installation

inotifywait will be used with the syntax shown above against the Documentum log directory ${DOCUMENTUM}/dba/log. Its output will be piped into a gawk script that will spawn “tail -F” commands when needed and keep a list of such processes so they can be killed after a timeout of inactivity, i.e. when they are following a file that did not get updated within a given time interval (let’s jokingly name this value “tail time to live”, or TTTL). This is to prevent a potentially unlimited number of such processes to hog the system’s resources, not that they consume much CPU cycles but it is pointless to leave hundreds of such idle processes sitting in memory. So the script will start a tail command on a file upon its CREATE event, or a MODIFY event if not already followed, and clean up existing idle tail commands. As said before, the code below could be included into a container’s entrypoint to make it run constantly in the background.

#!/bin/bash

export watcher_workdir=/tmp/watcher
export tail_on_off=${watcher_workdir}/tail_on_off
export heartbeat_file=${watcher_workdir}/heartbeat_file
export pause_duration=3

# comma-separated list of files watched but excluded from tailing;
export excluded_files="$tail_on_off,$heartbeat_file"

# make private copies of inotifywait and tail so they can be easily identified and killed from the list of running processes;
export private_inotify=~/dctm-inotifywait
export private_tail=~/dctm-tail

follow_logs() {
   # tail time to live, maximum duration in minutes an unmodified tailed file will stay tailed before the tail is killed, i.e. TTTL ;-);
   tail_timeout=1
 
   $private_inotify --quiet --monitor --event create,modify,attrib --recursive --timefmt "%Y/%m/%d-%H:%M:%S" --format "%T|%e|%w%f" ${DOCUMENTUM}/dba/log $watcher_workdir | gawk -v tail_timeout=$((tail_timeout * 60)) -v excluded_files="$excluded_files" -v tail_on_off=$tail_on_off -v heartbeat_file=$heartbeat_file -v env_private_tail=private_tail -v FS="|" -v Apo="'" 'BEGIN {
      # get the dedicated tail command from the environment;
      private_tail=ENVIRON[env_private_tail]

      # get current time;
      cmd_now = "date +\"%Y/%m/%d-%H:%M:%S\""
 
      # get the time of the next check, i.e. now + the timeout delay;
      cmd_future = "date +\"%Y/%m/%d-%H:%M:%S\" --date=\"" tail_timeout  " seconds\""
      cmd_future | getline next_check_date; close(cmd_future)
 
      # find the running private tail commands;
      # use the same FS defined in the outer gawk scope to guarantee thet $0 can be split smoothly;
      cmd_tail = "pgrep -fa " private_tail " | gawk " Apo "{print $1 \"" FS "\" $NF}" Apo
 
      # get the most recent time a running idle tail command is allowed to live;
      cmd_oldest = "date +\"%Y/%m/%d-%H:%M:%S\" --date=\"" tail_timeout  " seconds ago\""
 
      # command to get the modifiy date of a tailed file;
      cmd_file_stamp = "date +\"%Y/%m/%d-%H:%M:%S\" --reference="
 
      # files can be excluded from tailing if specified in parameter exclude_files;
      # convert the space-separated list of excluded files to an associative array, which is easier to search as it is indexed by file names;
      nb_files = split(excluded_files, tmp_excluded_files, ",")
      for (;nb_files > 0; nb_files--)
         tab_excluded_files[tmp_excluded_files[nb_files]] = 0

      bMust_tail = 1
   }
   {
      # skip directories (they have a trailing /);
      if (match($3, /\/$/)) next
 
      # check if logs must be tailed;
      if (tail_on_off == $3 && ("CREATE" == $2 || "MODIFY" == $2)) {
         if ((getline bMust_tail < tail_on_off) >  0) {
            close(tail_on_off)
            system("rm " tail_on_off " 2>/dev/null")
         }
      } else 
         # tailing on ?
         if (bMust_tail && !($3 in current_tails))
            # CREATE event ?
            if ("CREATE" == $2 && !($3 in tab_excluded_files)) {
               # newly created file not in exclusion list: tail it !
               system(private_tail " --follow=name --retry " $3 " 2>/dev/null &")
               # ... and keep track of it;
               current_tails[$3] = 0
            }
            # MODIFY event ?
            else if ("MODIFY" == $2 && !($3 in tab_excluded_files)) {
               # modified file not in exclusion list nor already tailed: tail it !
               system(private_tail " --follow=name --retry " $3 " 2>/dev/null &")
               # ... and keep track of it;
               current_tails[$3] = 0
            }
 
      # clean up logic starts here;
      # only upon a heartbeat event;
      if ("ATTRIB" == $2 && $3 == heartbeat_file) {
         # time to check ?
         cmd_now | getline now; close(cmd_now)
         if (now >= next_check_date) {
            # maximum time to live for idle tails;
            cmd_oldest | getline oldest_date; close(cmd_oldest)
            # loop though all the running tail commands;
            while ((cmd_tail | getline) > 0) {
               # cannot explain this spurious entry, ignoring it explicitly eventhough it will anyway; 
               if ("$NF}" Apo == $2) continue
               tail_pid = $1
               tailed_file = $2
               # is it one of the watched files ?
               if (tailed_file in current_tails) {
                  # get the current tailed file last modification date;
                  (cmd_file_stamp tailed_file " 2>/dev/null") | getline last_modif_date; close(cmd_file_stamp tailed_file " 2>/dev/null")
                  # tailed file not updated during time to live period ?
                  if (last_modif_date <= oldest_date) {
                     # yes, kill the tailing process;
                     system("kill -9 " tail_pid " 2> /dev/null")
                     # ... and update the list of tailed files;
                     delete current_tails[tailed_file]
                  }
                  else current_tails[tailed_file] = 1
               }
               # else it should not happen because private tail commands are only issues from here and get tracked;
            }
            close(cmd_tail)

            # resynchronize internal list with actual tailed files;
            for (f in current_tails)
               if (0 == current_tails[f])
                  # tracked file not tailed any more (this should not happen because private tail commands are killed from here only), untrack it;
                  delete current_tails[f]
               else
                  # tracked file was checked and is still alive;
                  current_tails[f] = 0
 
            # compute the next check time;
            cmd_future | getline next_check_date; close(cmd_future)
         }
      }
   }'
}
 
 
# -----------
# main;
# -----------

# create an inotifywait alias so it is easily identified to be stopped later;
ln -s /usr/bin/inotifywait $private_inotify 2> /dev/null
ln -s /usr/bin/tail $private_tail           2> /dev/null

# create the watcher's workdir;
mkdir -p $watcher_workdir 2> /dev/null

# enable following new or modified logs;
echo 1 > $tail_on_off
 
# start the watcher;
follow_logs &

while [ true ]; do
   # send heartbeat to inotify;
   touch $heartbeat_file
   # do something else here;
   # ...
   sleep $pause_duration
done

Admittedly, the fact that gawk relies a lot on external commands and pipes (because of its lack of libraries, but this can be arranged by extending it, see other articles in this blog such this one) obfuscates somewhat the statements’ purpose. Also, all the logic is contained in the not-so-obvious automatic loop which is executed each time an event is notified. Anyway, as usual the code is only provided as an illustration.
On line 3, a “technical” sub-directory is defined and created later on line 134. It will contain the heartbeat file (file heartbeat_file, see next paragraph for more details) and a file to switch tailing on and off (file tail_on_off). This directory is watched (see the call to inotifywait on line 19).
One line 12 and 13, private versions of the commands inotifywait and tail are defined and created later on line 130 and 131 as symlinks; this is to facilitate their identification in the list of running processes to kill them if needed.
This script is encapsulated in function follow_logs() started on line 15. It is launched in the background on line 140.
On line 46 the boolean bMust_tail is initialized; it gets updated on line 53 upon a CREATE event on the file $tail_on_off; after it has been read into bMust_tail, the file is remove. By writing 0 or a positive number in it, aggregation of new logs is respectively disabled or enabled:

# disable following new logs;
# from within a container or in a traditional installation:
$ echo 0 > $tail_on_off
 
# or, for a container, supposing $tail_on_off is defined in the container's current user's environment:
$ docker exec container_name /bin/bash -c "echo 0 > \$tail_on_off"
 
# enable following new logs;
# from within a container or in a traditional installation:
$ echo 1 > $tail_on_off
 
# or, for a container, supposing $tail_on_off is defined in the container's current user's environment:
$ docker exec container_name /bin/bash -c "echo 1 > \$tail_on_off"

Currently running tail commands are not impacted.
Note how the $ in $tail_on_off is escaped so it is not consumed by the docker’s host shell and is passed as-is to the container.
On line 63 and 70, private tails commands are started when new files have appeared in the watched sub-directory, or modified files which are not already tailed.
One line 79, a clean-up of idle (private) tail commands is performed and the internal associative array that keeps track of them is refreshed to make sure it reflects the currently running tails.
One line 142, the well-known container entrypoint’s never-ending loop is entered.
The above script can also be run stand-alone in a classic, non containerized installation.

Keeping the gawk script busy

A problem to solve is that the gawk script runs as a coroutine to inotify, i.e. it is synchronized with its output. If there is none, because no event were detected, the script blocks waiting for input, so no cleaning of idle tail commands gets done. This not such a big deal because no new tails commands are started either so the status quo is maintained. However, an easy work-around is possible: In order to force the script to enter the next iteration and give it a chance to perform the cleaning, we introduce a heartbeat with one watched file, e.g.:

...
while [ true ]; do
   # send heartbeat to inotify;
   touch $heartbeat_file
   # do something else here;
   # ...
   sleep $pause_duration
done

…
This explains the presence of the file $heartbeat_file in the list of inotifywait’s target (see line 19 in the code above).
Now, because its timestamp is updated, at least one event is always raised periodically, an ATTRIB on the heartbeat file $heartbeat_file, as shown before. Although ATTRIB events are listened to, no action is done upon them generally, except when occuring on $heartbeat_file in which case their sole purpose is to trigger the script’s execution, more precisely, the cleanup of inactive tail commands.

Let’s test it now.

Cet article Tracking Logs Inside a Documentum Container (part I) est apparu en premier sur Blog dbi services.

Testing the log watcher

This is part II of the article. Part I is here.
All the above code has to be included in the entrypoint script so it gets executed at container start up time but it can also be tested more simply in a traditional repository installation.
First, we’ll move the code into a excutable script, e.g. entrypoint.sh, and run it in the background in a first terminal. Soon, we will notice that lots of log messages get displayed, e.g. from jobs executing into the repository:

---------------------------------------
Job Arguments:
(StandardJobArgs: docbase_name: dmtest.dmtest user_name: dmadmin job_id: 0800c35080007042 method_trace_level: 0 )
window_interval=120
queueperson=null
max_job_threads=3
 
---------------------------------------
2020-01-02T07:03:15.807617 1600[1600] 0100c3508000ab80 [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab80 is owned by user dmadmin now."
Executing query to collect XCP automatic tasks to be processed:
select r_object_id from dmi_workitem where a_wq_name = 'to be processed by job' and r_auto_method_id != '0000000000000000' and (r_runtime_state = 0 or r_runtime_state = 1) order by r_creation_date asc
Report End 2020/01/02 07:03:15
2020-01-02T07:03:16.314586 5888[5888] 0100c3508000ab90 [DM_SESSION_I_SESSION_QUIT]info: "Session 0100c3508000ab90 quit."
Thu Jan 02 07:04:44 2020 [INFORMATION] [LAUNCHER 6311] Detected during program initialization: Version: 16.4.0080.0129 Linux64
2020-01-02T07:04:44.736720 6341[6341] 0100c3508000ab91 [DM_SESSION_I_SESSION_START]info: "Session 0100c3508000ab91 started for user dmadmin."
Thu Jan 02 07:04:45 2020 [INFORMATION] [LAUNCHER 6311] Detected while preparing job dm_Initialize_WQ for execution: Agent Exec connected to server dmtest: [DM_SESSION_I_SESSION_START]info: "Session 0100c3508000ab91 started for user dmadmin."
 
 
2020-01-02T07:04:45.686337 1600[1600] 0100c3508000ab80 [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab80 is owned by user dmadmin now."
2020-01-02T07:04:45.698970 1597[1597] 0100c3508000ab7f [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab7f is owned by user dmadmin now."
Initialize_WQ Report For DocBase dmtest.dmtest As Of 2020/01/02 07:04:45
 
---------------------------------------
Job Arguments:
(StandardJobArgs: docbase_name: dmtest.dmtest user_name: dmadmin job_id: 0800c3508000218b method_trace_level: 0 )
window_interval=120
queueperson=null
max_job_threads=3
 
---------------------------------------
Starting WQInitialisation job:
2020-01-02T07:04:45.756339 1600[1600] 0100c3508000ab80 [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab80 is owned by user dmadmin now."
Executing query to collect Unassigned worktiems to be processed:
select r_object_id, r_act_def_id from dmi_workitem where a_held_by = 'dm_system' and r_runtime_state = 0 order by r_creation_date
Total no. of workqueue tasks initialized 0
Report End 2020/01/02 07:04:45
2020-01-02T07:04:46.222728 6341[6341] 0100c3508000ab91 [DM_SESSION_I_SESSION_QUIT]info: "Session 0100c3508000ab91 quit."
Thu Jan 02 07:05:14 2020 [INFORMATION] [LAUNCHER 6522] Detected during program initialization: Version: 16.4.0080.0129 Linux64
2020-01-02T07:05:14.828073 6552[6552] 0100c3508000ab92 [DM_SESSION_I_SESSION_START]info: "Session 0100c3508000ab92 started for user dmadmin."
Thu Jan 02 07:05:15 2020 [INFORMATION] [LAUNCHER 6522] Detected while preparing job dm_bpm_XCPAutoTaskMgmt for execution: Agent Exec connected to server dmtest: [DM_SESSION_I_SESSION_START]info: "Session 0100c3508000ab92 started for user dmadmin."
 
 
2020-01-02T07:05:15.714803 1600[1600] 0100c3508000ab80 [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab80 is owned by user dmadmin now."
2020-01-02T07:05:15.726601 1597[1597] 0100c3508000ab7f [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab7f is owned by user dmadmin now."
bpm_XCPAutoTaskMgmt Report For DocBase dmtest.dmtest As Of 2020/01/02 07:05:15
 
---------------------------------------
 

Then, from a second terminal, we’ll start and stop several idql sessions and observe the resulting output. We will notice the familiar lines *_START and *_QUIT from the session’s logs:

---------------------------------------
2020-01-02T07:09:16.076945 1600[1600] 0100c3508000ab80 [DM_SESSION_I_ASSUME_USER]info: "Session 0100c3508000ab80 is owned by user dmadmin now."
Executing query to collect XCP automatic tasks to be processed:
select r_object_id from dmi_workitem where a_wq_name = 'to be processed by job' and r_auto_method_id != '0000000000000000' and (r_runtime_state = 0 or r_runtime_state = 1) order by r_creation_date asc
Report End 2020/01/02 07:09:16
2020-01-02T07:09:16.584776 7907[7907] 0100c3508000ab97 [DM_SESSION_I_SESSION_QUIT]info: "Session 0100c3508000ab97 quit."
2020-01-02T07:09:44.969770 8080[8080] 0100c3508000ab98 [DM_SESSION_I_SESSION_START]info: "Session 0100c3508000ab98 started for user dmadmin."
2020-01-02T07:09:47.329406 8080[8080] 0100c3508000ab98 [DM_SESSION_I_SESSION_QUIT]info: "Session 0100c3508000ab98 quit."
...

So, inotifywatch is pretty effective as a file watcher.

Let’se see how many tail processes are currently running:

$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 4818 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/agentexec.log
1 4846 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab80
1 4850 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab7f
1 8375 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c3508000218b
1 8389 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab99
1 8407 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/Initialize_WQDoc.txt
1 8599 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c35080000386
1 8614 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9a
1 8657 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9b
1 8673 24411 9328 pts/1 8723 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/DataDictionaryPublisherDoc.txt

And after a while:

$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 4818 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/agentexec.log
1 4846 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab80
1 4850 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab7f
1 8599 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c35080000386
1 8614 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9a
1 8657 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9b
1 8673 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/DataDictionaryPublisherDoc.txt
1 8824 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c35080007042
1 8834 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9c
1 8852 24411 9328 pts/1 9132 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/bpm_XCPAutoTaskMgmtDoc.txt


Again:

$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 10058 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/agentexec.log
1 10078 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c3508000218b
1 10111 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab9f
1 10131 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab80
1 10135 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab7f
1 10139 24411 9328 pts/1 10252 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/Initialize_WQDoc.txt


And again:

$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 10892 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/agentexec.log
1 10896 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/job_0800c3508000218b
1 10907 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000aba1
1 10921 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab80
1 10925 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab7f
1 10930 24411 9328 pts/1 11022 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/sysadmin/Initialize_WQDoc.txt


And, after disabling, the aggregation of new logs:

$ echo 0 > $tail_on_off
$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 24676 24277 9328 pts/1 26096 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/agentexec/agentexec.log
1 24710 24277 9328 pts/1 26096 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab80
1 24714 24277 9328 pts/1 26096 S 1001 0:00 tail --follow=name --retry /app/dctm/dba/log/0000c350/dmadmin/0100c3508000ab7f

And eventually:

$ psg tail | grep -v gawk
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
<nothing>

From now on, until the aggregation is turned back on, this list will be empty as no new tail commands will be started.
The number of tail commands grows and shrinks as expected, so the process clean up is working.
The agentexec.log file is often here because jobs are started frequently and therefore this file regularly triggers the MODIFY event. dmadmin and sysadmin owned session logs come and go for each started job. The DataDictionaryPublisherDoc.txt, bpm_XCPAutoTaskMgmtDoc.txt and Initialize_WQDoc.txt are a few followed job’s logs.
In the used test system, an out of the box docbase without any running applications, the output is obviously very quiet with long pauses in between but it can become extremely dense in a busy system. When investigating problems in such systems, it can be useful to redirect the whole output to a text file and use one’s favorite editor to search it at leisure. It is also possible to restrict it to an as narrow as desired time window (if docker logs is used) in order to reduce its noise, exclude files from being watched (see the next paragraph) and even to stop aggregating new logs via the $tail_on_off file so only the currently ones are followed as long as they are active (i.e. written in).

Dynamically reconfiguring inotifywait via a parameter file

In the preceding examples, the inotifywait command has been passed a fixed, hard-coded subtree name to watch. In certain cases, it could be useful to be able to watch a different, random sub-directory or list of arbitrary files in unrelated sub-directories with a possible list of excluded files; for even more flexibility, the other inotifywait’s parameters could also be changed dynamically. In such cases, the running inotifywait command will have to be stopped and restarted to change its command-line parameters. One could imagine to check a communication file (like the $tail_on_off file above) inside the entrypoint’s main loop, as shown below. Here the diff is relative to the precedent static parameters version:

10a11,13
> # file that contains the watcher's parameters;
> export new_inotify_parameters_file=${watcher_workdir}/inotify_parameters_file
> 
15a19,20
>    inotify_params="$1"
>  
19c24
    eval $private_inotify ${inotify_params} $watcher_workdir | gawk -v tail_timeout=$((tail_timeout * 60)) -v excluded_files="$excluded_files" -v bMust_tail=1 -v tail_on_off=$tail_on_off -v heartbeat_file=$heartbeat_file -v env_private_tail=private_tail -v FS="|" -v Apo="'" 'BEGIN {
122a128,142
> process_new_inotify_parameters() {
>    # read in the new parameters from file $new_inotify_parameters_file;
>    # first line of the $new_inotify_parameters_file must contains the non-target parameters, e.g. -m -e create,modify,attrib -r --timefmt "%Y/%m/%d-%H:%M:%S" --format "%T %e %w%f";
>    new_params=$(cat $new_inotify_parameters_file)
>    rm $new_inotify_parameters_file
>  
>    # kill current watcher;
>    pkill -f $private_inotify
> 
>    # kill the current private tail commands too;
>    pkill -f $private_tail
> 
>    # restart inotify with new command-line parameters taken from $new_inotify_parameters_file;
>    follow_logs "$new_params" &
> }
138,139c158,162
< # start the watcher;
 # default inotifywait parameters;
> # the creation of this file will trigger a restart of the watcher using the new parameters;
> cat - < $new_inotify_parameters_file
>    --quiet --monitor --event create,modify,attrib --recursive --timefmt "%Y/%m/%d-%H:%M:%S" --format '%T|%e|%w%f' \${DOCUMENTUM}/dba/log --exclude \$new_inotify_parameters_file
> eot
143a167
>    [[ -f $new_inotify_parameters_file ]] && process_new_inotify_parameters

To apply the patch, save the first listing into a file, e.g. inotify_static_parameters.sh, the above diff output into a file, e.g. static2dynamic.diff, and use the command below to create the script inotify_dynamic_parameters.sh:

patch inotify_static_parameters.sh static2dynamic.diff -o inotify_dynamic_parameters.sh

In order not to trigger events when it is created, the $new_inotify_parameters_file must be excluded from inotifywait’s watched directories.
If, for instance, we want later to exclude jobs’ logs in order to focus on more relevant logs, we could use the following inotifywait’s parameters:

# cat - <<-eot > $new_inotify_parameters_file
   --quiet --monitor --event create,modify,attrib --recursive --timefmt "%Y/%m/%d-%H:%M:%S" --format "%T|%e|%w%f" \${DOCUMENTUM}/dba/log --exclude '${DOCUMENTUM}/dba/log/dmtest/agentexec/job_.*' --exclude $new_inotify_parameters_file
eot

From the outside of a container:

docker exec <container> /bin/bash -c 'cat - < $new_inotify_parameters_file
   --quiet --monitor --event create,modify,attrib --recursive --timefmt "%Y/%m/%d-%H:%M:%S" --format "%T|%e|%w%f" \${DOCUMENTUM}/dba/log --exclude "\${DOCUMENTUM}/dba/log/dmtest/agentexec/job_.*" --exclude \$new_inotify_parameters_file
eot
'

where the new option −−exclude specifies a POSIX-compliant regular expression to be quoted so the shell does not perform its variable expansions.
After at most $pause_duration seconds, this file is detected (line 34), read (line 14), deleted (line 15) and the current watcher gets restarted in the background with the new parameters (line 24).
This approach is akin to polling the parameter file and it looks a bit unsophisticated. We use events to control the tailing through the $tail_on_off file and for the heartbeat. Why not for the parameter file ? The next paragraph will show just that.

Dynamically reconfiguring inotifywait via an event

A better way would be to use the watcher itself to check if the parameter file has changed ! After all, we shall put our money where our mouth is, right ?
In order to do this, and dedicated watcher is set up to watch the parameter file. For technical reasons (in order to watch a file, that file must already exist. Also, when the watched file is erased, inotifywait does not react to any events on that file any longer; it just sits there idly), instead of watching $new_inotify_parameters_file, it watches its parent directory. To avoid scattering files in too many locations, the parameter file will be stored along with the technical files in ${watcher_workdir} and, in order to avoid impacting the performance, it will be excluded from the main watcher (parameter –exclude \$new_inotify_parameters_file, do not forget to append it otherwise the main watcher will raise events for nothing; it should not impede the parameter file to be processed though).
When a MODIFY event on this file occurs, which includes a CREATE event if the file did not pre-exist, the event is processed as shown precedently.
Here the diff compared to the preceding polled parameter file version:

143a144,151
> # the parameter file's watcher;
> param_watcher() {
>    IFS="|"
>    inotifywait --quiet --monitor --event create,modify,attrib --timefmt "%Y/%m/%d-%H:%M:%S" --format '%T|%e|%w%f' $watcher_workdir --exclude \$tail_on_off \$heartbeat_file | while read timestamp event target_file; do
>       [[ $target_file == $new_inotify_parameters_file ]] && process_new_inotify_parameters
>    done
> }
> 
162a171,173
> 
> # starts the parameter file's watcher;
> param_watcher &

The delta is quite short with just the function param_watcher() to start the watcher on the parameter file and process its signal by invoking the same function process_new_inotify_parameters() introduced in the polling version.
To apply the patch, save the above diff output into a file, e.g. dynamic2event.diff, and use the command below to create the script inotify_dynamic_parameters_event.sh from the polling version’s script inotify_dynamic_parameters.sh created above:

patch inotify_dynamic_parameters.sh dynamic2event.diff -o inotify_dynamic_parameters_event.sh

The dedicated watcher runs in the background and restarts the main watcher whenever the latter receives new parameters. Quite simple.

Dynamically reconfiguring inotifywait via signals

Yet another way to interact with inotifywait’s script (e.g. the container’s entrypoint) could be through signals. To this effect, we’ll need first to choose suitable signals, say SIGUSR[12], define a trap handler and implement it. And while we are at it, let’s also add switching the watcher on and off, as illustrated below:

2a3,5
> trap 'bounce_watcher' SIGUSR1
> trap 'toggle_watcher' SIGUSR2
> 
7a11,14
> # new global variable to hold the current tail on/off status;
> # used to toggle the watcher;
> export bMust_tail=1
> 
144,149c151,162
< # the parameter file's watcher;
< param_watcher() {
<    IFS="|"
<    inotifywait --quiet --monitor --event create,modify,attrib --timefmt "%Y/%m/%d-%H:%M:%S" --format '%T|%e|%w%f' $watcher_workdir --exclude \$tail_on_off \$heartbeat_file | while read timestamp event target_file; do
<       [[ $target_file == $new_inotify_parameters_file ]] && process_new_inotify_parameters
 # restart the watcher with new parameters;
> bounce_watcher() {
>    [[ -f $new_inotify_parameters_file ]] && process_new_inotify_parameters
>    echo "parameters reloaded"
> }
> 
> # flip the watcher on/off;
> # as file $tail_on_off is watched, modifying it will trigger the processing of the boolean bMust_tail in the gawk script;
> toggle_watcher() {
>    (( bMust_tail = (bMust_tail + 1) % 2 ))
>    echo $bMust_tail > $tail_on_off
>    echo "toggled the watcher, it is $bMust_tail now"
172,173c185,187
< # starts the parameter file's watcher;
 process_new_inotify_parameters
> 
> echo "process $$ started"
178d191
<    [[ -f $new_inotify_parameters_file ]] && process_new_inotify_parameters

To apply the patch, save the above diff output into a file, e.g. event2signals.diff, and use the command below to create the script inotify_dynamic_parameters_signals.sh from the event version’s script inotify_dynamic_parameters_event.sh created above:

patch inotify_dynamic_parameters_events.sh event2signals.diff -o inotify_dynamic_parameters_signals.sh

The loop is simpler now as the functions are directly invoked by outside signals.
To use it, just send the SIGUSR[12] signals to the container, as shown below:

# write a new configuration file:
...
# and restart the watcher:
$ docker kill --signal SIGUSR1 <container>
 
# toggle the watcher;
$ docker kill --signal SIGUSR2 <container>

If testing outside a container, the commands would be:

/bin/kill --signal SIGUSR1 pid
# or:
/bin/kill --signal SIGUSR2 pid

where pid is the pid displayed when the script starts.
We won’t indulge more on signals, see How to stop Documentum processes in a docker container, and more (part I) for many more examples of using signals to talk to containers.

Conclusion

inotifywait is simple to configure and extremely fast, pratically instantaneous, at sending notifications. Although the aggregated output looks a bit confusing when the system is loaded, there are several ways to reduce its volume to make it easier to use. It is an interesting addition to an administrator to help troubleshooting repositories and their client applications. With suitable parameters, it can be used to support any containerized or traditional installations. It is one of those no frills tools that just work (mostly) as expected. If you ever happen to need a file watcher, consider it, you won’t be disappointed and it is fun to use.

Cet article Tracking Logs Inside a Documentum Container (part II) est apparu en premier sur Blog dbi services.

A few months ago, I was working on automating an LSS installation for a customer so I thought about sharing my experience around that topic… LSS is the Life Sciences product from OpenText (formerly EMC->Dell->OpenText). It can be split into four sub-products, which can be installed independently or all together, depending on the needs. These four products are the LSTMF (eTrial Master File Solution), LSQM (Quality and Manufacturing Solution), LSRD (Research and Development Solution) and finally the LSSSV (Submission Store & View Solution). When they are all bundled together, they form the Life Sciences Suite. In this blog, I will talk about the LSS product globally.

The LSS installation can be done either manually or silently. Contrary to most OpenText products which uses installers (.exe, .bin) to provide either a GUI or silent processing, the silent installation of the LSS is just a shell script or to be exact some shell scripts that you will need to execute one by one (depending on what you need exactly). In this case, my task was to automate the LSS installation to be able to integrate that into Kubernetes using Ansible playbooks. Therefore, OpenText provides some shell scripts to silently install the LSS but there is still some preparation work to be done before and some verifications after. The content of the LSS package looks like that:

[dmadmin@cspg-0 ~]$ workspace="/tmp/lss/"
[dmadmin@cspg-0 ~]$ package="LSSuite_16.6.1000.0126.tar"
[dmadmin@cspg-0 ~]$
[dmadmin@cspg-0 ~]$ mkdir -p ${workspace}
[dmadmin@cspg-0 ~]$ cd ${workspace}
[dmadmin@cspg-0 lss]$ cp /tmp/${package} ./
[dmadmin@cspg-0 lss]$ tar -xf ${package}
[dmadmin@cspg-0 lss]$
[dmadmin@cspg-0 lss]$ cd ./*/
[dmadmin@cspg-0 LSSuite]$ ls -l
total 216
drwxr-x--- 3 dmadmin dmadmin   152 Dec  2 17:09 CDFD2Plugins
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 ControlledPrintPluginJars
-rw-r----- 1 dmadmin dmadmin 13479 Oct  4 09:15 LSConfigImport.sh
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 SDK
drwxr-x--- 3 dmadmin dmadmin   152 Oct 21 09:52 Util
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 config
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 config_exports
-rw-r----- 1 dmadmin dmadmin   544 Oct  4 09:15 copyright.txt
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 dars
-rw-r----- 1 dmadmin dmadmin  4231 Oct  4 09:15 iHubConfigImport.sh
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 iHubDependentJar
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 iHubJDBCConnector
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 iHubReports
-rw-r----- 1 dmadmin dmadmin   294 Oct  4 09:15 ihubsolution.properties
-rw-r----- 1 dmadmin dmadmin  8467 Oct  4 09:15 install.properties
-rw-r----- 1 dmadmin dmadmin  8384 Oct  4 09:15 install.sh
-rw-r----- 1 dmadmin dmadmin 28838 Oct  4 09:15 install.xml
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 javadocs
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 lib
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 method_server_lib
-rw-r----- 1 dmadmin dmadmin  3096 Oct  4 09:15 myInsightPostInstall.sh
-rw-r----- 1 dmadmin dmadmin   224 Oct  4 09:15 nodmadmin.installparam.xml
-rw-r----- 1 dmadmin dmadmin   981 Oct  4 09:15 readme.txt
drwxr-x--- 3 dmadmin dmadmin  8192 Dec  2 17:09 scripts
drwxr-x--- 2 dmadmin dmadmin   152 Dec  2 17:09 sharepointconnector
-rw-r----- 1 dmadmin dmadmin  1779 Oct  4 09:15 solution.properties
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 updateVersion
-rw-r----- 1 dmadmin dmadmin    69 Oct  4 09:15 version.txt
drwxr-x--- 2 dmadmin dmadmin  8192 Dec  2 17:09 war
[dmadmin@cspg-0 LSSuite]$

Above, you have all the files/folders that come with the LSS package. There are several “.sh” scripts, that’s the shell scripts for the silent installation and they are using parameters/variables that have been defined in the different “.properties” files. What changes between the different LSS products – in addition to the DARs/JARs – is the value of the variables in these properties files.

In this blog, there are a few important things that I wanted to highlight:

1. You shouldn’t blindly trust what OpenText is providing as variables’ values. I have seen, several times, wrong values in properties files because of copy/paste from previous versions or worse, copy/paste from other LSS products. Therefore, before installing a new version/product, the first thing that I usually do is checking and validating that all variables have been set properly according to what I expect the soft to do. This obviously need some knowledge on how the LSS is behaving based on the parameters’ value. If you have some customization or some specific requirements, it’s also the moment to add/update the properties files to match your needs.
2. Before installing, you should really check carefully the release note. This is obvious for most people, but I know it can be quite painful to do for all products. Because the LSS “installer” is actually just shell script, it can very easily contain mistakes or typos, contrary to most Documentum binaries (or not!). So check the release note of the LSS because there are often known issues and bugs in the released products, even inside the shell scripts and variables’ values… Therefore, taking a few minutes for that will be worth it, that’s for sure, to make sure there is no known bugs with the silent scripts/variables before running them.
3. OpenText doesn’t always manage the return/exit codes (to not say never). If you don’t know what it is, return/exit codes can be seen as a numerical representation of the result of a command. In UNIX for example, each command that you will execute will return an exit code which can be checked. A value of 0 usually means that the command completed successfully while a value from 1 to 255 means the opposite, for different reasons (a value of 2 means a different issue compared to a value of 1). While scripting, that’s usually a very good and efficient (the best?) way to make sure everything is going as planned. The issue here is that OpenText is using the return codes at some places but clearly not everywhere… Therefore, you have maybe something like 75% of the scripting that can fail, and the installation will still continue anyway. So, what to do with that? Well, either you update the scripts provided by OpenText or you make sure to check thoroughly the log files generated by the installation. Updating the scripts isn’t really a good idea since they will be overwritten for each patch, so it only leaves you with not so much of a choice…

Let’s talk a little bit more about this third point. The simplest way to demonstrate that is to purposely try to execute the shell scripts with missing parameters and check the return code:

[dmadmin@cspg-0 LSSuite]$ cat version.txt; echo
OpenText Life Sciences Solution Suite
Build Version = 16.6.1000.0126
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ export LS_DOCKER_INSTALL=true
[dmadmin@cspg-0 LSSuite]$ ./install.sh
Installation started on
12022019
Please make sure the JMS is stopped before running the script
./install.sh: line 38: [: =: unary operator expected
./install.sh: line 38: [: =: unary operator expected
./install.sh: line 41: [: =: unary operator expected
./install.sh: line 41: [: =: unary operator expected
Please provide the supported Content Server version [7.1, 7.2, 7.3, 16.4].
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ echo $?
1
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ ./LSConfigImport.sh
Installation started on
12022019
Please provide the supported Content Server version
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ echo $?
0
[dmadmin@cspg-0 LSSuite]$

As you can see above, to silently install the LSS, you will need to set the variable “LS_DOCKER_INSTALL” to true first, otherwise it will prompt you to fill some variables’ value. If you are trying to execute the first shell script (install.sh), then it will fail because there are missing parameters (it expects 4 parameters) and the return code is 1, meaning that something went wrong: that’s the expected result so this is good! Doing the same thing for the second shell script (LSConfigImport.sh) will also fail because of missing parameters (it expects 14 parameters) but this time, the return code is 0, meaning that there were “no errors”… However, nothing has been done since it actually failed. How are you supposed to detect that the installation failed in this case? Well, that’s a good question! LSS will generate some logs that will be put in the working directory so a “solution” can be to make sure that all the expected logs are present and that their content is also “correct”… That’s not really a piece of cake but is there any other choice?

I have been installing LSS dozens of times and I saw several times errors reported on the logs that weren’t displayed on the screen during the execution of the scripts. And vice versa! Here is an example:

[dmadmin@cspg-0 LSSuite]$ cd ./working/logs/
[dmadmin@cspg-0 logs]$
[dmadmin@cspg-0 logs]$ grep -i _E_ *
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'jp-regional_1-0-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'stf-2-2-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
LSSuite_postInstall.log:[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'us-regional-2-3-normalize.xslt' due to Operating System error."
LSSuite_postInstall.log:[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."
[dmadmin@cspg-0 logs]$

The above errors for example are all linked to the same iAPI script but just looking at these messages, it’s not always that easy to find out where the issue is coming from. Most of the log files are just an aggregation of iAPI/iDQL session logging without any evidences of which script is being executed or what’s its purpose. For the above errors, finding the correct iAPI script was pretty easy because the messages contain three file names. Therefore, with just a simple grep command you can find that there is an iAPI script named “addRenditions.api” that seems to be the cause of these errors. I have seen cases where the errors were too generic and therefore, I had to count the number of iAPI/iDQL logins in the log (number of message “Connected to OpenText Documentum Server running…”). Then list all the iAPI/iDQL scripts that are being executed based on the properties files. Then loop through all these iAPI/iDQL scripts and count again the number of logins to find which command/script generated the errors… Easy, right?

Anyway, this particular issue is documented in the installation guide of the LSS. For some reasons, OpenText created an iAPI script and used a relative path for the files but it’s not correct because of how the LSS executes it… The solution is therefore to update the command to use the full path of the file:

[dmadmin@cspg-0 logs]$ cd ../../
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ cat ./scripts/scripts/addRenditions.api
retrieve,c,cd_ectd_xml_schema where object_name='jp-regional_1-0'
addrendition,c,l,jp-regional_1-0-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='stf_2-2'
addrendition,c,l,stf-2-2-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='us-regional_2-3'
addrendition,c,l,us-regional-2-3-normalize.xslt,xsl_ns
save,c,l
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ ls -l $(pwd)/scripts/scripts/stf-2-2-normalize.xslt
-rw-r----- 1 dmadmin dmadmin 3748 Oct  4 08:48 /tmp/lss/LSSuite/scripts/scripts/stf-2-2-normalize.xslt
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ iapi REPO1 -Udmadmin -Pxxx -r$(pwd)/scripts/scripts/addRenditions.api

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12345001568c started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Postgres
Session id is s0
API> ...
080f123450005514
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'jp-regional_1-0-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> ...
080f123450005518
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'stf-2-2-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> ...
080f12345000551e
API> ...
[DM_SYSOBJECT_E_CANT_ACCESS_FILE]error:  "Cannot access file 'us-regional-2-3-normalize.xslt' due to Operating System error."

API> ...
[DM_API_E_CANT_UPDATE]error:  "Cannot Save/Check-in object <> because the object is marked invalid.  This is most likely due to an error from a prior operation performed on the object."

API> Bye
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ sed -i "s@^addrendition,c,l,\([a-zA-Z]\)@addrendition,c,l,$(pwd)/scripts/scripts/\1@" ./scripts/scripts/addRenditions.api
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ cat ./scripts/scripts/addRenditions.api
retrieve,c,cd_ectd_xml_schema where object_name='jp-regional_1-0'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/jp-regional_1-0-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='stf_2-2'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/stf-2-2-normalize.xslt,xsl_ns
save,c,l

retrieve,c,cd_ectd_xml_schema where object_name='us-regional_2-3'
addrendition,c,l,/tmp/lss/LSSuite/scripts/scripts/us-regional-2-3-normalize.xslt,xsl_ns
save,c,l
[dmadmin@cspg-0 LSSuite]$
[dmadmin@cspg-0 LSSuite]$ iapi REPO1 -Udmadmin -Pxxx -r$(pwd)/scripts/scripts/addRenditions.api

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase REPO1
[DM_SESSION_I_SESSION_START]info:  "Session 010f12345001568e started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Postgres
Session id is s0
API> ...
080f123450005514
API> ...
OK
API> ...
OK
API> ...
080f123450005518
API> ...
OK
API> ...
OK
API> ...
080f12345000551e
API> ...
OK
API> ...
OK
API> Bye
[dmadmin@cspg-0 LSSuite]$

My experience with the OpenText Life Sciences product(s) in the past few years is that it’s always an adventure… It might be fun or interesting but before all, it’s most probably going to be a little bit challenging ;).

Cet article Documentum – A first experience with the Life Sciences Suite 16.6.1 silent installation est apparu en premier sur Blog dbi services.

D2-Smartview is a new UI that OpenText now provides starting with the version 16 of D2. It’s a lightweight UI that can perform some of the actions that D2 does. The list of features will probably increase with time but at the moment, I guess it’s more for simple users that have very basic needs, consumer like roles mainly. An interesting thing is that with a small configuration, users can switch between D2 and D2-Smartview on the fly.

The issue I wanted to talk about in this blog is a class cast exception that might while trying to download a document using D2-Smartview that has been deployed on a WebLogic Server:

2020-02-21 10:42:11,168 UTC [INFO ] ([ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)') - c.e.d.a.c.documentset.D2DocumentSetSwitch     : D2DocumentSetSwitch.getDocumentSetList end: 0.000s
2020-02-21 10:42:11,184 UTC [INFO ] ([ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.d2fs.dctm.utils.FormatUtils           : Compiled format pattern [a-zA-Z_\-0-9]+{1,32}
2020-02-21 10:42:11,405 UTC [INFO ] ([ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.common.dctm.utils.DfServerUtil        : loadBalancedContentServer set to false
2020-02-21 10:42:11,681 UTC [INFO ] ([ACTIVE] ExecuteThread: '95' for queue: 'weblogic.kernel.Default (self-tuning)') - c.e.d.d.w.services.config.D2X3ConfigService   : D2X3ConfigService.getAvailableWidgets end : 0.119s
2020-02-21 10:42:11,903 UTC [ERROR] ([ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.d2fs.dctm.servlets.D2HttpServlet      : Download could not parse request body
2020-02-21 10:42:11,908 UTC [ERROR] ([ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.d2fs.dctm.servlets.D2HttpServlet      : Request parsing failed
java.lang.ClassCastException: weblogic.security.principal.WLSUserImpl cannot be cast to com.emc.d2fs.authc.HttpAuthPrincipal
        at com.emc.d2fs.dctm.servlets.D2HttpContext.<init>(D2HttpContext.java:259)
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGetAndPost(D2HttpServlet.java:360)
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGet(D2HttpServlet.java:113)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3705)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3672)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:328)
        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
        at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
        at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1705)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1665)
        at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
2020-02-21 10:42:11,931 UTC [ERROR] ([ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)') - com.emc.documentum.rest.util.LogHelper        : LogId: 2516f8e0-f920-437c-86f3-502b3ec8ad14, Status: 500, code: E_INTERNAL_SERVER_ERROR, message: An internal server error occurs.
java.lang.ClassCastException: weblogic.security.principal.WLSUserImpl cannot be cast to com.emc.d2fs.authc.HttpAuthPrincipal
        at com.emc.d2fs.dctm.servlets.D2HttpContext.<init>(D2HttpContext.java:259)
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGetAndPost(D2HttpServlet.java:360)
        at com.emc.d2fs.dctm.servlets.D2HttpServlet.doGet(D2HttpServlet.java:113)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3705)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3672)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:328)
        at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
        at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
        at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
        at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2443)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2291)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2269)
        at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1705)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1665)
        at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
        at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
        at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
        at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
        at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:652)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
2020-02-21 10:42:13,972 UTC [ERROR] ([ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)') - c.e.d.d.w.s.b.D2BravaCSRAnnotationService     : Invalid object id:
2020-02-21 10:42:13,978 UTC [ERROR] ([ACTIVE] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)') - c.e.d.d.d.i.D2BravaCSRAnnotationsManagerImpl  : Error getting changemark for document 090f123480003979

D2-Smartview is using, in some part, REST services and therefore you might find several similarities between D2-REST and D2-Smartview for example. If you are familiar with the D2-REST application on WebLogic, you might know that OpenText usually asks you to remove a certain set of JARs, then disable some security, aso… It’s not surprising that the documentation for D2-Smartview also asks the same kind of thing. For D2-REST, I personally never had to remove JARs (except this one, which isn’t in the list from OpenText anyway) and I never had to disable the basic authentication. This is because I could always rely on LDAP authentication. Most companies rely on LDAP nowadays and therefore you usually don’t need to disable WebLogic basic authentication on D2-REST because you can just configure WebLogic to use the LDAP Server for authentication and D2-REST will use the same account from the Repository (assuming it is also integrated with the LDAP).

Therefore, when I started to do some engineering on D2-Smartview, to deploy and test it, I obviously didn’t follow all the documentation provided by OpenText, only the parts that I usually do for D2-REST because I know that’s sufficient. These are the things that OpenText asks you to do for D2-Smartview 16.5.1 that I simply ignored:

• Disable web service annotation scan
• Add “-Dweblogic.servlet.DIDisabled=true” to JVM parameters of the Managed Server
• Remove JARs files: jsr*.jar, stax-api-*.jar, stax2-api-*.jar, xml-apis-*.jar, xmlParserAPIs-*.jar
• Disable WebLogic basic authentication: <enforce-valid-basic-auth-credentials>false</enforcevalid-basic-auth-credentials>

With the above ignored, I could deploy successfully D2-Smartview 16.5.1. The login was working properly with my LDAP account, I could browse the repository, perform searches, see properties of documents, aso… However, there was one thing not working: the download of document which failed with the above exception. As you can probably tell yourself, the exception is related to class cast exception from WebLogic principal (weblogic.security.principal.WLSUserImpl) to D2-Smartview principal (com.emc.d2fs.authc.HttpAuthPrincipal). Therefore, this was most probably linked to the basic authentication that I kept enabled just like for D2-REST. It looked like for D2-Smartview, it was really needed to disable it.

Usually, I don’t like to disable security because you never know… This is especially true for the WebLogic basic authentication because this configuration applies to the whole WebLogic Domain! OpenText claims that disabling the basic authentication on WebLogic isn’t a security issue for their applications (DA, D2, D2-Config, D2-REST, D2-Smartview, d2ls, XMLViewer, DFS, …) because each of these handle the authentication directly. However, at some point in the future, some custom application might be deployed on that domain that expected WebLogic to perform the authentication so you might end-up with security holes. Unfortunately for D2-Smartview, it looks like there is no way around it at the moment (contrary to D2-REST), so disabling the basic authentication is needed if you expect it to work fully:

[weblogic@ws-0 ~]$ stopDOMAIN

The server 'msDA-01' has been stopped successfully.
The server 'msD2Conf-01' has been stopped successfully.
The server 'msD2SV-01' has been stopped successfully.
The server 'msD2-01' has been stopped successfully.
The server 'AdminServer' has been stopped successfully.
The NodeManager has been stopped successfully.

[weblogic@ws-0 ~]$
[weblogic@ws-0 ~]$ grep -A1 "valid-basic" $DOMAIN_HOME/config/config.xml
[weblogic@ws-0 ~]$
[weblogic@ws-0 ~]$ sed -i 's,.*</security-configuration>,    <enforce-valid-basic-auth-credentials>false</enforce-valid-basic-auth-credentials>\n&,' $DOMAIN_HOME/config/config.xml
[weblogic@ws-0 ~]$
[weblogic@ws-0 ~]$ grep -A1 "valid-basic" $DOMAIN_HOME/config/config.xml
    <enforce-valid-basic-auth-credentials>false</enforce-valid-basic-auth-credentials>
  </security-configuration>
[weblogic@ws-0 ~]$
[weblogic@ws-0 ~]$ startDOMAIN

The NodeManager has been started successfully.
The server 'AdminServer' has been started successfully.
The server 'msDA-01' has been started successfully.
The server 'msD2Conf-01' has been started successfully.
The server 'msD2SV-01' has been started successfully.
The server 'msD2-01' has been started successfully.

[weblogic@ws-0 ~]$

As you can see above, I updated the config.xml directly but the preferred way to do it is using WLST. There are several explanations on that on the internet. Once that’s done, the download of files is then working as expected. Maybe OpenText will change the source code so that it doesn’t need that at some point in the future but until then at least, no real other solution. The other three requirements from the documentation are still not in place on my side and I didn’t find anything not working so far so I guess I will just continue to ignore them (especially the “DIDisabled” one).

Cet article Documentum – D2-Smartview class cast exception est apparu en premier sur Blog dbi services.

As discussed in a previous blog, working with LSS might prove a little bit challenging from time to time. In this blog, I wanted to share an error I saw while installing LSS 16.6.1 on an Oracle database. Initially, I developed my silent installation for LSS (while encapsulate the LSS silent scripts provided by OpenText) using a PostgreSQL database because it’s usually easier to setup an environment on Kubernetes with PG because of licenses.

So, the silent installation scripts were created several months ago and working since then, apparently. Recently, I had to execute manually my silent install script of LSS on an environment which was using an Oracle database. The script completed properly, my automatic log file checking didn’t show any sign or errors or anything so for me it was fully installed. However, I still did a review of the logs printed on the screen to be sure and I did see a new “error” I wasn’t familiar with. I’m not sure you can call that an error because it’s just one line drowned in the flood of logs printed without any “ERROR” or “_E_” messages but it is clearly an error from a Linux point of view:

...
./registerOracleVersionView.sh: line 1: oracle: command not found
...

This message never appeared in the generated log file of the LSS installation, it’s only displayed on the screen, which makes it… quite difficult to see in automation. So, anyway, what’s the issue this time? Well looking at the message, it’s clear that the shell script has a wrong content because it is trying to execute a command “oracle” which doesn’t exist. Where is this file? What’s its content?

[dmadmin@cs-0 ~]$ workspace="/tmp/lss/"
[dmadmin@cs-0 ~]$
[dmadmin@cs-0 ~]$ cd ${workspace}/*/
[dmadmin@cs-0 LSSuite]$
[dmadmin@cs-0 LSSuite]$ ls -l *.sh
-rwxr-x--- 1 dmadmin dmadmin 13479 Oct  4 09:15 LSConfigImport.sh
-rwxr-x--- 1 dmadmin dmadmin  4231 Oct  4 09:15 iHubConfigImport.sh
-rwxr-x--- 1 dmadmin dmadmin  8384 Oct  4 09:15 install.sh
-rwxr-x--- 1 dmadmin dmadmin  3096 Oct  4 09:15 myInsightPostInstall.sh
[dmadmin@cs-0 LSSuite]$
[dmadmin@cs-0 LSSuite]$ find . -name registerOracleVersionView.sh
./scripts/registerOracleVersionView.sh
[dmadmin@cs-0 LSSuite]$
[dmadmin@cs-0 LSSuite]$ cd ./scripts/
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ cat registerOracleVersionView.sh
if  "$4" == "oracle"
then
        idql "$1" -U"$2" -P"$3" -R"./scripts/$4/oracleVersion.dql"
fi
[dmadmin@cs-0 scripts]$

If you are familiar with bash/shell scripting, you probably already saw what’s wrong with the script. It’s simply that this isn’t the correct way to write IF statements. I won’t go into the details of the correct formatting (one bracket, two brackets, with test command, aso…) because there are already plenty of documentation around that online but that’s definitively not a correct way to write IF statements. So, to correct this script, I opened the OpenText SR#4450083 and provided them the commands to fix it in a future patch/release. I didn’t receive a confirmation yet but it should be in the next LSS release. In the meanwhile, I put the workaround on my silent install script (if the correct format is already there it won’t be anything but if it’s not, then it will correct the file):

[dmadmin@cs-0 scripts]$ cat registerOracleVersionView.sh
if  "$4" == "oracle"
then
        idql "$1" -U"$2" -P"$3" -R"./scripts/$4/oracleVersion.dql"
fi
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ ./registerOracleVersionView.sh Repo01 dmadmin xxx oracle
./registerOracleVersionView.sh: line 1: oracle: command not found
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ sed -i -e 's,^if[[:space:]]*",if \[\[ ",' -e 's,^if \[\[ .*"$,& \]\],' registerOracleVersionView.sh
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ cat registerOracleVersionView.sh
if [[ "$4" == "oracle" ]]
then
        idql "$1" -U"$2" -P"$3" -R"./scripts/$4/oracleVersion.dql"
fi
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ ./registerOracleVersionView.sh Repo01 dmadmin xxx oracle

        OpenText Documentum idql - Interactive document query interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f1234800113af started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
1> 2> result
------------
T
(1 row affected)
1> 2> new_object_ID
----------------
190f1234800edc59
(1 row affected)
1>
[dmadmin@cs-0 scripts]$
[dmadmin@cs-0 scripts]$ cat ./scripts/oracle/oracleVersion.dql
execute exec_sql with query = 'create view oracle_version as select * from v$version'
go
REGISTER TABLE dm_dbo.oracle_version (banner String(80))
go[dmadmin@cs-0 scripts]$

As you can see above, the shell executes a DQL script “oracleVersion.dql”. This simply creates a new view “oracle_version”. I have no clue where this might be used in LSS but what I can tell you is that this script was already wrong in LSS 16.6.0 (released in Jul 2019 I believe) and nobody complained about it so far apparently, so maybe you can wait for the official fix from OpenText or you can fix it yourself like I did, up to you!

Cet article Documentum – LSS registerOracleVersionView script with wrong content est apparu en premier sur Blog dbi services.

Recently in a project, one of the Documentum environments we were working on started showing failure on the execution of the dm_DMFilescan job and only this one. After a couple minutes of investigation, I quickly found the root cause of the issue which wasn’t very important, but I thought I would share it because of the implications it has. In addition to that, it’s also not the first time I’m seeing this kind of issue so let’s dig into it.

The error shown in the job log was the following one:

[dmadmin@cs-0 ~]$ cd $DOCUMENTUM/dba/log/000f1234/sysadmin
[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat ../agentexec/job_080f12348000035c
Sun Mar 15 08:50:32 2020 [INFORMATION] [LAUNCHER 25934] Detected while preparing job dm_DMFilescan for execution: Agent Exec connected to server Repo01:  [DM_SESSION_I_SESSION_START]info:  "Session 010f123480032063 started for user dmadmin."

[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat DMFilescanDoc.txt
DMFilescan Report For DocBase Repo01 As Of 3/15/2020 09:50:37

Remove orphan content older than 168 hours.
  Generated DMFilescan script will be executed...
  The trace level is set to 0...

DMFilescan utility syntax: apply,c,NULL,DO_METHOD,METHOD,S,dmfilescan,ARGUMENTS,S,'-grace_period 168 '
Executing DMFilescan...
Unable to execute DMFilescan method...
[DM_METHOD_E_INVALID_MTHD_VERB]error:  "The dm_method named (dmfilescan) of type dmbasic has invalid method_verb (./dmfilescan)."


Exiting...
Report End  3/15/2020 09:50:37
[dmadmin@cs-0 sysadmin]$

Before explaining what happened, I think it’s necessary to take some time to explain how the execution of the dm_DMFilescan is done. The job “dm_DMFilescan” is a content type job which is used to scan storage areas in order to find files that do not have any relation to database objects. You can use it in report mode only (nothing done) or not for example. There are more details on the OpenText documentation but the thing to note here is that this job will execute the method of the same name: “dm_DMFilescan”. This method is a dmbasic one and it’s actually just an encapsulation. What I mean by that is it will actually execute a common dmbasic for several jobs, the only thing that changes are the parameters of the method. There are four jobs that will use the exact same dmbasic script and you can find the list easily:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,select object_name, method_verb from dm_method where method_verb like '%mthd1.ebs%'
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f123480032086 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> object_name      method_verb
--------------------  ---------------------------------------------------------
dm_ContentWarning     ./dmbasic -f../install/admin/mthd1.ebs -eContentWarning
dm_DMClean            ./dmbasic -f../install/admin/mthd1.ebs -eDMClean
dm_DMFilescan         ./dmbasic -f../install/admin/mthd1.ebs -eDMFilescan
dm_DMArchive          ./dmbasic -f../install/admin/mthd1.ebs -eDMArchive
(4 rows affected)

API> Bye
[dmadmin@cs-0 sysadmin]$

Take a look at these “mthdx.ebs” files if you aren’t familiar with them, it’s always good to know how it actually works. This dmbasic method will “only” prepare and execute another method which is then specific to each of the jobs. For the “dm_DMFilescan” dmbasic method for example, it will execute the “dmfilescan” method. This second method isn’t a dmbasic one, it’s a binary. So why am I saying all that? Well it’s simply to explain where the issue is coming from… When reading the error message from the log file, one might thing at first sight that the issue is with the “dm_DMFilescan” method but actually it’s not. The issue was with the second method and if you read it carefully, you actually have all key items to solve it. As said, the “dmfilescan” method is executing a binary but on the error message above, it is saying that this method is of type dmbasic currently. Comparing the final method for the “dm_DMFilescan” and “dm_DMClean” jobs, you can see the error:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,select object_name, method_type, method_verb from dm_method where object_name in ('dmfilescan','dmclean')
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f12348003209b started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> object_name      method_type    method_verb
--------------------  -------------  --------------
dmclean                              ./dmclean
dmfilescan            dmbasic        ./dmfilescan
(2 rows affected)

API> Bye
[dmadmin@cs-0 sysadmin]$

What happened, in my opinion, is that someone opened the properties of the “dmfilescan” method in Documentum Administrator but instead of using “Cancel” to close it, he clicked on “OK” which then saved the current configuration. Unfortunately, with DA, there are often cases where opening a property page can load values that aren’t currently configured and therefore when you click on “OK”, it will save back a value that wasn’t the initial one… I have seen that quite often on job’s target_server for example when working on HA environment: you have a job configured to run on ANY (‘ ‘) and while opening it on DA, it can show the Primary CS target (‘Repo.Repo@cs-0’ for example). Therefore, while clicking on “OK”, it will change the target_server of this job. I believe this is what happened here for this specific method because by default it’s a binary and therefore the method_type should be ‘ ‘ but in this case, it was changed recently to dmbasic. Changing the method_type back to the correct value and the job is working again:

[dmadmin@cs-0 sysadmin]$ iapi ${repo} -Udmadmin -Pxxx << EOC
> ?,c,update dm_method object set method_type=' ' where object_name='dmfilescan'
> EOC

        OpenText Documentum iapi - Interactive API interface
        Copyright (c) 2018. OpenText Corporation
        All rights reserved.
        Client Library Release 16.4.0170.0080

Connecting to Server using docbase Repo01
[DM_SESSION_I_SESSION_START]info:  "Session 010f1234800320a7 started for user dmadmin."

Connected to OpenText Documentum Server running Release 16.4.0170.0234  Linux64.Oracle
Session id is s0
API> objects_updated
---------------
              1
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "1 objects were affected by your UPDATE statement."

API> Bye
[dmadmin@cs-0 sysadmin]$
[dmadmin@cs-0 sysadmin]$ cat DMFilescanDoc.txt | grep -v ^#
DMFilescan Report For DocBase Repo01 As Of 3/15/2020 10:03:21

Remove orphan content older than 168 hours.
  Generated DMFilescan script will be executed...
  The trace level is set to 0...

DMFilescan utility syntax: apply,c,NULL,DO_METHOD,METHOD,S,dmfilescan,ARGUMENTS,S,'-grace_period 168 '
Executing DMFilescan...
Shell was successful!

Generated script from the DMFilescan method:
----- Start $DOCUMENTUM/dba/log/000f1234/sysadmin/080f12348000035c.bat output ------------------------

------- End $DOCUMENTUM/dba/log/000f1234/sysadmin/080f12348000035c.bat output ------------------------
Destroying DMFilescan script with ID 090f12348003d932...
Report End  3/15/2020 10:05:27
[dmadmin@cs-0 sysadmin]$

So, take care while using Documentum Administrator because it might surprise you.

Cet article Documentum – dm_DMFilescan fails with invalid method_verb? est apparu en premier sur Blog dbi services.

As you probably know, Documentum introduced the optional use of a Lockbox since version 7. It was done initially to contribute to the security of the AEK key which is one of the central components of a Content Server. Time showed that, well, it’s not so easy and since then, nothing much changed on the Lockbox layer. With the version 16.4, OpenText introduced some new things regarding the Lockbox like the upgrade from version 3 to 4. On this blog, I will talk about one of the changes and that is renaming the file on RCS/CFS in case there is already one present with the same name. This blog will not discuss a technical issue because it’s not an issue, but it might become one, so I just wanted to share some thoughts about that.

As you know, when installing a repository in Documentum, you can either choose to use the basic AEK key or use a Lockbox that will contain it. Usually, people tend to use the basic AEK key, most probably because of the Documentum history: most companies are using Documentum for a very long time and therefore, there were no other choice at the beginning and it’s only an optional step to upgrade the AEK key to the Lockbox so it’s often left on the side. The second trend is to use a single Lockbox for all repositories of a specific server (Global Registry Repository + normal Repository(ies)). If you are in this case, then installing a Primary/First CS is the same thing as installing a Remote/Second CS (aka CFS). Or rather it was the case until Documentum 7.3 included.

Before talking about what changed, it’s important to know how Documentum works exactly in regard to the Lockbox. While installing an RCS/CFS, the installer will connect to the Primary CS to read data from the Repository. In the installation process, it will also execute the script “$DM_HOME/install/admin/dm_rcs_copyfiles.ebs”. What this script does is:

1. Connect to the Primary CS
2. Create a dm_method named dm_rcs_copyfiles
3. Execute the dm_method dm_rcs_copyfiles on the Primary CS
4. The above dm_method will create a dm_document named dm_rcs_copydoc that will contain all the global files of the Repository:
   • AEK key or Lockbox file (page 0)
   • dbpasswd.txt (page 1)
   • server.ini (page 2)
   • webcache.ini (page 3)
   • rkmrootcert.pem – if any
   • rkm_config.ini – if any
   • ldap_*.cnt – if any (7.3+)
5. Download the files from the dm_document on the RCS/CFS to a temporary folder
6. Destroy the dm_document and dm_method

Therefore, in regard to the Lockbox, it will basically take the file from the Repository on the Primary CS and put it on the RCS/CFS for usage. This is needed because of encryption mechanisms that uses the Lockbox. So, what changed exactly? Well, until Documentum 7.3, the RCS/CFS did copy the Lockbox from the Primary CS for each and every Repositories but it was just overwriting any files with the same name at the same location. Therefore, if you are using the default $DOCUMENTUM/dba/secure/lockbox.lb name on the Primary CS and using it for all Repositories, then on the RCS/CFS, it would just create one Lockbox file with the same name at the same location and overwriting it each and every time a new RCS/CFS is created.

You can potentially see what could go wrong with that. Let’s say that you have two different environments:

• a first one with GR_A and RepoA, both using the same lockbox.lb
• a second one with GR_B and RepoB, both using the same lockbox.lb

With the above, if you were trying to install an RCS/CFS for RepoA and then for RepoB using the same binaries on the same server, then the Lockbox of RepoA would be overwritten by the one of RepoB. The change that has been introduced in 16.4 (it might have been backported to some latest patches of 7.3 but at least it wasn’t there in the early patches) is that when you are installing a RCS/CFS, if a Lockbox already exist with the same name (lockbox.lb), then the installer will rename it to <RepoName>_lockbox.lb on the RCS/CFS and it will then update the server.ini to match the new name obviously. This means that the first RCS/CFS installed on the remote server (usually the Global Registry Repository) will continue to use lockbox.lb because it doesn’t exist at that time BUT the second RCS/CFS (usually the normal Repository) will use the <RepoName>_lockbox.lb file because lockbox.lb is already taken… I assume that the above problem happened for some customers of OpenText and therefore, this solution/workaround was probably implemented. Another possibility is that they needed that for the cloud setup to make sure no repositories overlap each other’s.

So, in the end, this is a as-good-as-it-can-be solution for this specific installation problem. However, if you are part the second group of people that usually use a single Lockbox for all Repositories of a single host and you don’t mix-up environments, then you might want to keep the old name. There are already a lot of differences between a Primary CS and a CFS/RCS (start script name, server.ini name, log file name) so unless there is a good reason (i.e.: if it’s needed), I would personally continue to use lockbox.lb for all Repositories of the RCS/CFS. This is in the perspective of keeping the servers aligned as much as possible to simplify the maintenance work. Obviously, the final decision is up to you.

If you want to keep using the same Lockbox on the RCS/CFS for all Repositories, then after the installation, you can just update the server.ini. Please note that we are always using server.ini on all Content Servers, no matter if it’s a Primary or a Remote. On RCS/CFS, this is actually a symlink to the real file name to simplify our work and therefore the below command uses the “–follow-symlinks” option:

[dmadmin@cs-1 ~]$ repo=Repo1
[dmadmin@cs-1 ~]$ lockbox_name=lockbox.lb
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ ls -l $DOCUMENTUM/dba/config/${repo}/server.ini
lrwxrwxrwx 1 dmadmin dmadmin 67 Oct 14 12:03 $DOCUMENTUM/config/${repo}/server.ini -> $DOCUMENTUM/config/${repo}/server_cs-1_${repo}.ini
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ grep "crypto_" $DOCUMENTUM/dba/config/${repo}/server.ini
crypto_mode = AES256_RSA1024_SHA256
crypto_keystore = Local
crypto_lockbox = Repo1_lockbox.lb
crypto_keyname = CSaek
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ sed -i --follow-symlinks "s/^\(crypto_lockbox[[:space:]]*=[[:space:]]*\).*/\1${lockbox_name}/" $DOCUMENTUM/dba/config/${repo}/server.ini
[dmadmin@cs-1 ~]$
[dmadmin@cs-1 ~]$ grep crypto_ $DOCUMENTUM/dba/config/${repo}/server.ini
crypto_mode = AES256_RSA1024_SHA256
crypto_keystore = Local
crypto_lockbox = lockbox.lb
crypto_keyname = CSaek
[dmadmin@cs-1 ~]$

Then simply restart the Repository. In case you made a mistake and the file lockbox.lb wasn’t the same for all Repositories, then it just won’t start. That was just my 2-cents on this change, if you are a different opinion, feel free to share! I didn’t test because I usually use the Lockbox, but I assume OpenText implemented the same thing for the AEK key?

Cet article Documentum – Rename of Lockbox on RCS/CFS est apparu en premier sur Blog dbi services.