Intermittent 'cannot open database' errors using ODatabasePool with OrientDB 3.0.18

Occasional errors are seen when trying to query: saying ODatabaseException: Cannot open database ‘databases’

In my code the stack trace is:
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database ‘databases’
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleException(OChannelBinaryAsynchClient.java:338)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:285)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:307)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:196)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:158)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1953)
at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:348)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:407)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:331)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:365)
at com.orientechnologies.orient.client.remote.OStorageRemote.query(OStorageRemote.java:1028)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.query(ODatabaseDocumentRemote.java:380)

. . . . .

The code is like this:
There is an ODatabasePool instance that has been created:
pool = new ODatabasePool(orientDB, settings.getDatabase(), settings.getUsr(), settings.getPwd(), config);

Each time I want to execute a query I acquire an ODatabaseSession instance from this pool and call activateOnCurrentThread() on it.

Then I build a statement with some constant strings and some parameters and call query in a try block where the finally block closes both the result set and the ODatabaseSession:
e.g.
ODatabaseSession conn = null;
OResultSet rs = null;
try {
conn = orientDBFactory.getDB();
List results = new ArrayList<>();
String statement = MSHT_GUID_SELECT + GUID;
Map<String, Object> params = new HashMap<>();
params.put(GUID, guid);
rs = conn.query(statement, params);
while (rs.hasNext()) {
OResult item = rs.next();
results.add(fromDoc(item.toElement()));
}

	return (results.isEmpty() ? Optional.empty() : Optional.ofNullable(results.get(0)));
} finally {
	closeRs(rs);
	closeConn(conn);
}

where closeRs and closeConn are defined as follow:
private void closeRs(OResultSet rs) {
if (rs != null) {
rs.close();
}

}
private void closeConn(ODatabaseSession conn) {
if (conn != null && conn.isActiveOnCurrentThread()) {
conn.close();
}

}

fromDoc() just converts from OElement to the type of java POJO I want to return. It does not use do anything with orientdb connections or APIs.

hi @MaryPercival

can you try to upgrade to 3.0.22 and check if the error is still present?

Thanks

Thanks for your quick reply. I will try that.

We have been using 3.0.22 for nearly a week now and it is a lot better - instead of getting these errors about 12 times per working day we have only seen 1 occurrence in the last week, but it was the same error:
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database ‘databases’
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
. . .
at com.orientechnologies.orient.client.remote.OStorageRemote.query(OStorageRemote.java:1028)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.query(ODatabaseDocumentRemote.java:380)

Hi @MaryPercival

do you have the entire stacktrace of the exception?

Thanks

Do you see some exception on the server logs?

Yes there is a error in the server log:
2019-08-02 01:05:14:745 INFO Storage ‘plocal:/orientdb/databases’ is opened under OrientDB distribution : 3.0.22 - Veloce (build 8a
fc634a2ea9c351898ae57dee8d739ff4851252, branch 3.0.x) [OLocalPaginatedStorage]Exception 378C3157 in storage plocal:/orientdb/data bases/databases: 3.0.22 - Veloce (build 8afc634a2ea9c351898ae57dee8d739ff4851252, branch 3.0.x)
com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage ‘databases’ because it does not exist in path
: /orientdb/databases/databases
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:366)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:236)
at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:59)
at com.orientechnologies.orient.server.OServer.openDatabase(OServer.java:923)
at com.orientechnologies.orient.server.OClientConnection.init(OClientConnection.java:235)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.onBeforeOperationalRequest(ONetworkProtocolBinary.java:463)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:265)
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:211)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:69)

I included the last timestamped message from that log as well so you can see it is 3.0.22 but the actual error occurred at 10:05 as shown in this full stack from our application log:
2019-08-02 10:05:01.869 ERROR [msht-association-service,af70e7034c8f3c06,b9594ee462251c87,false] [ http-nio-8040-exec-9] []
o.s.c.s.i.web.ExceptionLoggingFilter : Uncaught exception thrown

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is com.orientechnologies.orient.cor
e.exception.ODatabaseException: Cannot open database ‘databases’
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:48)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at brave.servlet.TracingFilter.doFilter(TracingFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:155)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:123)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database ‘databases’
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleException(OChannelBinaryAsynchClient.java:338)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:285)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:307)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:196)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:158)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1953)
at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:348)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:407)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:331)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:365)
at com.orientechnologies.orient.client.remote.OStorageRemote.query(OStorageRemote.java:1028)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentRemote.query(ODatabaseDocumentRemote.java:380)

Hi @MaryPercival

i see that there is a space here

plocal:/orientdb/data bases/databases probably it’s due to that

That space was just due to the copy and paste of the text from the log file. Please see this screenshot

Hi @MaryPercival

can you paste here the ls of the directories

- orientdb
- orientdb/databases

thanks

root@ab00a62d69fa:/orientdb# ls
backup bin config databases history.txt lib license.txt log plugins readme.txt www
root@ab00a62d69fa:/orientdb# ls databases
OFunction.name_26.him OSystem _studio_49.cpm config_4.bd dictionary_14.cbt e_39.cpm manindex_11.cpm ofunction_24.pcl osequence_30.pcl v_34.pcl
OFunction.name_27.hit OUser.name_22.cbt collections_13_47.sbc config_5.nd dictionary_15.nbt index_8.pcl mshtentity_1_44.pcl ofunction_25.cpm osequence_31.cpm v_35.cpm
OFunction.name_28.hib OUser.name_23.nbt collections_14_46.sbc databases.429.wal dirty.fl index_9.cpm mshtentity_1_45.cpm orole_16.pcl ouser_20.pcl
OFunction.name_29.hnb _studio_1_50.pcl command-cache.json databases.wmr e_1_40.pcl internal_6.pcl mshtentity_42.pcl orole_17.cpm ouser_21.cpm
ORole.name_18.cbt _studio_1_51.cpm config_2.cd default_12.pcl e_1_41.cpm internal_7.cpm mshtentity_43.cpm oschedule_32.pcl v_1_36.pcl
ORole.name_19.nbt _studio_48.pcl config_3.ccm default_13.cpm e_38.pcl manindex_10.pcl name_id_map_v2.cm oschedule_33.cpm v_1_37.cpm

Note that orientdb is running as a docker image, I don’t know if that is relevant.

Hi @MaryPercival

that is the problem. the folder orientdb/databases is a container of OrientDB databases. You can see the OSystem folder which is an internal database. But then in the same folder “orientdb/databases” you have another database expanded (with all the files) inside.

You should create databases folder inside orientdb/databases and move all the file except the OSystem folder.

The layout should look like this


- orientdb/databases
   - OSystem
   - databases

Then it should solve your issue and you can access to your database with the name databases that i guess is the one you are using from your application

Let me now if this helps

OK thank you we will do that but it is strange that it works most of the time and just gives intermittent errors. Also I don’t think our devops guy did anything particular with the docker image to define the structure for the databases and the db probably got auto-created the first time the application connected to the image.

Also the application database is not called ‘databases’ . In the properties file it is given a different name:
orientdb.database=orientdb-msht
The db connection is done using ODatabasePool which picks up all the settings from a configuration class
pool = new ODatabasePool(orientDB, settings.getDatabase(), settings.getUsr(), settings.getPwd(), config);
The database name is picked up from the orientdb.database property in the properties file.

To give the full initialisation code it is like this:
OrientDBConfig config = OrientDBConfig.builder()
.addConfig(OGlobalConfiguration.DB_POOL_MIN, settings.getConnectionPoolMin())
.addConfig(OGlobalConfiguration.DB_POOL_MAX, settings.getConnectionPoolMax()).build();
orientDB = new OrientDB(settings.getUrl(), settings.getUsr(), settings.getPwd(), config);
pool = new ODatabasePool(orientDB, settings.getDatabase(), settings.getUsr(), settings.getPwd(), config);

Ok seems,
i don’t know how the db has been unfolded into the databases directory

but this

com.orientechnologies.orient.core.exception.OStorageException: Cannot open the storage ‘databases’ because it does not exist in path

means that the name of the database given was databases in the API

Thanks

We have not made any changes to the database folder structure in the docker container before or after deploying version 3.0.22 as I thought I would leave things as they were and see how many errors occurred over a longer period.

I have checked the logs today (21st August) and am pleased to say that we have not had any more of these ‘Cannot open database’ errors since the one on 2nd August (and that was the only one we had after deploying version 3.0.22 on 29th July) so moving to version 3.0.22 has resolved the issue for us.

1 Like

This issue can be closed