Wrong out().size() results after massive parallel insertion on ODB 3.0.2


#1

I’ve got database with transaction and entity vertex types and edges of different label that are outgoing from transactions to entities. My database is in ‘automerge’ conflict strategy, and i’m doing massive optimistic batch insertion in several threads in a way that same transaction and/or it’s edges could be in multiple batches.
I’m also using Bonsai structure for edge storage, i.e. ‘ridBag.embeddedToSbtreeBonsaiThreshold=-1’.
After massive insertion is completed, i’ve ran two queries:

  1. select sum(out().size()) from transaction
  2. select count(*) from E where inV() is not null and outV() is not null
    and it gave me different results (1st one returned less records, than a second, second result is matching total number of edges).
    “repair database” command didn’t fix anything, and reversed 2nd query (check for edges with null vertices) returned zero count.
    There are no errors during insertion whatsoever, neither on my Java application side nor on OrientDB server application side.
    Storage is running on ‘remote’ mode on localhost.
    When I’m running massive batch insertion in serial (one thread), both queries return the same number. Could it be because some vertices are not updated in terms of Bonsai structure pointer or something? Please advice.
    UPD: setting ‘ridBag.embeddedToSbtreeBonsaiThreshold=2000000000’ has led to extremely slow insertion in parallel, but both queries return same result, as well as in case with single-thread insertion. So I assume there is some problem with Bonsai structure pointers.

#2

Hi @biggysmallz

The first quick advice I can give you is to upgrade to latest v 3.0.12 and try again, we did a lot of fixes since V 3.0.2 and there is a significant chance that the problem is already fixed.
If it doesn’t work, having a reproducer to test it locally would allow us to do some debugging and see where the problem is

Thanks

Luigi


#3

Hi @luigidellaquila, thank you for answering!

Same thing reproduces on ODB 3.0.12.
In terms of reproducer - providing the code that makes an insertion, insertion batches dump (inside project RAR sources, each file is one batch) and test DB backup with predefined schema and conflict strategy.
https://dropmefiles.com/xjz79
I’ve tried this reproducer locally on ODB 3.0.12, 2 times out of 3 bug was reproduced.
Also there were changes in server properties in orientdb-server-config.xml, there is my properties tag:
entry value=“false” name=“profiler.enabled”
entry value=“512” name=“storage.wal.maxSize”
entry value=“67108864” name=“memory.chunk.size”
entry value=“65536” name=“network.binary.maxLength”
entry value=“4096” name=“storage.diskCache.bufferSize”
entry value="-1" name="ridBag.embeddedToSbtreeBonsaiThreshold"

Nothing else was changed, AFAIR.
Thank you in advance! Looking forward for your reply.


#4

Any news on this ticket? Also, please let me know the appropriate way for you to upload/give access to reproducer. Is it possible to post it with this community site tool, or you need some git repository with it?
Thank you in advance.