ETL importing JSON example file problem

Hi,
I am new to OrientDB, so please forgive me if not posting in he right group.

I am trying to understand the JSON importation module from ETL and tried to run the example given in the manual, with no success.
Thus, I installed Orientdb 3.0 on MacOSX (server and Studio).
Then, I slightly modified the configuration file, as follows (file import.json):
{
“config”: {
“log”: “debug”
},
“source” : {
“file”: { “path”: “/Users/armand/GIT/markdown-report/examples/orient/database.json” }
},
“extractor” : {
“json”: {}
},
“transformers” : [
{ “merge”: { “joinFieldName”: “id”, “lookup”: “Account.id” } },
{ “vertex”: { “class”: “Account”} },
{ “edge”: {
“class”: “Friend”,
“joinFieldName”: “friends”,
“lookup”: “Account.id”,
“unresolvedLinkAction”: “CREATE”
} }
],
“loader” : {
“orientdb”: {
“dbURL”: “remote:localhost/demodb”,
“dbUser”: “admin”,
“dbPassword”: “admin”,
“dbAutoDropIfExists”: true,
“dbAutoCreate”: true,
“standardElementConstraints”: false,
“tx”: false,
“wal”: false,
“batchCommit”: 1000,
“dbType”: “graph”,
“classes”: [{“name”: “Account”, “extends”:“V”}, {“name”: “Friend”, “extends”:“E”}, {“name”: ‘Enemy’, “extends”:“E”}],
“indexes”: [{“class”:“Account”, “fields”:[“id:integer”], “type”:“UNIQUE_HASH_INDEX” }]
}
}
}

and kept the data source file exactly as in the manual (file database.json):
[
{
“name”: “Joe”,
“id”: 1,
“friends”: [2,4,5],
“enemies”: [6]
},
{
“name”: “Suzie”,
“id”: 2,
“friends”: [1,4,6],
“enemies”: [5,2]
}
]

and finally ran ETL through the following command:
$ ~/Downloads/orientdb-3.0.21/bin/oetl.sh import.json
OrientDB etl v.3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x) https://www.orientdb.com

2019-07-15 17:06:31:570 INFO Default limit of open files (512) will be used. [ONative]Exception in thread “main” com.orientechnologies.orient.core.exception.OConfigurationException: Error on loading config file: import.json
at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:79)
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:116)
Caused by: java.io.FileNotFoundException: import.json (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
at com.orientechnologies.common.io.OIOUtils.readFileAsString(OIOUtils.java:145)
at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:69)
… 1 more

Any idea of what’s wrong? Maybe the dbURL field in the configuration file?
Thanks for any suggestion.
Armand

Hi,
the error:


means that the file import.json cannot be found.
Try to execute the command again with the correct path.

Hope it helps

Regards,
Michela

Thanks for the tip Michela. I put the argument with an absolute path, and changed the import.json file to indicate that the database is local and not remote, in the local section, as this:
“loader” : {
“orientdb”: {
“dbURL”: “plocal:/Users/armand/Downloads/orientdb-3.0.21/databases/demodb”,
“dbUser”: “admin”,
“dbPassword”: “admin”,

Not, running the same command again with the same arguments, leads to another error:

$ /Users/armand/Downloads/orientdb-3.0.21/bin/oetl.sh /Users/armand/GIT/markdown-report/examples/orient/import.json

OrientDB etl v.3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x) https://www.orientdb.com

2019-08-07 18:04:40:855 INFO Default limit of open files (512) will be used. [ONative]

2019-08-07 18:04:40:954 INFO 17179869184 B/16384 MB/16 GB of physical memory were detected on machine [ONative]

2019-08-07 18:04:40:954 INFO Detected memory limit for current process is 17179869184 B/16384 MB/16 GB [ONative]

2019-08-07 18:04:40:955 INFO JVM can use maximum 1963MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]

2019-08-07 18:04:40:956 INFO Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting ‘memory.leftToOS’ not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]

2019-08-07 18:04:40:958 INFO OrientDB auto-config DISKCACHE=12,454MB (heap=1,963MB os=16,384MB) [orientechnologies]

2019-08-07 18:04:40:959 INFO System is started under an effective user : armand [OEngineLocalPaginated]

2019-08-07 18:04:40:959 INFO Allocation of 189300 pages. [OEngineLocalPaginated]

2019-08-07 18:04:41:357 INFO WAL maximum segment size is set to 12,928 MB [OrientDBEmbedded]Exception 0BA2F4EC in storage plocal:/Users/armand/Downloads/orientdb-3.0.21/databases/demodb: 3.0.21 - Veloce (build 1d8b0c01ca8fe12a516bba355681e3ec8b218d1d, branch 3.0.x)

com.orientechnologies.orient.core.exception.OStorageException: Database is locked by another process, please shutdown process and try again

at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:102)

at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:127)

at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.checkIfStorageDirty(OLocalPaginatedStorage.java:432)

at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:375)

at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:236)

at com.orientechnologies.orient.core.db.OrientDBEmbedded.drop(OrientDBEmbedded.java:463)

at com.orientechnologies.orient.core.db.OrientDB.drop(OrientDB.java:264)

at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.createDatabasePool(OETLOrientDBLoader.java:446)

at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:402)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:158)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:199)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:129)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:92)

at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:116)

Exception in thread “main” com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:153)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:92)

at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:116)

Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Cannot open database ‘/demodb’

at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:246)

at com.orientechnologies.orient.core.db.OrientDBEmbedded.drop(OrientDBEmbedded.java:463)

at com.orientechnologies.orient.core.db.OrientDB.drop(OrientDB.java:264)

at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.createDatabasePool(OETLOrientDBLoader.java:446)

at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:402)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:158)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:199)

at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:129)

… 2 more

Caused by: com.orientechnologies.orient.core.exception.OStorageException: Database is locked by another process, please shutdown process and try again

at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.lockFile(OPaginatedStorageDirtyFlag.java:102)

at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedStorageDirtyFlag.open(OPaginatedStorageDirtyFlag.java:127)

at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.checkIfStorageDirty(OLocalPaginatedStorage.java:432)

at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:375)

at com.orientechnologies.orient.core.db.OrientDBEmbedded.openNoAuthenticate(OrientDBEmbedded.java:236)

… 9 more

armand@MacBook-Pro-de-Armand:~/GIT/markdown-report/examples/orient$ /Users/armand/Downloads/orientdb-3.0.21/bin/oetl.sh /Users/armand/GIT/markdown-report/examples/orient/import.json

2 errors are given:

  1. One seems to indicate that another client is still connected to the server, so I logged out from OrientDB Studio, but the error remains. I also removed some lock files located in the OrientDB install but that did not help.
  2. The other error message indicates that it cannot open the DB /demodb. The path seems correct through.

Any idea is welcome.
Best ,
Armand