Importing a csv: oetl Error in creating the ETL process #9993

francescarestante · 2023-07-10T18:54:46Z

francescarestante
Jul 10, 2023

Hello!! I'm new to OrientDB and I'm trying to upload on a database a csv file that is structured in this way:

BookName( ISBN, Title )

For Example:

ISBN;title
439785960;Harry Potter and the Half-Blood Prince (Harry Potter #6)
439358078;Harry Potter and the Order of the Phoenix (Harry Potter #5)
439554896;Harry Potter and the Chamber of Secrets (Harry Potter #2)
043965548X;Harry Potter and the Prisoner of Azkaban (Harry Potter #3)
...

I've created this Json file to exploit the oetl.bat:

{
  "config": {
     "log": "debug"
  },
  "begin": [],
  "source": {
    "file": {
      "path": "path/to/BookName.csv",
      "header": true,
      "separator": ";"
    }
  },
  "extractor": {
    "csv": {"separator": ";", "nullValue": "NULL", "skipFrom": 1, "skipTo": 3 }
  },
  "transformers": [
    {
      "vertex": {
        "class": "Isbn",
        "skipDuplicates": true
      }
    },
    {
      "vertex": {
        "class": "BookName",
        "skipDuplicates": true
      }
    },
    {
      "edge": {
        "class": "hasName",
        "joinFieldName": "Isbn",
        "lookup": "BookName.Isbn",
        "direction": "out"
      }
    },
    {
      "edge": {
        "class": "hasName",
        "joinFieldName": "BookName",
        "lookup": "BookName.Title",
        "direction": "in"
      }
    }
  ],
  "loader": {
    "orientdb": {
      "dbURL": "remote:localhost:2480/bookProject",
      "dbUser": <User>,
      "dbPassword": <dbPwd>,
      "dbAutoCreate": true ,
      "standardElementConstraints": false,
      "tx": false,
      "wal": false,
      "batchCommit" : 1000,
      "dbType": "graph",
      "classes": [
        {
          "name": "Isbn",
          "extends": "V"
        },
        {
          "name": "BookName",
          "extends": "V"
        },
        {
          "name": "hasName",
          "extends": "E"
        }
      ],
      "indexes": [
        {
          "class": "Isbn",
          "fields": ["ISBN"],
          "type": "UNIQUE"
        },
        {
          "class": "BookName",
          "fields": ["Title"],
          "type": "UNIQUE"
        }
      ]
    }
  },
  "end": []
}

where bookProject is the empty Db I created on OrientDB Community.

In the terminal once I try to execute the command: > oetl.bat path/to/bookName.json
I get this error:

Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:172)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParametersWithContext(OETLProcessorConfigurator.java:103)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:68)
        at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:121)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot connect to the remote server/database 'localhost:2480/bookProject/bookProject'
        at com.orientechnologies.orient.core.db.OrientDBRemote.networkAdminOperation(OrientDBRemote.java:620)
        at com.orientechnologies.orient.core.db.OrientDBRemote.connectAndExecute(OrientDBRemote.java:650)
        at com.orientechnologies.orient.core.db.OrientDBRemote.connectAndSend(OrientDBRemote.java:300)
        at com.orientechnologies.orient.core.db.OrientDBRemote.exists(OrientDBRemote.java:245)
        at com.orientechnologies.orient.core.db.OrientDB.exists(OrientDB.java:293)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.getDatabasePool(OETLOrientDBLoader.java:474)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:408)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:179)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:218)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:143)
        ... 3 more
Caused by: com.orientechnologies.common.io.OIOException: Error on connecting to localhost:2480
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:48)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:55)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:13)
        at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:107)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.acquire(ORemoteConnectionPool.java:81)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:116)
        at com.orientechnologies.orient.client.remote.OStorageRemote.getNetwork(OStorageRemote.java:1979)
        at com.orientechnologies.orient.core.db.OrientDBRemote.networkAdminOperation(OrientDBRemote.java:607)
        ... 12 more
Caused by: com.orientechnologies.orient.enterprise.channel.binary.ONetworkProtocolException: Cannot read protocol version from remote server localhost/127.0.0.1:2480: java.net.SocketTimeoutException: Read timed out
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.<init>(OChannelBinaryAsynchClient.java:105)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:34)
        ... 19 more

How can I fix it?

tglman · 2023-07-11T10:40:51Z

tglman
Jul 11, 2023
Maintainer

Hi,

you are using as connection url: remote:localhost:2480/bookProject which has the http port on it instead the binary protocol port, so the url should be remote:localhost:2424/bookProject the 2424 is the default port for the binary protocol.

1 reply

francescarestante Jul 11, 2023
Author

I tried to do on the 2424 port before but wasn't working, so I checked the DB and tried to do a plocal connection.
I replicated a simpler example on the OrientDB manual (https://orientdb.com/docs/3.0.x/etl/Import-from-CSV-to-a-Graph.html) regarding a post and the comment associated

The dbUrl in this case is the local path to a db that is created at the moment:

"loader": {
      "orientdb": {
         "dbURL": "plocal:/orientdb-community-3.2.10/databases/postDb",
         "dbType": "graph",
         "classes": [
           {"name": "Post", "extends": "V"},
           {"name": "Comment", "extends": "V"},
           {"name": "HasComments", "extends": "E"}
         ], "indexes": [
           {"class":"Post", "fields":["id:integer"], "type":"UNIQUE" }
         ]
      }

Now the problem is once I try to start the otl with a db that already exist, it doesn't give error if I create at the moment but if I want to access it another time trying to populate the db records it gives me this error:

Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:172)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParametersWithContext(OETLProcessorConfigurator.java:103)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:68)
        at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:121)
Caused by: java.lang.RuntimeException: java.io.IOException: Unable to access file. Another process has locked part of the file
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.logAndPrepareForRethrow(OAbstractPaginatedStorage.java:6661)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.logAndPrepareForRethrow(OAbstractPaginatedStorage.java:6641)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:529)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.getAndOpenStorage(OrientDBEmbedded.java:542)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.poolOpen(OrientDBEmbedded.java:585)
        at com.orientechnologies.orient.core.db.ODatabasePoolImpl$1.createNewResource(ODatabasePoolImpl.java:56)
        at com.orientechnologies.orient.core.db.ODatabasePoolImpl$1.createNewResource(ODatabasePoolImpl.java:52)
        at com.orientechnologies.common.concur.resource.OResourcePool.<init>(OResourcePool.java:66)
        at com.orientechnologies.orient.core.db.ODatabasePoolImpl.<init>(ODatabasePoolImpl.java:48)
        at com.orientechnologies.orient.core.db.OrientDBEmbedded.openPool(OrientDBEmbedded.java:953)
        at com.orientechnologies.orient.core.db.OrientDB.openPool(OrientDB.java:323)
        at com.orientechnologies.orient.core.db.ODatabasePool.<init>(ODatabasePool.java:85)
        at com.orientechnologies.orient.core.db.ODatabasePool.<init>(ODatabasePool.java:64)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.getDatabasePool(OETLOrientDBLoader.java:467)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:408)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:179)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:218)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:143)
        ... 3 more
Caused by: java.io.IOException: Unable to access file. Another process has locked part of the file
        at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:54)
        at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
        at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
        at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:223)
        at com.orientechnologies.common.io.OIOUtils.readByteBuffer(OIOUtils.java:362)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.StorageStartupMetadata.open(StorageStartupMetadata.java:212)
        at com.orientechnologies.orient.core.storage.disk.OLocalPaginatedStorage.checkIfStorageDirty(OLocalPaginatedStorage.java:514)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:382)
        ... 18 more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Importing a csv: oetl Error in creating the ETL process #9993

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Uh oh!

Importing a csv: oetl Error in creating the ETL process #9993

Uh oh!

francescarestante Jul 10, 2023

Replies: 1 comment · 1 reply

Uh oh!

tglman Jul 11, 2023 Maintainer

Uh oh!

Uh oh!

francescarestante Jul 11, 2023 Author

francescarestante
Jul 10, 2023

Replies: 1 comment 1 reply

tglman
Jul 11, 2023
Maintainer

francescarestante Jul 11, 2023
Author