Does someone has used the databricks component to push data into databricks ?
I tried to test it (writing a table with only 2 fields in string format), and it was way too slow (less than 100 records / minute). It appears that even whlle changing the batch size in the metadata and disabling the auto-commit, xDi still push data row by row to databricks.
So if anyone as a solution to push data into databricks with xDI, I will really appreciate it
Thanks;
Best Answer
A
Audric FREY
said
about 7 hours ago
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
1 Comment
A
Audric FREY
said
about 7 hours ago
Answer
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
Yann Courtet
Hi,
Does someone has used the databricks component to push data into databricks ?
I tried to test it (writing a table with only 2 fields in string format), and it was way too slow (less than 100 records / minute).
It appears that even whlle changing the batch size in the metadata and disabling the auto-commit, xDi still push data row by row to databricks.
So if anyone as a solution to push data into databricks with xDI, I will really appreciate it
Thanks;
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
Audric FREY
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
-
File external resolver with encrypted value
-
Timeout on Mapping/Webservice call
-
UNIQUE TEMP TABLE NAMES
-
Previous step name from a sub-process of execute delivery
-
Dropping multiple tables at once in XDI
-
Freeze Designer and corrupted workspace
-
FileWait/Move/Copy: FileInclude with space
-
Loop excel files
-
Change SVN user in the Designer
-
How to Specify the JVM used by the Designer?
See all 76 topics