Does someone has used the databricks component to push data into databricks ?
I tried to test it (writing a table with only 2 fields in string format), and it was way too slow (less than 100 records / minute). It appears that even whlle changing the batch size in the metadata and disabling the auto-commit, xDi still push data row by row to databricks.
So if anyone as a solution to push data into databricks with xDI, I will really appreciate it
Thanks;
Best Answer
A
Audric FREY
said
about 11 hours ago
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
1 Comment
A
Audric FREY
said
about 11 hours ago
Answer
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
Yann Courtet
Hi,
Does someone has used the databricks component to push data into databricks ?
I tried to test it (writing a table with only 2 fields in string format), and it was way too slow (less than 100 records / minute).
It appears that even whlle changing the batch size in the metadata and disabling the auto-commit, xDi still push data row by row to databricks.
So if anyone as a solution to push data into databricks with xDI, I will really appreciate it
Thanks;
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
Audric FREY
Hello,
Databricks load template natively loads data row by row.
Unfortunately, there is no native way to improve this.
You can imagine splitting your flow with a mapping that inserts data into a CSV file
And add a step in your process ( proces with the mapping to Csv at the frist step) a SQLoperation with a COPY statement
COPY INTO ma_base.mon_schema.ma_table
FROM '/chemin/vers/fichier.csv' -- ou 's3://...', 'abfss://...', '/Volumes/...'
FILEFORMAT = CSV
FORMAT_OPTIONS (
'header' = 'true',
'inferSchema' = 'true',
'delimiter' = ',',
'encoding' = 'UTF-8'
)
COPY_OPTIONS ('mergeSchema' = 'true');
-
File external resolver with encrypted value
-
Timeout on Mapping/Webservice call
-
UNIQUE TEMP TABLE NAMES
-
Previous step name from a sub-process of execute delivery
-
Dropping multiple tables at once in XDI
-
Freeze Designer and corrupted workspace
-
FileWait/Move/Copy: FileInclude with space
-
Loop excel files
-
Change SVN user in the Designer
-
How to Specify the JVM used by the Designer?
See all 76 topics