We need to load large files (approx. 10,000 lines) through the xDM user interface using an import action.
For traceability, we would like to store the source file's line number in a dedicated column (e.g., "line_number") within the golden table. Is this possible to achieve natively, or does it require a custom workaround?
Thanks for your feedback,
Jean-Christophe
Best Answer
A
Alexia SIMOND
said
5 months ago
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
If you store this value in the golden table, it becomes business data from xDM’s point of view, so you should define clearly which line number should survive when multiple source records consolidate into one golden record.
For fuzzy-matched entities in particular, several source/master records can contribute to the same golden record, so a single line_number in GD may become ambiguous unless you define a survivorship rule or a dedicated audit design.
1 Comment
A
Alexia SIMOND
said
5 months ago
Answer
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
If you store this value in the golden table, it becomes business data from xDM’s point of view, so you should define clearly which line number should survive when multiple source records consolidate into one golden record.
For fuzzy-matched entities in particular, several source/master records can contribute to the same golden record, so a single line_number in GD may become ambiguous unless you define a survivorship rule or a dedicated audit design.
Jean-Christophe Blanchard
Hello,
We need to load large files (approx. 10,000 lines) through the xDM user interface using an import action.
For traceability, we would like to store the source file's line number in a dedicated column (e.g., "line_number") within the golden table. Is this possible to achieve natively, or does it require a custom workaround?
Thanks for your feedback,
Jean-Christophe
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
https://www.semarchy.com/doc/semarchy-xdm/xdm/latest/Design/applications/action-sets.html#_data_size_limitations
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
Alexia SIMOND
Hello Jean-Christophe,
I need to preface my answer by saying that we do not recommend loading big data files through import in xDM.
This is even clearly documented in our official documentation:
https://www.semarchy.com/doc/semarchy-xdm/xdm/latest/Design/applications/action-sets.html#_data_size_limitations
That being said, you would like to load large files through the xDM user interface import action and keep the original source file line number in a dedicated column such as line_number in the golden table for traceability.
In xDM, this is not available as a native feature of the UI import itself.
So if you need the file line number in the golden data, you'll need a custom implementation.
I think the easier option would be to add a line_number column directly in the source file before import, then map/import it like any other business attribute.
A few points to keep in mind:
-
Import Data Into Entities via Azure Data Lake
-
Recover Deleted(soft Delete) Record and Configure in Application
-
Data Quality in batch mode and real-time integration
-
Integration with analytics tools
-
Query/Load/Delete data with the REST API
-
Does the Done Tab in Inbox have a limit?
-
How Can I Trigger Enricher or Sql Procedure when deleting?
-
Matching Rules But Only The Latest Record Creates a Golden Record
-
Unstructured and Semi Structured Data in Semarchy?
-
Read CSV files from AWS S3
See all 86 topics