How to improve duplicate review processing time in xDM
V
Vinoth KUMAR C
started a topic
10 months ago
Hi All,
Even though we don't have large match groups in our system, the usual review and merge process where stewards need to click the suggestion and the duplicate manager populates the details takes time.
Am looking for options to improve the duplicate review processing time in xDM and would really appreciate your inputs and suggestions from your experience.
Exporting the suggestions and reviewing them offline and run it through custom SQL load is one of the options am thinking, but it has its own issues and involves 2-3 extra steps. Thanks!
Best Answer
A
Alexia SIMOND
said
10 months ago
Hello Vinoth,
For those "performances" topics we usually advise to activate below loggers to capture the query ran on the DB:
Careful, they are very verbose, so we usually advise to:
Do this test on a low environment with little activity.
Prepare as much as possible the use case in the UI.
Clear the PDE.
Activate the loggers.
Reproduce the use case.
Once done, quickly revert the logging configuration to default.
Extract the pde.
And if the file begins with the mention "This is a continuation of log file .../work/Catalina/localhost/semarchy/eclipse/workspace/.metadata/.bak_xxx.log", then extract all the content of this metadata folder (because the files get rewritten with the content of each new entry in the pde).
Once you have the logs, extract the query in question.
You may need to replace the parameters (?) by their values that are stated just below the query.
And once you have a runnable query, then execute it on DB side with an explain plan to check where the there might be some possible improvement.
It could help you determine if maybe new indexes are needed, or maybe that statistics needs to be recomputed.
Or even that a specific join is taking time because it may be based on non-optimized attributes...
Many things could be at play here.
Of course, if you have a specific case you'd like to report us, please feel free to open a ticket for it.
Careful, they are very verbose, so we usually advise to:
Do this test on a low environment with little activity.
Prepare as much as possible the use case in the UI.
Clear the PDE.
Activate the loggers.
Reproduce the use case.
Once done, quickly revert the logging configuration to default.
Extract the pde.
And if the file begins with the mention "This is a continuation of log file .../work/Catalina/localhost/semarchy/eclipse/workspace/.metadata/.bak_xxx.log", then extract all the content of this metadata folder (because the files get rewritten with the content of each new entry in the pde).
Once you have the logs, extract the query in question.
You may need to replace the parameters (?) by their values that are stated just below the query.
And once you have a runnable query, then execute it on DB side with an explain plan to check where the there might be some possible improvement.
It could help you determine if maybe new indexes are needed, or maybe that statistics needs to be recomputed.
Or even that a specific join is taking time because it may be based on non-optimized attributes...
Many things could be at play here.
Of course, if you have a specific case you'd like to report us, please feel free to open a ticket for it.
Thank you
V
Vinoth KUMAR C
said
10 months ago
HI Alexia,
Thanks for the details, we will check and get back incase of additional help.
Vinoth KUMAR C
Hi All,
Even though we don't have large match groups in our system, the usual review and merge process where stewards need to click the suggestion and the duplicate manager populates the details takes time.
Am looking for options to improve the duplicate review processing time in xDM and would really appreciate your inputs and suggestions from your experience.
Exporting the suggestions and reviewing them offline and run it through custom SQL load is one of the options am thinking, but it has its own issues and involves 2-3 extra steps. Thanks!
Hello Vinoth,
For those "performances" topics we usually advise to activate below loggers to capture the query ran on the DB:
<Root level="TRACE">
<Logger level="TRACE" name="org.springframework.jdbc.core.JdbcTemplate"/>
<Logger level="TRACE" name="org.springframework.jdbc.core.StatementCreatorUtils"/>
Careful, they are very verbose, so we usually advise to:
Once you have the logs, extract the query in question.
You may need to replace the parameters (?) by their values that are stated just below the query.
And once you have a runnable query, then execute it on DB side with an explain plan to check where the there might be some possible improvement.
It could help you determine if maybe new indexes are needed, or maybe that statistics needs to be recomputed.
Or even that a specific join is taking time because it may be based on non-optimized attributes...
Many things could be at play here.
Of course, if you have a specific case you'd like to report us, please feel free to open a ticket for it.
Thank you
- Oldest First
- Popular
- Newest First
Sorted by Oldest FirstAlexia SIMOND
Hello Vinoth,
For those "performances" topics we usually advise to activate below loggers to capture the query ran on the DB:
<Root level="TRACE">
<Logger level="TRACE" name="org.springframework.jdbc.core.JdbcTemplate"/>
<Logger level="TRACE" name="org.springframework.jdbc.core.StatementCreatorUtils"/>
Careful, they are very verbose, so we usually advise to:
Once you have the logs, extract the query in question.
You may need to replace the parameters (?) by their values that are stated just below the query.
And once you have a runnable query, then execute it on DB side with an explain plan to check where the there might be some possible improvement.
It could help you determine if maybe new indexes are needed, or maybe that statistics needs to be recomputed.
Or even that a specific join is taking time because it may be based on non-optimized attributes...
Many things could be at play here.
Of course, if you have a specific case you'd like to report us, please feel free to open a ticket for it.
Thank you
Vinoth KUMAR C
HI Alexia,
Thanks for the details, we will check and get back incase of additional help.
Alexia SIMOND
Hi Vinoth,
Of course, good luck.
-
Extend a model with new entities or attributes
-
Data types in xDM
-
Effective date on entities
-
Search using wild cards
-
Export a model from production and import on a development environment
-
"Allow Delete" vs "Allow Removal" privileges
-
LOV label in Named Query
-
Select location on a map and save coordinates
-
Is there a way to set up a master-detail relationship on browse mode?
-
Choose Either a Stepper or A Workflow Based on The User Privileges
See all 346 topics