Overview
If you have to periodically cleanse and validate data within a specific time window, and keep an audit trail of your validation process, you can use Xplain’s anomaly detection module for market data or for valuation data.
You can also use our trade onboarding module, which is based on a similar methodology.
On this page, we will discuss:
- prerequisites that need to be in place before you can start a data cleansing workflow
- the phases of our two-stage data cleansing market data workflow and valuation data workflow or exception management (that we will also refer to as market data or valuation data XM).
Data Cleansing Workflow Prerequisites
There are three types of prerequisites for the data cleansing workflow:
- generic prerequisites
- market data XM specific prerequisites
- valuation data XM specific prerequisites
1. Generic Prerequisites
To perform data data cleansing in a fully audited and automated manner, you will need to:
- define applicable break tests (including applicable measure, scaling, threshold and data scope) which will be applied to identify potential outliers
- define the resolution and approval teams that will be responsible for i) resolving potential breaks and ii) approving the proposed resolutions, as part of the two-stage data cleansing workflow
- upload the ‘raw’ data that will be subject to the anomaly detection process
The market data XM specific and valuation data XM specific prerequisites are set out below.
2. Market Data XM Specific Prerequisites
Once the generic prerequisites are in place, you will need to define a default pricing environment for each company whose portfolios are valued in Xplain with cleansed data.
This will include defining:
- the portfolio(s)
- the associated curve configuration(s)
- the relevant market data environment
- the company/entity’s default valuation settings, which will set out the link between the market data group, the portfolios and the curve configuration
This will allow Xplain during the data preparation phase of the market data workflow to identify the relevant curve nodes and volatility points whose associated market data will be subject to the anomaly detection process.
3. Valuation Data XM Specific Prerequisites
Once the generic prerequisites are in place, you will need to define a valuation data environment for each company whose portfolio valuation will be subject to the anomaly detection process.
This will include defining:
- the portfolio(s)
- the relevant valuation data group(s)
- the company/entity’s default valuation data provider settings, which will set out the link between the pricing slot, the valuation data group(s) and the portfolios
This will allow Xplain during the data preparation phase of the valuation data workflow to identify the relevant trades whose valuation will be subject to the anomaly detection process.
Data Cleansing Workflow Phases
There are two types of data cleansing workflows related to the data types discussed above:
- market data data cleansing
- valuation data data cleansing
1. Market Data XM Workflow Phases
Market data data cleansing can be performed ahead of valuation in Xplain, to ensure that the market data used in Xplain is of high quality. Xplain’s valuations will be based on the cleansed market data and can subsequently be used in the valuation data XM workflow, where Xplain is one of the valuation data providers.
To start a market data XM workflow:
- create a market data dashboard
- specify the market data group that holds the ‘raw’ data to be validated
- specify the curve date
For a given market data group, the XM workflow phases are as follows:
%%{init:{
'flowchart':{
'nodeSpacing': 15,
'rankSpacing': 50,
'diagramPadding': 5
}
}}%%
flowchart LR
A["Identify all<br>companies/entities linked<br>to market data group"]
B["Identify all associated<br>curve configurations<br>(incl. data providers)"]
C["Identify unique list of all<br>market data linked to the curve configurations"]
subgraph DP[Data Preparation]
direction TB
A --> B
B --> C
end
D["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#marketDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">preliminary</a> <sup>(1)</sup><br>break tests<br>against raw data"]
E["Resolve<br>preliminary breaks <sup>(2)</sup>"]
F["Approve or reject<br>proposed resolutions"]
subgraph PC[Preliminary Data XM]
direction TB
D --> E
E --> F
F -.-> |reject|E
end
G["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#marketDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">overlay</a> <sup>(3)</sup><br>break tests<br>against preliminary data"]
H["Resolve<br>overlay breaks<sup>(2)</sup>"]
I["Approve or reject<br>proposed resolutions"]
subgraph OC[Overlay Data XM]
direction TB
G --> H
H --> I
I -.-> |reject|H
end
DP --> PC --> OC
L1[Performed by Xplain]
L2[Performed by you]
classDef subgraphStyle font-weight:bold,fill:none,stroke:#805CDD,stroke-width:1px;
classDef xplStyle fill:#805CDD,stroke:#333,stroke-width:1px,color:#fff;
class DP,PC,OC subgraphStyle;
class L1,A,B,C,D,G xplStyle;
(1) Preliminary break tests aim at identifying potential outliers on a standalone basis.
(2) The granularity used to split break test results into resolution tasks will be done according to the task granularity settings.
(3) Overlay break tests aim at identifying potential outliers on a comparison basis (e.g. day-on-day or source-to-source), and are applied on preliminary cleaned data on a curve configuration basis.
The dashboard will help you monitor the progress of the XM workflow. For more details, please refer to the market data data cleansing page.
2. Valuation Data XM Workflow Phases
Valuation data data cleansing can be performed on a standalone basis, based upon third-party data only.
If Xplain is one of the valuation data providers, we will also perform trade valuations as part of the XM workflow, based on company/entity’s default valuation settings and (cleansed) market data.
To start a valuation data XM workflow:
- create valuation data dashboard
- specify the pricing slot(s) (with further portfolio granularity options allowed) corresponding to the ‘raw’ data to be validated
- specify the valuation date
For a given valuation data group, the XM workflow phases are as follows:
%%{init:{
'flowchart':{
'nodeSpacing': 15,
'rankSpacing': 50,
'diagramPadding': 5
}
}}%%
flowchart LR
A["Identify all portfolios<br>with associated valuation data group(s) linked to pricing slot"]
B["Identify all trades<br>in those portfolios"]
C["Trade valuation when Xplain is a data source <sup>(1)</sup>"]
subgraph DP[Data Preparation]
direction TB
A --> B
B --> C
end
D["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#valuationDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">Overlay I</a> break tests<br>against Xplain data <sup>(1)</sup> and raw third-party data"]
E["Resolve<br>Overlay I breaks <sup>(2)</sup>"]
F["Approve or reject<br>proposed resolutions"]
subgraph PC[Overlay I Data XM]
direction TB
D --> E
E --> F
F -.-> |reject|E
end
G["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#valuationDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">Overlay II</a> <sup>(3)</sup><br>break tests<br>against Overlay I data"]
H["Resolve<br>Overlay II breaks<sup>(2)</sup>"]
I["Approve or reject<br>proposed resolutions"]
subgraph OC[Overlay II Data XM]
direction TB
G --> H
H --> I
I -.-> |reject|H
end
DP --> PC --> OC
L1[Performed by Xplain]
L2[Performed by you]
classDef subgraphStyle font-weight:bold,fill:none,stroke:#805CDD,stroke-width:1px;
classDef xplStyle fill:#805CDD,stroke:#333,stroke-width:1px,color:#fff;
class DP,PC,OC subgraphStyle;
class L1,A,B,C,D,G xplStyle;
(1) As applicable. Xplain’s trade valuation will be based upon company/entity’s default valuation settings and (cleansed) market data, and be automatically performed as a ‘dashboard calculation’.
(2) The granularity used to split break test results into resolution tasks will be done on a pricing slot basis, and further according to the task granularity settings.
(3) Only relevant when Overlay II break tests are applicable. In that case, the ‘primary provider’ data will be deemed to be the Overlay I outputs.
The dashboard will help you monitor the progress of the XM workflow. Valuation data data cleansing is discussed in detail on the valuation data data cleansing page.
