Overview
If you have to periodically cleanse and validate data within a specific time window, and keep an audit trail of your validation process, you can use Xplain’s anomaly detection module for market data or for valuation data.
You can also use our trade onboarding module, which is based on a similar methodology.
On this page, we will discuss:
- prerequisites that need to be in place before you can start a data cleansing workflow
- the phases of our two-stage data cleansing for market data and valuation data.
Data Cleansing Workflow Prerequisites
There are three types of prerequisites for the data cleansing workflow:
- generic prerequisites
- market data cleansing specific prerequisites
- valuation data cleansing specific prerequisites
1. Generic Prerequisites
To perform data cleansing in a fully audited and automated manner, you will need to:
- define applicable break tests (including applicable measure, scaling, threshold and data scope) which will be applied to identify potential outliers
- define automated resolution rules that will be applied to resolve potential breaks automatically (optional)
- define the resolution and approval teams that will be responsible for i) resolving potential breaks and ii) approving the proposed resolutions, as part of the two-stage data cleansing workflow
- upload the ‘raw’ data that will be subject to the anomaly detection process
The market data cleansing specific and valuation data cleansing specific prerequisites are set out below.
2. Market Data Cleansing Specific Prerequisites
Once the generic prerequisites are in place, you will need to define a default pricing environment for each company whose portfolios are valued in Xplain with cleansed data.
This will include defining:
- the portfolio(s)
- the associated curve configuration(s)
- the relevant market data environment
- the company/entity’s default valuation settings, which will set out the link between the market data group, the portfolios and the curve configuration
During the data preparation phase of the market data cleansing workflow, Xplain will identify the relevant curve nodes and volatility points whose associated market data will be subject to the anomaly detection process.
3. Valuation Data Cleansing Specific Prerequisites
Once the generic prerequisites are in place, you will need to define a valuation data environment for each company whose portfolio valuation will be subject to the anomaly detection process.
This will include defining:
- the portfolio(s)
- the relevant valuation data group(s)
- the company/entity’s default valuation data provider settings, which will set out the link between the pricing slot, the valuation data group(s) and the portfolios
During the data preparation phase of the valuation data cleansing workflow, Xplain will identify the relevant trades whose valuation will be subject to the anomaly detection process.
Data Cleansing Workflow Phases
In this section, we will discuss the phases of the two types of data cleansing workflows, which are as follows:
- market data cleansing
- valuation data cleansing
1. Market Data Cleansing Workflow Phases
Market data cleansing can be performed ahead of valuation, to ensure that the market data used in Xplain is reliable.
To start a market data cleansing workflow:
- create a market data dashboard
- specify the market data group that holds the ‘raw’ data to be validated
- specify the curve date
Generate Xplain Valuation Data
Once a market data cleansing workflow is completed, you will be able to use the cleansed market data to generate ‘Xplain valuation data’ for use in a valuation data cleansing workflow, if Xplain is one of the valuation data providers.
For a given market data group, the cleansing workflow phases are as follows:
%%{init:{
'flowchart':{
'nodeSpacing': 15,
'rankSpacing': 50,
'diagramPadding': 5
}
}}%%
flowchart LR
A["Identify all<br>companies/entities linked<br>to market data group"]
B["Identify all associated<br>curve configurations<br>(incl. data providers)"]
C["Identify unique list of all<br>market data linked to the curve configurations"]
D["Identify relevant market data linked to existing trades <sup>(4)</sup>"]
subgraph DP[Data Preparation]
direction TB
A --> B
B --> C
C --> |Optional| D
end
E["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#marketDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">preliminary</a> <sup>(1)</sup><br>break tests<br>against raw data"]
F["Resolve<br>preliminary breaks <sup>(2)</sup>"]
G["Approve or reject<br>proposed resolutions"]
P["Save as verified preliminary data"]
subgraph PC[Preliminary Data]
direction TB
E --> F
F --> G
G -.-> |reject|F
G --> P
end
H["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#marketDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">overlay</a> <sup>(3)</sup><br>break tests<br>against preliminary data"]
I["Resolve<br>overlay breaks<sup>(2)</sup>"]
J["Approve or reject<br>proposed resolutions"]
O["Save as verified<br>overlay data"]
subgraph OC[Overlay Data]
direction TB
H --> I
I --> J
J -.-> |reject|I
J --> O
end
K["Calculate Xplain valuations with preliminary (primary and secondary) and overlay market data <sup>(4)</sup>"]
L["Generate PV comparison metrics across market data sources"]
M["Validate Xplain overlay valuations as Xplain valuation data, based on the comparison metrics"]
subgraph MD[Xplain Valuation Data]
direction TB
K --> L
L --> M
end
DP --> PC --> OC --> |Optional| MD
L1[Performed by Xplain]
L2[Performed by you]
classDef subgraphStyle font-weight:bold,fill:none,stroke:#805CDD,stroke-width:1px;
classDef xplStyle fill:#805CDD,stroke:#333,stroke-width:1px,color:#fff;
class DP,PC,OC,MD subgraphStyle;
class L1,A,B,C,D,E,H,K,L,P,O xplStyle;
The dashboard will help you monitor the progress of the data cleansing workflow. For more details, please refer to the market data cleansing page.
2. Valuation Data Cleansing Workflow Phases
Valuation data cleansing can be performed on a standalone basis, based upon third-party data only.
Prior to starting a valuation data cleansing workflow, if Xplain is one of the valuation data providers, you will first need to generate ‘Xplain valuation data’ once the relevant market data cleansing workflow has been completed.
To start a valuation data cleansing workflow:
- create valuation data dashboard
- specify the pricing slot(s) (with further portfolio granularity options allowed) corresponding to the ‘raw’ data to be validated
- specify the valuation date
- specify the previous valuation date (for day-on-day tests)
For a given valuation data group, the data cleansing workflow phases are as follows:
%%{init:{
'flowchart':{
'nodeSpacing': 15,
'rankSpacing': 50,
'diagramPadding': 5
}
}}%%
flowchart LR
A["Identify all portfolios<br>with associated valuation data group(s) linked to pricing slot"]
B["Identify all trades<br>in those portfolios"]
subgraph DP[Data Preparation]
direction TB
A --> B
end
D["Apply <a href="/docs.xplainfinancial/docs/userGuide/preferences/breakTestDefinitions/#valuationDataBreakTests" style="color: white; border-bottom: 1px solid white; padding-bottom: 2px;">overlay</a> break tests<br>against primary valuation data provider"]
E["Resolve<br>overlay breaks <sup>(1)</sup>"]
F["Approve or reject<br>proposed resolutions"]
O["Save as verified<br>overlay data"]
subgraph PC[Overlay Data]
direction TB
D --> E
E --> F
F -.-> |reject|E
F --> O
end
DP --> PC
L1[Performed by Xplain]
L2[Performed by you]
classDef subgraphStyle font-weight:bold,fill:none,stroke:#805CDD,stroke-width:1px;
classDef xplStyle fill:#805CDD,stroke:#333,stroke-width:1px,color:#fff;
class DP,PC subgraphStyle;
class L1,A,B,C,D,G,O xplStyle;
The dashboard will help you monitor the progress of the datacleansing workflow. For more details, please refer to the valuation data cleansing page.
