System requirements

Functional:

1. System should generate a disaster recovery plan for IT system to minimize data loss and downtime.Plan should include following aspects
2. Data backup
3. System replication
4. Failover handling
5. Disaster recovery strategy should include following aspects
6. Risk assessment*
7. Identify critical assets.
8. Recovery objective
9. Establish clear role and responsibility
10. System should provide a detailed test plan for updating disaster recovery plan regularly.

Non-Functional:

1. Highly available.
2. Resilient.
3. Scalable.
4. Eventual consistent.
5. Good performance.

Capacity estimation

1. Number of IT Components/Infrastructure: Let's assume the disaster recovery planning service will manage disaster recovery for a total of 1000 IT components/infrastructure.
2. Expected Latency:

• Metadata Upload API: The expected latency for metadata upload API can be around 100ms.
• Plan Generation: The expected latency for plan generation can be around 200ms.

1. Size of Metadata Document: Considering the metadata of each component as semi-structured data with an average size of 500KB per document.

API design

API 1: Metadata creation API

POST v1/infrastructure/{:id}/component/metadata

content-type: application-drs/json

accept-type: application-drs/json

Payload: {

component-name:

deployemnt-details:{

replicas:

regions:

av:

}

dsrAPI:{

backup:{}

restore:{}

}

quorum:{

}

rebootOrder:{

}

backupOrder:{

}

}

respone:{

component_id:

status:

}

API 2: Metadata update API

PATCH v1/infrastructure/{:id}/component/metadata

content-type: application-drs/json

accept-type: application-drs/json

Payload: {

component-name:

deployemnt-details:{

replicas:

regions:

av:

}

dsrAPI:{

backup:{}

restore:{}

}

quorum:{

}

rebootOrder:{

}

backupOrder:{

}

}

API 3: Plan generation API

GET v1/infrastructure/{:id}/plan

content-type: application-drs/json

accept-type: application-drs/json

response:

{

infra_id:

strategy:{

component1:{

BackUpSource:{}

BackUpDestination:{}

BackUPOrder:{}

RestoreOrder:{}

RiskScore:{}

Suggestions:{}

}

}

}

Database design

Since data is like a document and all the fields may not be applicable to all the components in the infrastructure i prefer to use document database for storing metadata information. Final analysis disaster recovery solution also will be placed in the document database.

Document database is horizontally scalable.

High-level design

Refer to flow chart

Request flows

Refer to sequence diagram

Detailed component design

Planning service:

1. Reads the component metadata and parse it.
2. It will generate a directed acyclic graph of events to be executed on the metadata to develop disaster recovery strategy.
3. If there are independent events found it can be executed parallel but same or different service in order to achieve low latency.
4. After executing each task a risk score, Mitigation techniques will be generated.
5. A repository of best practice of well architected disaster recovery standards will be maintained and recommendation will be provided if there are deviation from standards.

Trade offs/Tech choices

1.Instead of Metadata service directly contacting Planner service meassage is added to kafka so that it helps in consumer end to do parallel processing. It enables reliablity and ensures event is not left unprocessed.

Failure scenarios/bottlenecks

1. Each component disaster recovery and failure analysis invlolves directed acyclic graph with many nodes. If node crash in between processing it may lead to rework and waster of resource. So i would like to introduce durable replicated scalable cache like Redis which holds under processing state.

Future improvements

1. Whole application monitoring and observability capabilities not discussed it should be handled.
2. Data security aspects is not discussed its important since wrong/tampered disaster recovery plan can have huge impact.