System requirements

Functional:

• Generate short url : Provided longurl ,generate shorturl
• short url can use characters A to Z ,a to z ,0 to 9 total of 62 characters
• Redirect:User provides short url allow redirection
• Analytics: Store link click count for analytics
• Custom short url : Allow user to create custom alias(optional)
• Expiration management (optional)
• Reporting can be provided -optional
• API security ( optional)
• Monitoring (Optional)

Non-Functional:

• Highly scalable
• Low latency
• Reliable & Fault tolerant

Capacity estimation

Total DAU : 1,00,000

1 URL Request per second

Read Ratio: 10:1 1,000,000 reads per day

Storage: URL 100 characters storage

100 bytes *365 days*10 years= 36.6GB

API design

Generate : http://api.urlshortner.com/v1/shorten

Return : 201 , 400, 500 codes

User : Provides short url , redirect to longurl

301 - permanent re-direction

302- Temporary redirection

Database design

ER Diagram

High-level design

High Level Design diagram

User Provides Long URL

Generate Unique ID

Convert ID to Base62

Create Short URL

Store Long URL & Short URL in Database

Redirect to Long URL

Request flows

1. User Provides Long URL: The process starts when a user submits a long URL.
2. Generate Unique ID: A unique numerical ID is generated for that URL.
3. Convert ID to Base62: This ID is converted into a Base62 string (7 characters).
4. Create Short URL: The Base62 string serves as the short URL.
5. Store: The long and short URLs are stored in the database.
6. Redirect: Upon accessing the short URL, the system redirects to the original long URL.

Detailed component design

Implementation Steps

1. Unique ID Generation:
   • Use a unique ID generator (such as an auto-incrementing ID in your database or a UUID generator) to create a unique identifier for each URL. This will ensure that every URL has a unique numerical ID.
2. Base62 Encoding:
   • Convert this unique ID to a Base62 string. This method would use digits (0-9), lowercase (a-z), and uppercase (A-Z) characters to represent the number in a compact format.
   • By using a 7-character long Base62 string, you get ( 62^7 ) combinations, which is roughly 353 billion unique short URLs.
3. Handling Collisions:
   • Since you are using a unique ID as the base to generate the short URL, the chance of collision is practically eliminated. However, you may still add a validation step to ensure that the generated short URL does not already exist in your database.

Trade offs/Tech choices

echnology Stack

1. Load Balancer (NGINX):
   • Manages incoming requests and distributes load across your application instances.
   • Ensures high availability and redundancy for your service.
2. API Gateway (Kong):
   • Acts as a reverse proxy, handling API requests.
   • Provides authentication, rate limiting, and additional security features.
   • Can manage multiple microservices if you decide to expand the functionality of your platform.
3. Containerized Deployment (Spring Boot on Kubernetes):
   • Using Spring Boot to build a microservice for the URL shortening functionality.
   • Deploying in Kubernetes for container orchestration, managing scaling, and ensuring resilience.
   • Enables seamless updates and rollbacks of your application.
4. Cache (Redis):
   • Used for caching both short URLs and analytics data.
   • Improves performance for read-heavy operations by serving data quickly.
5. Database (MySQL):
   • Stores long URLs, short URLs, user data, and click counts.
   • The relational nature of MySQL helps with structured queries and relationships.
6. Monitoring (Dynatrace):
   • Monitors application performance, user interactions, and the overall health of your system.
   • Provides insights into potential bottlenecks and performance issues.
7. Analytics and Reporting (Kibana, Elasticsearch):
   • Use Elasticsearch to store and search large volumes of analytics data.
   • Kibana provides a rich visual interface for reporting and visualizing click patterns, user engagement, and overall system usage metrics.

Failure scenarios/bottlenecks

If all DB nodes goes down , we should have Disaster recovery zone .Hence multi-dc deployment will be a good approach with circuit breaker in application for fallout.

Future improvements

Phase 1: Foundational Enhancements (1-2 Months)

Objective: Establish a strong foundation by implementing basic but high-value features that will immediately improve user experience and system reliability.

• Weeks 1-2: Planning and Requirements Gathering
  • User Surveys: Develop surveys and execute user interviews to finalize feature priorities.
  • Technical Specifications: Define clear technical requirements for each feature, focusing on scalability and security.
  • Project Management Setup: Utilize tools like Jira or Trello to manage tasks, assign teams, and track progress.
• Weeks 3-4: Custom Shortened URLs & Basic Analytics
  • Milestone: User-defined aliases and basic click tracking implemented.
  • Development: Build and deploy the custom URL feature. Integrate basic analytics for click tracking.
  • Testing: Conduct internal testing followed by beta testing with a select user group to refine usability.
• Weeks 5-6: Security Enhancements
  • Milestone: Basic security features such as malware detection and link expiry implemented.
  • Development: Integrate malware scanning APIs and establish link expiration policies.
  • Testing: Perform security tests and involve beta testers for feedback on security features.
• Weeks 7-8: Performance Optimization
  • Milestone: Optimize system performance to handle increased traffic.
  • Load Testing: Simulate traffic loads and optimize database queries.
  • Caching Implementation: Use Redis to maintain session state and cached responses.
  • Testing: Conduct performance testing under high-traffic simulations.

Phase 2: Advanced Features Rollout (2-3 Months)

Objective: Build upon foundational features with advanced functionalities to cater to power users and businesses.

• Weeks 9-10: Advanced Analytics & Reporting
  • Milestone: Expanded analytics functionality developed.
  • Implementation: Add detailed metrics like geographic distribution, device tracking, and time-based clicks.
  • Testing: A/B testing various dashboard layouts for user engagement metrics.
• Weeks 11-12: User Management and Permissions
  • Milestone: Multi-user accounts with role-based access controls created.
  • Development: Build a user management system for organizations and integrate API key capabilities.
  • Testing: User acceptance testing (UAT) with target business clients to ensure functionality.
• Weeks 13-14: Browser Extensions and Social Media Integrations
  • Milestone: Enhance accessibility through browser extensions.
  • Implementation: Create extensions for major browsers and enable social media sharing.
  • Testing: Ideate for beta testing with select users, followed by iterations based on feedback.
• Weeks 15-16: QR Code Generation and A/B Testing Features
  • Milestone: Marketing capabilities enhanced with QR code generation and A/B testing implemented.
  • Implementation: Develop automatic QR code generation and setup A/B testing capabilities.
  • Testing: Pilot testing of A/B testing features with select marketing teams.

Phase 3: Security, Scalability, and Compliance (2 Months)

Objective: Ensure the service is secure, scalable, and compliant with regulations, addressing concerns of reliability and data protection.

• Weeks 17-18: Enhanced Security Features
  • Milestone: Advanced security measures implemented.
  • Development: Introduce password protection for links and enhance malware detection.
  • Testing: Engage in penetration testing and rigorous security audits.
• Weeks 19-20: Scalability and Redundancy
  • Milestone: System capabilities for scaling enhanced.
  • Implementation: Establish load balancing solutions and redundancy mechanisms.
  • Testing: Conduct scalability simulations to ensure seamless user adoption.
• Weeks 21-22: Compliance and Privacy
  • Milestone: GDPR and other compliance regulations addressed.
  • Implementation: Enable data deletion features and anonymize user data for privacy.
  • Testing: Comprehensive compliance checks and legal reviews conducted.
• Weeks 23-24: Continuous Improvement and Final Rollout
  • Milestone: Full feature deployment and user engagement established.
  • Implementation: Roll out features to all users, promoting the updates effectively.
  • Feedback Mechanisms: Set up continuous feedback loops for ongoing improvements.

User Journey Focus

1. URL Creation and Management: Smooth, user-friendly URL creation experience.
2. Link Performance Analysis: Enhance tracking capabilities to help users analyze performance.
3. Security and Privacy: Maintain a strong emphasis on user security throughout.
4. Collaboration and Integration: Foster a collaborative environment and facilitate integrations.

Key Milestones to Track Progress:

• Custom Shortened URLs and Basic Analytics implemented.
• Security enhancements (malware detection, link expiry).
• Advanced analytics and user management features released.
• Browser extensions and social media integration complete.
• QR code generation and A/B testing capabilities added.
• Scalability, security, and compliance fully implemented.

By following this structured approach, you can ensure that each phase is actionable and aligned with user needs, while also allowing for flexibility based on real-time feedback and testing outcomes.