System requirements

Functional Requirements

User Management:
- User registration and authentication.
- User profiles to manage payment and delivery details.
- Role-based access (Buyer, Seller, Admin).
Item Management:
- Sellers can list items with details like title, description, images, starting bid price, reserve price, and auction duration.
- Sellers can manage active auctions (edit/cancel).
Auction Management:
- Real-time bidding functionality.
- Automatic bidding (proxy bidding) up to a max amount specified by the bidder.
- Notifications for bidders (outbid, auction win/loss, auction start/end).
- Reserve price enforcement.
Payment Processing:
- Secure payment processing for completed auctions.
- Transaction tracking for buyers and sellers.
- Refund and cancellation policies.
Search and Filters:
- Advanced search for auctions based on keywords, categories, price range, and auction status.
- Sorting options (e.g., ending soonest, highest bid).
Real-time Features:
- Live updates for bids and auction states.
- Push notifications for events.
Reporting and Analytics:
- Admin dashboards to monitor auction activity, fraud detection, and revenue statistics.
- User reports for bids, purchases, and sales.

Non-Functional Requirements

Scalability:
- Handle millions of concurrent users and real-time updates.
- Support global auctions across multiple time zones.
Reliability:
- Ensure consistent performance with minimal downtime.
- Failover mechanisms for critical services.
Security:
- Encrypt sensitive user data and transactions (e.g., payments).
- Protect against fraud and unauthorized access (e.g., bidding bots).
Performance:
- Ensure low latency for real-time updates (target <1s).
- API response time ≤200ms under normal load.
Usability:
- User-friendly and intuitive interfaces for both web and mobile.
- Localization support for different languages and currencies.

Capacity estimation

Estimate the scale of the system you are going to design...

User Base: Assume 10 million active users monthly, with 1 million concurrent users.

Auctions: 500,000 active auctions at any time.

Bid Traffic: Assume 5 bids/sec/auction at peak times → ~2.5 million bids/sec globally.

Storage:

Item details: 100 KB/item → ~50 GB for active items.
Bid history: 1 KB/bid → ~2.5 TB for one day's peak bids.

API design

Define what APIs are expected from the system...

User APIs:

POST /users/register: Register a new user.
POST /users/login: User authentication.
GET /users/{id}: Get user profile details.

Item APIs:

POST /items: List a new item for auction.
GET /items/{id}: Get item details.
PUT /items/{id}: Update item details.
DELETE /items/{id}: Cancel an auction.

Auction APIs:

POST /auctions/{id}/bids: Place a bid.
GET /auctions/{id}: Get auction details (e.g., current bid, status).
GET /auctions: Search/filter auctions.
PUT /auctions/{id}/end: End an auction (Admin or timer-based).

Notification APIs:

POST /notifications: Push notifications for events.

Payment APIs:

POST /payments: Process payments.
GET /payments/{id}: Get payment status.

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

1. Auction Database

Schema Details:


Table: Auctions
Columns:
  - auction_id (UUID, PK)
  - item_id (FK)
  - start_time (TIMESTAMP)
  - end_time (TIMESTAMP)
  - starting_price (FLOAT)
  - reserve_price (FLOAT)
  - current_bid (FLOAT)
  - status (ENUM: active, completed, cancelled)

Purpose: Store auction metadata.
Tech Used: PostgreSQL.
Tradeoff: Strong consistency ensures accurate bidding state but could impact write performance during peak traffic.

2. User Database

Schema Details:


Table: Users
Columns:
  - user_id (UUID, PK)
  - username (VARCHAR)
  - email (VARCHAR)
  - hashed_password (VARCHAR)
  - payment_info (JSON)

Purpose: Manage user profiles and credentials.
Tech Used: PostgreSQL.
Tradeoff: Using JSON for payment_info offers flexibility but could complicate queries.

3. Bid History Database

Schema Details:


Table: Bids
Columns:
  - bid_id (UUID, PK)
  - auction_id (FK)
  - user_id (FK)
  - bid_amount (FLOAT)
  - timestamp (TIMESTAMP)

Purpose: Track bidding history.
Tech Used: Cassandra.
Tradeoff: High write throughput and scalability, but lacks strong consistency across regions.

4. Notification Database

Schema Details:


Table: Notifications
Columns:
  - notification_id (UUID, PK)
  - user_id (FK)
  - message (TEXT)
  - status (ENUM: sent, pending)
  - timestamp (TIMESTAMP)

Purpose: Manage user notifications.
Tech Used: MongoDB.
Tradeoff: Flexible schema for varied notification types, but lacks relational capabilities.

5. Analytics Database

Schema Details:


Table: Analytics
Columns:
  - event_id (UUID, PK)
  - event_type (ENUM: bid_placed, auction_created, payment_completed)
  - user_id (FK)
  - metadata (JSON)
  - timestamp (TIMESTAMP)

Purpose: Log events for reporting and fraud detection.
Tech Used: Elasticsearch.
Tradeoff: Fast search capabilities but higher storage requirements.

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

1. User Management

Overview: Handles user registration, authentication, and profile management.
Key Features:
- Secure login/logout.
- Role-based permissions (buyer, seller, admin).
- Payment and address management.

2. Auction Management

Overview: Core component for creating and managing auctions.
Key Features:
- Create, update, and cancel auctions.
- Real-time updates on auction status and bids.

3. Bidding Engine

Overview: Manages real-time bidding, including automatic bidding logic.
Key Features:
- Tracks highest bid and handles outbid scenarios.
- Validates bids against auction rules (e.g., minimum increment).

4. Notification Service

Overview: Sends real-time notifications to users for critical events.
Key Features:
- Push notifications for outbid alerts, auction start/end, and payment confirmations.
- Integration with email/SMS services.

5. Payment Gateway

Overview: Facilitates secure transactions between buyers and sellers.
Key Features:
- Supports multiple payment methods.
- Processes refunds and payment disputes.
- Ensures compliance with PCI DSS standards.

6. Search and Discovery

Overview: Enables users to search and filter auctions.
Key Features:
- Full-text search across item titles and descriptions.
- Advanced filters for categories, price, and auction status.

7. Analytics and Reporting

Overview: Provides insights for admin monitoring and user activity reports.
Key Features:
- Tracks metrics like active users, bids per auction, and fraud detection.
- Generates seller/buyer activity reports.

8. Real-Time Communication

Overview: Supports live updates for bids and auction status.
Key Features:
- Uses WebSockets or Server-Sent Events (SSE) for bid updates.
- Ensures low-latency communication for end-users.

9. Admin Panel

Overview: Provides administrative control over the platform.
Key Features:
- Approves or bans users.
- Monitors suspicious activity.
- Configures platform settings (e.g., bid increment rules).

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

1. User Registration/Login

User Action: User registers or logs in via the frontend.
Frontend: Sends a request (POST /users/register or POST /users/login) to the backend.
Backend:
- Validates input (e.g., email format, password strength).
- Stores user data in the User Database (hashed passwords).
- Issues a session token or JWT.
Response: Token returned to the frontend for future authentication.

2. Creating an Auction

User Action: Seller submits item details via the frontend.
Frontend: Sends a request (POST /items) with auction details to the backend.
Backend:
- Validates input (e.g., title, starting price, auction duration).
- Creates a new auction entry in the Auction Database.
Response: Auction creation confirmation with auction ID.

3. Placing a Bid

User Action: Buyer places a bid on an auction.
Frontend: Sends a request (POST /auctions/{id}/bids) with bid details.
Backend:
- Verifies the bid against auction rules (e.g., higher than current bid, within auction duration).
- Updates the current bid in the Auction Database.
- Logs the bid in the Bid History Database.
Real-Time Update:
- Notification Service sends outbid alerts to other bidders.
- Real-Time Communication updates the frontend with the new bid.
Response: Bid placement confirmation.

4. Auction End

Trigger: Auction reaches its scheduled end time.
Backend:
- Marks the auction as "completed" in the Auction Database.
- Identifies the highest bidder and notifies them.
Notification Service:
- Alerts the seller and winning bidder.
Payment Gateway:
- Processes payment from the winning bidder.
- Transfers funds to the seller’s account after deducting platform fees.

5. Notifications

Trigger: Events like outbid, auction end, or payment confirmation occur.
Notification Service:
- Retrieves the user’s preferred notification method.
- Sends notifications via email, SMS, or push.

6. Searching Auctions

User Action: User searches or filters auctions via the frontend.
Frontend: Sends a request (GET /auctions) with query parameters.
Backend:
- Queries the Search Index or Auction Database.
- Returns a paginated list of matching auctions.
Response: Search results displayed to the user.

7. Analytics Report Generation

Admin Action: Admin requests a report via the admin panel.
Backend:
- Aggregates data from the Analytics Database and other sources.
- Formats the report (e.g., CSV, dashboard).
Response: Report displayed or downloadable.

Detailed component design

1. User Management Service

End-to-End Working

The User Management Service handles registration, authentication, and user profile management:

Registration:
- The user submits details like email, password, and profile information.
- The service validates inputs and stores user data in the User Database, hashing the password using secure algorithms like bcrypt.
Login:
- User credentials are validated against stored data. On success, a JWT (JSON Web Token) or session token is issued.
- The token includes metadata (e.g., roles and permissions) and is signed with a private key for integrity.
Profile Updates:
- Users can modify profile details such as payment methods or shipping addresses, which are securely updated in the database.

Communication Protocols

Frontend to Backend: Uses HTTPS for secure REST API calls.
Inter-Service Communication:
- gRPC for internal microservice calls, ensuring efficient serialization and low latency.
- OAuth 2.0 protocol for user authorization.

Data Structures & Algorithms

User Data Storage:
- A relational schema stores user data (e.g., PostgreSQL):

sql
Copy code
Table: Users
Columns:
  - user_id (UUID, PK)
  - email (VARCHAR, UNIQUE)
  - hashed_password (VARCHAR)
  - profile_data (JSON)

Hashing: Uses bcrypt, which is computationally expensive to deter brute force attacks.
Rate Limiting: Implements a token bucket algorithm to control API abuse and login attempts.

Scaling for Peak Traffic

Horizontal Scaling: Adds multiple instances of the service behind a load balancer to handle user spikes.
Caching: Session tokens are stored in Redis, reducing the need to query the database for each request.
CDN: Static user data (e.g., profile pictures) is served from a CDN to offload traffic.

Edge Cases

Duplicate Emails:
- Prevented via a UNIQUE constraint in the database.
Forgotten Passwords:
- Password reset workflows are secured with one-time tokens, valid for a limited time.
Concurrent Updates:
- Optimistic locking ensures updates are consistent and avoids overwriting.

2. Auction Management Service

End-to-End Working

The Auction Management Service handles auction creation, updates, and termination:

Auction Creation:
- Sellers provide item details, starting price, and auction duration.
- The service validates inputs and stores auction metadata in the Auction Database.
Auction Updates:
- Real-time updates (e.g., new bids) modify the current bid and status fields.
Auction Termination:
- Scheduled tasks mark auctions as completed and trigger payment requests for the highest bidder.

Communication Protocols

Frontend to Backend:
- REST APIs over HTTPS for creating and viewing auctions.
Backend to Services:
- Publishes auction events (e.g., start/end) to Kafka for downstream services like notifications and payments.

Data Structures & Algorithms

Auction Timer:
- Uses distributed task queues like Celery or AWS Step Functions to track auction durations.
Auction State:
- A finite-state machine (FSM) ensures valid transitions (e.g., active → completed).
Database:

sql
Copy code
Table: Auctions
Columns:
  - auction_id (UUID, PK)
  - item_id (FK)
  - start_time (TIMESTAMP)
  - end_time (TIMESTAMP)
  - current_bid (FLOAT)
  - reserve_price (FLOAT)

Scaling for Peak Traffic

Partitioning: Auctions are partitioned by categories or regions for load distribution.
Event-Driven Architecture: Event queues scale horizontally, ensuring auction events are processed independently.

Edge Cases

Unsold Items:
- Notify sellers and allow automatic relisting.
Auction Overlaps:
- Time synchronization ensures proper scheduling of start/end times.

3. Bidding Engine

End-to-End Working

The Bidding Engine ensures efficient and real-time bid handling:

Bid Placement:
- Validates user input, ensuring it meets criteria (e.g., minimum increment).
- Updates the current highest bid in the Auction Database.
- Logs the bid in the Bid History Database for auditing.
Real-Time Updates:
- Sends outbid notifications to previous bidders.
- Pushes new bid data to clients via WebSockets.

Communication Protocols

Frontend to Backend:
- WebSocket connections for real-time communication.
Backend to Services:
- Uses Kafka to publish bid events for analytics and notifications.

Data Structures & Algorithms

Priority Queue:
- Maintains the highest bid efficiently, ensuring constant-time access.
Proxy Bidding:
- Uses a custom implementation to track maximum bids and automatically outbid competitors.

Scaling for Peak Traffic

Sharding:
- Auctions are divided across multiple bidding instances to balance traffic.
Load Balancers:
- Distribute WebSocket connections across server pools.

Edge Cases

Simultaneous Bids:
- Conflict resolution uses timestamps and auction-specific rules.
Bid Expiry:
- Ensures bids are ignored once auctions end.

4. Notification Service

End-to-End Working

The Notification Service sends real-time alerts for important events:

Event Subscription:
- Subscribes to auction and bid events from a message broker.
Notification Dispatch:
- Processes events and sends messages via user-preferred channels (e.g., email, SMS).

Communication Protocols

Message Broker:
- Uses Kafka or RabbitMQ for event-driven communication.
Third-Party APIs:
- Integrates with providers like Twilio or SendGrid for message delivery.

Data Structures & Algorithms

Retry Queue:
- Implements exponential backoff for failed notification attempts.
Rate Limiting:
- Prevents spamming by capping the number of messages per user per hour.

Scaling for Peak Traffic

Worker Pools:
- Dynamically scale worker instances to handle high notification volumes.
CDN Integration:
- Ensures low latency for push notifications.

Edge Cases

API Rate Limits:
- Batch notifications and prioritize critical alerts.
User Preferences:
- Honors user settings to avoid over-notification.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Token-Based Authentication (JWT)

Advantage: Stateless authentication reduces server load and allows horizontal scaling.
Trade-Off: Token revocation is challenging, requiring additional mechanisms like token blacklists.

Eventual Consistency in Auction Data

Advantage: Improves system performance and scalability by not enforcing immediate consistency.
Trade-Off: Temporary inconsistencies might confuse users, such as bids not immediately reflected in the UI.

WebSocket for Real-Time Updates

Advantage: Enables low-latency communication for live auctions and bids.
Trade-Off: WebSocket connections require persistent resources, making it harder to scale compared to REST APIs.

Proxy Bidding Logic

Advantage: Enhances user experience by automating bids up to a user-specified maximum.
Trade-Off: Increases computational complexity and requires additional checks for bid validity.

Third-Party Notification Services

Advantage: Leveraging external providers (e.g., Twilio, SendGrid) simplifies implementation and ensures scalability.
Trade-Off: Reliance on third-party services introduces potential bottlenecks due to rate limits or outages.

Use of Multiple Databases

Advantage: Tailoring databases (e.g., Cassandra for bids, PostgreSQL for user data) improves performance for specific use cases.
Trade-Off: Increases operational complexity, as multiple database technologies require specialized maintenance and monitoring.

Priority Queue for Bidding

Advantage: Efficiently maintains the highest bids, ensuring rapid updates and retrieval.
Trade-Off: Adds complexity when scaling across distributed systems, requiring synchronization mechanisms.

Horizontal Scaling of Services

Advantage: Improves fault tolerance and allows the system to handle high traffic volumes.
Trade-Off: Requires careful load balancing and session management, especially for stateful services like WebSocket connections.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Database Bottlenecks:

High read/write contention during peak hours.
Use of sharding and replicas can help.

Service Downtime:

Service-level disruptions cause cascading failures.
Implement circuit breakers and retries.

Event Backlogs:

High event volumes overwhelm the message broker.
Scale brokers and optimize event processing.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

Database Improvements:

Transition to a multi-model database (e.g., CosmosDB) for better scalability.

Disaster Recovery:

Enhance failover mechanisms with geo-replication.

Real-Time Optimization:

Switch to QUIC protocol for faster real-time updates.

AI-Based Fraud Detection:

Use machine learning to detect and prevent suspicious bidding behavior.

Solutions

nectar4678 - 04/16/2024

1

iridescent_luminous693 - 01/07/2025

0

redzrdi - 01/08/2025

0