Loading...
generate new short url ( original url maxi 2KB -> short url 10B) since (26+10) ^ 10 will be 3,656,158,440,062,976 result enough to store
query short url to redirect to real url
metrics collecting for most frequent url for analysis
expiration policy
user can manage their generated url, delete or change ttl
fast （10ms for redirect）, 1 sec for write
stability (active-active or active-passive will discuss later)
consistency (DB write and read)
monitoring and alerts
scalability
qps:
100,000 users in 1 year
100 request read
1million request per day = 110 rps = 1 server
10ms per request, our server can handle 100 qps(improved by async)
10 request write
1million request per day = 11 rps = 1 server
1 sec per request, our server can handle 1qps( improved by async)
peak hour
double rps, still 1 server
1 million users in 5 years
1110 rps = 1 server since normally one server 5k qps to 10k qps
peak hour 2220 rps
data storage:
url length = 2K, short url length 10
average = 1K
100,000 * 10 * 100 write request per year = 100M url = 100G storage
storing maximum = 1 year
1 DB Server, replica = 2, 200G
network bandwidth:
avg:
121 qps * (1010B) = 121K/s
max = 242K/s
Let's assume user already passed validation phase
/generate?user_id={},url = {}, ttl = {}
return short Url(maybe existed url)
/redirect? short_url = {} for request
return original url or not existed
/manage?user_id={}, short_url={},ttl={} to change ttl
/delete?user_id={}, short_url={}
/list?user_id={}
/details?user_id={},short_url={}
mapping:
short_url varchar(10) PK
original_url varchar(2048)
date_created timestamp
date_expired timestamp
created_by FK user_id
build index on created_by
build index on date_expired(if we want to save storage, or lazy mode then don't need this index, we can talk about this later)
user:
user_id int
user_name varchar(20)
metrics: # store metrics for each 1 day
facet: short_url varchar(10)
clicks, timerange, etc.
backend server (master-master/master-slave discussed later), java, go, node.js
SQL database (master for write, slave for read)
for 1 year, we can have 1 SQL Db with backup enabled
for 5 years, we can have 1 write and 2 read slave for DB
metrics DB: influx DB or other time series DB that could do aggregation
write:
user-> server -> wirte to sql -> response ok or not
read:
user -> server -> get from sql db -> response orginal url or not exist
server- > influxDB- > grafana
backend server:
/generate?original url = {}, ttl = {}
hash_res = md5(user_id, timestamp)[:10]
store hash_res, original_url, ttl into DB
return domain_url + "/" +hash_res as res
/redirect:
check if short_url is existed in mapping table
if yes, returns original_url, else return none which should be handled by frontend
Backend server: active-active vs active-passive
active - active mode:
both are active so when 1 server down, the other server can take up quickly, we can use feature in cloud infra/ add a load balancer on the front for IP redirect, have a heart beat check enabled
very fast recovery, user feel nothing
active - passive mode:
when active node no heart beat, we will turn on passive node. save some money since only 1 node running. user will feel down time depending on the machine. sometime 1 minutes if starting EC2 is slow.
we can also deploy our app on AWS lambda that we don't need to care about anything except QPS if we want to save more money.
another active-active mode:
user is distributed into 2 server at all time and when 1 node down, all traffic redirect by LB.
money cost is same but we need LB.
Database write-read isolation:
no isolation:
save money, but read user will be impacted when large amount of write request. since write request is slower.
when node is down, it's down. we have to recover.
Need to backup data frequently e.g. 5minutes. or enable WAL log. Make sure disk won't lost like in EC2 + SSD configuration. down time will be very long for mysql db to restart, 1 min +
have isolation:
no impact of read request. better when 1 server is down, other server can choose a new lead quickly by ZK or election.
problem: consistency, depending on what consistency level we want. Maybe final consistency is acceptable in this case.
10sec down time.
metrics: pull/ push model from backend server to influxDB
pull model: some delay, mostly used. data lost if server restarted. pressure is on influxDB.
push model: have impact on server when pushing. can use async push since network bandwidth is ok. data will be sent when manually restarted. pressure is on Backend Server.
Hybrid: Backend use long poll to notify influxDB to collect data, influx pull data from backend
QPS concern.
when qps is too high, add a queue like kafka or aws sns/sqs, when user query backend, return them a caller_id of idempotent key, and use push/pull to query backend to see if data is processed.
only redirect to queue when qps reach a certain level. add complexity to maintain and develop so only when needed or we are forecasting a peak.
MySQL and DB server dead concern like when nodes down, discussed in above .
backend has to be async to enable max cpu power.
rate limit for one user generate too much request or redirect too much.
hearbeat check for backend server and mysql, create PD alert or datadog. if failed, bring up another server based on choices of model.
client can be hashed into different server if we are scaling to 10x, add load balancer in front
also mysql can be sharded as well, maybe depending on hash(original_url)
add in memory cache on backend side or another component like redis to cache when qps > 100 for mysql