Loading...
generate new short url ( maxi 2KB -> 10B)
query short url to redirect to real url
metrics collecting for most frequent url for analysis
expiration policy
fast （10ms for redirect）, 1 sec for write
stability (active-active or active-passive will discuss later)
consistency (DB write and read)
monitoring and alerts
scalability
qps:
100,000 users in 1 year
100 request read
1million request per day = 110 rps = 1 server
10ms per request, our server can handle 100 qps(improved by async)
10 request write
1million request per day = 11 rps = 1 server
1 sec per request, our server can handle 1qps( improved by async)
peak hour
double rps, still 1 server
1 million users in 5 years
1110 rps = 1 server since normally one server 5k qps to 10k qps
peak hour 2220 rps
data storage:
url length = 2K, short url length 10
average = 1K
100,000 * 10 * 100 write request per year = 100M url = 100G storage
storing maximum = 1 year
1 DB Server, replica = 2, 200G
network bandwidth:
avg:
121 qps * (1010B) = 121K/s
max = 242K/s
/generate?url = {}
return short Url(maybe existed url)
/redirect? short_url = {}
return original url or not existed
mapping:
short_url varchar(10) PK
original_url varchar(2048)
date_created timestamp
date_expired timestamp
created_by FK user_id
build index on created_by
build index on date_expired(if we want to save storage, or lazy mode then don't need this index, we can talk about this later)
user:
user_id int
user_name varchar(20)
metrics: # store metrics for each 1 day
short_url varchar(10)
clicks int
date date
backend server (master-master/master-slave discussed later), java, go, node.js
SQL database (master for write, slave for read)
for 1 year, we can have 1 SQL Db with backup enabled
for 5 years, we can have 1 write and 2 read slave for DB
metrics DB: influx DB or other time series DB that could do aggregation
write:
user->server -> wirte to sql -> response ok or not
read:
user -> server -> get from sql db -> response orginal url or not exist
server- > influxDB- > grafana
Backend server: active-active vs active-passive
active - active mode:
both are active so when 1 server down, the other server can take up quickly, we can use feature in cloud infra/ add a load balancer on the front for IP redirect, have a heart beat check enabled
very fast recovery, user feel nothing
active - passive mode:
when active node no heart beat, we will turn on passive node. save some money since only 1 node running. user will feel down time depending on the machine. sometime 1 minutes if starting EC2 is slow.
we can also deploy our app on AWS lambda that we don't need to care about anything except QPS if we want to save more money.
another active-active mode:
user is distributed into 2 server at all time and when 1 node down, all traffic redirect by LB.
money cost is same but we need LB.
Database write-read isolation:
no isolation:
save money, but read user will be impacted when large amount of write request. since write request is slower.
when node is down, it's down. we have to recover.
Need to backup data frequently e.g. 5minutes. or enable WAL log. Make sure disk won't lost like in EC2 + SSD configuration. down time will be very long for mysql db to restart, 1 min +
have isolation:
no impact of read request. better when 1 server is down, other server can choose a new lead quickly by ZK or election.
problem: consistency, depending on what consistency level we want. Maybe final consistency is acceptable in this case.
10sec down time.
metrics: pull/ push model from backend server to influxDB
pull model: some delay, mostly used. data lost if server restarted.
push model: have impact on server when pushing. can use async push since network bandwidth is ok. data will be sent when manually restarted.
QPS concern.
MySQL and DB server dead concern like when nodes down, discussed in above .
backend has to be async to enable max cpu power.
client can be hashed into different server if we are scaling to 10x
also mysql can be sharded as well, maybe depending on hash(original_url)