|
@@ -1,4 +1,4 @@
|
|
|
-*[English](README.md) ∙ [日本語](README-ja.md) ∙ [简体中文](README-zh-Hans.md) | [Brazilian Portuguese](https://github.com/donnemartin/system-design-primer/issues/40) ∙ [Italian](https://github.com/donnemartin/system-design-primer/issues/104) ∙ [Korean](https://github.com/donnemartin/system-design-primer/issues/102) ∙ [Persian](https://github.com/donnemartin/system-design-primer/issues/110) ∙ [Polish](https://github.com/donnemartin/system-design-primer/issues/68) ∙ [Russian](https://github.com/donnemartin/system-design-primer/issues/87) ∙ [Traditional Chinese](https://github.com/donnemartin/system-design-primer/issues/88) ∙ [Turkish](https://github.com/donnemartin/system-design-primer/issues/39) | [Add Translation](https://github.com/donnemartin/system-design-primer/issues/28)*
|
|
|
|
|
|
|
+*[English](README.md) ∙ [日本語](README-ja.md) ∙ [简体中文](README-zh-Hans.md) | [Brazilian Portuguese](https://github.com/donnemartin/system-design-primer/issues/40) ∙ [Italian](https://github.com/donnemartin/system-design-primer/issues/104) ∙ [Korean](https://github.com/donnemartin/system-design-primer/issues/102) ∙ [Persian](https://github.com/donnemartin/system-design-primer/issues/110) ∙ [Polish](https://github.com/donnemartin/system-design-primer/issues/68) ∙ [Russian](https://github.com/donnemartin/system-design-primer/issues/87) ∙ [Traditional Chinese](https://github.com/donnemartin/system-design-primer/issues/88) ∙ [Turkish](https://github.com/donnemartin/system-design-primer/issues/39) ∙ [Vietnamese](https://github.com/donnemartin/system-design-primer/issues/127) | [Add Translation](https://github.com/donnemartin/system-design-primer/issues/28)*
|
|
|
|
|
|
|
|
# The System Design Primer
|
|
# The System Design Primer
|
|
|
|
|
|
|
@@ -556,7 +556,7 @@ Services such as [CloudFlare](https://www.cloudflare.com/dns/) and [Route 53](ht
|
|
|
### Disadvantage(s): DNS
|
|
### Disadvantage(s): DNS
|
|
|
|
|
|
|
|
* Accessing a DNS server introduces a slight delay, although mitigated by caching described above.
|
|
* Accessing a DNS server introduces a slight delay, although mitigated by caching described above.
|
|
|
-* DNS server management could be complex, although they are generally managed by [governments, ISPs, and large companies](http://superuser.com/questions/472695/who-controls-the-dns-servers/472729).
|
|
|
|
|
|
|
+* DNS server management could be complex and is generally managed by [governments, ISPs, and large companies](http://superuser.com/questions/472695/who-controls-the-dns-servers/472729).
|
|
|
* DNS services have recently come under [DDoS attack](http://dyn.com/blog/dyn-analysis-summary-of-friday-october-21-attack/), preventing users from accessing websites such as Twitter without knowing Twitter's IP address(es).
|
|
* DNS services have recently come under [DDoS attack](http://dyn.com/blog/dyn-analysis-summary-of-friday-october-21-attack/), preventing users from accessing websites such as Twitter without knowing Twitter's IP address(es).
|
|
|
|
|
|
|
|
### Source(s) and further reading
|
|
### Source(s) and further reading
|
|
@@ -727,9 +727,7 @@ Additional benefits include:
|
|
|
<i><a href=http://lethain.com/introduction-to-architecting-systems-for-scale/#platform_layer>Source: Intro to architecting systems for scale</a></i>
|
|
<i><a href=http://lethain.com/introduction-to-architecting-systems-for-scale/#platform_layer>Source: Intro to architecting systems for scale</a></i>
|
|
|
</p>
|
|
</p>
|
|
|
|
|
|
|
|
-Separating out the web layer from the application layer (also known as platform layer) allows you to scale and configure both layers independently. Adding a new API results in adding application servers without necessarily adding additional web servers.
|
|
|
|
|
-
|
|
|
|
|
-The **single responsibility principle** advocates for small and autonomous services that work together. Small teams with small services can plan more aggressively for rapid growth.
|
|
|
|
|
|
|
+Separating out the web layer from the application layer (also known as platform layer) allows you to scale and configure both layers independently. Adding a new API results in adding application servers without necessarily adding additional web servers. The **single responsibility principle** advocates for small and autonomous services that work together. Small teams with small services can plan more aggressively for rapid growth.
|
|
|
|
|
|
|
|
Workers in the application layer also help enable [asynchronism](#asynchronism).
|
|
Workers in the application layer also help enable [asynchronism](#asynchronism).
|
|
|
|
|
|
|
@@ -761,7 +759,7 @@ Systems such as [Consul](https://www.consul.io/docs/index.html), [Etcd](https://
|
|
|
<p align="center">
|
|
<p align="center">
|
|
|
<img src="http://i.imgur.com/Xkm5CXz.png">
|
|
<img src="http://i.imgur.com/Xkm5CXz.png">
|
|
|
<br/>
|
|
<br/>
|
|
|
- <i><a href=https://www.youtube.com/watch?v=vg5onp8TU6Q>Source: Scaling up to your first 10 million users</a></i>
|
|
|
|
|
|
|
+ <i><a href=https://www.youtube.com/watch?v=w95murBkYmU>Source: Scaling up to your first 10 million users</a></i>
|
|
|
</p>
|
|
</p>
|
|
|
|
|
|
|
|
### Relational database management system (RDBMS)
|
|
### Relational database management system (RDBMS)
|
|
@@ -827,7 +825,7 @@ Both masters serve reads and writes and coordinate with each other on writes. I
|
|
|
<p align="center">
|
|
<p align="center">
|
|
|
<img src="http://i.imgur.com/U3qV33e.png">
|
|
<img src="http://i.imgur.com/U3qV33e.png">
|
|
|
<br/>
|
|
<br/>
|
|
|
- <i><a href=https://www.youtube.com/watch?v=vg5onp8TU6Q>Source: Scaling up to your first 10 million users</a></i>
|
|
|
|
|
|
|
+ <i><a href=https://www.youtube.com/watch?v=w95murBkYmU>Source: Scaling up to your first 10 million users</a></i>
|
|
|
</p>
|
|
</p>
|
|
|
|
|
|
|
|
Federation (or functional partitioning) splits up databases by function. For example, instead of a single, monolithic database, you could have three databases: **forums**, **users**, and **products**, resulting in less read and write traffic to each database and therefore less replication lag. Smaller databases result in more data that can fit in memory, which in turn results in more cache hits due to improved cache locality. With no single central master serializing writes you can write in parallel, increasing throughput.
|
|
Federation (or functional partitioning) splits up databases by function. For example, instead of a single, monolithic database, you could have three databases: **forums**, **users**, and **products**, resulting in less read and write traffic to each database and therefore less replication lag. Smaller databases result in more data that can fit in memory, which in turn results in more cache hits due to improved cache locality. With no single central master serializing writes you can write in parallel, increasing throughput.
|
|
@@ -841,7 +839,7 @@ Federation (or functional partitioning) splits up databases by function. For ex
|
|
|
|
|
|
|
|
##### Source(s) and further reading: federation
|
|
##### Source(s) and further reading: federation
|
|
|
|
|
|
|
|
-* [Scaling up to your first 10 million users](https://www.youtube.com/watch?v=vg5onp8TU6Q)
|
|
|
|
|
|
|
+* [Scaling up to your first 10 million users](https://www.youtube.com/watch?v=w95murBkYmU)
|
|
|
|
|
|
|
|
#### Sharding
|
|
#### Sharding
|
|
|
|
|
|
|
@@ -1077,7 +1075,7 @@ Sample data well-suited for NoSQL:
|
|
|
|
|
|
|
|
##### Source(s) and further reading: SQL or NoSQL
|
|
##### Source(s) and further reading: SQL or NoSQL
|
|
|
|
|
|
|
|
-* [Scaling up to your first 10 million users](https://www.youtube.com/watch?v=vg5onp8TU6Q)
|
|
|
|
|
|
|
+* [Scaling up to your first 10 million users](https://www.youtube.com/watch?v=w95murBkYmU)
|
|
|
* [SQL vs NoSQL differences](https://www.sitepoint.com/sql-vs-nosql-differences/)
|
|
* [SQL vs NoSQL differences](https://www.sitepoint.com/sql-vs-nosql-differences/)
|
|
|
|
|
|
|
|
## Cache
|
|
## Cache
|
|
@@ -1259,8 +1257,8 @@ Refresh-ahead can result in reduced latency vs read-through if the cache can acc
|
|
|
### Disadvantage(s): cache
|
|
### Disadvantage(s): cache
|
|
|
|
|
|
|
|
* Need to maintain consistency between caches and the source of truth such as the database through [cache invalidation](https://en.wikipedia.org/wiki/Cache_algorithms).
|
|
* Need to maintain consistency between caches and the source of truth such as the database through [cache invalidation](https://en.wikipedia.org/wiki/Cache_algorithms).
|
|
|
-* Need to make application changes such as adding Redis or memcached.
|
|
|
|
|
* Cache invalidation is a difficult problem, there is additional complexity associated with when to update the cache.
|
|
* Cache invalidation is a difficult problem, there is additional complexity associated with when to update the cache.
|
|
|
|
|
+* Need to make application changes such as adding Redis or memcached.
|
|
|
|
|
|
|
|
### Source(s) and further reading
|
|
### Source(s) and further reading
|
|
|
|
|
|
|
@@ -1485,12 +1483,12 @@ REST is focused on exposing data. It minimizes the coupling between client/serv
|
|
|
|
|
|
|
|
| Operation | RPC | REST |
|
|
| Operation | RPC | REST |
|
|
|
|---|---|---|
|
|
|---|---|---|
|
|
|
-| Signup | **POST** /signup | **POST** /persons |
|
|
|
|
|
-| Resign | **POST** /resign<br/>{<br/>"personid": "1234"<br/>} | **DELETE** /persons/1234 |
|
|
|
|
|
|
|
+| Signup | **POST** /signup | **POST** /persons |
|
|
|
|
|
+| Resign | **POST** /resign<br/>{<br/>"personid": "1234"<br/>} | **DELETE** /persons/1234 |
|
|
|
| Read a person | **GET** /readPerson?personid=1234 | **GET** /persons/1234 |
|
|
| Read a person | **GET** /readPerson?personid=1234 | **GET** /persons/1234 |
|
|
|
| Read a person’s items list | **GET** /readUsersItemsList?personid=1234 | **GET** /persons/1234/items |
|
|
| Read a person’s items list | **GET** /readUsersItemsList?personid=1234 | **GET** /persons/1234/items |
|
|
|
| Add an item to a person’s items | **POST** /addItemToUsersItemsList<br/>{<br/>"personid": "1234";<br/>"itemid": "456"<br/>} | **POST** /persons/1234/items<br/>{<br/>"itemid": "456"<br/>} |
|
|
| Add an item to a person’s items | **POST** /addItemToUsersItemsList<br/>{<br/>"personid": "1234";<br/>"itemid": "456"<br/>} | **POST** /persons/1234/items<br/>{<br/>"itemid": "456"<br/>} |
|
|
|
-| Update an item | **POST** /modifyItem<br/>{<br/>"itemid": "456";<br/>"key": "value"<br/>} | **PUT** /items/456<br/>{<br/>"key": "value"<br/>} |
|
|
|
|
|
|
|
+| Update an item | **POST** /modifyItem<br/>{<br/>"itemid": "456";<br/>"key": "value"<br/>} | **PUT** /items/456<br/>{<br/>"key": "value"<br/>} |
|
|
|
| Delete an item | **POST** /removeItem<br/>{<br/>"itemid": "456"<br/>} | **DELETE** /items/456 |
|
|
| Delete an item | **POST** /removeItem<br/>{<br/>"itemid": "456"<br/>} | **DELETE** /items/456 |
|
|
|
|
|
|
|
|
<p align="center">
|
|
<p align="center">
|
|
@@ -1677,9 +1675,10 @@ Handy metrics based on numbers above:
|
|
|
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
|
| Google | [Google architecture](http://highscalability.com/google-architecture) |
|
|
|
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
|
| Instagram | [14 million users, terabytes of photos](http://highscalability.com/blog/2011/12/6/instagram-architecture-14-million-users-terabytes-of-photos.html)<br/>[What powers Instagram](http://instagram-engineering.tumblr.com/post/13649370142/what-powers-instagram-hundreds-of-instances) |
|
|
|
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
|
| Justin.tv | [Justin.Tv's live video broadcasting architecture](http://highscalability.com/blog/2010/3/16/justintvs-live-video-broadcasting-architecture.html) |
|
|
|
-| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf) |
|
|
|
|
|
|
|
+| Facebook | [Scaling memcached at Facebook](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf)<br/>[TAO: Facebook’s distributed data store for the social graph](https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf)<br/>[Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf)<br/>[How Facebook Live Streams To 800,000 Simultaneous Viewers](http://highscalability.com/blog/2016/6/27/how-facebook-live-streams-to-800000-simultaneous-viewers.html) |
|
|
|
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
|
| Flickr | [Flickr architecture](http://highscalability.com/flickr-architecture) |
|
|
|
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
|
| Mailbox | [From 0 to one million users in 6 weeks](http://highscalability.com/blog/2013/6/18/scaling-mailbox-from-0-to-one-million-users-in-6-weeks-and-1.html) |
|
|
|
|
|
+| Netflix | [Netflix: What Happens When You Press Play?](http://highscalability.com/blog/2017/12/11/netflix-what-happens-when-you-press-play.html) |
|
|
|
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
|
| Pinterest | [From 0 To 10s of billions of page views a month](http://highscalability.com/blog/2013/4/15/scaling-pinterest-from-0-to-10s-of-billions-of-page-views-a.html)<br/>[18 million visitors, 10x growth, 12 employees](http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html) |
|
|
|
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
|
| Playfish | [50 million monthly users and growing](http://highscalability.com/blog/2010/9/21/playfishs-social-gaming-architecture-50-million-monthly-user.html) |
|
|
|
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
|
| PlentyOfFish | [PlentyOfFish architecture](http://highscalability.com/plentyoffish-architecture) |
|
|
@@ -1688,7 +1687,7 @@ Handy metrics based on numbers above:
|
|
|
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
|
| TripAdvisor | [40M visitors, 200M dynamic page views, 30TB data](http://highscalability.com/blog/2011/6/27/tripadvisor-architecture-40m-visitors-200m-dynamic-page-view.html) |
|
|
|
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
|
| Tumblr | [15 billion page views a month](http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html) |
|
|
|
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)<br/>[How Twitter Handles 3,000 Images Per Second](http://highscalability.com/blog/2016/4/20/how-twitter-handles-3000-images-per-second.html) |
|
|
| Twitter | [Making Twitter 10000 percent faster](http://highscalability.com/scaling-twitter-making-twitter-10000-percent-faster)<br/>[Storing 250 million tweets a day using MySQL](http://highscalability.com/blog/2011/12/19/how-twitter-stores-250-million-tweets-a-day-using-mysql.html)<br/>[150M active users, 300K QPS, a 22 MB/S firehose](http://highscalability.com/blog/2013/7/8/the-architecture-twitter-uses-to-deal-with-150m-active-users.html)<br/>[Timelines at scale](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)<br/>[Big and small data at Twitter](https://www.youtube.com/watch?v=5cKTP36HVgI)<br/>[Operations at Twitter: scaling beyond 100 million users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)<br/>[How Twitter Handles 3,000 Images Per Second](http://highscalability.com/blog/2016/4/20/how-twitter-handles-3000-images-per-second.html) |
|
|
|
-| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html) |
|
|
|
|
|
|
|
+| Uber | [How Uber scales their real-time market platform](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html)<br/>[Lessons Learned From Scaling Uber To 2000 Engineers, 1000 Services, And 8000 Git Repositories](http://highscalability.com/blog/2016/10/12/lessons-learned-from-scaling-uber-to-2000-engineers-1000-ser.html) |
|
|
|
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
|
| WhatsApp | [The WhatsApp architecture Facebook bought for $19 billion](http://highscalability.com/blog/2014/2/26/the-whatsapp-architecture-facebook-bought-for-19-billion.html) |
|
|
|
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
|
| YouTube | [YouTube scalability](https://www.youtube.com/watch?v=w5WVu624fY8)<br/>[YouTube architecture](http://highscalability.com/youtube-architecture) |
|
|
|
|
|
|
|
@@ -1742,9 +1741,9 @@ Handy metrics based on numbers above:
|
|
|
|
|
|
|
|
#### Source(s) and further reading
|
|
#### Source(s) and further reading
|
|
|
|
|
|
|
|
-* [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs)
|
|
|
|
|
|
|
+Looking to add a blog? To avoid duplicating work, consider adding your company blog to the following repo:
|
|
|
|
|
|
|
|
-The list of blogs here will be kept relatively small and [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs) will contain the larger list to avoid duplicating work. Do consider adding your company blog to the engineering-blogs repo instead.
|
|
|
|
|
|
|
+* [kilimchoi/engineering-blogs](https://github.com/kilimchoi/engineering-blogs)
|
|
|
|
|
|
|
|
## Under development
|
|
## Under development
|
|
|
|
|
|
Chang Liu