Design Web crawler: coming to Tailspin

Web crawler is coming to Tailspin. One website gets every request.

What it will show: Politeness for each website; The frontier backlog and duplicate pages.