Looks like there's some kind of peering problem between Cable & Wireless and LJ's hosts

[andrewducker]

C:\Users\Andy>tracert www.livejournal.com

Tracing route to www.livejournal.com [208.93.0.128]
over a maximum of 30 hops:

1 10 ms 11 ms 11 ms cpc17-sgyl33-2-0-gw.sgyl.cable.virginmedia.com [94.174.128.1]
2 8 ms 11 ms 11 ms sgyl-core-1b-ge214.network.virginmedia.net [81.97.49.117]
3 22 ms 11 ms 11 ms sgyl-core-2b-ae2-0.network.virginmedia.net [195.182.178.94]
4 15 ms 15 ms 14 ms leed-bb-1b-ae5-0.network.virginmedia.net [81.97.48.81]
5 20 ms 24 ms 38 ms 62.252.224.238
6 21 ms 21 ms 23 ms xe-5-3-0-xcr1.lnd.cw.net [195.2.21.209]
7 29 ms 35 ms 23 ms xe-3-1-0-xcr1.lns.cw.net [195.2.28.134]
8 21 ms 43 ms 23 ms xe-2-2-0-xcr1.lnt.cw.net [195.2.30.106]
9 * * * Request timed out.
10 * * * Request timed out.
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

And so I shall not be replying to LJ comments until they fix it.

Comments

[pete stevens (from livejournal.com)]

Livejournal runs it's own AS number and takes transit from Prolexic (a DDOS prevention company) amongst others but I think prefers all inbound traffic via them.

Our routing tables show Prolexic peered at LINX but rather surprisingly they don't seem to peer with Virgin Media (this may be they lacked the time to set it up - we had to phone them daily for months to get our session up) so Virgin send the traffic via Cable & Wireless rather than direct over LINX.

We know that LINX was up at the time and also that Prolexic have a single 10Gbps port there which frankly is a bit shit for a DDOS prevention company[*]. We've seen upwards of 5Gbps to a single site under DDOS so I doubt they'd last that long against someone who was properly determined (e.g. someone with a credit card and enough amazon VMs - Amazon have 30Gbps of London peering).

So I'd guess Prolexic were under DDOS (that is their sole purpose after all) and they ran out of network at LINX as theirs was the next hop that would usually be visible on the trace.

[andrewducker]

Thank you! Interesting to know!

[pete stevens (from livejournal.com)]

Interesting Prolexic claim to have 500Gbps of capacity, which assuming that's marketing capacity means they probably have 200Gbps or so. In which case they're mostly connected via transit / private-peering. I can assume that they're forcing virgin through their transit rather than LINX in order that Virgin bottlenecks out rather than overwelhming Prolexic as Virgin has 100Gbits into LINX by itself. So it may be there were enough botnet customers connected to vigin DoSing a Prolexic customer that it took out Virgin rather than Prolexic, but that's still a bit lame.