The holidays are here and tis the season for DDoS attacks if you host popular websites. Last week I was deep in firefighting mode with a team I am advising. 

One of the more vigilant team members noticed that Latency was lower than integration Latency. 

At a glance, this can be really confusing as “Latency” is a measurement of a HTTP request from the time it hits the AWS APIGateway to the time we respond back and close the connection.

The “Integration Latency” metric is actually a measurement of the time that passes from when the request passes through the APIGateway to our application layer until our application layer responds.

It is typical to see a Latency response of 100ms and an “Integration Latency” response of 90ms because the APIGateway is wrapping the request and there is about 5ms the APIGateway takes to decide where to route the request and another 5 ms after it gets back that it decides how to transform the request.

So how in the world would the Integration Latency be slower than the Latency? That would be impossible for the APIGateway proxying the request to somehow finish faster than the Application layer since the APIGateway is waiting for the Application Layer to finish processing the request before it can finish its work and respond back to the internet.

Take a minute and ponder that. Let me know your guess in the comments. I will wait….

—-

If you guessed magic elves then you need to lay off the egg nog.

The answer is WAF(AWS Web Application Firewall). APIGateway checks every request proxied through it to see if the request appears to be malicious or comes from a block source. If the request is deemed to be malicious the APIGateway responds with a 403 without ever sending the request to the Application Layer. This is great as it saves us the cost of processing the malicious request. 

The tricky part is because of this there is no data point recorded for the Integration Latency. Just the latency.

Think of it Latency and Integration Latency are being graded on a curve and Latency keeps getting all these good fast scores which drags the curve really high. Integration Latency is only getting a small fraction of the scores because they are blocked by the APIGateway so the curve doesn’t have the good scores throwing off the baseline.

I will admit it is complicated. If you have any questions on how to read these complicated metrics feel free to ask!

Good luck hunting these malicious DDoS attackers down and a happy holiday season to you all.

PS: Soon there might be a video course on AWS Security, that includes WAF, which will include a bunch of pretty pixel art animations… If you are good this year. Otherwise, you get coal. :)