--- Log opened Mon Jan 20 00:00:34 2020
07:40 -!- Mirage [mirage@ra.thehippo.net] has joined #se2600
07:40 -!- mode/#se2600 [+o Mirage] by ChanServ
07:49 -!- NotLarry [~NotLarry@c-68-52-173-111.hsd1.tn.comcast.net] has quit [Ping timeout: 265 seconds]
07:52 -!- NotLarry [~NotLarry@c-68-52-173-111.hsd1.tn.comcast.net] has joined #se2600
07:52 -!- mode/#se2600 [+o NotLarry] by ChanServ
12:34 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has joined #se2600
12:38 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
12:40 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has joined #se2600
13:30 <@Evilpig> fuckin' network security DoS'ing us internally.  
13:30 <@Evilpig> their security scanner has found a way to crash one of our distributed file systems somehow, and they scan every node in that cluster at the same time and it just destroyed it
13:37 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
13:38 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has joined #se2600
13:42 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
13:42 -!- rpifan [~rpifan@p200300D26743051E892025A97E6701E7.dip0.t-ipconnect.de] has joined #se2600
13:42 <@Corydon76> Evilpig: it's too bad you can't find a traffic magnification exploit against their scanner.
13:43 <@Corydon76> Imagine if their scanner causes a DDoS against itself.
14:03 <@Evilpig> just locked myself out of a box while blocking them
14:04 <@Evilpig> doing a reject on x.x.x.x/3 instead of x.x.x.x/32 will do that
14:24 <@Mirage> Be glad you don't have a mgr like Kendra...you'd wind up being written up for that typo
14:27 <@Evilpig> it got caught in our staging area when I locked myself out of about 6 servers
14:27 <@Evilpig> bad part was it blocked our configuration management servers from getting there to undo it
14:36 <@Corydon76> They're using a single IP scanner?  Odd.
14:44 <@Evilpig> they have multiple scanning systems
14:47 <@Mirage> Dunno about now but there used to be 4 internal and one external.  Internal ones were in a cluster, each with multiple bonded uplinks to make being able to annihilate target hosts that much easier.
14:48 <@Evilpig> there's a shitload of them now.  I found a few more this morning by searching through splunk and finding where they were pounding nrpe and it was saying "you don't have permission to talk to me!"
14:49 <@Mirage> We had rate-limiting rules in iptables on the web, mysql, etc servers to keep them from demolishing them because the security guys either couldn't figure out how, or Qualys just doesn't have a rate-limiting functionality
14:49 <@Evilpig> not sure how but their scanner kills glusterd 
14:50 <@Mirage> What was super fun was when the internal and external scans would both hit something at the same time.  Didn't happen often, but it wasn't pretty when it did happen.
14:51 <@Evilpig> Jan 20 08:02:32 gls3001lt nrpe[21210]: Host 10.158.184.25 is not allowed to talk to us!
14:51 <@Evilpig> Jan 20 08:09:24 gls3001lt bricks-ems-brick[10609]: time of crash:
14:51 <@Evilpig> Jan 20 08:09:24 gls3001lt bricks-ems-brick[10609]: 2020-01-20 14:09:24
14:52 <@Mirage> Machine would be on it's knees crying and whimpering.  Couldn't SSH into it because the load was so high.  You'd have a few seconds to login after a reboot before everything started up again.  Only if you were luck enough to have logged in before that and were looking at the logs could you tell the case.
14:52 <@Mirage> er, cause
14:53 <@Mirage> Granted I did my fair share of destroying web servers when we first got the google search appliances, but at least you could tune the spiders on them.
14:55 <@Mirage> Who here has had their upper management decide "SRE is the way to go" and told everyone to start learning it?
14:57 <@Mirage> Does anyone who has, or has at least looked into it, think that it's some new wonderous system or are you like me and think that it's mostly just outlining all the different shit that people in large environments and few admins have already figured out for themselves and been doing for years?
14:58 <@Evilpig> we had SRE positions at tumblr when I was there but luckily it's still just called sysadmin here
15:02 <@Corydon76> We don't bother with titles here.
15:02 <@Corydon76> It's everyone's job to make sure that shit can be shipped.  If something can't be shipped, drop everything else, and fix that, first.
15:10 -!- rpifan_ [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has joined #se2600
15:11 -!- rpifan [~rpifan@p200300D26743051E892025A97E6701E7.dip0.t-ipconnect.de] has quit [Disconnected by services]
15:11 -!- rpifan_ is now known as rpifan
15:15 <@Mirage> https://landing.google.com/sre/sre-book/chapters/introduction/
15:15 < PigBot> Google - Site Reliability Engineering (at landing.google.com) http://tinyurl.com/y42q44va
15:16 <@Mirage> LOL.. "Running a service with a team that relies on manual intervention for both change management and event handling becomes expensive as the service and/or traffic to the service grows, because the size of the team necessarily scales with the load generated by the system."
15:18 <@Corydon76> Yeah...
15:19 <@Corydon76> Do they go on to explain _why_ they think a larger team is required for a larger load?
15:45 -!- rpifan_ [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has joined #se2600
15:45 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has quit [Read error: Connection reset by peer]
15:55 -!- rpifan_ is now known as rpifan
16:54 < aestetix> heh
16:54 < aestetix> yeah I have that book on a shelf at work
16:54 < aestetix> annnnnnnnnd never touched it
16:54 < aestetix> which I think is par for most books on the shelf at work
17:31 -!- rpifan [~rpifan@p4FCA28B3.dip0.t-ipconnect.de] has quit [Remote host closed the connection]
--- Log closed Tue Jan 21 00:00:36 2020