-
Oct 31, 2012
We’ve been using Cassandra more and more lately. It’s pretty good write-wise, but it isn’t a very verbose Database and it’s bothering me. Here’s a transcript of a little conversation we had over SSH, nodetool and DataStax OpsCenter recently. No data was harmed in the recording of this incident.
Act I
Cassandra: Cassandra Butterflies-cluster, node 101 speaking, how may I help you?
Me: Yeah, Hi. I’m the administrator. I was just checking on you a little bit.
C: Oh.. Well, it’s been a little hectic latelty. It looks like someone..
Me: … someone killed a couple of your nodes. Right. Eh.. That was me. Sorry about that.
C: Oh. Oh! Oh, that’s actually good news. ‘cause me and the gang we’re sort of starting to worry that you were about to shut us down.
Me: Oh no. Dude. Nothing like that. I was just playing around with the servers and I.. Look, it was an accident and it won’t happen again. You’re still very much our production DB env.
C: Ok, cool.
Me: Great. I’m glad to hear you’re all right.
C: Yeah, no worries. We’re redistributing the data right now. So.. You know. It’s a couple of terabytes, so it’ll be a little while.
Me: Cool. Cool. I’ll try and pull the servers back online as fast as I can.
C: No rush. We’re serving requests and everything’s ticking.
Me: Great. So, no data loss then?
C: Nope
Me: Awesome. Talk to you soon. Bye.Act II
Cassandra: Cassandra Butterflies-cluster, node 101 speaking, how may I help you?
Me: Yeah, Hi. Me again. Listen, I got the nodes back up and everything …
C: Yeah, we noticed. Thanks. Really boosted the morale.
Me: … thaat’s great. So. The thing is.. There’s a lot less disk usage.
C: Oh?
Me: Yeah and it’s sort of bugging me a little bit.
C: M-hm?
Me: The thing is, you used to have about half a terabyte of data per node, and now there’s closer to 250 gig. Sooo what’s up with that?
C: Well.. You know, we replicate all data to at least two other nodes and the cluster didn’t completely die when we lost the nodes.
Me: Riiight. Riight. Still though, the diskspace is really worrying me. Are you sure you’re ok?
C: I’m fine.
Me: Sure?
C: Yep.
Me: Well.. How can you be so sure?
C: Dude, I’m fucking sure. I just know, alright?
Me: You just know?
C: I just know.
Me: … How can you ‘just know’? Isn’t there some sort of a counter? Is there somepleace I can get like a list of all the rows and data, or some counts or some shit like that?
C: Not really, no. If you ask me for a particular row, I can go and check if I have it.
Me: Dude, there’s literally billions of rows with more or less random keys. That’s no good.. I need some sort of a statistic.
C: Hey listen. I’m not the one zapping nodes willy-nilly. If it was that important, why didn’t you have a backup?
Me: There’s terabytes of data, updated thousands of times per second! There’s no backup. You ARE the backup. The whole point is to build a system… You know what, nevermind.
C: Look, I’m sorry, but..
Me: Don’t be. It’s not your fault. We’ll just have to figure something out..
C: Alright. Later then.
Me: *sigh*. Later. =
