SRECon2017 Training New SRES Panel

Skills To Look For SREs

  • Engineer background
  • Should be able to teach themselves*
  • Focus -> Learn about the tech
  • Shouldn't resistance to change
  • Impose changes
  • Being able to learn new skills
  • Good reverse engineers
  • No backward thinkers
  • See the big pictures
  • Jack of all trades
  • Make decisions with data

Good SREs

  • Finding the root cause
  • How are you going to software yourself out of the problem
  • Follow the scientific method for solving an issue
  • Ability to say No

New SREs

  • Read documentation and updating it
  • Understand what skills you need to learn for your day to day operation
  • Ask questions
  • Everyone is doing SRE. Learn what your team is doing first and the work yourself outerwear.
  • Trust everyone to be doing the right thing.
  • Learn the tools
  • Make sure you join a company with SRE training
  • Find friendly Sr. SRE to learn from them. A SRE should want to share all his knowledge to grown other people.
  • Learn how engineers develop. Learn their tooling

Most Important Training

  • Bootcamps
  • Make them use your product
  • Teach some of the soft parts.
  • Working with a distributed team.
  • Make sure people can run the scary scripts. Make it easy to rerun script in production and feel safe.
  • Chapter 28 in SRE book
  • Allow people to fail.

Team Factors

  • Guide to the answer, don’t give them the answer.
  • Don’t assume they know everything.
  • Feedback is very important
  • Be ready to take the feedback
  • Don’t give the a task. Give them a problem
  • Give them solutions and make they can pick between them.
  • Over Explain
  • Reverse engineer a service
  • What would it happen games?
  • Share wars stories with new SRE. It will make the feel that is okay to failed.

Other

  • Training new SRE is valuable to finding holes.
  • What is the learning plan for this before
  • Things that are important for SREs need to happen a cross an organization. If this is not the case you don’t really have a SRE team.
  • Start working on things that you don’t want to be doing 6 months down the line.
  • Make sure the CEO knows what the SRE team is and doing for the company.
  • Use software engineers to help with the backlog.
  • Tell people the benefit of a SRE. Use case studies
  • Make it easy for engineer to rotate through SRE (Catch and Release)
    • Most of the time it’s will be a catch.

Cassandra Data Modeling Notes

in

Modeling principles

  • Know your data
  • Know your query
  • Nest Data
  • Duplicate data

Modeling Good to know

  • Query-driven data modeling. Application queries should be build before creating a data model.
  • Every query should have its own table.
  • Name tables after queries ex (videos_by_tag_added_year).
  • It’s okay to have duplicated data across multiple tables. We don’t care about space, we care about speed.
  • Cassandra will do an upsert if the primary key and clustering key in a cell are not unique.
  • Primary keys, clustering keys and clustering order can’t be change after a table is create. You will need need to create a new table and move old data to new table.
  • You can only do range queries on clustering keys.
  • Make sure data is duplicate at a constant duplication. If data is duplicated 25 x N, limit N to make the duplication factor constant.
  • A partition can only have 2 billion cells. You are likely to hit performance issues before hitting this limit.
  • A partition should only be hundreds of megabytes on disk
  • A cell or column is a key-value pair.
  • Trade-off between efficiency and space. Use client side joins and don’t duplicate data.
  • There are no JOINs in cassandra.
  • Use BATCH statement to update or insert all duplicated data.
  • Cassandra has SQL like sytanx call CQL for creating and manipulating.

Calculate partition size

The formula below can be use to calculate how big a partition will get to overtime. If a partition gets bigger than 2 billion cells, performance will be affected.

  • Nr -> Number of rows
  • Nc -> Number of regular colums
  • Npk -> Number of primary keys
  • Ns -> Number of static columns
  • Nv -> Number of values
D4E82075A7038CC2E116798E111DB94D.png
  • Nv = 40000 x (7 - 3 - 0) + 0
  • Nv = 40000 x 4 + 0
  • Nv = 160,000

Estimate table disk space

  • Ck -> Partition key
  • Cs -> Static columns
  • Cr -> Regular column
  • Cc -> Clustering column
  • Nr -> Number of rows
  • Nv -> See formula above
  • St -> Size of a table
  • sizeOf -> Estimate size of column
  • St = 16 + 0 + 40000 x ((55 + (8 + 16)) + (12 + (8 + 16)) + (30 + (8 + 16)) + (2340 + (8 + 16))) + 8 x 160000
  • St = 16 + 0 + 40000 x 2533 + 8 x 160000
  • St = 16 + 0 + 101320000 + 1280000
  • St = 102,600,016 bytes
Resources

Cassandra Architecture and Operation

Architecture

  • Cassandra can be configure to be rack and data center aware.
  • Cassandra usages Gossip protocol to failure detection. It uses an algorithm for distributed computing called Phi Accrual Failure Detection. The white paper can be found here.
  • Cassandra uses snitch to determine relative host proximity for each node in a cluster and to gather information about the network topology. Cassandra implements different types of snitch that can be configure. It is possible to build your own snitch as well.
  • Partitions are distributed by Cassandra assigning each nodes a token. Tokens are reassign when a node is taken out of the cluster by using vnode.
  • Cassandra keyspace can be replicated by setting the replication factor. Cassandra can replicate data depending on the network topology. Replicas can be set across data centers and racks.
  • Consistency levels can be control by the client:
    • One -> One replica node must response to WRITE or READ request.
    • Two -> Two replica nodes must response to WRITE or READ request.
    • Three -> Three replica nodes must response to WRITE or READ request.
    • Quorum -> (replication factor / 2 + 1) to WRITE or READ request.
    • All -> All replica nodes must response to WRITE or READ request.
    • Queries can be run in any of the Cassandra nodes in a cluster. This node will become the coordinator node and coordinator node will identify which node the data has to be written or read from.
  • Cassandra keeps internal data in 3 places Memtables, SSTables and commit logs.
    • Commit logs are immediately written when a WRITE operation happen. This logs are use to recover data when Cassandra is restarted and the data has not been saved to disk.
    • Memtables is Cassandra's in-memory storage. This data is not kept in JVM but in a native memory implementation. Each commit log has a one bit flag to determine if data need to be flush to disk.
    • SSTables are use to keep Cassandra data after it has been flush from memory. SStables are immutable. SSTables are only removed after a mergesort and the old files are remove on success.
    • Memtables and SSTables are read to find data values for a READ query.
    • Cassandra Caching layers
    • Key Cache stores a map of partition key to row index entries making it easy to read access into SSTables. This cache is kept in the JVM heap.
    • Row Cache stores a full row to improve reads for frequently access rows. This cache is kept in an off-heap memory. This is off by default.
    • Counter Cache is use to reduce lock contention for the most frequently access counters. This is store in the JVM heap.
  • If a node is offline where a write needs to happen the coordinator node will hold the write as part of a hinted handoff. This write will later be send to the offline node after it gets back online. Only applies to ANY consistency level.
  • Cassandra support lightweight transaction by using IF NOT EXISTS
  • Cassandra doesn't delete rows but marks it for deleting and this tombstones are GC after 10 days. 10 days is the default setting. The GC is delayed to give offline nodes enough time to come back online. If the node down then the node is treated as failed and replaced.
  • Cassandra Compacts larger files into one and create new index of the new sorted data. A new SSTables is created. Compaction increase performance as well by reducing the number of required seeks. During compacting disk IO spikes.
  • Cassandra reads data from multiple replicas in order to achieve the requested consistency level, and detects if any replicas have out of date values. If the number of nodes have a out of date values, a read repair is immediately done. Otherwise, the repair happens in the background.
  • Anti-entropy repair is done manually after a node has been offline for a while. This is done using the nodetool repair.

Performance

  • Create a performance goal and use cassandra-stress tool to generate test load. It is better to do this in the production hardware.
  • It is recommend to keep commit logs and SSTables in separate hard drives to improve performance.
  • Use tracing to see how a query is behaving.
  • Make sure to use caching when possible. Caching can be configure per table
  • Caches can be save to disk and be reuse after a node is restarted.
  • Cassandra will use JVM heap memory and off-heap memory of Memtables. You can influence cassandra to use different type of memory with memtable_allocation_type. off-heap memory is affected less by the JVM GC.
  • It is better to use multiple processing cores over one or two very fast ones.
  • Cassanra has write_survey mode. This mode allows the live traffic to hit this node but won't affect the cluster. The node never joins and will never handle real traffic.

Maintenance

  • Flush Memtables to disk with nodetool flush on the node. You can also flush specific keyspace with nodetool flush keyspace_name.
  • nodetool cleanup scans all nodes and discards any data that is no owned by the node.
  • If a node gets out of sync and the cluster has been gc, you will need to repair the cluster with nodetool repair. The repair command can be limited to a data center with -in-local-dc or -dc . After repairing a table you should rebuild the index with nodetool rebuild_index. Both repair and rebuild_index are CPU and I/O intensive.
  • You can decommission a node by running nodetool decommission on the node. You can see the progess of the decommission by using nodetool netstats.
  • To remove a down node you can use nodetool removenode.
  • When upgrading a cluster make sure to read NEWS.txt and follow the upgrading instructions there.
  • In most cases Cassandra can be upgraded by a rolling upgrade.
    • Run nodetool drain to flush writes to disk and stop the node processing writes. Another node will take over the writes.
    • Stop the node.
    • Make sure to backup cassandra.yml and cassandra-env.sh.
    • Install new version.
    • update new cassandra.yml and cassandra-env.sh to match your backup configuration.
  • Backing up Cassandra
    • Full backups
    • Incremental backups
  • You can configure Cassandra to take auto snapshots when a DROP keyspace, DROP table or TRUNCATE.

Monitoring

A Project Per Week

 

I noticed that every time I started a new personal project, I got bored or ran out of ideas. (I’ve been trying unsuccessfully to code this website for the past four months.) So, I decided to start rotating between projects on a weekly basis. I write down any ideas I get about shelved projects and implement them when the week of that project comes up. So far this system is working out.