Introduction to Python – Class 7 – Functions

Dear Reader,

This is the 8th week of classes that I have taught at the CIC for health innovators, a health incubator/accelerator.  There are currently 3 lessons remaining for the series to be completed.

Last class, we covered basic concept of functions:

Class Slides 7

Class Code 7

In the class, we start with a very basic function with no return statement or parameters.  We then add more “features” to the function.  We begin with return statements, then go through different types of parameters and end with a discussion of variable scope.  Variable scope explores how variables interact with functions.  The last topic covered is import statements and packages.  We get into a discussion of how pip works and what PyPI is.

I didn’t cover the advanced section of the course, which goes over lambdas, decorators, recursion, memoization and closures.  Class 8 will cover these subjects and then go through 2 examples: 1. Medical Records processing example and 2. CLI builder tool called “Click”, which extensively uses decorators.  The later two projects will be considered part of Class 8.

I really wanted to get into some of Peter Norvig’s work.  Since I didn’t cover it in class, I will add it as suggested article below:

Peter Norvig’s Sudoku

 

Best,

Chris

 

Monitor: Graphite, Collectd, Statsd and Graphana

Dear Reader,

Need to monitor many computers at the same time? Are you worried about a pegged CPU, memory paging, lots of swap space activity or low disk space? What about high network traffic? These things are readily available on Amazon through AWS CloudWatch.

I decided to implement 1 of 2 major monitoring stacks. The first stack is: Logstash, Elasticsearch and Kibana (ELK stack). The second group of technologies is: Collectd/Statsd, Carbon/Whisper and Graphite/Graphana.  I implemented the later.

My pre-mature conclusion is:

  1. ELK stack: Is great for text heavy documents.  Elastic Search is based on Lucene, a popular open-source search engine.
  2. Graphana/Carbon: is great for time series analysis with an emphasis on near realtime data feeds.  Collectd provides a convenient plugin/interface to operating system statistics.

Below is the Front-end component monitoring the monitoring computer (not WordPress):

Graphana Server

Graphana Server

 

The components of the system are:

  1. Collectd – data collection software with a plugin architecture.  Common plugins seen above include: CPU, Memory, Disk Usage, Processes, Network Traffic, Apache metrics and much more.
  2. Statsd – Used more for application monitoring.  You can send custom metrics based on set intervals.  The common 4 metrics I saw: gauges, counts, sets and intervals.  The first one takes the last of measurement within an interval and reports it.  Counts aggregate data over 10 seconds.  Sets return a unique count of values encountered (via UDP).  Intervals are time-based calculations (like rates).
  3. Carbon/Whisper – A data store focused on time series aggregation.  Data is aggregated based on two configuration files.  The first sets up regex matches (used to categorize TCP traffic) followed by a data retention policies as well as specifying polling time-frames (typically in seconds or minutes).  The second file specifies the type of aggregations available: sum, min, and max.
  4. Graphite/Graphana – Graphite is a front-end dashboard tool for carbon.   Graphana serves a similar role, but with a sleek black UI (featured above).  Both are similar to Kibana.  They both provided dashboard and graphing utilities to their respective data stores.  With Graphana being able to access other sources (like ElasticSearch).

For my Graphana localhost dashboard, I used CPU, Memory, DF and Processes modules from Collectd.

If interested in trying out the service, I can recommend the following 4 articles (Ubuntu):

digital ocean: metric tracking tutorial

For Graphana, you have to follow these instructions:

Graphana Install

Best,

Chris