Category Archives: Software Systems

Fall 2018 – Readings and Online Courses

Leave a reply

Dear Reader,

I decided to write quickly about 6 programs/books I took: AI Programming with Python (Udacity), Google Cloud Platform Architecting (Coursera), Creating Kivy Apps by Dusty Philips, Linux Kernel Development by Robert Love, Terraform: Up and Running by Yevgeniy Brikman and the Design of Everyday Things by Donald A. Norman.

Udacity: AI Programming with Python

This is the entry-level course in AI for Udacity.  Udacity is an online course-ware provider that focuses on providing content on cutting-edge topics: AI, Deep Learning, Self-driving cars and Node/React.  It also has classes on data analysis and marketing.  AI program starts by teaching you python, brushing up on some linear algebra and finishing off with 4 sections on AI: Two theoretical units on gradient descent and neural networks, one on PyTorch and a final project that classifies flowers.  PyTorch is Facebook’s AI framework built in Python.

AI Class

The course is well prepared overall and should be manageable for someone with intermediate programming skills.  Best in Python.  It’s a bit on the expensive side for online material: $500-$1000, but Udacity tends to make up for the price tag with more interesting content and topics.  Overall the quality is good.  I’ve heard FastAI is a free alternative.

My Repo: https://github.com/Silber8806/udacity-AI-programming-with-python

Coursera: Architecting with Google Cloud Platform

Offered by Google Cloud

Google Cloud Platform, GCP, is googles answer to AWS, Amazon Web Services.  Overall, the platform has many of the same services as Amazon including Cloud Functions, which looks to be AWS Lambda for Google Cloud.  The only glaring issues I’ve noticed so far: 1. No equivalent to SNS for e-mail and some UI issues in the console concerning Google Cloud Storage (S3).  The first part is painful in that Google by default blocks the smtp ports forcing you to use a 3rd party provider (with limited support in stackdriver).

Google outsourced most of it’s training for GCP to coursera.  The courses are broken into 6 parts.  The first course covers all of the services on a 40,000 foot level, 4 courses proceeding this are in-depth technical reviews of virtual machines, networking, storage, database technologies, container services and autoscaling solutions.  It also includes a host of managed services including the very neat sounding: “Spanner”, a supposedly horizontally scaling RDBMS service.

The last course, Reliable Cloud Infrastructure: Design and Process,  was the most interesting.  It provides a practical overview of how to construct infrastructure as well as going over Google’s philosophy on designing scalable software.

Something that really stood out about this program is the labs.  They provide you a temporary Google Cloud Account for your tutorials.  You get to practice on the actual Google console and shell prompts.  This program is reasonably priced at $49.99 per month.  You can take the material without labs for free, but I’d recommend the labs!

Creating Apps in Kivy:

Kivy is Python’s Mobile Application development framework.  It’s used to make cross-platform applications for: Android, IPhone and Ipads.  It’s something I’ve wanted to try out for a while just out of curiosity.  I picked up Creating Apps in Kivy by Dusty Phillips, a book on the subject.

Creating Apps in Kivy

The book covers two applications: A weather application using an open weather api and a video game involving spaceships.  The weather app takes the block of the book with significant amount of time spent on UI and UI interactions.  Kivy generally is split into two files: a main file in python that is event-driven and a .kv file that describes the component layout.  A single chapter is dedicated to graphics.  The best part of the graphics section was developing animated snowflakes that fell from a single component.  The book also covers databases and advanced UI components like a carousel.

Overall, I’m happy with the quality of this book with the exception of 2 things.  It was hard to read the indentation between pages and there were a few chapters where I had to add imports or features he didn’t mention to get the examples to run.  For this reason, I’d recommend being somewhat well versed in Python before attempting the sample projects in the book.

Github Repository: https://github.com/Silber8806/kivy_tutorial

Linux Kernel Development (3rd Edition) by Robert Love:

I started reading this book after my colleague at Salesforce.com, a DevOps Engineer, began talking to me more about the Linux Kernel and strange behaviors in Bash.  I read a previous book to get a better understanding of the OS and am really happy that I skimmed through this book.

Kernel Book

Linux Kernel Development covers the Linux source code by going through different subsystems and explaining how it works.  It’s a deep-dive into things like: process management, memory, I/O, Virtual File System, Cacheing, Timesharing… It’s inspirational to see how much thought has gone into the Operating System and how many improvements are mentioned.  You really feel like you are standing on the shoulders of giants after reading this book.

Linux Kernel Development feels like a technical reference and I’d recommend reading a book on Operating Systems before getting too deep into it (or taking courses in C).  I skimmed through this book, because I use Linux daily and feel a bit clueless about what is happening under the covers and wanted to learn more about it.

Terraform: Up and Running by Yevgeniy Brikman:

Terraform: Up and Running covers… Terraform, the cloud provisioning software by HCL corp.  .  Terraform is a framework for deploying IT services to AWS, GCP, Azure and Digital Oceans in an automated fashion.

Terraform is cool in that you write your infrastructure as a series of templates.   These templates are in the form of a declarative language called HCL, which describes your cloud infrastructure typically in a 1-to-1 fashion with the services you have deployed.  What’s special about Terraform is that after you apply a HCL template to your cloud environment, it writes the current state into a file (immutable data structure).  Every additional change begins with your last state and modifies it.  A lot of devops systems are not aware of their previous configurations.  Terraform reminds me a little of Redux, but applied to Infrastructure instead of web development (I might be wrong).

Things I noticed from the book.  They suggested deploying these state files to S3, expect strong concurrency controls around the Terraform library and advice that you don’t mix other devops tools with Terraform.  That way if a user manual creates an account that Terraform isn’t aware of, it won’t create an error (since that manually created user isn’t part of Terraform’s state).  It seems Terraform works best as an isolated service and the “main” provisioning tool for a specific subspace of your infrastructure.

I skimmed through this book as I was interested in Terraform and have heard of a lot of adoptions in industry of this technology.  I didn’t get to test the code in this book and can’t really speak about it.

The Design of Everyday Things by Donald Norman:

The Design of Everyday Things by Donald A. Norman is a book about usability and user design.  It’s the book that I’ve been listening to on Audible, an audio book app by Amazon.com.

The book goes over the design of everyday things: ovens, doors, cars, computers, phone systems, radios and details both design flaws and positive things about different objects.  He focuses a lot on understanding differences between knowledge in the world vs in the mind, how we map controls (buttons etc) to actions and how overloading them can lead to confusion and how to constrict user settings to prevent too many options flooding the users consciousness.

He’s a big proponent of mapping controls to physical aspects of the world.  For example, a car seat control should move the seat forward if a switch is moved forward and backwards if a switch is flipped backwards.  A switch shouldn’t move forward and backwards if a button is pressed left or right as that will confuse a user.  He also favors designs were cues exist in the world that indicate what to do and where each control (button) does one and only one thing.  A button on a watch should only turn off the alarm not adjust the alarm or answer phone calls as well.  If these aspects of designs can’t be applied, he suggests standardization across an industry.  A water facet for example should have cold water knob always to the right of the hot water knob.

This is a great book.  I would suggest reading it on Kindle or as a paperback book. My big issue with Audible and Audio books in general is that I typically listen to them while walking or multi-tasking.  That means I lose focus on certain sections (last 2 chapters).  Overall, I really enjoyed the content I heard from this book.

Best,

Chris

Monitor: Graphite, Collectd, Statsd and Graphana

Leave a reply

Dear Reader,

Need to monitor many computers at the same time? Are you worried about a pegged CPU, memory paging, lots of swap space activity or low disk space? What about high network traffic? These things are readily available on Amazon through AWS CloudWatch.

I decided to implement 1 of 2 major monitoring stacks. The first stack is: Logstash, Elasticsearch and Kibana (ELK stack). The second group of technologies is: Collectd/Statsd, Carbon/Whisper and Graphite/Graphana.  I implemented the later.

My pre-mature conclusion is:

  1. ELK stack: Is great for text heavy documents.  Elastic Search is based on Lucene, a popular open-source search engine.
  2. Graphana/Carbon: is great for time series analysis with an emphasis on near realtime data feeds.  Collectd provides a convenient plugin/interface to operating system statistics.

Below is the Front-end component monitoring the monitoring computer (not WordPress):

Graphana Server

Graphana Server

 

The components of the system are:

  1. Collectd – data collection software with a plugin architecture.  Common plugins seen above include: CPU, Memory, Disk Usage, Processes, Network Traffic, Apache metrics and much more.
  2. Statsd – Used more for application monitoring.  You can send custom metrics based on set intervals.  The common 4 metrics I saw: gauges, counts, sets and intervals.  The first one takes the last of measurement within an interval and reports it.  Counts aggregate data over 10 seconds.  Sets return a unique count of values encountered (via UDP).  Intervals are time-based calculations (like rates).
  3. Carbon/Whisper – A data store focused on time series aggregation.  Data is aggregated based on two configuration files.  The first sets up regex matches (used to categorize TCP traffic) followed by a data retention policies as well as specifying polling time-frames (typically in seconds or minutes).  The second file specifies the type of aggregations available: sum, min, and max.
  4. Graphite/Graphana – Graphite is a front-end dashboard tool for carbon.   Graphana serves a similar role, but with a sleek black UI (featured above).  Both are similar to Kibana.  They both provided dashboard and graphing utilities to their respective data stores.  With Graphana being able to access other sources (like ElasticSearch).

For my Graphana localhost dashboard, I used CPU, Memory, DF and Processes modules from Collectd.

If interested in trying out the service, I can recommend the following 4 articles (Ubuntu):

digital ocean: metric tracking tutorial

For Graphana, you have to follow these instructions:

Graphana Install

Best,

Chris

 

SEO for Growth – Bend Google to your Way!

Leave a reply

SEO for Growth

SEO for Growth Book

SEO for Growth

Bloggers often make fortunes with financialsamurai.com being a great example.  The question, how do you become a successful blogger and more importantly a successful marketer.  SEO for Growth by John Jantsch and Phil Singleton discusses how you can optimize your blog to beat those Google Search rankings and be listed on the first page.  That can be the difference between getting no traffic and being like this cute pug watching money fall from the sky.

Cute pug watching $1 bills fall

Isn’t he cute!

What does Google hate?

Spammy pages that look more advertisement then content!

You forgot smart phones exist!

You cover half your webpage with Eye Junk!  The other half is infinite scrolling.

You are a copy-paste king/queen.  Very original!

What Google Likes!

Google is in the business of answering search queries.  Anything that detracts from that is bad business.  Google is interested in original content that best answers search queries and penalizes websites that produce noise or distraction. To simplify marketers and web designers lives Google provides resources on best practices curate great content and provides tools to analyze your website (site maps, simulations of google crawls, viewing data and more).  Google also graciously provides paid advertisement as well:

Adwords!

Each day, 3.5 billon people ask Google a question.  Over a single year, that is 1.2 trillion searches.  That’s as many searches a day as Donald Trump is estimated having in cold hard cash.  You as an advertiser wants a slice of those people’s attention, which is where Adwords comes into (word) play.  Adwords let’s you bid on search terms, the questions people ask.  Rarer search terms have less demand.  SEO for Growth says this is fertile ground as you can bid for more targeted terms that better represent your customer.  In turn, your customer is more likely to convert into a purchase or blog reader.

Google Analytics! 

Everyone loves a dashboard.  Don’t they?  Google Analytics might be a Marketers Mecca in that it prevents a view of your audience.  Do people coming to your websites like your article on clothing or would rather stalk your About me page?  Google Analytics also tracks other useful information.  For example, where did the person originate?  Did they start on another 3rd party website or where they intensively Googling for an answer?  This type of information can help you fine-tune what appeals to your audience and, if you are clever, be utilized for A/B split test.  A/B split testing is when you send people to two webpages that differ by a single feature.  You then see who stays around, buys or clicks on more ads (group A or B).

Google of course provides a lot more tools than presented in this blog.  I will most likely get into more details in the future.  For now, I have a laundry list of site improvements mentioned in SEO for Growth!

Hopefully, with these tools, I can finally have my website pay for my next Latte!

Best Luck Blogging,

Chris

 

Learn Web Development: Backend Certificate

Leave a reply

Dear Readers,

In the last few posts, I explored freecodecamp.org, a non-profit that assists people in learning how to develop websites for free.  I also show cased 10 quickly developed front-end prototypes.  Why is free code camp awesome, it’s a free curriculum that prepares you for a Junior Web Developer role (Salary).  It’s also great practice if you want to learn JavaScript, HTML and CSS.  As a database engineer, I really wanted to learn more about these technologies as almost everything touches the web in some way.  In this post, I will talk a bit more about backend-certificate and my current progress towards getting the certificate.

FreeCodeCamp.org provides 3 certificates.  The first one is called the front-end certificate.  It is the one that I would recommend starting with.  It begins with very simple elements of a webpage like text, images and buttons.  It then progresses through 200 lessons until you can develop TicTacToe and the Simon Game.  The first 150 or so lessons will take the average person approximately 5 minutes each and so can be done during your daily commute (train).  The remaining ones take significantly more time, but are well worth going through.

The next certificate I’m attempting is the backend certificate.  Front-end development involves the actual webpage, the buttons, text and images that are presented in your browser.  Front-end applications tend to utilize your computer and the resources tied to it.  The backend certificate focuses more on sending and processing data stored on your a computer to your browser.  Why would anyone want to store data on a separate computer, process it and send it to your web browser?  First, you can process things in one location preventing a bunch of other computers from doing the work.  This is the focus of cacheing.  Second, you want a guarantee that your data doesn’t change, which is near impossible to do if you don’t have control of the computer.  Third, you can take all this data and often aggregate it to look at interesting group behavior.  These are just a few examples why you’d want a “backend” server.

The current curriculum focuses on a framework called Node.js.  Node.js is written in JavaScript and is a framework for dealing with network connections.  A good example of a network connection is going to your facebook profile and making a post.  It involves, getting your profile page from a Facebook backend server called a http get request and then providing data to facebook in the form of your post called a http post request.  Both these requests need to be handled by some form of network software, one to get your profile information and the other to store the new data in the database, permanent place for your data.  Node.js specializes in handling these types of connections.  In fact, it can handle hundreds of connections in a few seconds from many different people.

What other types of technology are covered and what would you learn from this curriculum:

 

 

npm – Node Package Manager, when you are dealing with websites there are a lot of moving pieces.  You want to be able to set up node.js to handle connections.  You also want to build logic around those connections, are you a bank processing credit card balances or is it just retrieving your blog post.  NPM allows you to save a copy of that logic and makes useful logic available for others to use.  It also provides a way to do automated tests.  If you decide to for example change the functionality of your blog, you can run tests to see if it breaks your websites.  Broken website often results in a blank page or missing elements.  In the course, you do one whole module consisting of about 12 tasks that teach you how to manage your code.

node.js – Node as mentioned above is a library that deals with network connections.  Many people use Node specifically for websites, but you can use it for other situations as well (maybe forwarding database connections).  In this curriculum, there are two lessons that utilize node.  One of the things you will end up building is a service where you type in a url with a date and it will convert it into different timezones.  These backend services can of course become significantly more complicated.

express.js – Express is another JavaScript library, but focuses more on building websites.  Express.js would set up for example a template of how a Facebook profile would look like (Picture on the top left, a background picture, short summary and posts below) and bind it to a specific url.  It would also create separate urls for your facebook profile vs your activity feed.  The lessons focus on very simple webpages and templating.

mongodb – mongodb is a database and is not written in JavaScript.  It focuses on storing data on the backend server and is optimized for quick retrieval.  Mongo itself is called a document store, which means that it does not store things in tables like a spreadsheet, but rather as a set of keys and collections.  One neat thing about this is that you can store a set of collections within a collection, which makes it very flexible.  An example of this type of data can be seen here (it is called JSON):

Example Google Maps JSON File

MongoDB section has about a dozen tasks and focuses on storing data, retrieving it and using it in a web application.  This is important in that you can retrieve the data for a specific person and just put it in a template (or webpage).  Hence, your name, summary and posts in facebook are all stored in the database and are unique to the individual (database like Mongo), but are presented in the same way as a profile page (node.js/express.js provide template and logic).

In the next few posts, I will talk about the second part of the backend certificate, which are the 10 projects that utilize the tutorials mentioned previously to produce products people can use.  After doing those 10 projects, a person at freecodecamp.org could potentially develop backends similar to quora.com or facebook.com.  One thing to note, even though that website would look similar to facebook, it would be missing the messaging and not operate at the scale of Facebook.  Handling webpages for billions of users is a much harder problem.

Best,

Chris

Powerful Accessories to Excel – Introduction

Leave a reply

Dear Readers,

I wanted to start a quick tutorial targeted at Excel users and highlighting cool technologies that are relatively easy to learn and can provide a lot of benefit.  Basically, they are technologies that provide a lot of bang for their “effort” bucks.  The technologies I want to cover in order of ease to learn are:

Bash/Batch – Bash and Batch are shells for Linux/Mac and Windows respectively.  Shells interact with your operating system to allow you to do things like read files, manipulate data, create remote login sessions and schedule tasks on your computer.  One of the great things about bash and batch is that if you study it long enough, you can start using “cloud” computers, which allow computers to do work, while you are sleeping.

SQL/RDBMS – Relational databases allow you to store data compactly on your computer and provide a convenient interface for doing data manipulation.  One great reason you should pick up this tool is that it allows you to specify relationships between data: Student – Class -Teacher – School being a good example and then make statements like: I want only class taught by John Naughton that are considered math classes and only students whose first name starts with A.  One great thing about databases is scale.  You can work with 100s of millions of rows. Since SQL is a standard for data access, there are SQL-esque access to big data tools like Hadoop and Spark, which is software that allows you to take dozens of computer and have them process things in parallel.

Python – Python is a general purpose programming language that provides countless libraries for different purposes.  Do you want to develop a website or automate networking?  Python can do this.  Do you want to scrape a website, travel through a social network or do analysis on wikipedia?  Python is a great tool for it.  Do you want to be on the cutting edge of machine learning and AI by having direct interface to more advanced libraries?  Python…  Python can also be used for data manipulation and processing, which I will focus on.

I focus on the above three based on my knowledge.  Other noteworthy technologies include R, which I hope to cover in a future article.  During these tutorials, I might also mention other cool concepts or technologies.

If you are interested in this lecture series, I will put it under tutorial categories.

Best,

Chris