Category Archives: Languages

Data Camp: Week Adventure into Python Data Science

Image result for data camp

Introduction:

Noah, a former TechTarget colleague, mentioned  datacamp.com  to me, while we were discussing an issue with remotely hosted files (probably rdp associated) and pandas DataFrames.  On February 10th, I decided to give it a try and today, 6 days later, I finished up the Python Programmer Track (10 classes)!

What is Data Camp?  Data Camp is an online education company that offers data science specific courses.  They focus mostly on two lines of technology: R and Python.  They offer some auxiliary courses in other topics: SQL, Linux and git (as well as a few derivatives).  All the courses are hosted on their online platform, which overall is beautiful to interact with.  A bit more about the topics:

Data Camp Topics:

R – Statistical Programming Language

A statistical programming language inspired by LISP and developed by professors at the University of Auckland.  R has become famous amongst statistics and machine learning researchers where they prototype cutting-edge research and provide it as packages to CRAN, a online index for hosting R code.

Python – General Purpose Language

A general purpose programming language developed by Guido van Rossum.  It’s a dynamically typed, interpreted language that has been used for rapid prototyping and scripting.  Python as a general purpose programming language that can be used for: web development, networking, devops, data analysis and machine learning.  Python has gained popularity over the last few years with increased interest in it’s scientific computing platform.  Pandas, a popular Data Science framework, was inspired by R’s data frame.

SQL – Structured Query Language

Structured Query Language, SQL, is a domain-specific language typically associated with relational database management systems (RDBMS aka SQL databases), but has been coopted by other solutions (BI platforms, Hadoop and Spark).  SQL is easy to learn and revolves around a few common entities: Servers, Databases and Tables (Views etc).  Tables can be viewed as tabular data where rows are entities (employee) and columns are attributes of the entity (salary).  Most SQL is used to retrieve data hosted in tables mediated by RDBMS.

Context

R and Python are used to manipulate data in a procedural way, typically through the use of things like Data Frames (R), Pandas (Python) or Numpy (Python).  These libraries create tabular, matrix, vector or scalar data often rely on vectorized operations to do computation.  Both R and Python have extensive data visualization frameworks to create graphs.  They also have great libraries for statistics, machine learning and artificial intelligence.  SQL is primary associated with data retrieval and is used to get “data” back from a hard drive (through the database).  Most database are more limited in functionality when it concerns more finite data manipulation, graphing and machine learning (though extensions do exist).  They make up for it by being able to store large quantities of data without relying extensively on memory (volatile and limited).  Generally, most software engineers and analysts use both a programming language like (R/Python) and SQL (to access data).

Data Camp:

Summary:

Data Camp breaks down Python/R courses on career tracks: Python Programmer, Data Analyst and Data Scientist.  Each track is composed of a number of courses: 10, 13 and 20 respectively.  Topics covered: basic programming, data manipulation techniques, graphing, statistics, machine learning, network analysis and ai (1 course).

Each course is composed of 3-5 segments.  Each segment has a set of lectures followed by exercises.  You can expect around 3 lectures and 10 exercises per segment.  Most courses build on themselves intuitively beginning with the basics and gradually building up in complexity.  I was surprised to find lectures on generators, closures and how they relate to data frames within the first 4 classes.

Lectures:

datacamp lecture

Lecture portion of the website is beautiful.  Presentations have nice transitions and the website background is not distracting.  Lecturers were clear and easy to follow.  You get a sense that al to of effort was put into the curriculum.  There were multiple lecturers in the 10 courses I took (around 5-6).  Some of these lecturers work for esteemed companies like Anaconda, published books on the topic they spoke on or had a background in software engineering/consulting.  Overall, the lectures were high quality with very few mistakes.

Practice Sets:

datacamp practice problems.

The practice problems were conducted in what looks like a modified ipython notebook embedded in the website.  They have 4 panels, the 2 on the left: exercise and instructions and 2 on the right: Scrapt.py and ipython Shell.

The exercise and instructions provide guidance on how to complete the exercise.  The Exercise section explains the topic.  The instructions tell you what steps need to be completed before submission is excepted.

Script.py is where you write your code.  The run code button let’s you execute script.py and see the output in the IPYTHON SHELL.  It’s pretty interactive.  When you submit answer, it checks if the solution matches the instruction section.  For the most part, there are few cases where code I submitted was marked wrong when it was in fact right.  That’s great!

What if you get stuck on a programming problem?  There is a button called hint.  This provides some extra guidance typically in the form of small chunks of code.  If you press the hint button, a new button called show solution will appear.  Clicking the show solution button will overwrite the SCRIPT.PY window with the correct solution, which you can then run and submit.

In rare occasions, you might end up with a multiple choice question.  They offer an interactive shell in this case.  It always proceeds the more free-form question variant.

Gamification:

Data Camp has one really great feature that I liked.  Each exercise has an amount of xp that you can collect.  Each lecture is worth 50xp, multiple choice is worth 50xp and problem sets are worth 100xp.  If you click the show hint button, the problem set xp is reduced to 70xp.  Show solution gives you 0xp.  When logging in, you will see the total xp you got that day as well as how many days in a row you have utilized data camp (streak). This is a great way to motivate you to do the exercises.

Another layer of gamification comes from certificates you can collect (and post on linkedin) as well as career tracks you can complete.

Cost/Summary:

I think data camp is very friendly to beginners interested in learning about data analysis in Python and R, both marketable skills.  I think the course layouts, lectures and practice problems are well thought out.  I would suggest that beginners also read books and online documentation on subjects like Pandas and Numpy.  I found data camp focused more on practicing skills and less on implementation details (which is a good thing).

Data Camp is currently on sale for $180/year (usually $300/year).  You can also buy it as a monthly subscription for $30/month.  Data Camp is similar to Udemy.  There are 3 advantages Data Camp has over Udemy:

  1. Practice problems make up a larger percent of the curriculum.
  2. $30/month you have access to any one of 100+ courses.  The Udemy equivalent is $15/course.  A Udemy course is equivalent to 3 Data Camp courses (in material)
  3. Data Camp specializes in data science and has really thought about how to naturally progress through the data science material.
  4. If you like structured programs, this is better.

Advantages of Udemy:

  1. Overall larger selection of content, variety and topics.  If you want to practice Python, but want to tackle multiple topics like: web programming, networking, penetration testing or a specific subset of machine learning.  It might be a better option.
  2. There is no subscription fee.  It’s fixed cost.  If you are not sure you want to invest a lot of time into python programming this is a better choice.
  3. Lectures are not as uniform.  You might find some lectures that are more theoretical.  Others are more practical.  That means you can try out different lecture styles to see what works.

Best,

Chris

 

 

Interactive Programming Diagrams

Python Tutors

Introduction:

Philip J. Guo is a professor of Cognitive Science at UC San Diego, who focuses on teaching programming at interactively online.  He has produced great products for both programming beginners and those interested in Python internals (10-hour lecture).

He impressed me with Python Tutors website: python tutors. Python Tutors is: Python, Java, JavaScript, TypeScript, Ruby, C and C++.  Python tutors converts Python written on the left into data structures on the right.  This provides a deeper view of what Python is actually doing.  The diagrams are step-based, which means you can see the execution of each line of code and what happens beneath the covers.  Red and Green arrows show you the line to be executed and the line being executed.  An example below.

Example (click image to go to interactive page):


Related Posts:

introduction to python – health innovators class

great python books – beginner to intermediate

Happy Holidays,

Chris

Introduction to Python Courses

Dear Reader,

I am providing free lectures on intro-to-python.  Salesforce supporting me through their voluntary time off program.  Salesforce.com under it’s 1-1-1 program gives each employee 7 days to work on voluntary projects.

You can find the lectures here:

Lectures

Code

Application Class

The 4th class will be held at Cambridge Innovation Center, 1 Broadway, Cambridge Ma on Saturday November 18th 2017.  They are held every 2nd Saturday afterwards.  Possible projects:

  1. A mini Q/A program.  It introduces the concept of regex using re.
  2. Opening a file in python, reading it’s lines and analyzing words.
  3. Utilizing SimpleHttp server host a basic webpage with <p>, <h1> and <div> tags.  This is a simple introduction to a one-line web server.

The curriculum is free and I encourage people to submit practice problems to the GitHub repository.

Moodle Platform

Moodle is an open-source learning platform. Often, Moodle is used in universities.  I plan on implementing a Moodle instance to host lectures online.

Spammy E-mails, Great!

Moodle implementation has stalled.  I decided to host e-mail service myself.  I did get smtp and e-mail server up.  The obstacle now is getting Google and other e-mail providers to realize my E-mail isn’t spam.

Why is it considered SPAM?

Evidently, you can send an E-mail from chriskottmyer.com, but claim it originated from john smith.com.  Web industry has developed two processes to prevent this: SPF and DKIM.  SPF creates a guarantee that a message from johnsmith.com originates from johnsmith.com.  DKIM encrypts SMTP header preventing snoopers from changing that in transit.

Apache and VPL!

After resolving the spam crisis, I will have to deal with an annoying URL issue.  Moodle loves my IP address.  It loves it so much, it’s bound it to all the URLs.  I don’t like!  I’ll have to make either application-level change or re-route in Apache to resolve.

Having Moodle is great.  It doesn’t support programming assignments out of the box.  Luckily, some wonderful academics invented a plugin called the VPL.  It takes code presented to the web, submits it to a restrictive JVM-based sandbox and runs it.  It should prevent any malicious hackers from hijacking the server (crossing fingers).  It also supports automated grading of coding exercises (yay!).

Both issues aren’t blocking the lectures!  Hopefully everyone can enjoy those!

Best,

Chris

 

Learn Web Development: Backend Certificate

Dear Readers,

In the last few posts, I explored freecodecamp.org, a non-profit that assists people in learning how to develop websites for free.  I also show cased 10 quickly developed front-end prototypes.  Why is free code camp awesome, it’s a free curriculum that prepares you for a Junior Web Developer role (Salary).  It’s also great practice if you want to learn JavaScript, HTML and CSS.  As a database engineer, I really wanted to learn more about these technologies as almost everything touches the web in some way.  In this post, I will talk a bit more about backend-certificate and my current progress towards getting the certificate.

FreeCodeCamp.org provides 3 certificates.  The first one is called the front-end certificate.  It is the one that I would recommend starting with.  It begins with very simple elements of a webpage like text, images and buttons.  It then progresses through 200 lessons until you can develop TicTacToe and the Simon Game.  The first 150 or so lessons will take the average person approximately 5 minutes each and so can be done during your daily commute (train).  The remaining ones take significantly more time, but are well worth going through.

The next certificate I’m attempting is the backend certificate.  Front-end development involves the actual webpage, the buttons, text and images that are presented in your browser.  Front-end applications tend to utilize your computer and the resources tied to it.  The backend certificate focuses more on sending and processing data stored on your a computer to your browser.  Why would anyone want to store data on a separate computer, process it and send it to your web browser?  First, you can process things in one location preventing a bunch of other computers from doing the work.  This is the focus of cacheing.  Second, you want a guarantee that your data doesn’t change, which is near impossible to do if you don’t have control of the computer.  Third, you can take all this data and often aggregate it to look at interesting group behavior.  These are just a few examples why you’d want a “backend” server.

The current curriculum focuses on a framework called Node.js.  Node.js is written in JavaScript and is a framework for dealing with network connections.  A good example of a network connection is going to your facebook profile and making a post.  It involves, getting your profile page from a Facebook backend server called a http get request and then providing data to facebook in the form of your post called a http post request.  Both these requests need to be handled by some form of network software, one to get your profile information and the other to store the new data in the database, permanent place for your data.  Node.js specializes in handling these types of connections.  In fact, it can handle hundreds of connections in a few seconds from many different people.

What other types of technology are covered and what would you learn from this curriculum:

 

 

npm – Node Package Manager, when you are dealing with websites there are a lot of moving pieces.  You want to be able to set up node.js to handle connections.  You also want to build logic around those connections, are you a bank processing credit card balances or is it just retrieving your blog post.  NPM allows you to save a copy of that logic and makes useful logic available for others to use.  It also provides a way to do automated tests.  If you decide to for example change the functionality of your blog, you can run tests to see if it breaks your websites.  Broken website often results in a blank page or missing elements.  In the course, you do one whole module consisting of about 12 tasks that teach you how to manage your code.

node.js – Node as mentioned above is a library that deals with network connections.  Many people use Node specifically for websites, but you can use it for other situations as well (maybe forwarding database connections).  In this curriculum, there are two lessons that utilize node.  One of the things you will end up building is a service where you type in a url with a date and it will convert it into different timezones.  These backend services can of course become significantly more complicated.

express.js – Express is another JavaScript library, but focuses more on building websites.  Express.js would set up for example a template of how a Facebook profile would look like (Picture on the top left, a background picture, short summary and posts below) and bind it to a specific url.  It would also create separate urls for your facebook profile vs your activity feed.  The lessons focus on very simple webpages and templating.

mongodb – mongodb is a database and is not written in JavaScript.  It focuses on storing data on the backend server and is optimized for quick retrieval.  Mongo itself is called a document store, which means that it does not store things in tables like a spreadsheet, but rather as a set of keys and collections.  One neat thing about this is that you can store a set of collections within a collection, which makes it very flexible.  An example of this type of data can be seen here (it is called JSON):

Example Google Maps JSON File

MongoDB section has about a dozen tasks and focuses on storing data, retrieving it and using it in a web application.  This is important in that you can retrieve the data for a specific person and just put it in a template (or webpage).  Hence, your name, summary and posts in facebook are all stored in the database and are unique to the individual (database like Mongo), but are presented in the same way as a profile page (node.js/express.js provide template and logic).

In the next few posts, I will talk about the second part of the backend certificate, which are the 10 projects that utilize the tutorials mentioned previously to produce products people can use.  After doing those 10 projects, a person at freecodecamp.org could potentially develop backends similar to quora.com or facebook.com.  One thing to note, even though that website would look similar to facebook, it would be missing the messaging and not operate at the scale of Facebook.  Handling webpages for billions of users is a much harder problem.

Best,

Chris

Learn Web Development: Front End Certificate – Finished!!!

Dear Reader,

I finally finished the JavaScript front-end course.  That involved two last projects, which I’ll present here.  Warning these are not mobile optimized and look ugly on a phone:

Pomodoro Clock:

Pomodoro clock is count-down clock, which allows you to switch between work and break sessions.  You can adjust the work and play dials to choose how many minutes you want to work or play.  If you press the large Session button, the count down begins.  This project wasn’t that difficult to complete.  The hardest thing was figuring out how to create a circle, center it and then start a timer.

TicTacToe:

This is a tictactoe board with a very “dumb” AI.  It starts by allowing you to choose Xs or Os.  It then continues until either you win, the computer does or a draw occurs.  The AI plays a rather simple strategy:

  1.  If the AI can win that turn it.  It plays the winning move.
  2. If the AI will lose this turn, it will block the player from winning.
  3. If the AI can’t win or block a losing move, it finds the diagonal, column or row with 1 of it’s own tokens in it and no opposing token.
  4. If the previous 3 positions do not occur, it picks a random diagonal, column or row with no opposing token in it.

At first when I played this AI, I thought it was pretty smart.  I soon realized that playing the bottom 2 corners and one of the top corners results in a win for the player every time (at least for O).

Simon Game:

Simon is an old 1990s Toy that plays a sequence of colors that the player has to match.  It starts with a sequence of 1 color and increases it by one if the player matches the entire sequence without an error.  It does this up to a sequence length of 20 and then declares the player a winner.  The Simon game has a strict mode, which will reset the sequence to 1 on a player error.

This specific implementation took me around 4-5 hours.  The difficulty came from the fact that JavaScript is asynchronous.  This means code that looks sequential in the file executes at the same time.  This came as a shocker to me when I pressed the start button, was immediately declared a winner and then watched the entire circle turn lightblue 2 seconds later.  What had happened is all 20 sequences had played out at the same time including all… 20 + 19 + 18 … 1 button presses (2 seconds later).

To resolve this specific issue. I created an event on the color buttons.  If the player had pressed the correct sequence of colored buttons to match the computer’s color sequence, it would add a new color to the color sequence and repeat it back to the player.  To make sure the play timing of the color/sounds  was correct, I iterated over the index multiplying each index value i by 2000 milliseconds.  This ends up solving all those strange asynchronous bugs and makes the game playable.

After this project, I finally got the front-end certificate for free code camp.  It was a cool 6 weeks of part-time coding.  The last 2 weeks were especially exciting, because I got to do a deeper dive into JavaScript itself.

Best,

Chris