Category Archives: Python

Quick Experiment with Networks (Python)

Python has some really great libraries for networking.  Going from multi-threaded asynchronous services to one-time use cases to more generalized services to run commands over many servers.  With several hundred virtualized instances to manage, utilities to deal with some of the more common tasks from the command line become a lot more useful.  So, a list of cool libraries I’ve recently looked into:

fabric – A cool python library that abstracts hosts into a single managed list and then allows you to execute commands over the entire host list.  Used in Salt (devops) library and uses paramiko under the cover.  More on that later.  Has a single entry point called a fabfile.  Great for developing a set of tasks for a central computer.

http://www.fabfile.org

paramiko – A library that makes a peer-to-peer ssh connection relatively easy through SSHClient class.  The SSHClient class allows you to set up policies for dealing with unknown hosts etc.  Connecting is pretty easy, you use a connection method and pass in some basic information about ports, ip address, username and key files.  To execute a command, you can then just execute exec_command on the instance and it provides 3 file like objects: stdin, stdout and stderr with typical file reading operations.  Great for setting up single remote connections.

http://www.paramiko.org

sshtunnel – A library for creating ssh tunnels quickly.  Provides a class just for port forwarding.  Worth checking out if you do this occasionally as it has a quick hand solution.  Great for forwarding information, like a database connection.

https://sshtunnel.readthedocs.io/en/latest/

sockets – A library for doing socket manipulation.  Lower level than the other protocols.  You have to do things like send/receive from a given socket.  Provides a bunch of different protocols you can use.  More extensible.  Great for dealing with lower level problems or making network more customizable.

https://docs.python.org/2/library/socket.html

socket server – A library that allows you to create a socket server that handles networking events via a handler.  A series of mixins and servers are available, including ones that make the handler asynchronous.  I used this to implement my own version of port forwarding service.  Great for setting up quick server to do connections.

https://docs.python.org/2/library/socketserver.html

Conch, twisted framework – twisted is a asynchronous network framework in python.  It’s pretty cool project and is similar to Tornado.  Twisted has a client called conch that allows you to handle ssh traffic.  Cool project, but only been through the tutorials.  I’m a big fan of the project, but haven’t done that much with it.

http://twistedmatrix.com/documents/current/conch/howto/conch_client.html

 

 

Getting Hands Dirty…

After about 6 months or so reading up 2-3 books on Django and going through the tutorials, I’m finally at the point where I comfortably can pursue a website with confidence.  Over the last two days, I’ve been setting up a small Amazon instance for demos.  Doing all the server, network and security configuration was awesome as I’ve learned most of that in the last 6 months +.  I’ve got nginx, postgres and django set up, tonight I form the first url and templates.  Exciting!  Once I have a domain set up and production web pages, I’ll post a url on my blog this blog and people can test out the application (nature app).

Machine Learning in Action: Part 1

 

eigenvector14b

I’m interested in learning more about computer programming.  Recently, I’ve picked up a book on algorithms and data structures as well as looked into Greenplum and Postgres.  I wanted to have a slight change of focus this weekend and picked up Machine Learning in Action over the weekend.

The book has been great so far.  It’s written using Python and implements many common machine learning algorithms from scratch.  Currently, I’ve gone through 2 chapters, one on KNN and the other on ID3 trees.  The later was a bit more challenging then the first, requiring quiet a bit of recursion due to the tree structure involved with that methodology.  I like this book so far in that it does a lot of the implementations from scratch, which makes it easier to understand.  I still want to get deeper into Shannon entropy and that up to get a better understanding of it.

For those interested in the code behind the book, it can be found here:

https://github.com/pbharrin/machinelearninginaction

Hopefully, I will get to try out the next few chapters.  Chapter 4 covers Bayesian methodology.

Best,

Chris