Using Protobuf With PyPy

In case you're not familiar with PyPy yet, it's a Python interpreter with JIT compiler.

PyPy has many advantages. It can run Python code faster then any existing interpreter, has a lower memory footprint then CPython, it allows you to choose the right level of language abstraction for your code depending on how fast you need it to be since it supports running Python, RPython and allows you to provide C and C++ extensions and It's 99% compatible with CPython.

One of the major differences and what makes PyPy adoption a non-trivial process is the why PyPy interfaces with CPython extensions. If you're interfacing with C code you need to ensure that you use the right bindings.

CPython extensions use Python's C API which PyPy does not have. PyPy can run some CPython extensions using cpyext but those extensions will run slower on PyPy due to the need to emulate reference counting. That also means that Cython generated extensions will be slower.

PyPy on the other hand has support for the not very commonly used ctypes and CFFI which can be run on both interpreters. CFFI runs much faster on PyPy than ctypes so if you're targeting PyPy it's better to use it. I was told that CFFI has been known to be a bit slower than C extensions on CPython because it does not access the C API directly but it wraps it. I could not find evidence for that claim although it makes sense.

If you're interfacing with C++ code, there has been a discussion to port boost::python to PyPy but I don't think it has been done yet.

PyPy provides cppyy which is written in CPython and thus able to JIT out most of the binding related code. cppyy has multiple backends it can use. In this post I'm going to cover Reflex since it's the default and it is currently the most stable backend.

In this post I'm going to demonstrate how to run Protobuf 2.5.1 on PyPy. Protobuf's compiler generates Python code which relies on a CPython extension in order to interface with the Protobuf implementation and is I mentioned before this is not going to be as fast as it should be for PyPy. Google provides a pure python implementation of Protobuf in version 3.0 which is not yet release and breaks compatibility in some aspects from Protobuf 2.x.

Read More
/

Git Hooks (Part II) – Implementing Git hooks using Python

This post describes how to implement Git hooks using Python. If you have some commit hooks in Python or any other languages please do share them in the comments as Gists or repositories.

Read More
/

Measuring Coverage Accurately with Coveralls.io

If you've ever developed an open source project on Github you probably have heard about coveralls.io already but you, like me until recently are probably not measuring code coverage accurately enough if you have multiple testsuites (e.g.. unit, functional, integration).

How did I reach this conclusion? Keep reading.

Read More
/

I am Now a nose2 Core Commiter!

I am now part of the nose2 core development team. nose2 is the next generation of nicer testing for Python, based on the plugins branch of unittest2.

/

More Information About Steps To Install The New Setuptools

I had some problems with packages that still depend on Distribute directly. I solved them by installing Distribute 0.7.x compatibility layer.

I updated the original post with the information on how to do so.

You can find it here

/

Enforce Plugin Dependency Requirements Using Vagrant Plugin Bundler

 

We like package managers. They help us maintain our dependencies easily.

Vagrant has a package manager for it's plugins which make them very easy to install, upgrade or remove but before Vagrant Plugin Bundler we had no way to ensure our Vagrantfile is using the right version of each plugin.

With Vagrant Plugin Bundler you can do so very easily. 

Read More
/

Distribute Is Now Merged Back To Setuptools

As of Setuptools 0.7 Distribute is now merged back into Setuptools.

This means that you should remove Distribute if you have it installed already and you probably do. 

You can find the instructions to do so here but I have encountered some problems during the process so I'll share my solution here.

Read More
/ /Source

Travis-CI Does Not Work When You Package Vagrant Boxes As A Build Step (Yet)

I'm learning how to use Salt Stack so I started a new project that provisions a baseline Vagrant Box for Python development.

I wanted to check that my vagrant box can be provisioned & packaged every single time for every single change I committed into the repository so I created a .travis.yml configuration file in order to do so.

That attempt has failed unfortunately.  Read this post to understand why.

Read More
/

Vagrant Providers - VMWare Support Out Of The Box

Since version 1.1.0 Vagrant supports multiple virtualization solutions through a new concept called providers. They are easy to install and use.

The first paid providers that were developed are for VMWare Fusion & VMWare Workstation.  There are other providers for the cloud. 

Read More
/

A customized shell for Git

If you are working with Git heavily like me you probably want a customized shell.

​It lets you know exactly where you stand without typing anything.

It saves you time typing git branch to know what is the correct branch. It let's you know how many files are uncommitted/added, what revision are you looking at and much more.​

If it saves you time you should probably use it. Because if it does it allows you to focus on what's important - your product.

Read More
/

Git Hooks (Part I) - The Basics

Hooks are scripts that help you enforce policies on your repository by triggering scripts at certain events that enforce policies such as 'The commit message must be longer than 20 characters' or 'The test suite must be run before a commit and if it fails, reject the commit'.

Hooks are available in every VCS I have used before. They are also available in Visual Sourcesafe which is not a VCS.

Read More
/

A Vagrant Base Box For Django Development

I stumbled into this project which is a project template that provides some integration with Vagrant. It also provides a base box that can be customized for your needs.

Read More
/

Finally, a package manager for windows

I have finally found a good package manager for windows and installed node.js pretty easily with it. Add it to your tool chain.

It's called Chocolatey.

Read More
/

Vagrant & Puppet Best Practices

During my usage of Vagrant I've discovered some best practices that will help you develop puppet projects more efficiently.

Here are 4 best practices to act upon when using Vagrant.

Read More
/

Setting Up A Development Workstation (Part II) - Scripting The Installation

Last time we talked about selecting and installing the OS. I have referred you to resources about preseeding and Kickstart in order for you to have a better grasp about unattended installations. You've probably attempted (and even succeeded) to perform one already. In this post I am going to further elaborate on how to improve the installation and configuration process of a new development machine.

Read More
/

Setting Up A Development Workstation (Part I) - Selecting and Installing the OS

Whenever a new developer arrives to the team he and/or the IT department usually spends some time on setting up their development workstation. This time is wasted as the man-hours spent on setting up a new workstation could have been spent on creating a better product (aka adding value).

Read More
/