Using Protobuf With PyPy

In case you're not familiar with PyPy yet, it's a Python interpreter with JIT compiler.

PyPy has many advantages. It can run Python code faster then any existing interpreter, has a lower memory footprint then CPython, it allows you to choose the right level of language abstraction for your code depending on how fast you need it to be since it supports running Python, RPython and allows you to provide C and C++ extensions and It's 99% compatible with CPython.

One of the major differences and what makes PyPy adoption a non-trivial process is the why PyPy interfaces with CPython extensions. If you're interfacing with C code you need to ensure that you use the right bindings.

CPython extensions use Python's C API which PyPy does not have. PyPy can run some CPython extensions using cpyext but those extensions will run slower on PyPy due to the need to emulate reference counting. That also means that Cython generated extensions will be slower.

PyPy on the other hand has support for the not very commonly used ctypes and CFFI which can be run on both interpreters. CFFI runs much faster on PyPy than ctypes so if you're targeting PyPy it's better to use it. I was told that CFFI has been known to be a bit slower than C extensions on CPython because it does not access the C API directly but it wraps it. I could not find evidence for that claim although it makes sense.

If you're interfacing with C++ code, there has been a discussion to port boost::python to PyPy but I don't think it has been done yet.

PyPy provides cppyy which is written in CPython and thus able to JIT out most of the binding related code. cppyy has multiple backends it can use. In this post I'm going to cover Reflex since it's the default and it is currently the most stable backend.

In this post I'm going to demonstrate how to run Protobuf 2.5.1 on PyPy. Protobuf's compiler generates Python code which relies on a CPython extension in order to interface with the Protobuf implementation and is I mentioned before this is not going to be as fast as it should be for PyPy. Google provides a pure python implementation of Protobuf in version 3.0 which is not yet release and breaks compatibility in some aspects from Protobuf 2.x.

Software DevelopmentOmer KatzSeptember 11, 2015Python, PyPy, Protobuf, Reflex, cppyy, C++

Git Hooks (Part II) – Implementing Git hooks using Python

This post describes how to implement Git hooks using Python. If you have some commit hooks in Python or any other languages please do share them in the comments as Gists or repositories.

Git, SCMOmer KatzSeptember 22, 2014git, vcs, hooks, Python, software configuration management, source control

Measuring Coverage Accurately with Coveralls.io

If you've ever developed an open source project on Github you probably have heard about coveralls.io already but you, like me until recently are probably not measuring code coverage accurately enough if you have multiple testsuites (e.g.. unit, functional, integration).

How did I reach this conclusion? Keep reading.

Continuous Integretion, SCMOmer KatzOctober 13, 2013Travis CI, Coveralls

I am Now a nose2 Core Commiter!

General AnnouncementsOmer KatzAugust 11, 2013

More Information About Steps To Install The New Setuptools

Packaging, SCMOmer KatzJune 25, 2013Distribute, Setuptools, Python

Enforce Plugin Dependency Requirements Using Vagrant Plugin Bundler

We like package managers. They help us maintain our dependencies easily.

Vagrant has a package manager for it's plugins which make them very easy to install, upgrade or remove but before Vagrant Plugin Bundler we had no way to ensure our Vagrantfile is using the right version of each plugin.

With Vagrant Plugin Bundler you can do so very easily.

Packaging, Virtualization, SCMOmer KatzJune 19, 2013Vagrant, Vagrant Plugins

Distribute Is Now Merged Back To Setuptools

As of Setuptools 0.7 Distribute is now merged back into Setuptools.

This means that you should remove Distribute if you have it installed already and you probably do.

You can find the instructions to do so here but I have encountered some problems during the process so I'll share my solution here.

PackagingOmer KatzJune 19, 2013Python, Distribute, Setuptools

Travis-CI Does Not Work When You Package Vagrant Boxes As A Build Step (Yet)

I'm learning how to use Salt Stack so I started a new project that provisions a baseline Vagrant Box for Python development.

I wanted to check that my vagrant box can be provisioned & packaged every single time for every single change I committed into the repository so I created a .travis.yml configuration file in order to do so.

That attempt has failed unfortunately. Read this post to understand why.

Virtualization, Continuous IntegretionOmer KatzJune 15, 2013Vagrant, Travis CI

Vagrant Providers - VMWare Support Out Of The Box

Since version 1.1.0 Vagrant supports multiple virtualization solutions through a new concept called providers. They are easy to install and use.

The first paid providers that were developed are for VMWare Fusion & VMWare Workstation. There are other providers for the cloud.

VirtualizationOmer KatzMay 29, 2013Vagrant, VMWare, Base Box, Cloud, Provisioning

A customized shell for Git

If you are working with Git heavily like me you probably want a customized shell.

It lets you know exactly where you stand without typing anything.

It saves you time typing git branch to know what is the correct branch. It let's you know how many files are uncommitted/added, what revision are you looking at and much more.

If it saves you time you should probably use it. Because if it does it allows you to focus on what's important - your product.

Omer KatzMay 4, 2013git, git-sh, Posh-Git, customized shells, tips, timesavers

Git Hooks (Part I) - The Basics

Hooks are scripts that help you enforce policies on your repository by triggering scripts at certain events that enforce policies such as 'The commit message must be longer than 20 characters' or 'The test suite must be run before a commit and if it fails, reject the commit'.

Hooks are available in every VCS I have used before. They are also available in Visual Sourcesafe which is not a VCS.

Git, Source ControlOmer KatzFebruary 15, 2013git, hooks, scm, software configuration management, source control, vcs

A Vagrant Base Box For Django Development

I stumbled into this project which is a project template that provides some integration with Vagrant. It also provides a base box that can be customized for your needs.

VirtualizationOmer KatzFebruary 8, 2013Django, Vagrant, Base Box

Finally, a package manager for windows

I have finally found a good package manager for windows and installed node.js pretty easily with it. Add it to your tool chain.

It's called Chocolatey.

SCMOmer KatzJuly 12, 2012Package Manager, Windows, Chocolatey

Vagrant & Puppet Best Practices

During my usage of Vagrant I've discovered some best practices that will help you develop puppet projects more efficiently.

Here are 4 best practices to act upon when using Vagrant.

VirtualizationOmer KatzJuly 12, 2012Vagrant, Puppet, Librarian-Puppet, Best Practices, Provisioning

Setting Up A Development Workstation (Part II) - Scripting The Installation

Last time we talked about selecting and installing the OS. I have referred you to resources about preseeding and Kickstart in order for you to have a better grasp about unattended installations. You've probably attempted (and even succeeded) to perform one already. In this post I am going to further elaborate on how to improve the installation and configuration process of a new development machine.

Setting Up A Development Workstation (Part I) - Selecting and Installing the OS

Whenever a new developer arrives to the team he and/or the IT department usually spends some time on setting up their development workstation. This time is wasted as the man-hours spent on setting up a new workstation could have been spent on creating a better product (aka adding value).