Pyplot in Jupyter inside pyenv on El Capitan

Jan 25, 2016

With the new root protection feature in El Capitan, it’s hard to install pip packages into the system python. Besides, it’s generally considered bad practice to clutter up your system python’s library with packages. Most developers prefer to set up their own virtualenv for python and then install packages into that local environment. I did the same for setting up Jupyter Notebook and everything worked as expected. Well, mostly everything.

When I tried to import matplotlib.pyplot, I got a rather long error:

RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see ‘Working with Matplotlib in Virtual environments’ in the Matplotlib FAQ

So it seems that pyplot doesn’t work well with python inside a virtualenv, at least on OS X. The FAQ describes a way to work around this by adding a script to your virtualenv and then invoking that script as your interpreter instead of invoking python directly. That’s half the problem solved, the other half being getting IPython to use this script. A little googling provided a solution to that as well.

So here’s the combined solution, including how to create and use the virtualenv.

Install required packages

# Install pyenv

brew install pyenv

# A convenient way to create virtualenvs
brew install pyenv-virtualenv

# Install the desired version of python
pyenv install 2.7.10

# Create a virtualenv, I'm calling mine "jupyter"
pyenv virtualenv 2.7.10 jupyter

# Activate the virtualenv
# You'll have to do this everytime before running "jupyter notebook"
pyenv activate jupyter

# Install all the packages your heart desires
pip install notebook matplotlib numpy scipy sklearn

Create the wrapper around python

Find the path to your virtualenv’s python binary:

# Only run this if you're not already in the jupyter virtualenv
pyenv activate jupyter

# Where's python?
pyenv which python

This will give you a path that looks like /Users/amey/.pyenv/versions/jupyter/bin/python. Create a new file named frameworkpython (or whatever you want, this is the name suggested by the matplotlib FAQ) and put the following inside:

#!/bin/bash

# what real Python executable to use
PYVER=2.7
PATHTOPYTHON=/usr/bin/
PYTHON=${PATHTOPYTHON}python${PYVER}

# find the root of the virtualenv, it should be the parent of the dir this script is in
ENV=`$PYTHON -c "import os; print os.path.abspath(os.path.join(os.path.dirname(\"$0\"), '..'))"`

# now run Python with the virtualenv set as Python's HOME
export PYTHONHOME=$ENV
exec $PYTHON "$@"

Remember to use the correct version in the line PYVER=2.7 and the correct path to your system’s python in the line PATHTOPYTHON=/usr/bin/. The system python directory can usually be found by running which python in a new shell session. If you get a path to pyenv (like /Users/amey/.pyenv/shims/python), you’ll have to go hunting. It should be at /usr/bin/python or /usr/local/bin/python.

Mark the new file as executable with chmod +x /path/to/frameworkpython, and that’s the wrapper taken care of. Next we need to create a new kernel for jupyter and tell it to use the newly-created frameworkpython as the interpreter. For this, we need to find out where jupyter is reading its configuration files from. Run jupyter --paths inside the virtualenv, you should get output similar to the following:

config:
    /Users/amey/.jupyter
    /Users/amey/.pyenv/versions/jupyter/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /Users/amey/Library/Jupyter
    /Users/amey/.pyenv/versions/jupyter/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
/Users/amey/Library/Jupyter/runtime

The first data directory is usually the one we’d like to use. So create a directory there for the new kernel. I decided to call my new kernel matplotlib to make it easy to identify, and so I ran

mkdir -p ~/Library/Jupyter/kernels/matplotlib

And then create a configuration file named kernel.json for the new kernel inside the directory you just created. For me, this was ~/Library/Jupyter/kernels/matplotlib/kernel.json. Paste the following inside this file:

{
 "argv": [ "/Users/amey/.pyenv/versions/jupyter/bin/frameworkpython", "-m", "ipykernel",
          "-f", "{connection_file}"],
 "display_name": "matplotlib",
 "language": "python"
}

Tweak the path inside argv and the display_name variable according to where your wrapper is and the name of your new kernel respectively. And that should do it. When you activate the virtualenv and launch jupyter, you should see an option to create a notebook with your new kernel and Matplotlib and Pyplot should work fine inside it if everything went according to plan.

Tunnel vision

Oct 25, 2015

For the past few years I’ve been looking forward to an extended stay in the States. I had a long list of things I’d do, people I’d meet, places I’d see, dishes I’d try, etc.

I’ve been here for about six weeks now, and the only thing (well, one of two things) that’s consistently on my mind is.. the next quiz. The next homework. The assignment that’s due three weeks from now. The homework that I did poorly in. The career fair I didn’t go for. The https://www.google.com/foobar tab that’s been sitting open in my browser for two weeks now. Internships I need to apply for (and land) before December.

It’s easy to get tunnel vision. While all of the above are undoubtedly important, I’m falling into the same trap I fell in a few years ago. Instead of living my life, I’m worrying about everything else so that I can live a better life sometime in the future. And when that future gets here, I’ll still be busy making my future life better. Hence I’m making this post to remind myself: while worrying about tomorrow, don’t forget about today.

First week in the states

Sep 21, 2015

Time to pull back the veil a bit. I’ve been absent for the past few months because I couldn’t write about what I was up to. And that’s because I was busy applying for college, getting admitted and then working out the logistics of paying fees, finding a place to stay and moving to a whole new country.

Now that all that is done and I’m sitting in a lounge on campus, I can write about it. I’ve enrolled for a Masters in Computer Science at UC San Diego. I reached San Diego on the 10th, and it’s been a fun time so far with lots of ups and downs.

One thing’s for sure, it’s really hard to get around. I have a bus pass sticker on my student ID so that helps a bit, but the bus routes don’t seem to lead where I need to go and the only bus stop close to my place is about a mile away. The closest grocery stores are also a few miles away, so getting a car seems like the best thing to do. But maybe I should be more realistic and just get a bike instead.

I definitely like the city though. It’s a lot like Sunnyvale (which I also liked) in terms of how there are very few tall buildings and everything is spread apart with lots of space. Can’t complain about the weather, although it’s been on the hotter side the past few days.

Right next to the campus
No tall buildings!

Orientation week is almost over. Today’s the last day of orientation where they perform a documentation check to make sure everything’s in order. Classes start on the 24th.

Somewhere I belong

May 12, 2015

In my last post I talked about how members of a community are expected to follow some basic rules in order to make that community thrive. And that’s one of the requirements for a successful community. And what do members of a community get out of said membership? A sense of belonging. Humans are social animals by design. It is in our nature to thrive in a group of like-minded individuals rather than in a solitary setting. And while we’re surrounded by people, we subconsciously seek their approval and confirmation. Studies have shown that individuals are more likely to answer a question a particular way when informed that their neighbors have picked that particular answer. And while some individuals can thrive under solitary conditions, most people when left alone quickly yearn for company, any company. Which is why solitary confinement is the most severe form of punishment for misbehaviour in (most) prisons.

I haven’t truly felt a sense of belonging anywhere since primary school. I was a popular kid back then and had lots of friends. Then my parents made me change schools to another one. And that school was supposed to be the continuation-school of a different primary school, and they had a policy by which they would never accept any students who hadn’t attended that particular primary school. But my parents obstinately refused to back down, and I suppose my academic record also helped convince the school officials to grant me admission. And so it came to be that I was literally the only new kid in a class composed of guys who’d known one another for four years. And it didn’t help that I did consistently better than their poster boys and supplanted them at the top of the class. A friend of mine told me in confidence after we’d graduated that even my handful of friends didn’t actually like me and found me callous and arrogant.

When the time came for me to go to college, I’d topped the entrance exam to BITS Pilani (not tooting my own horn here, I did horribly in the IIT entrance) and wanted to study Computer Science. But my parents were convinced that computers were just a “bubble” (hah!) and signed me up for electronics instead, a subject I wasn’t interested in at all. So I spent four years hanging out with the electronics crowd, studying topics that put me to sleep and surrounded by people talking about things I couldn’t understand. The only glimpses I got of the people I wanted to be with were during the optional computer science courses I took.

After college, I’ve been employed twice; once at Infinera and currently at Amazon. There are no reasons why I shouldn’t feel a sense of belonging now, but maybe it’s simply some baggage that I’ve carried over from the earlier years. That’s my second-greatest fear at the moment: that I’ll never find a place where I belong. And my greatest fear? That my life won’t amount to anything meaningful.

Black and White

May 11, 2015

There was a time - back in college - when I could see the world in black and white. Things people did could be unilaterally classified as “good” or “bad, and anyone doing even a single bad thing was equivocally bad. A friend of mine rebuked me for taking such a hard stance; she insisted that one could not look at the world in monochrome, that things only exist in shades of grey. She had close to her several people I believed were bad, and I could not reconcile that with the kind of person she was. To me, she was good beyond a doubt, and I could not understand how she could be friends with people who had done some terrible things. And I kid you not, they did do some very bad things like bribing college professors, embezzling money from the students’ mess and beating up their competitors in the presidential elections; but that’s beside the point.

A lot of atheists don’t see the point to religion. They believe that people who believe in a higher power are dumb and weak-willed, that giving up control of your life to something whose existence cannot be proved is short-sighted and just plain idiotic. I disagree with this point of view. I believe that since its inception religion has served a very clear and obvious purpose: it helps people differentiate between right and wrong. Now, one might argue that it’s really not that hard to know that theft and murder are wrong, and that one doesn’t need religion to teach people that. While that might be true for the extremes, it’s the shades of grey that tend to trip people up. Stealing is bad, yes; but is stealing food to survive wrong? Lying is bad, true; but lying to save someone heartache? To spare someone’s life? I tend to look at religion as a set of moral guidelines that exists to guide people onto a path that will allow them to coexist peacefully in a community and to live full lives. For most people, that is enough; they pray to their God(s) and they live their lives following the code laid down by their religion, expecting that at the end of their journey they will be justly rewarded with a seat in heaven. For people who are inherently evil in nature but are weak-willed in their convictions, religion wields the stick rather than the carrot, promising wrongdoers an eternity of torment and suffering in hell. These two facets to religion serve to control probably 3/4ths of the religious populace, leaving the rest to instead spend their lives trying to dodge the long arm of the law.

But what of atheists? We, who do not believe in God or a higher power, what of us? Are we inherently immoral?

My take - and this is solely my own uninformed opinion - is that people who are inquisitive enough to question the blind faith other people have towards God, are naturally also inquisitive enough to take a close look at what makes society work. Any functional society is founded on the belief that its members will find a way to coexist amicably and with a minimum of discord. The law can only do so much when it comes to maintaining the peace; in the end, it is the responsibility of each and every member of society to do their own part in upholding the law. Having realized this, any sane atheist would then also recognize that anyone who is not bound by either a yearning for heaven or a fear of hell needs some sort of a moral compass to help guide their actions. And finally, my hope is that these individuals will arrive at a set of guidelines that are aligned closely with the overall commandments of any non-fanatic religion. Because following the law isn’t enough for a well-functioning community. There’s no law that tells you to be nice to your neighbour and to lock their front door if you see them leave the house in hurry with the door open. There’s no law that requires you to help cover for a colleague because she has to pop out in the middle of the day to pick up her kids from school. And there’s no law commanding you to call up a friend to enquire about their health when they take a sick day. These are things that are usually learnt from religions - well, technically from your parents, but the roots can be traced back to religions - and atheists usually end up at a similar-enough set of principles that they live by. They decide where to draw the line between good and bad, and they decide which side of the line they wish to stand on.

And just as the position of this line on the X-axis varies across religions, so does it vary across atheists. Some people might consider as acceptable the things that others look down upon as being bad. And even for the same individual, the line might move a bit based on the time of day or the cycle of the moon. This is a lesson I learnt slightly too late in my life. Who knows how my life might have turned out if I’d been able to see my friend’s point of view back when it actually mattered.