I broke down and jailbroke my phone last night. Partially it was just to try it, but also because I was getting sick of the subpar 3rd party apps that were inundating the App Store. After following the instructions via Lifehacker to install Cydia, I was able to install OpenSSH as well as other cool things, like Python.

Then I saw that you could install iPython on the iPhone so I thought, let’s try it.

So hard was it? With the python package installed it was

easy_install ipython

Seriously.

To lay it out in terms of steps…

1. Install Cydia (The only caveat here is that I got a different SHASUM when I checked the pwnage tool from the macgeekblog site, I then redownloaded from the pwnage mirrors)
2. Follow the instructions to get openSSH up and running.
3. Go into Cydia and under “sections” got to “scripting”. There they have Python (among others).
4. I also installed a terminal
5. Now you can either go in through the terminal on the iPhone(touch) or SSH in from a differnet computer. Either way, su to root and then you can
6. easy_install ipython

Next of course would be to install Numpy and do folding at home (I’m kidding!), but this just shows some serious possibilities.

Did I also mention that I installed the NES frontend which can use all the public domain ROMs that are out there? Someone mentioned ROM world and The Old Computer but I haven’t checked them out yet.

Cool stuff.

Tags: ipodcomputers iphone itouch pwnage

It was tougher than I thought, and while I have seen these types of plots quite a bit in the fisheries world, there didn’t seem to be any examples of how to do this. So I thought I would post up here what I did, both for myself in the future and for anyone else in the world who may be interested in this type of plot.

Basically what I want to do is to make a dummy plot with a pie chart centered at every 5×5 degree box with a different size outer circle based on the total number in the box, and wedges representing percentages of that total amount.

For this exercise I am using the PXSY routine of GMT4.2.0 with my default MEASURE_UNIT = in.

To make this chart you have to have 5 columns of data:

Longitude — Latitude — Radius — StartAngle — EndAngle

So say I want a pie chart to represent how many types of widgets I sold in the area from 160-155W, 18-23N, with each wedge a portion of the widgets and centered in the middle of the 5×5 box. Here’s the data:

#lon lat blueWidget greenWidget redWidget
-157.5 20.5 200 200 400

So let’s say I make a map where each inch = 1 degree. The largest that I want my pie wedge diameter is 1 inch, so I know that for this example (only one data point) I will make the radius 0.5. I also know that the total for this example point is 800, so I convert this into angles. I actually have to make 3 rows of data now since I have three widgets. I also converted to 0-360 degrees.

$>cat pienc.xy
#lon lat radius startAngle endAngle
202.5 20.5 0.5 0 90 #end angle is 360 * (200/800)
202.5 20.5 0.5 90 180 #start angle is row-1 end angle
202.5 20.5 0.5 180 360 #Finish circle

So, a nice shell script to plot this up with the output below:

$>cat pienc
#!/bin/ksh
psfile=pie.ps
psbasemap -Jm1 -R200/206/17/23 -Bf1a1g1/f1a1g1WeSn -X1.5 -Y4 -P -K > $psfile
pscoast -Jm -R -O -K -Di -G200/200/200 -W1/0/0/0 >> $psfile
psxy pienc.xy -Sw -Jm -R -O -K >> $psfile
echo “203 24 12 0 0 6 Pie Chart Example” | pstext -Jm -R -O -N >> $psfile

$>display pie.ps

Pie Wedge Plot no Color

So that’s all well and good, except it would be nice to have different colors for each wedge representing a different widget. To get color in there you have to give a new column of values that will be mapped to a color value in a color lookup table (a cptfile in GMT). This column must be in the third row and then the -C switch must be given in the psxy call.

$>cat pie.xy
#lon lat COLORVALUE radius startAngle endAngle
202.5 20.5 1 0.5 0 90 #end angle is 360 * (200/800)
202.5 20.5 2 0.5 90 180 #start angle is row-1 end angle
202.5 20.5 3 0.5 180 360 #Finish circle

And my cptfile:

$>cat pie.cpt
0 0 0 255 1.1 0 0 255
1.1 0 255 0 2.1 0 255 0
2.1 255 0 0 3.1 255 0 0

The adjusted script:

$>cat pie
#!/bin/ksh
psfile=pie.ps
psbasemap -Jm1 -R200/206/17/23 -Bf1a1g1/f1a1g1WeSn -X1.5 -Y4 -P -K > $psfile
pscoast -Jm -R -O -K -Di -G200/200/200 -W1/0/0/0 >> $psfile
psxy pie.xy -Sw -Jm -R -O -K -Cpie.cpt >> $psfile
echo “203 24 12 0 0 6 Pie Chart Example with Color” | pstext -Jm -R -O -N >> $psfile

$>display pie.ps

Pie Wedge plot with color

And that’s pretty much it. Now to go sell some more widgets.

Then the fun began. Not only did Parallels hang when I tried to start the virtual machine, I got the grey screen of death! “YOU MUST REBOOT YOUR COMPUTER NOW!” Crap.

Well, looking up the problem in the “Knowledge base” I found that “Errors occur when you try to install or update Parallels Desktop, create or open virtual machines, load the required drivers to the guest OS” The handy solution? Reboot. Repair disk permissions. Reinstall. Why? Because evidently “Working in Mac OS for long periods of time without restart may lead to some minor glitches to appear in the system as a whole.”

Yup, feels like XP!

Tags: computers, osx

# Load in required modules
sage: from scipy.io.netcdf import *
sage: from pylab import *
sage: from scipy.stats.stats import nanmean
sage: import datetime

sage: ind = where(raw<0)
sage: data[ind] = nan


And here was the first real bottleneck, as things just slowed to a crawl as python tried to find all the instances where the data was less than zero. This is something that is instantaneous in Matlab, and took over 30 seconds to go through 124*50*151 values. There must be a faster way to do this.

data2=data.copy()
#Take out monthly averages
sage: mclim = ones((50,151))
sage: for i in range(1,13):
….index = where(months==i)[0]
….mclim = nanmean(data[index,:,:])
….data2[index,:,:] = data[index,:,:] - mclim

sage: EOF = nan_to_num(anom) #push land back to zero
sage: [u,s,v] = linalg.svd(EOF)
sage: for i in range(0,ncycles):#build array so that we can project eigenvalues back onto timeseries
….s2[i,i] = s[i]
sage: amp = dot(s2.transpose(),u.transpose()) #get amplitude
sage: spatial = v[0:4,:]# pull out spatial fields
sage: ratios = pow(s,2)/sum(pow(s,2))*100 #get %variance explained for each mode
sage: temp = spatial[0,:]
sage: temp.shape = (ny,nx) #push back to original dims
sage: plot(amp)
sage: savefig(’amplitude.png’)
sage: imshow(flipud(temp))
sage: savefig(’spatial.png’)


Success!

I actually really felt positive about this whole example as I really learned a lot more. This also was probably too large of an array to test out (measure twice cut once!) but it’s what I was working with so I wanted a real world example. The more that I worked in sage the more comfortable I felt as well. The geographic projection issue is still there, as well as some indexing speed issues, but overall, I was really impressed with the Sage/SciPy/NumPy experience today. Overall I feel that more of a transition was made for me last night/today. Which was great timing as a co-worker actually called me and asked if I knew of any free replacements for Matlab…

Technorati Tags:
Computers, linux, matlab, programming, python, sage, scientific programming

# Load in required modules
sage: from scipy.io.netcdf import *
sage: from pylab import *
sage: from scipy.stats.stats import nanmean
sage: import datetime

sage: raw = ncfile.variables[varnames[3]][:,0:50,:] #I only want 50 records in Y
sage: data = raw.copy() #make a copy
sage: data.shape
(124, 50, 151)
sage: (ncycles, ny, nx) = data.shape
sage: ind = where(raw<0)
sage: data[ind] = nan


And here was the first real bottleneck, as things just slowed to a crawl as python tried to find all the instances where the data was less than zero. This is something that is instantaneous in Matlab, and took over 30 seconds to go through 124*50*151 values. There must be a faster way to do this.

data2=data.copy()
#Take out monthly averages
sage: mclim = ones((80,151))
sage: for i in range(1,13):
….index = where(months==i)[0]
….mclim = nanmean(data[index,:,:])
….data2[index,:,:] = data[index,:,:] - mclim

sage: EOF = nan_to_num(anom) #push land back to zero
sage: [u,s,v] = linalg.svd(EOF)
sage: for i in range(0,ncycles):#build array so that we can project eigenvalues back onto timeseries
….s2[i,i] = s[i]
sage: amp = dot(s2.transpose(),u.transpose()) #get amplitude
sage: spatial = v[0:4,:]# pull out spatial fields
sage: ratios = pow(s,2)/sum(pow(s,2))*100 #get %variance explained for each mode
sage: temp = spatial[0,:]
sage: temp.shape = (ny,nx) #push back to original dims
sage: plot(amp)
sage: savefig(’amplitude.png’)
sage: imshow(flipud(temp))
sage: savefig(’spatial.png’)


Success!

I actually really felt positive about this whole example as I really learned a lot more. This also was probably too large of an array to test out (measure twice cut once!) but it’s what I was working with so I wanted a real world example. The more that I worked in sage the more comfortable I felt as well. The geographic projection issue is still there, as well as some indexing speed issues, but overall, I was really impressed with the Sage/SciPy/NumPy experience today. Overall I feel that more of a transition was made for me last night/today. Which was great timing as a co-worker actually called me and asked if I knew of any free replacements for Matlab…

Technorati Tags:
Computers, linux, matlab, programming, python, sage, scientific programming

# Load in required modules
sage: from scipy.io.netcdf import *
sage: from pylab import *
sage: from scipy.stats.stats import nanmean
sage: import datetime

sage: raw = ncfile.variables[varnames[3]][:,0:50,:] #I only want 50 records in Y
sage: data = raw.copy() #make a copy
sage: data.shape
(124, 50, 151)
sage: (ncycles, ny, nx) = data.shape
sage: ind = where(raw<0)
sage: data[ind] = nan


And here was the first real bottleneck, as things just slowed to a crawl as python tried to find all the instances where the data was less than zero. This is something that is instantaneous in Matlab, and took over 30 seconds to go through 124*50*151 values. There must be a faster way to do this.

data2=data.copy()
#Take out monthly averages
sage: mclim = ones((80,151))
sage: for i in range(1,13):
….index = where(months==i)[0]
….mclim = nanmean(data[index,:,:])
….data2[index,:,:] = data[index,:,:] - mclim

sage: EOF = nan_to_num(anom) #push land back to zero
sage: [u,s,v] = linalg.svd(EOF)
sage: for i in range(0,ncycles):#build array so that we can project eigenvalues back onto timeseries
….s2[i,i] = s[i]
sage: amp = dot(s2.transpose(),u.transpose()) #get amplitude
sage: spatial = v[0:4,:]# pull out spatial fields
sage: ratios = pow(s,2)/sum(pow(s,2))*100 #get %variance explained for each mode
sage: temp = spatial[0,:]
sage: temp.shape = (ny,nx) #push back to original dims
sage: plot(amp)
sage: savefig(’amplitude.png’)
sage: imshow(flipud(temp))
sage: savefig(’spatial.png’)


Success!

I actually really felt positive about this whole example as I really learned a lot more. This also was probably too large of an array to test out (measure twice cut once!) but it’s what I was working with so I wanted a real world example. The more that I worked in sage the more comfortable I felt as well. The geographic projection issue is still there, as well as some indexing speed issues, but overall, I was really impressed with the Sage/SciPy/NumPy experience today. Overall I feel that more of a transition was made for me last night/today. Which was great timing as a co-worker actually called me and asked if I knew of any free replacements for Matlab…

Technorati Tags:
Computers, linux, matlab, programming, python, sage, scientific programming

Anywho, I decided to continue trying to map the data from the netcdf file onto a projection, and here’s what I ran into.

It looks like the basemap module is installed (as basemap) but that it depends on matplotlib > 0.98 and 0.91 is installed. I tried to be tricky and move my locally installed matplotlib over to the sage/local/lib/python2.5/site-packages directory but then that version of matplotlib needed a newer version of numpy than what was installed. At this point I tried

hostname $> sage -upgrade

to see if updated packages/modules were available. This started a huge chain reaction of downloads and source compiling to get to the latest, greatest versions. This process took exactly 59m10.482s to complete (I know because it told me!).

But once again, I get this error:

sage: from basemap import basemap

ImportError: your matplotlib is too old - basemap requires version 0.98 or higher, you have version 0.91.1

At this point though, it’s not working on either the linux or OSX platforms due to outdated dependencies, so either I need to find another way to plot mapped projections or use something else.

Again, this isn’t a knock against Sage, because I really don’t think that is an ideal test for this software. But honestly, a lot of why I went for this approach was to avoid having to use separate approaches for data manipulation and visualization, and this would be a common task. Matlab’s mapping toolbox is useless to me for plotting, so I end up using m_map, which is still not as good as GMT, but it gets the job done in house.

My main thoughts at this point are that it seems easy to get into dependency hell here, as one module upgrade can force another, and so on. At this point it’s another block of time spent on setup, and no result. Time to stop for the time being.

Technorati Tags:
Computers, linux, Mac OS X, programming, python, R, sage, scientific programming

So for me, I really just wanted to be able to do a few different examples which would be close to “real world applications” for me.

Some things that I would like to be able to do in sage:

1. Load in a 2-D NetCDF satellite data file and display it as a map projection. This should be really simple. I would usually just use GMT for this (a small shell script wrapping psbasemap, grdimage, and pscoast).

2. Load in a data series with dates and locations, and match this to corresponding satellite data in time and space. Normally I would use a perl script that I wrote many moons ago to do this. I would basically sort the data, then match a block of data at a time using GMT’s grdtrack function. I know that this is inefficient, and really I would like to be able to pull extra data in x,y, or t and take the mean or median value, which would be more CPU intensive, but better than matching just one point in space and time to the nearest pixel.

3. Load in a multivariate data series and do multivariate statistics (e.g. LME, GLM/GAM, RDA). This is where the R interface would come into play. Normally I would prepare the data elsewhere, then import the flat table into R and use the R functions. This may involve installing more packages (nlme, mgcv, etc).

4. Load in a 3-D set (x,y,t) of satellite data files and perform an EOF analysis on them (akin to SVD in Matlab). Normally I would do this in Matlab or Ferret. I’m just curious how easy it would be to do this here.

There are other things that I could do, but these are a few off the top of my head, and things that I am doing now, so it would be incentive to try Sage out with. For tonight, I’ll just work on #1, which should be really fast.

The data file I’m using is just a NetCDF file (created by GMT) which I can read with pupynere in python. Here I’m going to use the scipy.io.netcdf module (which is actually based on pupynere I believe).

sage: from scipy.io.netcdf import *
sage: from pylab import *

# just plot sst to test 2D image plotting
sage: plot(sst)
[<matplotlib.AxesImage instance at 0xc03636c>]


Nice, but it’s upside down. Let’s flip it vertically.


sage: clf
sage: plot(flipud(sst))
[<matplotlib.AxesImage instance at 0xb86a2ac>]
sage: savefig(’temp.png’)


Easy, but I want to put this on a projection. Normally I would use the basemap tools which are an add on to matplotlib. I don’t see these installed, and I didn’t see them in the extra sage packages on line, so I downloaded them from SourceForge and installed them.

The first step you have to do is to install the geos package, just read the README in the geos folder and hit

./configure
make

and then we get our first epic fail. Something in the geos chain won’t compile, and I’m just about fried enough to call it quits for this evening.

At this point I’ve been playing with this for more than 2 hours, and I still have yet to make a simple map on a projection. There has to be something I’m missing, but at this point I’m going to pause until tomorrow. So not the best testing evening, but there are some positives so far. The bundling of most packages is a plus, and the ease of loading in NetCDF files is nice. Data displays well using the Pylab interface, even though I am still forced to save to a file at this point.

So immediate goals:

1. Get a backend working for viewing plots in widgets (akin to ipython -pylab)

2. Get the basemap tools installed so that I can make a map with a projection!

Technorati Tags:
Computers, linux, Mac OS X, python, R, scientific programming, sage

As Sage is “a free mathematics software system which combines the power of many existing open-source packages into a common Python-based interface”, that means that there are many libraries bundled together. These include ipython, numpy, scipy, R, RPy, and more. While I already had many of these installed on my Macbook, rather than look for a solution to cobble things together to save space, I just took the easy route and downloaded the .dmg file to keep everything together in one environment. I’m not sure if it’s possible (or even wise) to use existing components (probably not) but I figured that it would be more hassle than it was worth.

The .dmg file is fairly large (~256 MB), but smaller than the last Matlab release, which can put things in perspective. Download also only took about 6 minutes over wireless (>500 KB/sec). Installation was as simple as dragging the sage folder from the mounted .dmg file into my Applications folder. Expanded the folder was pretty hefty (~780 MB), which was larger than the last Matlab release (~680 MB), but not by much.

The next step in the installation instructed me to double click the sage icon (inside the Sage folder), and then change some preferences. When I double clicked it however a terminal fired up and I sage started to work

[user@localbox ~]$ /Applications/sage/sage ; exit;
———————————————————————-
| SAGE Version 3.0.2, Release Date: 2008-05-24 |
| Type notebook() for the GUI, and license() for information. |
———————————————————————-
The SAGE install tree may have moved.
Regenerating Python.pyo and .pyc files that hardcode the install PATH (please wait at most a few minutes)…
Please do not interrupt this.

Setting permissions of DOT_SAGE directory so only you can read and write it.

After this step I was presented with the sage prompt. Typing in

sage: notebook()

prompted me to create an admin password, and then opened the sage notebook GUI in my default web browser. I closed this down and then went back to the command prompt to start to play with the Tutorial.

I’m going to stop here for now, because it’s late, but I’ll post more after playing around with sage a bit. I’ve been through the tutorial a bit, and plotting works, yet on the surface there seems to be an inability to view plots “on the fly” with matplotlib, instead having to save and view with an external viewer. Just an extra step, but I as a it used to having graphics on the fly with ipython -pylab. As I said though, I just wanted to talk about installation. Once I give it a fair shake I’ll post up what I think so far.

Technorati Tags:
Computers, Mac OS X, programming, python, scientific programming

I couldn’t pass up a good deal on a slightly used 8 GB iPod touch. I mean, it still had the plastic sticky cover on it!

One thing that bugged me was that they seem to sell them now in the store with the January update included, yet this one didn’t have it. And I can’t seem to find the upgrade anywhere. I’m too much of a scaredy-cat to jailbreak it just yet, but if the 2.0 update doesn’t come out soon I’m going to be sorely tempted!

Update 7/17/08

Well, I was able to download the 2.0 firmware upgrade last Friday night, and I feel that it was well worth the $10 charge. It’s pretty much a whole new level, and while the quality of applications from the app store may vary, I think that this was a real step in the right direction. Too early to really review specific apps, but I know that within the last week I have been using Remote, NetNewsWire, Evernote, Twitterific, SFNetNews, Aurora Feint, WritingPad, and Moonlight Mahjong Lite. I just started playing with MochaVNC Lite as well and there is definite potential. I was able to access computers running Ultra VNC on both XP and Vista.