Sujee’s blog

I just started tinkering with Hadoop - ‘(Java) distributed computing platform’ from Apache. Even though it is a pretty nice platform, I wasted lot of time, chasing trivial / silly issues. Here they are, so some one else might find them useful.

hadoop version : 0.16.3

A) The dreaded ‘Port Out of Range’ exception in ‘NameNode’

2008-04-20 19:27:34,241 ERROR org.apache.hadoop.dfs.NameNode: java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:118)
at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:65)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1180)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:53)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1191)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:148)
at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:122)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:94)
at org.apache.hadoop.fs.Trash.<init>(Trash.java:63)
at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:134)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:176)
at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:162)
at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:846)
at org.apache.hadoop.dfs.NameNode.main(NameNode.java:855)

2008-04-20 19:27:34,243 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9000: starting
2008-04-20 19:27:34,243 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG:
SHUTDOWN_MSG: Shutting down NameNode at sanfrancisco/127.0.1.1

This issue caused me so much aggravation. The issue is - my hostnames.

/etc/hosts

#hadoop
192.168.0.12 hadoop_master
192.168.0.2 hadoop_slave_1
hadoop-site.xml:

<property>
<name>fs.default.name</name>
<value>hadoop_master:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>hadoop_master:9001</value>
</property>
For some reason, hostnames with underscores ( ‘hadoop_master’ or ‘hadoop_slave_1′ ) are causing this weired error.

Solution: after replacing all instances of

hadoop_master —-> master

hadoop_slave_1 —> slave

Every thing just worked !!!

Or you could use IP-addresses as well.

B)java.io.IOException: Incompatible namespaceIDs

At least this issue and a workaround is some what easier to find than the previous one.

go to the end of this tutorial : http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)

http://issues.apache.org/jira/browse/HADOOP-1212

*Whew*

I have reverted back to hadoop v0.15.3, so the namespace issue went away
C) Map/Reduce tasks dying out

I kick off a simple wordcout across two machines, and my reduce tasks will die out, and process will hang for ever. Trawling through the logs revealed this is caused by some really weired reverse hostname lookups by Hadoop (or Java.net) framework.

For example, my IP address (192.168.0.10)  was resolved as ’somehost.comcast.net’.  After tweaking my ‘/etc/hosts’  things worked like a charm!

Ahh… why go to the bother of doing a reverse IP lookup?  Just use the IP address please..

These issues were trivial / silly, but time-consuming nonetheless.

Looks like Hadoop - the Elephant - can scale mountains, but slips on a banana peel

I have some  geocode data (GPS co-ordinates) for US Zips or Cities.

* Read the article *

Thunderbird is a great mail application, but this problem nagged me enough to warrant a post!

I run Kubuntu - KDE based flavor of Ubuntu. Thunderbird is a GNOME application and doesn’t seem to take the default BROWSER values set in KDE control panel or in my BASH profile (BROWSER=/usr/bin/firefox). I tried this - http://www.knoppix.net/forum/viewtopic.php?t=20315 - but no luck

Here is one way of fixing it:

1) Install gnome-control-center
sudo apt-get install gnome-control-center
or do it through Adept Manager UI.
this will install a bunch of Gnome libraries, that is okay

2) Once the install is done, launch ‘gnome-control-center‘ and goto Preferred Applications. Select FIREFOX as browser and choose ‘Open in Tab’

Here is a bash script I came up with to create playlists on Sansa music player, on Linux

** read more **

Fleet week is in San Francisco. And Blue Angles are back, roaring through crisp autumn skies, ’shock-and-awe’ing crowds, scaring dogs and ticking off some San Francisco Board of Supervisors (see here and here )

I love watching these precision flying machines and love photographing them even more! I went to the practice runs on Thursday and Friday [they practice for 2 days - Thu & Fri, and fly for the crowds during the weekend]. Here are some photos:

Checkout Fleet-Week Gallery for more

I have an article that talks about logistics of photographing Fleet Week in San Francisco, and detailing some of the best places to ’shoot’ Blue Angles, including GPS co-ordinates plotted on a Google Map.

Here are some pictures from 2005, around Fort Point of Golden Gate Bridge

And SFChronicle photographers produce some amazing photos of the event. See here and here.

Today San Francisco Bay Area was blanketed by smoke/haze from wild fires from Sierra. (news : link1, link2, link3). The sky was completely gray and dull. Predicting that the sunset would be extra color full, I went by Pacifica Pier hoping to take some photos. Here they are:

First 4 photos, show the fiery sun sinking into the ocean. Last two photos are about 15,20 minutes after sunset. Last image is my favorite.

Notes

• Time & Location : I go to shoot sunsets when there are high clouds or the sky is hazy. Clouds sprinkle colors on the sky, and the haze accentuates redness of the sunset.
  Pacifica is almost always fogged in during Summer months; so you won’t even see sunset. During Spring and Fall skies will be clear & crisp; but this is not make for a grand sunset either. During Winter, storm clouds hang low on horizon and you can get dramatic sunsets. Or look for days when there are lots of high clouds in Western sky
• I use a telephoto, to zoom in and compress the image. The size of the Sun is ‘magnified’ when doing this
• For the last photo, I used a 0.6 (2 stop) ND Gradual filter (explanation, buy) to mask the sky, so I could get smooth, even colors on sky and water. Without the filter, the sky would have been too bright, and water would be too dark
• I used longest shutter speed I could possibly get (by turning up Aperture to F16,F22), to smooth out the waves and water movement
• used a Tripod with cable release
• bracketed from -1.0 ev to +1.0ev
  When the Sun is in the pictures (first pictures) the camera will underexpose because the scene is very bright. So compensate by ++ (+1.0 or +1.5ev works for me)
  After the sunset, camera will over-expose (not enough light); to maintain silhouettes underexpose (-1.0ev or -1.5ev works for me)
• Gear:
  Camera : Nikon D200
  Lens : Nikor 80-200
  White Balance : Shade
  Tripod + cable release.
  images were shot in JPG and no editing is done, just resizing

<plug>Checkout My Gallery, Latest Images and Writeups </plug>

After having to put up with co-workers and friends gushing ‘ooh, wanna hang out with me and my cool iPhone (in Santana Row)’, I am coining a new term:

iFone :

• When applied to a person:
  Used to describe a person so full of himself but very little substance/depth
• When applied to a thing / coding methodology / language / operating system
  When a thing is so hyped and doesn’t live up to it

Examples:

• oh, he is such an iFone
• oh, it is such an iFone

Fans of Sienfield will remember the quote from Jerry : ‘He is such a Phony’ from episode ‘Parking Space’

http://www.tvsquad.com/2006/09/06/seinfeld-the-parking-space/

Note : The word is iFone, not iPhone (which is, I am sure a trademark by Apple Inc)

This article walks through examples of processing Epcis XML events in Java, using Xmlbeans library.

*Go to Article*

Hi all,

I had a blog before every cat and dog had one. But I was getting inundated with so much spam. Pretty soon it became not worthwhile to fight it and I shut the blog down (besides I didn’t have very much interesting stuff to say anyway)

Now, with the excellent Akismeet, it is possible to just write a blog without worrying to much about the spam. So I am trying, once again.

I want to write mostly about tech and photography.