Excellent post indeed, thank you for taking the time! Also please consider linking it into the HBase Wiki, so that other can find it easily (if you have not done so already) – simply register and edit the appropriate page. Much appreciated!
Hey Sujee!! Really nice and helpful post, thanks a lot!!! Though I have a little doubt, I am still learning Hadoop MapReduce and I was wondering why I can’t see the jobs I am executing on my localhost:50030?? I tried to executed from eclipse using the cluster I set up, but I got an error that said
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
Hi I tried this: http://wiki.apache.org/hadoop/Hbase/MapReduce
with no success. Though I am able to execute my job regularly from the command line, and as a java program, any ideas or suggestions are highly appreciated. Thanks in advanced.
@Renato,
the end goal is to be able to run in command line using
hadoop jar your.jar your.mr.class.name
so it will run on all cluster machines.
So if you can do this, that is pretty good
Not sure why you aren’t able to run it within Eclipse…
Hi,
I really like this post and has helped me in my project..thanks for the same..
I am now motivated to make a HBase admin UI interface…
do ping me if u want to help/get involved..
thanks once again
thank for your tutorial. That is the only one that I found that does what I need: read from the HBase in MR and write back in MR! It works and uses the latest API. What more can a person want?
@Mark,
the row key = random_user_id + incrementing_counter
so it will be unique.
timestamps are automatically created by Hbase for versioning cell data. I think it is milli-second accuracy for now. And it is possible you get a few entries with the same TS. But it is okay in most cases (in this case too)
Really nice post, thanks a lot !!! Its really helpful to understand job creation and configuration.But i m getting some exception while executing through eclipse.I m using HBase 0.20.6 APIs. HDFS is in a cluster(multi machines)
Exception in thread “main” java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)
at com.exilant.hbase.FreqCounter1.main(FreqCounter1.java:81)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:319)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
at java.lang.ClassLoader.loadClass(ClassLoader.java:254)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:399)
… 8 more
@swagat,
I am guessing Eclipse can not find some Hadoop / Hbase jars. Try running the program as a Map-Reduce job (use hadoop command to submit) from command line. That might fix the issue
@Amarjeet
not sure what do you mean by using-memory data as Mapreduce input. are you talking about doing ‘data joins’ ?
If that is the case, then you might try this:
- load your data xref data into a memory cache like memcached (this can be done in the driver code of mapreduce)
- in your mapper, access memcached to xref
I Mean:
lets take this example in your blog -
you are reading input from Hbase table – ‘access_logs’ and then process it and put your output in ‘summary’ table.
What I want is:
I wish I could take input from some Map or list (present in-memory only or create myself in code).
Excellent tutorial Sujee!
With the filter, someone should also use scan.setCaching so speed up the job.
J-D
Hi Sujee!
Excellent post indeed, thank you for taking the time! Also please consider linking it into the HBase Wiki, so that other can find it easily (if you have not done so already) – simply register and edit the appropriate page. Much appreciated!
Lars
Hey Sujee!! Really nice and helpful post, thanks a lot!!! Though I have a little doubt, I am still learning Hadoop MapReduce and I was wondering why I can’t see the jobs I am executing on my localhost:50030?? I tried to executed from eclipse using the cluster I set up, but I got an error that said
Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
any idea??? Thanks a lot in advance (=
Renato
@Renato
Your hadoop installation doesn’t know about Hbase classes.
Check ‘step 0′ under ‘Hbase setup’ section. Or you can follow this:
http://hadoop.apache.org/hbase/docs/r0.20.3/api/org/apache/hadoop/hbase/mapreduce/package-summary.html
Once you modify your file, make sure to DISTRIBUTE the hadoop-env.sh across the cluster.
Hi I tried this:
http://wiki.apache.org/hadoop/Hbase/MapReduce
with no success. Though I am able to execute my job regularly from the command line, and as a java program, any ideas or suggestions are highly appreciated. Thanks in advanced.
Renato M.
@Renato,
the end goal is to be able to run in command line using
hadoop jar your.jar your.mr.class.name
so it will run on all cluster machines.
So if you can do this, that is pretty good
Not sure why you aren’t able to run it within Eclipse…
Hi,
I really like this post and has helped me in my project..thanks for the same..
I am now motivated to make a HBase admin UI interface…
do ping me if u want to help/get involved..
thanks once again
Worked perfectly. This tutorial was so helpful. Thank you so much.
Even the official Apache tutorial in the Hadoop docs use the deprecated APIs, it’s nice to see an up to date tutorial like this one.
Hi, Sujee,
thank for your tutorial. That is the only one that I found that does what I need: read from the HBase in MR and write back in MR! It works and uses the latest API. What more can a person want?
Cheers,
Mark
By the way, your timestamp is not unique, because a few records are created for the same time stamp – or my computer is too fast
, but you get this:
\x00\x00\x00\x02\x00 column=details:page, timestamp=1296845269865, value=/a.htm
\x00\x07i l
\x00\x00\x00\x02\x00 column=details:page, timestamp=1296845269865, value=/a.htm
\x00\x07j l
@Mark,
the row key = random_user_id + incrementing_counter
so it will be unique.
timestamps are automatically created by Hbase for versioning cell data. I think it is milli-second accuracy for now. And it is possible you get a few entries with the same TS. But it is okay in most cases (in this case too)
glad you found the article useful
Thanks for great tutorials !
Dear Sujee,
Thanks for your great tutorial. One may need to download google-collect-1.0-rc1.jar and jsr305.jar
I am glad that I found your tutorial too.
Hi Sujee,
Really nice post, thanks a lot !!! Its really helpful to understand job creation and configuration.But i m getting some exception while executing through eclipse.I m using HBase 0.20.6 APIs. HDFS is in a cluster(multi machines)
Exception in thread “main” java.lang.NoClassDefFoundError: org/codehaus/jackson/map/JsonMappingException
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapreduce.Job.connect(Job.java:476)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:464)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:495)
at com.exilant.hbase.FreqCounter1.main(FreqCounter1.java:81)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.map.JsonMappingException
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:319)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
at java.lang.ClassLoader.loadClass(ClassLoader.java:254)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:399)
… 8 more
any suggestion will be a great help for me.
Thanks
Swagat
@swagat,
I am guessing Eclipse can not find some Hadoop / Hbase jars. Try running the program as a Map-Reduce job (use hadoop command to submit) from command line. That might fix the issue
Can you please guide me how to use HBase in Java programs, I tried running your example but it gives this error
11/05/02 17:44:40 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 0 time(s).
11/05/02 17:44:42 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 1 time(s).
11/05/02 17:44:44 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 2 time(s).
11/05/02 17:44:46 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 3 time(s).
11/05/02 17:44:48 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 4 time(s).
11/05/02 17:44:50 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 5 time(s).
11/05/02 17:44:52 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 6 time(s).
11/05/02 17:44:54 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 7 time(s).
11/05/02 17:44:56 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 8 time(s).
11/05/02 17:44:58 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:60000. Already tried 9 time(s).
11/05/02 17:44:59 INFO client.HConnectionManager$TableServers: Attempt 0 of 10 failed with . Retrying after sleep of 2000
please let me know where am i going wrong. please suggest a simple way to use HBase. I just want to use HBase for simple java programming
I tried your example, and these are the errors I am getting
The project cannot be built until build path errors are resolved hbase_mapreduce Unknown Java Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/conf’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/hbase-0.20.3.jar’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/lib/commons-logging-1.0.4.jar’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/lib/hadoop-0.20.1-hdfs127-core.jar’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/lib/log4j-1.2.15.jar’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Unbound classpath variable: ‘C:cygwin/usr/local/hbase-0.90.2/lib/zookeeper-3.2.2.jar’ in project ‘hbase_mapreduce’ hbase_mapreduce Build path Build Path Problem
Please let me know where I am going wrong
This is an awesome tutorial. Gave me a good feel for what HBase brings to Hadoop. Good stuff!
Thank you Sujee, Really useful.
It’s working fine. But If I check the table “access_logs”, It is present only in one of the machine(cluster of 3).
How can I make it as distributed?
Thank you Sujee, Really useful.
I did this, in a cluster which contains 3 Region Servers
It’s working fine. But If I check the table “access_logs”, It is present only in one of the Region Server.
How can I make it as distributed?
@Anil
the table will split as it exceeds the ‘region max size’ (default 256M, I think).
Hi Sujee,
Thanks for great tutorial!!
I am able to run the mapreduce application using HDFS and HBase both.
But now I want to use in memory data to be used as input to MapReduce application.
e.g. – I have few data maps in my memory and want to use those in spite of Hbase table as input.
Could you please guide me for this??
Timely reply will be appreciated very much.
Thanks in advance!!
@Amarjeet
not sure what do you mean by using-memory data as Mapreduce input. are you talking about doing ‘data joins’ ?
If that is the case, then you might try this:
- load your data xref data into a memory cache like memcached (this can be done in the driver code of mapreduce)
- in your mapper, access memcached to xref
Hi Sujee,
by memory data I dont mean ‘DATA JOINS’.
I Mean:
lets take this example in your blog -
you are reading input from Hbase table – ‘access_logs’ and then process it and put your output in ‘summary’ table.
What I want is:
I wish I could take input from some Map or list (present in-memory only or create myself in code).
I hope you understand now.
Hi Sujee ,
really very good doc …..
thanks