HBase is able to connect to ZooKeeper but the connection closes immediately

I got following error when i tried to connect hbase from hadoop nodes. Zookeeper servers runs normally but hadoop nodes can not connect remote zookeeper cluster.

ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Then i fixed problem copying hbase-site.xml to hadoop/conf directory.

JUDCon2013: UnitedStates

When: June 9-11, 2013
Where: Hynes Convention Center, Boston, Massachusetts
Why: Because developers need to get their hands dirty in code.

Early registration is open, and you can save 40% if you register now!

The ninth JUDCon - JUDCon2013: UnitedStates - will be a 3-day, 3-track extravaganza including a welcome reception, 3 tracks of valuable sessions, with 2 additional workshop tracks. And this year, one low price will get you access to CamelOne and Red Hat Exchange as well!

JUDCon sessions will include:

• Building HTML5 applications, the JBoss Way
  Peter Muir
• 59 mins to (geo)satellite search in your Hibernate apps... 58 mins...
  Emmanuel Bernard
• Re-thinking Mobile/Web Applications, Getting it Done, Keeping it That Way
  Lincoln Baxter, III
• What's new in Drools 6.0
  Mark Proctor

The activities start Sunday evening with a JUDCon, CamelOne and Red Hat Exchange reception. Then Monday and Tuesday, there will be 3 tracks of sessions and 2 workshop tracks. During dinner Monday evening, the JBoss Community Recognition Award winners will be recognized. The evening will also include a JBoss Core Developer panel, a live recording of the JBoss Community Asylum, and, yes, beer.

The hotel blocks are filling up so please register now to reserve your place! See you there!

The JBoss Team

How to check hadoop health from commandline

from the hadoop home directory:
# ./bin/hadoop fsck /

FSCK started by root from /10.1.1.2 for path / at Mon Apr 15 14:16:20 EEST 2013

....................................................................................................

....................................................................................................

....................................................................................................

....................................................................................................

....................................................................................................

....................................................................................................

....................................................................................................

..............................................................Status: HEALTHY

Total size:                               6907330009 B

Total dirs:                               491

Total files:                               1162 (Files currently being written: 3)

Total blocks (validated):          1215 (avg. block size 5685045 B)

Minimally replicated blocks:     1215 (100.0 %)

Over-replicated blocks:           0 (0.0 %)

Under-replicated blocks:         0 (0.0 %)

Mis-replicated blocks:             0 (0.0 %)

Default replication factor:         3

Average block replication:       3.0

Corrupt blocks:                       0

Missing replicas:                      0 (0.0 %)

Number of data-nodes:            3

Number of racks:                    1

FSCK ended at Mon Apr 15 14:16:20 EEST 2013 in 116 milliseconds

The filesystem under path '/' is HEALTHY

How to check Hbase health from commandline?

from the hbase home directory:

# ./bin/hbase hbck

Version: 0.90.5

Number of Tables: 1

Number of live region servers: 3

Number of dead region servers: 0

Number of empty REGIONINFO_QUALIFIER rows in .META.: 0

Summary:

  -ROOT- is okay.

    Number of regions: 1

    Deployed on: server1.internal:60020

  .META. is okay.

    Number of regions: 1

    Deployed on: server2.internal:60020

  webpage is okay.

    Number of regions: 41

    Deployed on: server3.internal:60020 server2.internal:60020 server1.internal:60020

0 inconsistencies detected.

Status: OK

Open Source Load Balancing Solutions

There are plenty of open source tcp load balancing software.

HAproxy
site: http://haproxy.1wt.eu/

Linux Virtual Server
site: http://www.LinuxVirtualServer.org/

Linux Network Load Balancing
site: http://lnlb.sourceforge.net/

Distributor Load Balancer
site: http://distributor.sourceforge.net/

Balance
site:http://www.inlab.de/balance.html

Load Balancer Project
site: http://www.jmcresearch.com/projects/loadbalancer/

Pen
site: http://siag.nu/pen

Zen Load Balancer
site: http://zenloadbalancer.sourceforge.net/

Crossroads
site: http://crossroads.e-tunity.com/

Core Balance
http://core-balance.sourceforge.net/

Open Source System Logging Solutions

For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.

Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.

logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.

Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.

Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.

Network performance testing in Linux

IPERF is a network speed testing tool. It is reliable if you have two servers, and you want to measure network speed between them.

It can be installed using yum
# yum install iperf

For testing, one iperf instance runs as a server on one server and another iperf instance runs as a client on the other server.

For running iperf as a server
# iperf -s
Server listening on TCP port 5001

And for running iperf as a client
# iperf -c 192.168.0.11
Client connecting to 192.168.0.11, TCP port 5001 TCP window size: 16.0 KByte (default) 
------------------------------------------------------------ 
[3] local 192.168.0.12 port 47326 connected with 192.168.0.11 port 5001 
[ID] Interval Transfer Bandwidth 
[3] 0.0-10.0 sec 267 MBytes 228 Mbits/sec