Skip to main content

Posts

Showing posts from April, 2013

HBase is able to connect to ZooKeeper but the connection closes immediately

I got following error when i tried to connect hbase from hadoop nodes. Zookeeper servers runs normally but hadoop nodes can not connect remote zookeeper cluster.

ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Then i fixed problem copying hbase-site.xml to hadoop/conf directory.

How to check hadoop health from commandline

from the hadoop home directory:
# ./bin/hadoop fsck /
FSCK started by root from /10.1.1.2 for path / at Mon Apr 15 14:16:20 EEST 2013 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ..............................................................Status: HEALTHY Total size:                               6907330009 B Total dirs:               …

How to check Hbase health from commandline?

from the hbase home directory: # ./bin/hbase hbck
Version: 0.90.5
Number of Tables: 1 Number of live region servers: 3 Number of dead region servers: 0 Number of empty REGIONINFO_QUALIFIER rows in .META.: 0 Summary:   -ROOT- is okay.     Number of regions: 1     Deployed on: server1.internal:60020   .META. is okay.     Number of regions: 1     Deployed on: server2.internal:60020   webpage is okay.     Number of regions: 41     Deployed on: server3.internal:60020 server2.internal:60020 server1.internal:60020 0 inconsistencies detected. Status: OK

Open Source Load Balancing Solutions

There are plenty of open source tcp load balancing software.

HAproxy
site: http://haproxy.1wt.eu/

Linux Virtual Server
site: http://www.LinuxVirtualServer.org/

Linux Network Load Balancing
site: http://lnlb.sourceforge.net/

Distributor Load Balancer
site: http://distributor.sourceforge.net/

Balance
site:http://www.inlab.de/balance.html

Load Balancer Project
site: http://www.jmcresearch.com/projects/loadbalancer/

Pen
site: http://siag.nu/pen

Zen Load Balancer
site: http://zenloadbalancer.sourceforge.net/

Crossroads
site: http://crossroads.e-tunity.com/

Core Balance
http://core-balance.sourceforge.net/

Open Source System Logging Solutions

For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.


Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.
logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.
Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.
Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.