Skip to main content

Posts

Showing posts from 2013

Chrome Custom URL Redirector Extension

Working as a IT professional sometimes you need to access remote systems using vpn connection. Throughout the vpn connection dns records and hosts records may be overwritten and control panel url addresses are inaccessible. To overcome this you need to use custom browser extension like greasemonkey, tempermonkey etc.
But i don't want to use these third party extensions so i wrote a simple chrome plugin which translates url addresses to ip addresses.
You need two files, manifest.json and background.js
Examples of these files are(add urls and ip addresses to urlDict variable as required!):
manifest.json: {   "name": "Custom URL Redirector",   "version": "0.1",   "description": "Checks URL and redirects as required.",   "manifest_version": 2,   "background": {     "scripts": ["background.js"]   },   "permissions":["http://*/*","https://*/*","webReques…

How to create hive external table for nutch's hbase webpage schema?

In order to query hbase table using hive, an external table should be created.
CREATE EXTERNAL TABLE webpage_hive (key string, baseUrl string, status int, prevFetchTime bigint, fetchTime bigint, fetchInterval bigint, retriesSinceFetch int, reprUrl string, content string, contentType string, protocolStatus string, modifiedTime bigint, prevModifiedTime bigint, batchId string, title string, text string, parseStatus int, signature string, prevSignature string, score int, headers map<string,string>, inlinks map<string,string>, outlinks map<string,string>, metadata map<string,string>, markers map<string,string>) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f:bas,f:st,f:pts#b,f:ts#b,f:fi#b,f:rsf,f:rpr,f:cnt,f:typ,f:prot,f:mod#b,f:pmod#b,f:bid,p:t,p:c,p:st,p:sig,p:psig,s:s,h:,il:,ol:,mtdt:,mk:") TBLPROPERTIES ("hbase.table.name" = "webpage");
after…

Howto disable IPv6 Stack Ubuntu Linux

Sometimes network infrastructures are misconfigured to use ipv6 stack. Then you want to disable it at all. For ubuntu distribution it is easy as changing sysctl.conf and reboot your machine. Here is how to do it.

First check whether it is enabled or not:
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
if output is 0 it is enabled and 1 it is disabled.

Adding the following rows is enough to disable ipv6 stack then restart your computer to take effect:
# vi /etc/sysctl.conf
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1
# shutdown -r now


When your computer is up again, you can check it:
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6


Pig Latin Error 2998 when loading data from HBaseStorage

ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/WritableByteArrayComparable
Probably your PIG_CLASSPATH does not contain hbase jars. PIG_CLASSPATH should contain them like:
export PIG_CLASSPATH=/<hbase_home>/<hbase-your_version.jar>:<hbase_home>/lib/<zookeeper-your_version.jar>:$PIG_CLASSPATH

Hive exception when executing a query: java.lang.Integer cant be mapped for this datastore

When executing query in the hive shell following error is shown.
hive> show tables; FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOFatalInternalException: JDBC type integer declared for field "org.apache.hadoop.hive.metastore.model.MTable.createTime" of java type java.lang.Integer cant be mapped for this datastore.
NestedThrowables: org.datanucleus.exceptions.NucleusException: JDBC type integer declared for field "org.apache.hadoop.hive.metastore.model.MTable.createTime" of java type java.lang.Integer cant be mapped for this datastore.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
That is because of your language settings. To solve the problem export lang variable before launching hive. # export LANG=C

Saltstack as a Configuration Management Software

Salt is an infrastructure management system and takes a new approach for developing system administration software that is to get running in minutes, scalable to tens of thousands of servers. The Salt open source project was started in 2011. Salt provides a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management. SaltStack is the company responsible for keeping the Salt project healthy and dynamic. Salt remains 100% open.
Installing Salt on RHEL/Centos Systems:
From the 0.9.4 release Salt has been available in EPEL repos. To enable EPEL: # rpm -Uvh http://ftp.linux.ncsu.edu/pub/epel/6/i386/epel-release-6-8.noarch.rpm
Salt is divided into two packages. Minion and the master. It is necessary only to install the suitable package for the role the machine will play. In general, there is one master and multiple minions.
On the master, install salt-master: # yum install salt-master 
On each slave node, install salt-minion: # yum install salt-…

How to install flash player on fedora linux 18

To install flash player on fedora linux 18 operating system:

32-bit installation:
# rpm -ihv http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

HBase is able to connect to ZooKeeper but the connection closes immediately

I got following error when i tried to connect hbase from hadoop nodes. Zookeeper servers runs normally but hadoop nodes can not connect remote zookeeper cluster.

ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Then i fixed problem copying hbase-site.xml to hadoop/conf directory.

How to check hadoop health from commandline

from the hadoop home directory:
# ./bin/hadoop fsck /
FSCK started by root from /10.1.1.2 for path / at Mon Apr 15 14:16:20 EEST 2013 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ..............................................................Status: HEALTHY Total size:                               6907330009 B Total dirs:               …

How to check Hbase health from commandline?

from the hbase home directory: # ./bin/hbase hbck
Version: 0.90.5
Number of Tables: 1 Number of live region servers: 3 Number of dead region servers: 0 Number of empty REGIONINFO_QUALIFIER rows in .META.: 0 Summary:   -ROOT- is okay.     Number of regions: 1     Deployed on: server1.internal:60020   .META. is okay.     Number of regions: 1     Deployed on: server2.internal:60020   webpage is okay.     Number of regions: 41     Deployed on: server3.internal:60020 server2.internal:60020 server1.internal:60020 0 inconsistencies detected. Status: OK

Open Source Load Balancing Solutions

There are plenty of open source tcp load balancing software.

HAproxy
site: http://haproxy.1wt.eu/

Linux Virtual Server
site: http://www.LinuxVirtualServer.org/

Linux Network Load Balancing
site: http://lnlb.sourceforge.net/

Distributor Load Balancer
site: http://distributor.sourceforge.net/

Balance
site:http://www.inlab.de/balance.html

Load Balancer Project
site: http://www.jmcresearch.com/projects/loadbalancer/

Pen
site: http://siag.nu/pen

Zen Load Balancer
site: http://zenloadbalancer.sourceforge.net/

Crossroads
site: http://crossroads.e-tunity.com/

Core Balance
http://core-balance.sourceforge.net/

Open Source System Logging Solutions

For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.


Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.
logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.
Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.
Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.


Network performance testing in Linux

IPERF is a network speed testing tool. It is reliable if you have two servers, and you want to measure network speed between them.

It can be installed using yum
# yum install iperf

For testing, one iperf instance runs as a server on one server and another iperf instance runs as a client on the other server.

For running iperf as a server
# iperf -s
Server listening on TCP port 5001


And for running iperf as a client
# iperf -c 192.168.0.11
Client connecting to 192.168.0.11, TCP port 5001 TCP window size: 16.0 KByte (default) 

------------------------------------------------------------ 
[3] local 192.168.0.12 port 47326 connected with 192.168.0.11 port 5001 
[ID] Interval Transfer Bandwidth 
[3] 0.0-10.0 sec 267 MBytes 228 Mbits/sec

What is OpenID

OpenID is an open standard that allows users to be authenticated by certain co-operating sites using a third party service, eliminating the need for webmasters to provide their own ad hoc systems and allowing users to consolidate their digital identities.

Users may create accounts with their preferred OpenID identity providers, and then use those accounts as the basis for signing on to any website which accepts OpenID authentication.

Linux bash string manipulation

Length of a string${#str} str="abcdefg" echo ${#str} #gives output 7
Substring extraction ${str:pos} extracts substring from $str at $pos str="abcdefg" echo ${str:2} #gives output cdefg
${str:pos:len} extracts substring from $str at $pos length of $len str="abcdefg" echo ${str:2:3} #gives output cde
Substring Removal${str#substr} removes shortest match of $substr from front of $str str="abcdefgabcdefg" echo ${str#a*c} #gives output defgabcdefg
${str##substr} removes longest match of $substr from front of $str str="abcdefgabcdefg" echo ${str##a*c} #gives output defg
${str%substr} removes shortest match of $substr from back of $str str="abcdefgabcdefg" echo ${str%b*g} #gives output abcdefga
${str%%substr} removes longest match of $substr from back of $str str="abcdefgabcdefg" echo ${str%%b*g} #gives output a

libXext.so problem when installing oracle

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-01-22_10-21-26AM. Please wait ...[oracle@orcltest database]$ Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/OraInstall2013-01-22_10-21-26AM/jdk/jre/lib/amd64/xawt/libmawt.so: libXext.so.6: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1649) at java.lang.Runtime.load0(Runtime.java:769)

This may be encountered when installing oracle database on a redhat linux server. That's because libXp package was not installed on the RHEL5/6 system. Just install the libXp packege.

SOLR4 how to abort running dataimport command

First you can get current dataimport status with the following url in a multicore environment: http://<solr4_server_ip>:<port_number>/solr/<core_name>/dataimport?command=status
Probably you may have seen the following result: <str name="importResponse">A command is still running...</str>
Then this command can be aborted with the following url: http://<solr4_server_ip>:<port_number>/solr/<core_name>/dataimport?command=abort

bash script include config file

With source command static variables may be included from the main bash script like:
source /etc/script.config

And content of config file may like:
server_ip="5.5.5.5"
app_version=3.5
app_list=('app1' 'app2' 'app3')