Identify user from python mapper in Hadoop

If you are running a mapreduce job using hadoop streaming with python, and want to know the user the job runs as or other parameters in the OS environment on the task tracker node, use

if(envdict.has_key('user_name')):
           user = os.environ['user_name']
            sys.stdout.write('Job runs as username: '+user)

To make sure what variables you have, use this

       envdict = os.environ
       keylist = envdict.keys()
       sys.stdout.write('Variables available'+str(keylist)+'\n')

Moving Hbase heap dump locations

Occasionally HBase can generate heap dumps on OOM and put it in the bin folder (which is default). This can cause quite a lot of issues, if the system partitioning is not prepared for huge gigabyte files in the bin folder for hbase. To move the dump cores to another folder ,

Do this in the file  hbase-ver-conf/hbase-env.sh, add  -XX:HeapDumpPath=/path/to/dump to the line

>> export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError $HBASE_GC"

Restart HBase.

Now you can sleep well without fears of heap dumps filling up the hbase binary partitions :)

Compare list of files to identify files and directories

Say xyz is a file with a list of files and directories. (The files and directories are all on the working system itself)
You need to print only the directories amongst them.

for i in `cat xyz`; do if [ -d $i ] ; then echo $i ;fi; done

will print out only directories among them.

Hp Openview Errors

Last week, I had a few issues with bringing up the HP Openview Operations Manager. The snap-in wouldn't load .It just gave this error.

"The Console is unable to connect to the server. Please make sure that all services including the database are up and running."

Now on the services.msc in Windows, the database services and related services were all up and running. I could also connect to the db using the SQL studio. ( My DB is MS SQL 2005)

Then i tried to run this
C:\Program Files\HP\HP BTO Software\support\OVMigration\Binn\osql -E openview -S DBserverName\InstanceName

That too showed that things were running fine.

Next I tried running the command on the OVO server
vpstat -8

It showed that all services except OvEpMsgActSrv was running fine.

You can also use ovc -status to check the services.

Now this turned out to be failing due to a huge message queue file at this location.
"\Program Files\HP Openview\Data\Shared\tmp\queues"

Once that was dumped, I restarted the services with
vpstat -8 -r

and things were fine again. This is how a good configuration should look.


D:\HP\OM\support\OVMigration\Binn>osql.exe -E openview -s SERVERNAME\DBNAME
1>
2>
3>
4>
5> quit

D:\HP\OM\support\OVMigration\Binn>
D:\HP\OM\support\OVMigration\Binn>ovc -status
ovcd OV Control CORE (5624) Running
opcmsga OVO Message Agent AGENT,EA (2148) Running
opcmsgi OVO Message Interceptor AGENT,EA (6480) Running
opcle OVO Logfile Encapsulator AGENT,EA (428) Running
opcacta OVO Action Agent AGENT,EA (5580) Running
agtrep OV Discovery Agent AGENT,AgtRep (5776) Running
coda OV Performance Core COREXT (6456) Running
opcmona OVO Monitor Agent AGENT,EA (6576) Running
ovconfd OV Config and Deploy COREXT (9720) Running
opcwbemi OVO WMI Interceptor AGENT,EA (5572) Running
ovbbccb OV Communication Broker CORE (4424) Running
ovcs OV Certificate Server SERVER (1524) Running
ovtomcatB OV Tomcat(B) Servlet Container WEB (6128) Running

D:\HP\OM\support\OVMigration\Binn>
D:\HP\OM\support\OVMigration\Binn>vpstat -3
============================================================
running: D:\HP\OM\bin\vpstat.exe
version: A.22.0.56.30 on SERVERNAME
now: Sun May 01 11:01:25 2011
============================================================

============================================================
==================== Services Information ==================
============================================================

Configuration file: D:\HP\OM\data\conf\vpstat\vpstat.conf
Ver: 0.8.10.074
Date: Thu Jun 19 15:40:36 2008
Remark: HP Operation Manager - Performance.

Services on : LOCAL MACHINE

(NT Services) :

OK: All services : SERVICE_ACTIVE


D:\HP\OM\support\OVMigration\Binn>vpstat -8
============================================================
running: D:\HP\OM\bin\vpstat.exe
version: A.22.0.56.30 on UP-OVO01
now: Sun May 01 11:01:28 2011
============================================================

============================================================
==================== Processes Information =================
============================================================

Configuration file: D:\HP\OM\data\conf\vpstat\vpstat.conf
Ver: 0.8.10.074
Date: Thu Jun 19 15:40:36 2008
Remark: HP Operation Manager - Performance.


OK:ALL CRITICAL PROCESSES ARE RUNNING

D:\HP\OM\support\OVMigration\Binn>

PIX and SMTP troubles

Twice in our current environment, we've had to face issues with PIX interfering with very specific SMTP communications. The first time, one of our users could not receive a particular type of pdf attachments from one source. After a hell of troubleshooting, it finally turned out to be that PIX ESMTP filtering was the source of the problem.

This time, there was another similar issue. Some domains with a big list of entries in the "ADDITIONAL SECTION" of the DNS response could not me sent emails. The error being

Delivery Failure
Attempted Delivery to: Unknown  Error: 554 5.4.4 [internal] domain lookup failed

This error though was not correct, since the DNS lookups were successful in the nslookup utility on the messaging gateway.

Digging further in pointed to the issue being related to large DNS query responses which were being rejected by PIX DNS restrictions. DNS UDP Packets of size greater than 512 bytes were not allowed by PIX. So Once that was removed, things started working well.

Autocad network license manager error

I was having a tough time with the following error in many systems on a trusted domain within our network. The Autocad licensing server was on our domain and could be accessed without issues from the client desktop which was on the trusted domain, but AutoCAD wouldn't start. It just kept popping up this error

The network license manager was not able to get a valid license. If this problem continues, please contact your system administrator. Error [1.5.-18]

Turned out that, in the AutoCAD server license file, the server name provided was not accessible from the client desktop since it was not in a FQDN format. When the hosts file was modified to do the name mapping, it worked fine. I then created a host record in the trusted domain with the same name to point to the licensing server and it worked.

Anonymous users in proxy logs

Ever since we installed a new Microsoft TMG server for proxy authentication and web filtering, we've seen random users being denied access for no good reason. The rules have been set to allow access to these users and it works for most of the other users.

The logs seem to show that the users, who are logged in as domain users still show up as anonymous accounts at the TMG server.

In one such instance, this step fixed it.
In IE-> tools-> Internet Options-> Advanced-> Security
Toggle the "Enable integrated windows authentication."