Category Archives: IT Stuff

Route a message to MongoDB

The requirement is very simple. Route an XML message from rabbitMQ to MongoDB. MongoDB BSON as the data storage and network transfer format for “documents”. BSON is a binary-encoded serialization of JSON-like documents. So, the source message is in a XML format, after getting it from rabbitMQ is necessary to translate into a JSON format compatible with MongoDB.

To translate a message from XML to JSON is possible to use Marshalling function available in Camel extension XmlJson. The pom.xml file needs this new entry:

<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-xmljson</artifactId>
	<version>${org.camel.version}</version>
</dependency>

Of course, to send communicate with MongoDB another entry has to be added to pom.xml:

<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-mongodb</artifactId>
	<version>${org.camel.version}</version>
</dependency>

Now, is possible to change the route on camel ApplicationContext file:

<!-- Camel route -->    
<camelContext xmlns="http://camel.apache.org/schema/spring">
	<dataFormats>
		<xmljson id="xmljson"/>
	</dataFormats>
	<route> 
		<from uri="spring-amqp:TPDirect:TPQueue:TPRouting?type=direct&amp;autodelete=true&amp;durable=true" />
		<marshal ref="xmljson"/>
		<log message="From XML to Json: DONE!" />
		<convertBodyTo type="java.lang.String"/>
		<to uri="mongodb:myDb?database=flights&amp;collection=tickets&amp;operation=save" />
	</route>                                 
</camelContext>
 
 
<!-- Mongo DB -->
<bean id="myDb" class="com.mongodb.Mongo">
	<constructor-arg index="0" value="localhost"/>
</bean>

Bean myDB contains the information to reach MongoDB. It’s also possible to define it using the full url:

<bean id="myDb" class="com.mongodb.Mongo"> 
    <constructor-arg index="0"> 
        <bean class="com.mongodb.MongoURI"> 
            <constructor-arg index="0" value="mongodb://username:password@host:port/db" /> 
        </bean> 
    </constructor-arg> 
</bean>

To avoid this exception:

Caused by: No type converter available to convert from type: byte[] to the required type: org.apache.camel.component.mongodb.converters.MongoDbBasicConverters with value

The JSON translated message has to be converted into String.

That’s all!

Tagged , , ,

How to switch between different JVM in Ubuntu

Sometime is needed to install different JVM on Ubuntu server due to application compatibility or software development purposes. To switch between different JVM in ubuntu from command line, use this command:

sudo update-alternatives --config java

and this menu will appear:

There are 2 choices for the alternative java (providing /usr/bin/java).
 
  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      auto mode
  1            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      manual mode
  2            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071      manual mode
Tagged ,

Camel and RabbitMQ : Finally, how to!

Define a RabbitMQ broker endpoint in Camel is possible with the Bluelock camel-spring-amqp (https://github.com/Bluelock/camel-spring-amqp) library. It’s an Apache Camel component that allows to natively communicate with a RabbitMQ broker and it’s implemented using Spring’s AMQP.

For first, with Eclipse IDE create a new Maven Project with Artifact ID camel-arthetype-spring. This allows using Spring DSL to configure Camel route and execute the run:camel goal of Camel Mavel Pluing (Camel Maven Plugin) in a forked JVM from Maven.

To resolve the dependencies, these entries are mandatory:

<dependency>
	<groupId>com.bluelock</groupId>
	<artifactId>camel-spring-amqp</artifactId>
	<version>1.2</version>
</dependency>
 
<!-- Camel dependencies -->
<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-test</artifactId>
	<version>${org.camel.version}</version>          
</dependency>
<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-spring</artifactId>
	<version>${org.camel.version}</version>        
	<exclusions>
		<exclusion>
			<artifactId>spring-tx</artifactId>
			<groupId>org.springframework</groupId>
		</exclusion>
		<exclusion>
			<artifactId>spring-context</artifactId>
			<groupId>org.springframework</groupId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-xstream</artifactId>
	<version>${org.camel.version}</version>            
</dependency>

At this point edit the camel-context.xml available on src/main/resource/META-INF/spring folder:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Configures the Camel Context-->
 
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:camel="http://camel.apache.org/schema/spring"
       xmlns:rabbit="http://www.springframework.org/schema/rabbit"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
       http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
       http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit-1.0.xsd">
 
	<camelContext xmlns="http://camel.apache.org/schema/spring">
		<route>	
			<from uri="spring-amqp:KipcastDirect:KipcastQueue:KipcastRouting?type=direct&amp;autodelete=true&amp;durable=true" />
			<log message="Message available on a RabbitMQ Queue" />			
			<process ref="processorTest" />
		</route>
	</camelContext>
 
	<rabbit:connection-factory id="amqpConnectionFactory" />
	<rabbit:template id="amqpTemplate" connection-factory="amqpConnectionFactory" message-converter="messageConverter" exchange="KipcastBean" />
	<rabbit:admin connection-factory="amqpConnectionFactory"/>
 
	<bean id="amqpConnectionFactory" class="org.springframework.amqp.rabbit.connection.CachingConnectionFactory">
	    <property name="host" value="10.211.55.20"/>
	    <property name="port" value="5672"/>
	    <property name="username" value="guest"/>
	    <property name="password" value="guest"/>
	    <property name="virtualHost" value="/"/>
	</bean>
 
    <bean id="jsonMessageConverter" class="amqp.spring.converter.XStreamConverter"/>
    <bean id="textMessageConverter" class="amqp.spring.converter.StringConverter"/>
    <bean id="messageConverter" class="amqp.spring.converter.ContentTypeConverterFactory">
        <property name="converters">
            <map>
                <entry key="application/json" value-ref="jsonMessageConverter"/>
                <entry key="application/xml" value-ref="textMessageConverter"/>
            </map>
        </property>
        <property name="fallbackConverter" ref="textMessageConverter"/>
    </bean>
 
</beans>

In this case, a route starting from a RabbitMQ queue to a system log and a listener on your queue that will be active until you terminate your maven came:run process has been created.

It’s very important to note that, on Spring XML & has to be quoted &amp;

The following Java code should be used send a message to the Exchange defined on Camel Route:

@Test
public void test() throws KeyManagementException, NoSuchAlgorithmException, URISyntaxException, IOException {
	ConnectionFactory factory = new ConnectionFactory();
	factory.setHost("10.211.55.20");
	factory.setPort(5672);
	factory.setVirtualHost("/");
	factory.setUsername("guest");
	factory.setPassword("guest");
	Connection connection = factory.newConnection();
	Channel channel = connection.createChannel();
 
	channel.exchangeDeclare("KipcastDirect", "direct", 
		   true, 	/* durable */
		   true, 	/* autodelete */
		   null); 	/* */
 
	byte[] messageBodyBytes = "Hello, world!".getBytes();
 
	AMQP.BasicProperties.Builder basic = new AMQP.BasicProperties.Builder();
	AMQP.BasicProperties minBasic = basic.build();
 
	minBasic = basic.priority(0).deliveryMode(1).build();
 
	channel.basicPublish("KipcastDirect", "KipcastRouting", minBasic, messageBodyBytes);
	System.out.println(" [x] Sent ");
 
	channel.close();
}

To test if listener works, for first run the queue listener and, after that, run the Junit class to send message. The output will be:

[pache.camel.spring.Main.main()] MainSupport                    INFO  Apache Camel 2.10.3 starting
[pache.camel.spring.Main.main()] SpringCamelContext             INFO  Apache Camel 2.10.3 (CamelContext: camel-1) is starting
[pache.camel.spring.Main.main()] ManagementStrategyFactory      INFO  JMX enabled.
[pache.camel.spring.Main.main()] DefaultTypeConverter           INFO  Loaded 177 type converters
[pache.camel.spring.Main.main()] SpringAMQPComponent            INFO  Found AMQP ConnectionFactory in registry for 10.211.55.20
[pache.camel.spring.Main.main()] SpringAMQPComponent            INFO  Found AMQP Template in registry
[pache.camel.spring.Main.main()] SpringAMQPComponent            INFO  Found AMQP Administrator in registry
[pache.camel.spring.Main.main()] SpringAMQPEndpoint             INFO  Creating endpoint for KipcastDirect:KipcastQueue:KipcastRouting
[pache.camel.spring.Main.main()] SpringAMQPConsumer             INFO  Declared exchange KipcastDirect
[pache.camel.spring.Main.main()] SpringAMQPConsumer             INFO  Declared queue KipcastQueue
[pache.camel.spring.Main.main()] SpringAMQPConsumer             INFO  Declaring binding KipcastRouting
[pache.camel.spring.Main.main()] SpringAMQPConsumer             INFO  Started AMQP Async Listeners for spring-amqp://KipcastDirect:KipcastQueue:KipcastRouting?autodelete=true&amp;durable=true&amp;type=direct
[pache.camel.spring.Main.main()] SpringCamelContext             INFO  Route: route1 started and consuming from: Endpoint[spring-amqp://KipcastDirect:KipcastQueue:KipcastRouting?autodelete=true&amp;durable=true&amp;type=direct]
[pache.camel.spring.Main.main()] ultManagementLifecycleStrategy INFO  StatisticsLevel at All so enabling load performance statistics
[pache.camel.spring.Main.main()] SpringCamelContext             INFO  Total 1 routes, of which 1 is started.
[pache.camel.spring.Main.main()] SpringCamelContext             INFO  Apache Camel 2.10.3 (CamelContext: camel-1) started in 0.505 seconds
[     SimpleAsyncTaskExecutor-1] route1                         INFO  Message available on a RabbitMQ Queue

At this point, you can have fun with Camel and RabbitMQ!!!!!

NOTES:

1 – Please be careful: the URI (from and to) on Camel Spring DSL context and JUnit class must refer to same Exchange and Queue to prevent a reply-text=PRECONDITION_FAILED – parameters for queue ‘QUEUE’ in vhost ‘/’ not equivalen error or similar. To check the queues/exchanges configuration parameter using:

rabbitmqadmin -V / list queue
rabbitmqadmin -V test list exchanges

if you like this post, please click on the advertise :)

Tagged , ,

How to get Color Sidebar icons back in MacOS 10.7+

I was really sad looking at my gray finder sidebar. I thought that this release of Finder was designed by some engineer from Soviet Union or North Korea but, finally, I found a solution to get back my colored finder and also my smile!

Take a look to this link: http://osxdaily.com/2011/08/25/get-color-sidebar-icons-back-in-mac-os-x-10-7-lion-finder-windows/

Tagged

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

If the database is running, probably the database hasn’t registered yet with the listener. This occurs when the database or listener just starts up. To fix, connect as sysdba:

SHOW parameter LOCAL listener

If something or nothing appears, reset it with:

ALTER system SET local_listener='(address=(protocol=tcp)(host=localhost)(port=1521))';
ALTER system register;

At this point test if it’s working fine:

tnsping xe
 
TNS Ping Utility for Linux: Version 11.2.0.2.0 - Production on 26-NOV-2012 17:04:16
 
Copyright (c) 1997, 2011, Oracle.  All rights reserved.
 
Used parameter files:
 
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))
OK (0 msec)
Tagged ,

Listener refused the connection with the following error:12505

No listener.ora file is required for LISTENER to operate. Try to move, rename, remove, or delete listener.ora file & then restart the listener.

Every Oracle DB tries once every 60 seconds to register itself with LISTENER as shown below:

tail -f /u01/app/oracle/product/11.2.0/xe/log/diag/tnslsnr/XXXXX/listener/alert/log.xml 
<msg time='2012-10-25T01:15:50.522-07:00' org_id='oracle' comp_id='tnslsnr'
 type='UNKNOWN' level='16' host_id='host.domain'
 host_addr='XX.XX.XX.XX'>
 <txt>25-OCT-2012 01:15:50 * service_update * XE * 0
 </txt>
</msg>
Tagged

Row count for all tables in schema

Counting all of the rows in a schema can require code that actually counts the table rows so, how do you count up all of the rows for all tables in a schema?

For Oracle with system user:

SELECT TABLE_NAME, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = UPPER('{schema}');

For mySql:

SELECT TABLE_NAME, table_rows FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = '{schema}';
Tagged ,

Put SQL result into a map

To extract two or more object (in this sample two Integer) from a DB and putting into a map, use this:

final Map<Integer,Integer> topic = new HashMap<Integer,Integer>();
 
getJdbcTemplate().query(sqlCommand, new Object[] {query_parameter}, new RowMapper<Object>(){
	public Object mapRow(ResultSet rs, int arg1) throws SQLException {
		topic.put(rs.getInt("integer1"), rs.getInt("integer2"));
		return null;
	}
});

To read all values:

Iterator<Map.Entry<Integer, Integer>> users = userScheduledToday.entrySet().iterator();
while ( users.hasNext() ) {
	Map.Entry<Integer, Integer> entry = users.next();
	logger.debug("Key: " + entry.getKey() + " - Value: " + entry.getValue());
}
Tagged , ,

Linux: mow much RAM is used per program

The script is a very helpful to show how much RAM is used by single process. This script is written by Pádraig Brady, He can be contacted at P@draigBrady.com. The script itself follows:

#!/usr/bin/env python
 
# Try to determine how much RAM is currently being used per program.
# Note the per program, not per process. So for example this script
# will report mem used by all httpd process together. In detail it reports:
# sum(all RSS for process instances) + max(shared mem for any process instance)
#
# The shared calculation below will factor out shared text and
# libs etc. within a program, but not between programs. So there
# will always be some overestimation. This will be the same for
# all processes that just use libc for e.g. but more for others
# that use larger shared libs like gnome, kde etc.
 
# Author: P@draigBrady.com
 
# V1.0      06 Jul 2005    Initial release
# V1.1      11 Aug 2006    root permission required for accuracy
# V1.2      08 Nov 2006    Add total to output
#                          Use KiB,MiB,... for units rather than K,M,...
# V1.3      22 Nov 2006    Ignore shared col from /proc/$pid/statm for
#                          2.6 kernels up to and including 2.6.9.
#                          There it represented the total file backed extent
# V1.4      23 Nov 2006    Remove total from output as it's meaningless
#                          (the shared values overlap with other programs).
#                          Display the shared column. This extra info is
#                          useful, especially as it overlaps between programs.
# V1.5      26 Mar 2007    Remove redundant recursion from human()
# V1.6      05 Jun 2007    Also report number of processes with a given name.
#                          Patch from riccardo.murri@gmail.com
 
# Notes:
#
# All interpreted programs where the interpreter is started
# by the shell or with env, will be merged to the interpreter
# (as that's what's given to exec). For e.g. all python programs
# starting with "#!/usr/bin/env python" will be grouped under python.
# You can change this by changing comm= to args= below but that will
# have the undesirable affect of splitting up programs started with
# differing parameters (for e.g. mingetty tty[1-6]).
#
# For 2.6 kernels up to and including 2.6.13 and later 2.4 redhat kernels
# (rmap vm without smaps) it can not be accurately determined how many pages
# are shared between processes in general or within a program in our case:
# http://lkml.org/lkml/2005/7/6/250
# A warning is printed if overestimation is possible.
# In addition for 2.6 kernels up to 2.6.9 inclusive, the shared
# value in /proc/$pid/statm is the total file-backed extent of a process.
# We ignore that, introducing more overestimation, again printing a warning.
#
# I don't take account of memory allocated for a program
# by other programs. For e.g. memory used in the X server for
# a program could be determined, but is not.
#
# This script assumes threads are already merged by ps
 
# TODO:
#
# use ps just to enumerate the pids and names
# so as to remove the race between reading rss and shared values
 
import sys, os, string
 
if os.geteuid() != 0:
    sys.stderr.write("Sorry, root permission required.\n");
    sys.exit(1)
 
PAGESIZE=os.sysconf("SC_PAGE_SIZE")/1024 #KiB
our_pid=os.getpid()
 
#(major,minor,release)
def kernel_ver():
    kv=open("/proc/sys/kernel/osrelease").readline().split(".")[:3]
    for char in "-_":
        kv[2]=kv[2].split(char)[0]
    return (int(kv[0]), int(kv[1]), int(kv[2]))
 
kv=kernel_ver()
 
def getShared(pid):
    if os.path.exists("/proc/"+str(pid)+"/smaps"):
        shared_lines=[line
                      for line in open("/proc/"+str(pid)+"/smaps").readlines()
                      if line.find("Shared")!=-1]
        return sum([int(line.split()[1]) for line in shared_lines])
    elif (2,6,1) <= kv <= (2,6,9):
        return 0 #lots of overestimation, but what can we do?
    else:
        return int(open("/proc/"+str(pid)+"/statm").readline().split()[2])*PAGESIZE
 
cmds={}
shareds={}
count={}
for line in os.popen("ps -e -o rss=,pid=,comm=").readlines():
    size, pid, cmd = map(string.strip,line.strip().split(None,2))
    if int(pid) == our_pid:
        continue #no point counting this process
    try:
        shared=getShared(pid)
    except:
        continue #ps gone away
    if shareds.get(cmd):
        if shareds[cmd] < shared:
            shareds[cmd]=shared
    else:
        shareds[cmd]=shared
    #Note shared is always a subset of rss (trs is not always)
    cmds[cmd]=cmds.setdefault(cmd,0)+int(size)-shared
    if count.has_key(cmd):
      count[cmd] += 1
    else:
      count[cmd] = 1
 
#Add max shared mem for each program
for cmd in cmds.keys():
    cmds[cmd]=cmds[cmd]+shareds[cmd]
 
sort_list = cmds.items()
sort_list.sort(lambda x,y:cmp(x[1],y[1]))
sort_list=filter(lambda x:x[1],sort_list) #get rid of zero sized processes (kernel threads)
 
#The following matches "du -h" output
#see also human.py
def human(num, power="Ki"):
    powers=["Ki","Mi","Gi","Ti"]
    while num >= 1000: #4 digits
        num /= 1024.0
        power=powers[powers.index(power)+1]
    return "%.1f %s" % (num,power)
 
def cmd_with_count(cmd, count):
    if count>1:
      return "%s (%u)" % (cmd, count)
    else:
      return cmd
print " Private  +  Shared  =  RAM used\tProgram \n"
for cmd in sort_list:
    print "%8sB + %8sB = %8sB\t%s" % (human(cmd[1]-shareds[cmd[0]]), human(shareds[cmd[0]]), human(cmd[1]),
                                      cmd_with_count(cmd[0], count[cmd[0]]))
print "\n Private  +  Shared  =  RAM used\tProgram \n"
 
#Warn of possible inaccuracies
#1 = accurate
#0 = some shared mem not reported
#-1= all shared mem not reported
def shared_val_accurate():
    """http://wiki.apache.org/spamassassin/TopSharedMemoryBug"""
    if kv[:2] == (2,4):
        if open("/proc/meminfo").read().find("Inact_") == -1:
            return 1
        return 0
    elif kv[:2] == (2,6):
        if os.path.exists("/proc/"+str(os.getpid())+"/smaps"):
            return 1
        if (2,6,1) <= kv <= (2,6,9):
            return -1
        return 0
    else:
        return 1
 
vm_accuracy = shared_val_accurate()
if vm_accuracy == -1:
    sys.stderr.write("Warning: Shared memory is not reported by this system.\n")
    sys.stderr.write("Values reported will be too large.\n")
elif vm_accuracy == 0:
    sys.stderr.write("Warning: Shared memory is not reported accurately by this system.\n")
    sys.stderr.write("Values reported could be too large.\n")
Tagged , ,

Send message from C# to rabbitMQ

For first download and install the .NET/C# AMQP client library

On Visual Studio, include the RabbitMQ.Client reference import the library into the class:

using RabbitMQ.Client;
using RabbitMQ.Client.Framing.v0_8;

At this point initialise the queue and connection to the RabbitMQ broker in the constructor:

var connectionFactory = new ConnectionFactory();
connectionFactory.HostName = HostName;
Connection = connectionFactory.CreateConnection();
Model = Connection.CreateModel();
Model.QueueDeclare(QueueName, false, false, false, null);

Now the connection factory and model are definied and is possible to send a message to the queue called “QueueName”:

IBasicProperties basicProperties = Model.CreateBasicProperties();
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
Model.BasicPublish("", QueueName, basicProperties, encoding.GetBytes(message));

Note that BasicPublish use byte[] for the message. The parameters are:

BasicPublish(String exchange, String routingKey, IBasicProperties basicProperties, byte[] body)
Tagged , ,