Tag Archives: MongoDB

Return Distinct Values for an Array Field

The following example returns the distinct values for the field sizes from all documents in the Collection1 collection:

db.getCollection('inventory').distinct('storeCode')

This will be the expected result:

[
     "1502", "1002", "747"
]

Distinct, as for relational database, finds the distinct values for a specified field across a single collection or view and returns the results in an array.

Tagged

Mongo Replica set with docker-compose

replica set in MongoDB is a group of mongod processes that maintain the same data set. Replica sets provide redundancy and high availability and are the basis for all production deployments. Replication provides redundancy and increases data availability. With multiple copies of data on different database servers, replication provides a level of fault tolerance against the loss of a single database server. A replica set contains several data bearing nodes and optionally one arbiter node. Of the data-bearing nodes, one and only one member is deemed the primary node, while the other nodes are deemed secondary nodes.

The primary node receives all write operations. A replica set can have only one primary capable of confirming writes with { w: "majority" } write concern. By default, clients read from the primary [1]; however, clients can specify a read preference to send read operations to secondaries.

Doing it with docker-compose is pretty simple. The first step is to create the docker-compose.yml configuration file:

version: "3"
services:
  mongo1:
    hostname: mongo1
    container_name: localmongo1
    image: mongo:latest
    volumes:
      - mongodb1-data:/data/db
      - mongodb1-config:/data/configdb
    expose:
    - 27017
    ports:
      - 27011:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]
  mongo2:
    hostname: mongo2
    container_name: localmongo2
    image: mongo:latest
    volumes:
      - mongodb2-data:/data/db
      - mongodb2-config:/data/configdb
    expose:
    - 27017
    ports:
    - 27012:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]
  mongo3:
    hostname: mongo3
    container_name: localmongo3
    image: mongo:latest
    volumes:
      - mongodb3-data:/data/db
      - mongodb3-config:/data/configdb
    expose:
    - 27017
    ports:
    - 27013:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]

volumes:
  mongodb1-data: {}
  mongodb1-config: {}
  mongodb2-data: {}
  mongodb2-config: {}
  mongodb3-data: {}
  mongodb3-config: {}
  proftpd-home: {}

At this point, docker must be started:

$ docker-compose up 
# or
$ docker-compose up -d

At this point, enter in one mongo bash and access to mongo console:

$ docker exec -it localmongo1 /bin/bash
$ mongo

The last step is to run the DB replica set initialization:

rs.initiate(
  {
    _id : 'rs0',
    members: [
      { _id : 0, host : "mongo1:27017" },
      { _id : 1, host : "mongo2:27017" },
      { _id : 2, host : "mongo3:27017" }
    ]
  }
)

Now, mongo is ready to accept a connection on port 27011 and, as soon as a DB / collection / document is created or updated, it will be replicated on secondary servers.

Tagged ,

Find on different collections to create a document using mongoose

I need to query multiple collections to prepare a MongoDB document and then save it using Mongoose and NodeJS. The solution is to use async.parallel
I have two source collections, Robot, Target and the destination collection Activity.

For first, async must be added:

var async = require('async');

then:

async.parallel({
 
    robotFind: function(cb) { Robot.find({ "_id": jsonContent.robotId }).exec(cb); },
    targetFind: function(cb) { Target.find({ "_id": jsonContent.targetId }).exec(cb); }
 
}, function(err, result) {
 
    activity.robot = result.robotFind[0];
    activity.action = result.actionFind[0];
    activity.target = result.targetFind[0];
 
    activity.execution_date = jsonContent.execution_date;
    activity.alert = jsonContent.alert;
    activity.result = executionResult;
 
    activity.description = jsonContent.description;
 
    activity.save(function(err) {
        if (err) {
            console.log('[postActivity] ' + err)
            res.status(500).json({ error: err.message })
        } else {
            console.log('[postActivity] Saved!')
            res.status(200).json({ message: activity })
        }
    })
}

So, first part queries the MongoDB and fill the object result. Second part consumes result object, create the new document and save it.

PS: if you like it please, click on the banner :)

Tagged , ,

Date based query using milliseconds (Java long) time on MongoDB

Let’s suppose I need to search all records that match a date condition. On MongoDB I’ve a bunch of data like this:

{
    "_id" : "9ed3b937-0f43-4613-bd58-cb739a8c5bf6",
    "userModels" : {
        "5080" : {
            "generated_date_timestamp" : NumberLong(1413382499442),
            "model_id" : 5080,
        },
    }
    "values" : {}
}

This is the query:

db.anonProfile.find({ 
   "userModels.5080.generated_date_timestamp" : { 
      "$gte" : ISODate("2013-10-01T00:00:00.000Z").getTime() 
   }
});
.getTime()

allows to translate ISODate into a NumberLong timestamp.

Tagged

Query date based using milliseconds time on MongoDB

I need to search all records that match a date condition. On MongoDB I’ve a bunch of data like this:

{
    "_id" : "9ed3b937-0f43-4613-bd58-cb739a8c5bf6",
    "userModels" : {
        "5080" : {
            "generated_date_timestamp" : NumberLong(1413382499442),
            "model_id" : 5080,
        },
    }
    "values" : {}
}

With this query, is possible to do date / time based search:

db.anonProfile.find({ 
   "userModels.5080.generated_date_timestamp" : 
      { 
         "$gte" : ISODate("2013-10-01T00:00:00.000Z").getTime() 
      }
});

Tagged

Results pagination with MongoDB and Java

To implement the MongoDB results pagination in Java, there are few parameters that need to be collected:

1) the order of results (ascending / descending)
2) the field used to order the results
3) the number of element and the page selected
4) the field and the value used to filter the results

As well as the results of query, the method needs to return the total number of elements. All returned elements will be saved in a HashMap.

HashMap<String, Object> resultMap = new HashMap<String, Object>();
 
Direction direction = Sort.DEFAULT_DIRECTION;
if (sortDirection > 0) {
	direction = Sort.Direction.ASC;
} else { 
	direction = Sort.Direction.DESC;
}
 
List

If a pagination is required, skip and limit are used

if (pageSize > 0) {
	query.skip((pageNum - 1 ) * pageSize);
	query.limit(pageSize);
}
 
if ( sortField != null && !sortField.equals("") ) {				
	query.with(new Sort(direction, sortField));
}
 
results = mongoTemplate.find(query, Object.class);

If a pagination is required, queryCounter (basically a version of query without pagination an limit) is used to calculate the total number of results. Of course, if pagination is not required, is possible to use directly the size of results.

if ( pageSize > 0 ) {
	resultMap.put("RESULT_SIZE", (int) mongoTemplate.count(queryCounter, Object.class));
} else {
	// If pagination is not required, the query is not re-executed
	resultMap.put("RESULT_SIZE", results.size());
}

mongoTemplate is a spring bean defined in this way on context configuration:

<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"  c:mongo-ref="mongo" c:databaseName="${mongo.db.name}">
	<property name="writeConcern" value="SAFE" />
</bean>

Tagged ,

MongoDB query with logical and condition in Java

Suppose you need to apply some filters to your MongoDB query, for example to extract some _ids that match a regex condition. This is the way to do that:

Query query;
query.addCriteria(Criteria.where("_id").in(IDs).and(query_field).regex(".*" + query_value + ".*", "i"));

In this example I used the Query (see here) and Criteria (see here) classes

And, this is the query you can use in mongoDB (Robomongo* or command line):

Query: { 
	"_id" : { "$in" : [ "ID1" , "ID2" , "ID3" ]} , 
	"detail.medicationBrandName" : { "$regex" : ".*x.*" , "$options" : "i"}}, 
	Fields: null, Sort: { "medicationGenericName" : -1}
}

*Robomongo: is a shell-centric cross-platform open source MongoDB management tool (i.e. Admin GUI). Robomongo embeds the same JavaScript engine that powers MongoDB’s mongo shell. You can download it here.

Tagged , ,

Spring-data: Cannot use a complex object as a key value

I was trying to figured out how to solve this issue. Basically I’m saving a user profile bean that contains multiple occurrences of other beans inside him. Something like date:

public class UserProfile {
 
	List<Class1> classes1;
	List<Class2> classes2;
 
	Integer int1;
	Map<String, Class3> classes3;
 
}

I was working fine until I changed a getter adding some business functionalities and surrounding all lines with a try / catch. Of course, I putted the exception message on the log using adding this definition inside the bean:

protected final Log logger = LogFactory.getLog(getClass());

Good point for me, everything worked fine until I updated the object in Mongo. Holy crap, I got the “Cannot use a complex object as a key value” exception and I spent two hours trying to change the beans (of course I did a lot of other changes after the fatal adding of business logic inside my bean). Exhausted, I contacted my colleague Max (@maxfarnea) and he told me, “when I got this error, I removed the log inside a bean”. Ipso facto, I removed the log entry from my bean, added a throws exception instead of a try / catch and tested! Well done @maxfarnea, it works!

I learned two lesson today: one, don’t put log into beans that need to be stored using spring-data and, two, if you are in pain, ask, talk, don’t be silly, don’t waste your time!!! Stackoverflow.com is a good point to start (and my blog either, of course) but never is more helpful than a quick chat with a colleague!

Tagged , ,

Route a message to MongoDB

The requirement is very simple. Route an XML message from rabbitMQ to MongoDB. MongoDB BSON as the data storage and network transfer format for “documents”. BSON is a binary-encoded serialization of JSON-like documents. So, the source message is in a XML format, after getting it from rabbitMQ is necessary to translate into a JSON format compatible with MongoDB.

To translate a message from XML to JSON is possible to use Marshalling function available in Camel extension XmlJson. The pom.xml file needs this new entry:

<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-xmljson</artifactId>
	<version>${org.camel.version}</version>
</dependency>

Of course, to send communicate with MongoDB another entry has to be added to pom.xml:

<dependency>
	<groupId>org.apache.camel</groupId>
	<artifactId>camel-mongodb</artifactId>
	<version>${org.camel.version}</version>
</dependency>

Now, is possible to change the route on camel ApplicationContext file:

<!-- Camel route -->    
<camelContext xmlns="http://camel.apache.org/schema/spring">
	<dataFormats>
		<xmljson id="xmljson"/>
	</dataFormats>
	<route> 
		<from uri="spring-amqp:TPDirect:TPQueue:TPRouting?type=direct&amp;autodelete=true&amp;durable=true" />
		<marshal ref="xmljson"/>
		<log message="From XML to Json: DONE!" />
		<convertBodyTo type="java.lang.String"/>
		<to uri="mongodb:myDb?database=flights&amp;collection=tickets&amp;operation=save" />
	</route>                                 
</camelContext>
 
 
<!-- Mongo DB -->
<bean id="myDb" class="com.mongodb.Mongo">
	<constructor-arg index="0" value="localhost"/>
</bean>

Bean myDB contains the information to reach MongoDB. It’s also possible to define it using the full url:

<bean id="myDb" class="com.mongodb.Mongo"> 
    <constructor-arg index="0"> 
        <bean class="com.mongodb.MongoURI"> 
            <constructor-arg index="0" value="mongodb://username:password@host:port/db" /> 
        </bean> 
    </constructor-arg> 
</bean>

To avoid this exception:

Caused by: No type converter available to convert from type: byte[] to the required type: org.apache.camel.component.mongodb.converters.MongoDbBasicConverters with value

The JSON translated message has to be converted into String.

That’s all!

Tagged , , ,