Showing posts with label mongo. Show all posts
Showing posts with label mongo. Show all posts

Tuesday, January 27, 2015

Reading mongodump bson file from Spark in scala using mongo-hadoop

I couldn't find a complete Scala version using mongo-hadoop v1.3.1 to read a mongodump bson file, so here's one I prepared earlier:

val bsonData = sc.newAPIHadoopFile(
"file:///your/file.bson",
classOf[com.mongodb.hadoop.BSONFileInputFormat].asSubclass(classOf[org.apache.hadoop.mapreduce.lib.input.FileInputFormat[Object, org.bson.BSONObject]]),
classOf[Object],
classOf[org.bson.BSONObject])


Note that (for v1.3.1) we need to subclass com.mongodb.hadoop.BSONFileInputFormat to avoid this compilation error: "inferred type arguments do not conform to method newAPIHadoopFile's type parameter bounds".  This isn't required if reading from Mongo directly using com.mongodb.hadoop.MongoInputFormat.

Also, you can pass a Configuration object as a final parameter if you need to set any specific conf values.

For more bson examples see here: https://github.com/mongodb/mongo-hadoop/blob/master/BSON_README.md

For Java examples see here: http://crcsmnky.github.io/2014/07/13/mongodb-spark-input/

Wednesday, November 28, 2012

Decode a mongo BinData field into a Guid with C#

This C# snippet will decode a mongo BinData field into a Guid

got this from Mongo:
{
  "_id" : ObjectId("50b40a5db14ea8902cd5d5ed"),
   "MyGuid : new BinData(3, "MC/0P9nkEeGgjYQrK2V3pQ=="),

   ...
}

but want to know the value of MyGid?  Try this little snippet:

using System;
namespace MongoGuidDecoder
{
    class Program
    {
        static void Main(string[] args)
        {
            var input = args[0];
            var bytes = Convert.FromBase64String(input.Trim());
            var guid = new Guid(bytes);
            Console.WriteLine(input);
            Console.WriteLine(guid);
        }
    }
}


Now run to get the Guid representation:
>MongoGuidDecoder MC/0P9nkEeGgjYQrK2V3pQ==
MC/0P9nkEeGgjYQrK2V3pQ==
3ff42f30-e4d9-e111-a08d-842b2b6577a5