Jb in a nutshell: Category db4o

LINQ Expression Trees on the Compact Framework

Jb Evain — Mon, 22 Sep 2008 19:59:00 +0200

A week ago, I was writing about the ability to use Mono’s System.Core to run C# 3 and LINQ applications on .net 2.

A few months ago, I was writing about the new LINQ provider that friends at db4o released. This LINQ provider uses LINQ expression trees to optimize LINQ queries, turning them into db4o’s native query language.

Today I’m going to write about a cute new hack from our friends at db4o.

If the 3.5 version of the Compact Framework comes with a System.Core, and a LINQ to objects implementation, it doesn’t contain the namespace System.Linq.Expressions, which is used by all optimized LINQ providers. What those fine folks did, was simply to take Mono’s implementation of this namespace. There was one issue with that approach. Expression Trees can be compiled at runtime, and that uses System.Reflection.Emit, that the Compact Framework doesn’t have.

A nice thing is that folks at Mainsoft contributed to Mono’s System.Core an expression interpreter, which allows you to use the full LINQ expression trees without using System.Reflection.Emit. They sent me the patches they have for the expression interpreter to make it pass the full test suite we have, so I’ll integrate it shortly.

And voilà, just by using Mono’s implementation of System.Core, and by hacking a little in it, to make it compile on the Compact Framework, they’ve been able to produce a new assembly, System.Linq.Expressions.dll. You just have to reference it in your Compact Framework project, and it will let you use LINQ expression trees, which allows you to write optimized LINQ providers for the Compact Framework.

Of course, this is completely open-source and licensed under the MIT/X11. For now, you can find the code in db4o’s repository, but I’ll backport the changes they did to the LINQ expression interpreter, so that people that want to use it can use our code base directly.

Congrats to you guys for making the finest choice in the third party code you ship!

Sharpen, an Open Source Java to C# converter

Jb Evain — Tue, 20 May 2008 21:18:00 +0200

Today folks at db4o released Sharpen, an open source Java to C# converter.

Am happy to see this coming, as it is soming I’ve pushed strongly for when I was working at db4objects, and was hacking on it, mentored by its main developer, Rodrigo. There I had the chance to improve the generated code, to make it look like hand written code, that follows .net conventions.

It’s nothing magic, and requires compromises on both the Java and the generated C#, but it’s pretty handy, easily hackable, and all in all, damn useful.

Rodrigo wrote a nice tutorial on Sharpen.

Time to port your Java applications to the amazing Mono!

An elegant LINQ to db4o provider, and a few LINQ tricks

Jb Evain — Wed, 06 Feb 2008 18:30:00 +0100

Last year, when I was working at db4objects on db4o, I insisted on the need for db4o to act as a LINQ provider, and with that in mind, prototyped one. I’am happy today to see a LINQ provider making it to db4o’s SVN, especially as I had the chance to influence its design, as I was also researching LINQ for Mono. For that matters, I find LINQ to db4o’s design particularly elegant.

I’ll use that occasion to describe a few LINQ tricks. Hopefully, the folks at db4o won’t mind if I steal the show. Actually I think it will even unburden my good friend Rodrigo, who’s apparently tired of blogging.

Making a type queryable with LINQ

Let’s take db4o’s example. What you do first is getting an IObjectContainer, with the idea to store and retrieve objects.

using (var container = Db4oFactory.OpenFile ("data.db4o")) {
    // ...
}

So the goal was to be able to write somethings either like:

var peoples = from container.Query<Person> ()
             where p.Name == "jb" && p.Age == 25
             select p

or like:

var peoples = from Person p in container
             where p.Name == "jb" && p.Age == 25
             select p

There’s a few things that we didn’t like with the first method. First an IObjectContainer already have a Query<T> method, which is used for native queries, and secondly, aesthetically, it doesn’t look as good as the second one.

So we went for the second possibility. The issue is that IObjectContainer doesn’t implement any interface that the compiler could use to perform LINQ queries on it.

The trick here is that when the compiler encounters the form:

from Person p in container

It will try to compile it as:

container.Cast<Person> ()

And expect that this Cast<T> returns an IEnumerable<T>, on which he can perform LINQ query. The good news is that this Cast method can very well be an extension method.

With that in mind, the entry point of LINQ to db4o is really simple:

public static IDb4oLinqQuery<T> Cast<T>(this IObjectContainer container)
{
    return new Db4oQuery<T>(container);
}

Now comes the question of what is this IDb4oLinqQuery<T>.

Implementing a lightweight LINQ provider

LINQ is still a pretty new technology, so it’s not always easy to find good advices from people that are already experienced in the fine art of writing LINQ providers. If I don’t have the pretension to be one of those experienced guys, I know of at least one folk who used the following trick, and which is happy with it.

So today, the standard way of writing a LINQ provider seems to be implementing the interface System.Linq.IQueryable, and to implement a full fledged LINQ provider. Matt Warren have written an excellent series of posts about it. LINQ to db4o uses some code from his posts. Bart de Smet also wrote a nice one.

My very first prototype was also following this path, but at that time, we changed it quickly for an excellent reason.

Thing is that the IQueryable path is an all or nothing solution. It keeps building an expression tree, until you decide to work on its result, either by calling GetEnumerator, or by calling AsEnumerable on it.

Another thing is that db4o is only able to optimize a (comparatively with something like LINQ to SQL) very little subset of the possible LINQ operations. Actually, it can really optimize only Where, Count, OrderBy and ThenBy.

And again, here comes the magic of the extension methods. All LINQ operations are actually extension methods on IEnumerable<T>, or on a child of it, like IQueryable<T>.

The trick here, was simply to re-define the supported LINQ operations, on a custom interface. And that’s what the IDb4oLinqQuery interface we talked before is. It’s simply a marker interface:

public interface IDb4oLinqQuery<T> : IEnumerable<T>
{
}

And thanks to the C#3 magic, we can redefine the LINQ operations on it:

public static class Db4oLinqQueryExtensions
{
    public static IDb4oLinqQuery<TSource> Where<TSource>(this IDb4oLinqQuery<TSource> self, Expression<Func<TSource, bool>> expression)
    {
        // ..
    }

    public static int Count<TSource>(this IDb4oLinqQuery<TSource> self)
    {
        // ..
    }
}

The major advantage of this solution, is that it allows the LINQ query to fallback to LINQ to Objects, anytime a LINQ operations is not supported. So by default, is something cannot be optimized, the LINQ to db4o will return all the types for a certain type, and as we fall back to LINQ to Objects, the user will at least get the result it wanted, even if it doesn’t run at full speed. I think it’s something that makes this solution really elegant.

Let’s take a detailed example:

var peoples = from Person p in container
             where p.Name == "jb" 
             orderby p.Age
             select new { p.Age }

The compiler here sees four steps.

First it use the first Cast trick, that creates an IDb4oLinqQuery of T,
Secondly, it call the Where methods defined for IDb4oLinqQuery,
Then it does the same for OrderBy, which can be optimized,
And then only, as db4o cannot optimize the creation of an anonymous type, it will fall back to LINQ to Objects, but only for the select part, while the two first parts run completely optimized, and almost as fast as the db4o low level query mechanism.

That’s really nice if you ask me.

Using an expression tree as a dictionary key

That one is simple.

LINQ to db4o caches how a query is created, depending on the expression tree it gets. Thing is that System.Linq.Expression doesn’t override Equals and GetHashCode, and as the compiler re-creates a whole new tree every time it sees one, hence Expressions are poor candidate for being keys in Dictionary. The trick here is simply to have an IEqualityComparer for Expressions. It’s seems that it’s not trivial to implement, but they managed to get a pretty nice one.

Analyzing properties body’s

This trick is also a simple one, and it’s the same one that db4o uses for its Native Query functionality, LINQ to db4o uses the Cecil.FlowAnalysis library to read the body of the properties, so they know which fields are impacted by the LINQ query.

That’s all folks

Kuddos to db4o for such a nice addition, hopefully it’ll run on Mono soon!

It's not Linux, it's COCA-COLA!

Jb Evain — Tue, 27 Feb 2007 09:49:00 +0100

FOSDEM was really nice!

You can find my db4o presentation for my lightning talk here.

Interviews

Jb Evain — Fri, 09 Feb 2007 15:04:00 +0100

Here are two interviews in French I gave after my talk at Microsoft’s TechDays in Paris.

on Mono.
on db4o.

It has been the best way for me to realize that I really need an haircut. Now.

My friend Patrice and his team are uploading all their interviews here.

Back from Hong-Kong

Jb Evain — Sat, 20 Jan 2007 16:57:00 +0100

It’s been five days since I’m back from Hong-Kong, where I was attending the db4o Developer Conference, and I had wonderful time there. After such a nice trip, it’s really hard to be excited by the idea of returning home, in Mulhouse, France.

You’re welcome to visit my flickr set for this trip

Now I have to catch up with work, pending patches, and life :)

Interview of Christof Wittig, CEO of db4objects, Inc.

Jb Evain — Mon, 04 Dec 2006 16:31:00 +0100

My good friend Patrice interviewed my CEO, Christof Wittig during the db4o RoadShow. It’s a very nice introduction to what is db4o, and to how we are working on it. Go see the video!

db4o 6.0 released

Jb Evain — Tue, 14 Nov 2006 18:10:00 +0100

I’ve very thrilled to announce the release today of db4o 6.0, our brand new release.

Some lucky folks have seen at the Mono Meeting some bits of what’s new in it. Rodrigo and I’ve been mainly busy bringing some love to the public API, to make it look like a real .net/Mono library. And now that’s what it really is.

using Db4objects.Db4o;
using Db4objects.Db4o.Query;

class Test
{
    public static void Main()
    {
        using (IObjectContainer store = Db4oFactory.OpenFile("test.yap"))
        {
            IQuery q = store.Query();
            q.Constrain(typeof(Person));

            foreach (Person p in q.Execute())
            {
                Console.WriteLine (p);
            }
        }
    }
}

Here is the sample extracted from my blog post where I describe all the changes.

That’s the release I’ve been the more excited with. You can feel that it’s all packed with love. Go give it a try.

Mono Meeting

Jb Evain — Wed, 13 Sep 2006 19:31:00 +0200

Thanks to db4o who is sponsoring my trip, I’ll be attending the Mono Meeting at the end of October. I’ll have two talks here:

One about db4o, the leading object database,
The other about Cecil, our assembly manipulation library.

See you there!

db4objects is Hiring

Jb Evain — Tue, 23 May 2006 14:18:00 +0200

We have a bunch of open positions here at db4objects.

We are mostly looking for a developer to handle the replication and synchronization part of db4o.

If you are willing to work on an Open Source product, with lots of exciting technologies, and good looking developers ;), here are the open positions.