Jb in a nutshell

The Thing About ExpressionVisitor

Jb Evain — Tue, 16 Dec 2008 14:25:00 +0100

There was a discussion recently about why the class ExpressionVisitor, which every programmer that developed a LINQ provider or code manipulating .net 3.5 expression trees knows about.

I certainly agree that this class could have been made public, as basically everyone ends up embedding it. Even if it could use a little refactoring to visit lists.

Thing is that I have a naming issue with this class ever since I saw it on Matt’s blog. To me it does more than just visiting an expression tree. It allows you to transform it. And when I’m facing an expression tree, I often like to sometimes visit it, and at some other times, transform it.

That’s basically the reason why I have both a:

class ExpressionVisitor {

    public void Visit (Expression expression)
    {
        // ..
    }

    // ...
}

And a:

class ExpressionTransformer {

    public Expression Visit (Expression expression)
    {
        // ..
    }

    // ...
}

Simply because I write a lot of code that would look like:

SomethingExpression VisitSomethingExpression (SomethingExpression expression)
{
    if (Foo (expression))
        return expression;

    if (Bar (expression))
        return expression;

    Baz (expression);
    return expression;
}

And it looks a lot better (and is also a tiny bit faster) this way:


void VisitSomethingExpression (SomethingExpression expression)
{
    if (Foo (expression))
        return;

    if (Bar (expression))
        return;

    Baz (expression);
}

So if they make the current ExpressionVisitor public, having a real non transforming ExpressionVisitor will introduce an interesting naming issue.

I submitted a bug to the DLR CodePlex, as the namespace System.Linq.Expressions will come from the DLR in .net 4. What about voting for it?

Decompiler CodeCamp

Jb Evain — Tue, 16 Dec 2008 10:55:00 +0100

As I was writing in my last entry, announcing the most awesome Cecil.Decompiler library, I took my last week off to host a CodeCamp event at my parents place in the quiet place of Ardèche, France.

I had the pleasure to host, by order of arrival,

While I was working on the core of the decompiler, I paired with Romain to work on improving our pattern matching story, to refine a basic AST to something more evolved. We still clearly need to do some work here, as we’re not completely satisfied with what we came with. Robert with his experience of F# and pattern matching then helped Romain.

Patrice worked on a NUnit addin which allow us to write tests that are executed for both optimized and non optimized code. He then joined Vincent to work on the Visual Basic language support. Vincent also wrote a ColoredConsoleFormatter, which adds some color to my tests in the console routines.

Robert started a Windows.Forms front tend in F# a while ago, but we decided to switch to C#, to lower the entry barrier for contributions. The inimitable Sébastien took the responsibility to write the front-end for the next millennium. I’ll force him to blog on it ASAP.

Mathieu took the pictures. He sadly was busy working on a presentation he had to do for a client next week.

On the next monday, Vincent told me something like: wow it’s quite depressing, I barely had a single laugh today, while we were laughing every five minutes during the last few days.

So we maybe weren’t the most productive crowd of week-end hackers, but at least we had tons of fun.

I guess that if I had a lesson to take from the event, it would have been to prepare it a little bit more, and have more fine grained tasks, but I also loved the organic way it evolved.

Thanks guys for the week-end, hopefully we’ll manage to organize more of those CodeCamp.

Cecil.Decompiler

Jb Evain — Mon, 15 Dec 2008 23:40:00 +0100

I know I’m quite late to present what I’ve been working on during the last Hack-Week we had at Novell, but better late than never, right?

Cecil.Decompiler is a library which relies on Cecil, takes one of its method, and give you back an object graph representing the decompiled body of the method, an AST, depending on the language you target. It also provides you ways to pretty print this AST at your convenience.

It takes its roots in Cecil and is actually a refactoring of the Cecil.FlowAnalysis library that folks at db4o contributed to the Mono project.

During the last Hack-Week, I started refactoring Cecil.FlowAnalysis, and since then, I’ve been working pretty seldom on it. It was last month that I decided to give it a kick, and even took a week of vacations to organize a CodeCamp with friends to give it a boost and have fun altogether.

The next blog post will be the occasion to describe the last week, and to thank my friends for their help.

Let me first warn you. It’s still very young, and doesn’t support a lot of constructions. On the other hand, it can already do pretty neat stuff, and its architecture makes it easy to use and improve, or add new languages.

The mandatory picture of a little program using the decompiler to decompile itself:

Don’t pay attention to the code as it’s full of debug code. Basically only the last two lines are interesting.

The code can be pulled from SVN:

http://anonsvn.mono-project.com/source/trunk/cecil/decompiler

I guess this will trigger a few interesting questions, so I cooked a little list of questions I already had, and answers I provided.

Q: What it is usable for?

A: It can be very useful for tools that perform static analysis on the assembly. For instance, Gendarme could use it to detect more complex patterns in your code.

Q: Why would you work on such a project while there’s obviously a better known tool that seems to do the same (and which have been working for a long time)?

A: Cecil.Decompiler is Open Source, and licensed under the very liberal MIT/X11. Also it’s a library, which makes it easy to use from your own IDE, tool, or whatever project you can think of.

Q: But then what’s a good library without a good GUI?

A: I’ll write more on that later, but my good friend Sébastien, author of the fantastic Reflexil project, started a Windows Forms GUI on top of it. And from what I understand, he plans to write the core of his tool using in platform agnostic fashion, so that plugins for his GUI could be re-used by a Gtk# GUI for instance.

Q: Why don’t you push a binary distribution for now?

A: The project is still very new, and we’re changing it a lot. So if someone is interested in having a look at it, he should rather directly pull the sources from SVN regularly.

Q: Should I start filing bugs ?

A: It’s not worth it from now, we’re pretty much aware of what’s missing and failing.

Twitter

Jb Evain — Wed, 12 Nov 2008 15:59:00 +0100

Alright, I surrender.

Mono 2.0

Jb Evain — Mon, 06 Oct 2008 09:00:00 +0200

Mono’s 2.0 grand release has finally arrived.

LINQ Expression Trees on the Compact Framework

Jb Evain — Mon, 22 Sep 2008 19:59:00 +0200

A week ago, I was writing about the ability to use Mono’s System.Core to run C# 3 and LINQ applications on .net 2.

A few months ago, I was writing about the new LINQ provider that friends at db4o released. This LINQ provider uses LINQ expression trees to optimize LINQ queries, turning them into db4o’s native query language.

Today I’m going to write about a cute new hack from our friends at db4o.

If the 3.5 version of the Compact Framework comes with a System.Core, and a LINQ to objects implementation, it doesn’t contain the namespace System.Linq.Expressions, which is used by all optimized LINQ providers. What those fine folks did, was simply to take Mono’s implementation of this namespace. There was one issue with that approach. Expression Trees can be compiled at runtime, and that uses System.Reflection.Emit, that the Compact Framework doesn’t have.

A nice thing is that folks at Mainsoft contributed to Mono’s System.Core an expression interpreter, which allows you to use the full LINQ expression trees without using System.Reflection.Emit. They sent me the patches they have for the expression interpreter to make it pass the full test suite we have, so I’ll integrate it shortly.

And voilà, just by using Mono’s implementation of System.Core, and by hacking a little in it, to make it compile on the Compact Framework, they’ve been able to produce a new assembly, System.Linq.Expressions.dll. You just have to reference it in your Compact Framework project, and it will let you use LINQ expression trees, which allows you to write optimized LINQ providers for the Compact Framework.

Of course, this is completely open-source and licensed under the MIT/X11. For now, you can find the code in db4o’s repository, but I’ll backport the changes they did to the LINQ expression interpreter, so that people that want to use it can use our code base directly.

Congrats to you guys for making the finest choice in the third party code you ship!

C# 3 and LINQ on .net 2

Jb Evain — Sun, 14 Sep 2008 19:08:00 +0200

My friend Patrick is wondering what’s the best way to use the type System.Collections.Generic.HashSet, in NDepend, that he wants to deploy on computers with only .net 2 installed.

Well it’s pretty simple, the best way to get it, is to get it from Mono. The code is licensed under the MIT/X11 license, allowing multiple usages.

And that’s not all. If you’re dying to use C#3 or LINQ to objects in your application, but that you want to be able to release on .net 2 only, you can simply compile and ship Mono’s implementation of System.Core. Microsoft’s C# compiler, csc 3.5, won’t have any issue using it, et voilà, you’ll get an application using LINQ or C#3, and deployable on .net 2.

Mono’s implementation of System.Core is available in SVN here.

Cecil performance issues

Jb Evain — Tue, 02 Sep 2008 08:45:00 +0200

At the beginning, Cecil was written to be an assembly manipulation library. The initial goal was to be able to read assemblies without loading them in an AppDomain, and also, to expose more that what .net 1.1 provided.

I implemented the writing parts of Cecil during my first Summer of Code, and barely touched it after, at it turns out it was working.

But this summer, two great hackers, Jeroen, author or IKVM, and Rolf, our VB 8 compiler author, decided that they gave enough blood and sweat to work around System.Reflection.Emit issues, and prototyped a version of their project based on Cecil instead.

If Rolf decided to go the full Cecil way, and replaced the usage of System.Reflection and System.Reflection.Emit (his branch is available), Jeroen decided to write a layer on top of Cecil that mimics the System.Reflection.Emit API.

Both wrote a sum-up about their findings, here for Jeroen, and there for Rolf. And both came to the conclusion that Cecil performs a lot less well that System.Reflection.Emit, both in terms of speed and memory consumption.

Rolf also uses the delay loading branch of Cecil, created by Mainsoft for their CIL to Java bytecode translator.

So the point is that Cecil uses too much memory, and is not fast enough in those scenarios, while it performs better in the reading only or reading + manipulating.

The good news is that, thanks to those hackers, we have now two amazing test cases to work on and to optimize. I’ve also started a few weeks ago, as a night hack, a refactoring of Cecil, that removes the intermediate structures that the current version of Cecil uses to read and write assemblies. If that makes reading and writing a little bit more complicated, it should also save a lot of memory.

And who knows, maybe one day, when Cecil will be optimized properly, and that the SRE on top of Cecil layer will be sufficient, it will require only a couple of changes to base mcs on Cecil.

Anyway, kudos to those hackers, let see what I can do to catch up.

We’ll let you know!

System.Scripting

Jb Evain — Thu, 31 Jul 2008 10:36:00 +0200

Two days ago, I was sharing what I was foreseeing for C# 4 and .net 4, and the integration with the DLR.

Turns out that today, by reading this post, my eye was caught by the namespace I was thinking about: System.Scripting.

But this post is about the beta 4 of IronPython 2, that hasn’t been released yet. But heh, it’s easy to forget that latest IronPython code can be downloaded from CodePlex’s TFS.

And it indeed turns out, that in the latest revision, Microsoft.Scripting.Core namespace has been renamed to System.Scripting, and that the DLR AST has been merged into System.Linq.Expressions.

So yeah, I guess we can safely admit that C# 4 will have statements support inside expression trees.

And that .net 4 will contain very cute features for dynamically generating programs, without resorting to emitting plain IL, simply using the DLR, contained in a new assembly (or namespace, if they move some to parts to System.Core), System.Scripting.

.net 4, C# 4, and the DLR

Jb Evain — Tue, 29 Jul 2008 17:07:00 +0200

UPDATE: turns out that there’s some true in those previsions, a small follow up.

I’ve been thinking a bit about the discussion in this video, about the future of C#, and specifically about its next version. If nothing is said clearly and loudly, as they’re most probably waiting for the PDC, you can still get a glimpse of what C# 4 and .net 4 will get as new features.

I’m mostly interested in one aspect of those new features. It’s the possible integration of the DLR inside the framework, and in the C# language.

The DLR, as we used to have it in IronPython, has been split into two separate assemblies, Microsoft.Scripting, and Microsoft.Scripting.Core a few months ago.

Microsoft.Scripting.Core is probably the part that will make it into the framework. It’s the part that contains a generalist AST, a compiler for this AST, using both LCG and traditional SRE for the debug scenarios. It also comes with a fast dynamic call site implementation, and an hosting API. So are we going to have a new System.Scripting assembly?

An interesting change in the DLR, is the naming of the AST factory and nodes. I’ve spent the last months working on LINQ and the LINQ compiler in Mono, and comparing the LINQ expression AST with the DLR AST is an interesting thing to do.

The DLR AST is now basically the LINQ expression AST, with statements added. And it looks like the current DLR AST has been tinkered around what has been already shipped inside .net 3.5 in the namespace System.Linq.Expressions.

Here’s a small example. In System.Linq.Expressions, pretty much all node from the AST extends the type Expression. Each Expression has a NodeType property, of type ExpressionType.

And you can find the same thing now in the DLR, but with some stuff added.

public enum ExpressionType {
    // some traditional expressions

    Add,
    AddChecked,

    // ...
    Call,

    // ...

    IfStatement,
    SwitchStatement,   

    // ...
}

So for the DLR, a statement is an expression. It makes kind of sense for a dynamic language you’ll tell me, but it looks a bit weird to be implemented this way.

Anyway, C# 3 already has a way to create expression trees to support some basic meta programming features, I guess we can only wonder if C# 4 will extend the meta programming features to support not only expressions but also statements, using the DLR tree.

I have no idea how they’ll deal with the fact that the existing stuff is in System.Linq.Expressions. Are they going to obsolete the whole namespace, and redirect everything to a possible System.Scripting.Ast (sounds less likely)? Are they moving the DLR AST to System.Linq.Expressions (sounds more likely)? Who knows? Not me (I’ve never lost control).

Interesting speculations anyway :)

So for now, the only change I can foresee for C# 4, is that you’ll be able to get full code blocks instead of only expressions when asking for an Expression of T.

But here’s another interesting one we can have a glimpse here, is duck typing support based the DLR.

At the language level, it’s difficult to say if they’re going to rely

on a dynamic code block, like the example shows in this blog entry,
on a custom marker type,
on another syntax for message passing to dynamic objects,

All in all, it looks like C# 4 will have some nifty stuff, let’s wait for the PDC.