Simplistic benchmarking with 2D vector graphics

I have some ideas bubbling for an upcoming project so I was toying around with a simple way of measuring graphics performance, in particular, very primitive 2D vector rendering - i.e. drawing a bunch of lines (1000 in this case) to the backbuffer and flipping it. I rendered 250 frames and measured the result.

I was mostly interested in how close to the native OpenGL implementation I was in each language.

I excluded C++ and anything that felt like it would hinder my productivity.  I want to finish this project QUICKLY to prove a point.

Language	Real	User	Sys	Working Set (MB)
Java2D	0:02.69	4.04	0.38	500.39
Python2 OpenGL	0:04.84	3.67	0.29	199.36
Python3 OpenGL	0:04.89	4.38	0.25	198.83
Java LWJGL	0:05.08	0.94	0.32	326.48
JavaFX	0:05.67	2.35	0.57	546.78
BlitzMax	0:05.52	0.45	0.23	84.28
Java2D (Anti-aliased Software Rendering)	0:11.07	12.79	0.49	500.39
Python2 SDL	0:11.69	10.71	0.28	196.14
Python3 SDL	0:12.64	11.53	0.29	209.59

The results are pretty clear to me. It's either LWJGL or BlitzMax for this project. I'm slightly torn in all honesty - the IDE for BlitzMax is missing a LOT of necessary features like refactoring. Java has a very comprehensive API to call upon...

However, BlitzMax is very quick to get something going.

Update: I've added plain old Java2D - and it flies. Very interesting.

Language benchmarking - Update with memory consumption

Like some sort of amateur I totally missed the opportunity to calculate memory consumption in addition (maximum resident set size) in addition to CPU usage.

As I'm using the gtime (gnu time) command on Mac OS X I can use the %M format specifier.

So, with literally 3 seconds of find/replace I've updated my benchmarks and present the results here below.

PS. As my next project is going to be based around 2D vector graphics I'm going to do some performance comparisons of several languages - likely a much reduced set though, Haskell for instance doesn't really lend itself to lots of side effects. I'll probably do a collection of graphics primitives and measures FPS or time per frame. Watch this space!

Edit: I just realised I was running a very old version of the rust compiler (1.1.0), I've just upgraded to 1.9.0 and rerun the results. The old number is in parentheses.

Language	Real	User	Sys	Working Set (MB)
Rust	0.12.05 (0:12.47)	11.47 (11.94)	0.17 (0.14)	4.875 (5.98)
C++	0:12.73	12.05	0.21	191.33
Monkey-X	0:15.84	15.17	0.24	517.59
PyPy2	0:16.76	16.11	0.20	313.17
Go	0:23.41	23.19	0.17	24.55
BlitzMax	0:24.87	23.99	0.24	27.76
C#	0:27.87	26.50	0.60	282.44
Java	0:30.41	30.43	0.25	428.95
Scala	0:31.63	31.25	0.66	2804.08
D	0:32.22	31.98	0.15	9.20
PyPy3	0:32.44	31.34	0.34	278.56
Haskell	0:38.96	38.27	0.50	2088.64
O'Caml	0:43.50	43.19	0.20	16.64
F# Mono	0:58.05	57.32	0.52	144.59
Clojure	1:01.01	62.52	1.01	2306.17
C# Mono	1:01.38	58.84	0.83	128.00
Ruby	2:31.46	149.54	0.54	28.91
Python2	4:46.03	282.67	1.43	26.38
Python3	4:26.41	264.80	0.72	33.45

I've also put a funky chart together - left is faster, the taller the bar the higher the memory consumption.

Rust appears to be some sort of perfect language for this. Very interesting.

I read an interesting article this morning around Rust and the apparent advantages/disadvantages of the language.

http://www.viva64.com/en/b/0324/

(I appreciate some things have changed with Rust since the article was written.. It also reeks slightly of C++ elitism)

This is of particular relevance because in my performance comparison Rust came out top, beating C++. Now, neither my Rust or C++ implementations were hard-core optimised, but maybe that's the point. They were straightforward code versions of my algorithm and Rust was quicker. In addition, Rust is a safe language.

The point of the article is that C++ does a pretty great job at what it does and there has never been a viable replacement. For safe languages, there is a large choice, Go/Scala/Java/C# and so on.

Why is this interesting? Simply because after multiple diversions I'm ready to settle on a single language for some upcoming projects. Java is calling me as I wrote my fledgling, uncompleted game in it.

Edit: This is an interesting counter: http://cantrip.org/rust-vs-c++.html.

Latest language benchmarking

This originally started as an attempt to compare functional language performance before embarking on a new project. It was all about immutable, lists, recursion and all those glorious functional programming constructs.

It's a straightforward piece of code, with tight loops and LOTS of floating point calculation. It should benefit JITs well and is small enough to fit in the level 1 CPU cache.

However, it changed into something different, which was a general language performance comparison. This does however suggest that I should revisit the original intention and do a proper functional language comparison using the specific idioms of each. That WOULD be interesting.

I've tidied up the process and fixed some bugs in the code. Without further ado, here is the latest results (all run on my 2012 MacBook Pro in Mac OS X):

Language	Real	User	Sys
Rust	0:12.47	11.94	0.14
C++	0:12.73	12.05	0.21
Monkey-X	0:15.84	15.17	0.24
PyPy2	0:16.76	16.11	0.20
Go	0:23.41	23.19	0.17
BlitzMax	0:24.87	23.99	0.24
C#	0:27.87	26.50	0.60
Java	0:30.41	30.43	0.25
Scala	0:31.63	31.25	0.66
D	0:32.22	31.98	0.15
PyPy3	0:32.44	31.34	0.34
Haskell	0:38.96	38.27	0.50
O'Caml	0:43.50	43.19	0.20
F# Mono	0:58.05	57.32	0.52
Clojure	1:01.01	62.52	1.01
C# Mono	1:01.38	58.84	0.83
Ruby	2:31.46	149.54	0.54
Python2	4:46.03	282.67	1.43
Python3	4:26.41	264.80	0.72

Very interesting results. Rust is the quickest, with PyPy2 showing a very impressive performance.

The source code is at https://github.com/zolbatar/functional-comparison.

KineticJS + CoffeeScript

After a fairly long break I'm back to tinkering with things. Today, I spent a couple of hours reacquainting myself with CoffeeScript. I really enjoy coding in this language, it's clean and straightforward and compiles right down to Javascript automatically.

As usual, I like to see things happening graphically so I combined it with KineticJS. I'd not used this before today and the fact I managed to get something going so quickly speaks volumes as to its simplicity.

Anyway, I put together something with some animation and a bunch of classes.

It's not much to look at, but like all things I do it's just to scratch the surface and get a feel for it (-:

BTW. The quality of the video is awful, not entirely sure why - must investigate next time I'm bored.

PyPy Performance - Wow!

For those who follow my (very) occasional posts you know I've been working on measure performance of functional languages. I've been expanding this recently to include a more imperative comparison.

What I've done is use a simple algorithm to assign activities to resources. I have a data file with 500 activities and 10 resources. I assign 50 jobs to each resource in turn, selecting the closest each time. The result should be the same for all languages, however, the order in which I process the resources matter. Some languages (notably Go) have a random sequence for maps so I need to take this into account.

This is VERY numerically intensive and makes heavy use of collections (lists and maps).

The following table is a rundown of the results:

C++   -> 1.473s
Go           -> 2.966s
PyPy (Maps)       -> 3.180s
Scala (Imper.)    -> 3.283s
Haskell           -> 3.955s
C# (Adjusted)     -> 3.997s
O'Caml            -> 4.892s
F# (Adjusted)     -> 5.442s
Scala (Func/Imm)  -> 6.836s
PyPy              -> 6.987s
C# (Mono 3.0)     -> 10.137s
C# (Mono 2.10)    -> 10.554s
F# (Mono 3.0)     -> 13.802s
F# (Mono 2.10)    -> 16.031s
Ruby 2.0          -> 16.563s
Python 3.3        -> 25.274s
Python 3.3 (Maps) -> 27.642s
Python 2.7 (Maps) -> 31.736s
Cython            -> 32.745s
Ruby 1.9          -> 37.843s
Python 2.7        -> 43.798s
Clojure           -> 2m25.605s

I'm not going to go into the specifics of each, but where possible each is using the same basic algorithm but in an idiomatic style for each language.

From the title of this post you can see where I'm going with this. Python was by far the quickest to get to a working solution, was the simplest code to understand and extremely easy to profile and optimise. Running the standard interpreter (either 2.7 or 3.3) gives reasonable results but running PyPy gives a performance in the league of Go/C++/Scala/Haskell and O'Caml.

This was very interesting!

RISC OS - An interesting retro diversion.

My experience so far on using & coding on RISC OS is that it is a little bit like "the old days."

What I mean is that there isn't a great deal out there on the internet, Amazon has I think 2 books - apart from a bunch of old 90's second hand ones.

I've been coding in BBC Basic, which feels really retro; however, I did forget how easy it was to get things going and it's relatively powerful, for its age at least. I need to use an external editor to make it work efficiently, hardly a problem for me these days. In fact back on the BBC Master I used to use *VIEW to write my code anyway.

I've also been dabbling in some ARM assembly language. Now, I always had an idea about the utopian beauty of its RISC instruction set. I'm now discovering a few annoying limitations of it, partially due the fixed 32-bit wide op-code width.

So...it's been a fun journey, but RISC OS is most definitely stuck in the 80's/90's. It has given me some great inspiration though, which ultimately is the point of it all!

The pic below is my first BBC Basic program in ~25 years! You can see the Raspberry Pi just underneath the monitor:

An OS in 6MB!:

My RISC OS desktop running:

My venerable INPUT magazines (52 issues on programming for 8-bit machines from the early 80's), this is a page on Lisp coding: