Tuesday, 31 May 2016

Simplistic benchmarking with 2D vector graphics

I have some ideas bubbling for an upcoming project so I was toying around with a simple way of measuring graphics performance, in particular, very primitive 2D vector rendering - i.e. drawing a bunch of lines (1000 in this case) to the backbuffer and flipping it. I rendered 250 frames and measured the result.

I was mostly interested in how close to the native OpenGL implementation I was in each language.

I excluded C++ and anything that felt like it would hinder my productivity.  I want to finish this project QUICKLY to prove a point.




Language Real User Sys Working Set (MB)
Java2D 0:02.69 4.04 0.38 500.39
Python2 OpenGL 0:04.84 3.67 0.29 199.36
Python3 OpenGL 0:04.89 4.38  0.25 198.83
Java LWJGL 0:05.08 0.94 0.32 326.48
JavaFX 0:05.67 2.35 0.57 546.78
BlitzMax 0:05.52 0.45 0.23 84.28
Java2D
(Anti-aliased Software Rendering)
0:11.07


12.79 0.49 500.39
Python2 SDL 0:11.69 10.71 0.28 196.14
Python3 SDL 0:12.64 11.53 0.29 209.59

The results are pretty clear to me. It's either LWJGL or BlitzMax for this project. I'm slightly torn in all honesty - the IDE for BlitzMax is missing a LOT of necessary features like refactoring. Java has a very comprehensive API to call upon...

However, BlitzMax is very quick to get something going.

Update: I've added plain old Java2D - and it flies. Very interesting.

Sunday, 29 May 2016

Language benchmarking - Update with memory consumption

Like some sort of amateur I totally missed the opportunity to calculate memory consumption in addition (maximum resident set size) in addition to CPU usage.

As I'm using the gtime (gnu time) command on Mac OS X I can use the %M format specifier.

So, with literally 3 seconds of find/replace I've updated my benchmarks and present the results here below.

PS. As my next project is going to be based around 2D vector graphics I'm going to do some performance comparisons of several languages - likely a much reduced set though, Haskell for instance doesn't really lend itself to lots of side effects. I'll probably do a collection of graphics primitives and measures FPS or time per frame. Watch this space!

Edit: I just realised I was running a very old version of the rust compiler (1.1.0), I've just upgraded to 1.9.0 and rerun the results. The old number is in parentheses.

Language Real User Sys Working Set (MB)
Rust 0.12.05
(0:12.47)
11.47
(11.94)
0.17
(0.14)
4.875
(5.98)
C++ 0:12.73 12.05  0.21 191.33
Monkey-X 0:15.84 15.17 0.24 517.59
PyPy2 0:16.76 16.11 0.20 313.17
Go 0:23.41 23.19 0.17 24.55
BlitzMax 0:24.87 23.99 0.24 27.76
C# 0:27.87 26.50 0.60 282.44
Java 0:30.41 30.43 0.25 428.95
Scala 0:31.63 31.25 0.66 2804.08
D 0:32.22 31.98 0.15 9.20
PyPy3 0:32.44 31.34 0.34 278.56
Haskell 0:38.96 38.27 0.50 2088.64
O'Caml 0:43.50 43.19 0.20 16.64
F# Mono 0:58.05 57.32 0.52 144.59
Clojure 1:01.01 62.52 1.01 2306.17
C# Mono 1:01.38 58.84 0.83 128.00
Ruby 2:31.46 149.54 0.54 28.91
Python2 4:46.03 282.67 1.43 26.38
Python3 4:26.41 264.80 0.72 33.45

I've also put a funky chart together - left is faster, the taller the bar the higher the memory consumption.

Rust appears to be some sort of perfect language for this. Very interesting.



Friday, 27 May 2016

I read an interesting article this morning around Rust and the apparent advantages/disadvantages of the language.

http://www.viva64.com/en/b/0324/

(I appreciate some things have changed with Rust since the article was written.. It also reeks slightly of C++ elitism)

This is of particular relevance because in my performance comparison Rust came out top, beating C++. Now, neither my Rust or C++ implementations were hard-core optimised, but maybe that's the point. They were straightforward code versions of my algorithm and Rust was quicker. In addition, Rust is a safe language.

The point of the article is that C++ does a pretty great job at what it does and there has never been a viable replacement. For safe languages, there is a large choice, Go/Scala/Java/C# and so on.

Why is this interesting? Simply because after multiple diversions I'm ready to settle on a single language for some upcoming projects. Java is calling me as I wrote my fledgling, uncompleted game in it.

Edit: This is an interesting counter: http://cantrip.org/rust-vs-c++.html.

Wednesday, 25 May 2016

Latest language benchmarking

This originally started as an attempt to compare functional language performance before embarking on a new project. It was all about immutable, lists, recursion and all those glorious functional programming constructs.

It's a straightforward piece of code, with tight loops and LOTS of floating point calculation. It should benefit JITs well and is small enough to fit in the level 1 CPU cache.

However, it changed into something different, which was a general language performance comparison. This does however suggest that I should revisit the original intention and do a proper functional language comparison using the specific idioms of each. That WOULD be interesting.

I've tidied up the process and fixed some bugs in the code. Without further ado, here is the latest results (all run on my 2012 MacBook Pro in Mac OS X):

Language Real User Sys
Rust 0:12.47 11.94 0.14
C++ 0:12.73 12.05  0.21
Monkey-X 0:15.84 15.17 0.24
PyPy2 0:16.76 16.11 0.20
Go 0:23.41 23.19 0.17
BlitzMax 0:24.87 23.99 0.24
C# 0:27.87 26.50 0.60
Java 0:30.41 30.43 0.25
Scala 0:31.63 31.25 0.66
D 0:32.22 31.98 0.15
PyPy3 0:32.44 31.34 0.34
Haskell 0:38.96 38.27 0.50
O'Caml 0:43.50 43.19 0.20
F# Mono 0:58.05 57.32 0.52
Clojure 1:01.01 62.52 1.01
C# Mono 1:01.38 58.84 0.83
Ruby 2:31.46 149.54 0.54
Python2 4:46.03 282.67 1.43
Python3 4:26.41 264.80 0.72


Very interesting results. Rust is the quickest, with PyPy2 showing a very impressive performance.

The source code is at https://github.com/zolbatar/functional-comparison.