The blog of Gergely Orosz on IT related topics: web, software design, .NET, PHP and the rest. Follow me As part of my thesis I measured the performance of some .NET ORM frameworks including NHibernate and Entity Framework. Measuring was done by implementing two simple applications using the same table structure and doing the same operations on the same data.
Note: before reading this article please see my views on comparing ORM tools. Also see the follow up post to this comparison for revised results.
I measured the time it took for each framework to complete these operations:
- store
- read over relations
- read by ID
- update
- delete
The results were somewhat interesting. Here is a short summary of what I’ve found out.
| Operation \ Number of operations | NHiberante - 4K | Entity Framework - 4K | NHiberante - 40K | Entity Framework- 40K | Winner |
| Store | 37,37 | 9,19 | 1500 | 98 | Entity Framework |
| Read over relations | 1,01 | 0,54 | 10,13 | 4,18 | Entity Framework |
| Read by ID | 3,06 | 25,22 | 246 | 230 | NHibernate with smaller amount of objects |
| Update | 6,61 | 7,34 | 77 | 72 | Both |
| Delete | 3,35 | 16,76 | 58 | 1824 | NHibernate |
Read on if you’re interested in the details of the performance measurements.
Update: the source code of the program used to measure is now available for download.
How performance was measured
The data model I’ve set up to measure with was pretty simple. I used three entites: a Company entity which had multiple Employees all of which had multiple Reports all with a 1-n relation.
The detailed results for each operation are as follow (note that the graphs use logarithmic scale):
Storage
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 4,95 | 20,64 | 112 | 280 | 575 | 1107 | 15000 |
| Entity Framework | 2,11 | 6,18 | 18,4 | 31,3 | 51,7 | 77 | 98 |
Update (2009. 08. 24.):
After NHibernate improvements by Tuna Toksoz (download source) this metric changed the following way:
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 1,22 | 3,25 | 8,98 | 13,11 | 28 | 38 | 46 |
| Entity Framework | 2,11 | 6,18 | 18,4 | 31,3 | 51,7 | 77 | 98 |
Read over relations
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 0,21 | 0,66 | 1,81 | 3,31 | 5,0 | 8,24 | 10,13 |
| Entity Framework | 0,17 | 0,4 | 0,96 | 1,76 | 2,47 | 3,91 | 4,18 |
Read by ID
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 0,28 | 1,31 | 13,0 | 37,6 | 80 | 175,5 | 246 |
| Entity Framework | 5,56 | 17,03 | 48 | 83 | 132 | 195 | 230 |
Update
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 1,74 | 4,54 | 17,1 | 33,2 | 43,47 | 68 | 77 |
| Entity Framework | 1,99 | 5,29 | 14,2 | 24,2 | 41,9 | 59 | 72 |
Delete
| Framework/ No. of operations | 1K | 3K | 8K | 14 K | 23K | 33K | 40K |
| NHibernate | 0,78 | 2,31 | 8,83 | 20,6 | 28 | 46 | 58 |
| Entity Framework | 1,78 | 8,19 | 52,2 | 133 | 320 | 1014 | 1895 |
Conclusion
The performance measurements only provided significant differences in two cases.
- When it comes to storing data, Entity Framework proved to be significantly faster than NHibernate
- When deleting data, NHibernate was much faster than Entity Framework
My conclusion however after the test was that in those scenarios when the number of the operations is not too high performance does not differentiate that much. I would choose between these two tools based on which ones provides faster, more efficient and more straightforward development with the current project.
Give me the code
As many of the commenters pointed out a comparison is not really valid without publishing the code enabling validation of this measurement. The program I’ve written to do the test can therefore be downloaded: NHibernate vs Entity Framework source code.
To set up and run the project after downloading an unzipping the package follow these steps:
- Create two MSSQL Databases (e.g. EntityFrameworkTest and NHibernateTest)
- Run CreateTables.sql on both databases to create the database structure
- In the App.config section of the MeasurementController project set the connection strings for NHibernate and Entity Framework (replace the !!! marks)
- Build the solution
- Run the MeasurementController project
- Run times are written to the output and also saved in files in the bin/Data folder
Some additional comments for understanding the source:
- Upon every run test data is created in an XML file. From this XML file Company, Emlpoyee and Report objects are created in-memory as the test structure
- This test structure is passed to the NHibernate and Entity Framework implementations of the Store, Read, Read by ID, Update and Delete tests (in the source additionally search is also measured) on each test run.
» NHibernate vs Entity Framework: a performance test - Greg Does IT…
Thank you for submitting this cool story - Trackback from DotNetShoutout…
DotNetShoutout
August 18th, 2009
In order for this benchmark to make sense, you should publish some code.
Tuna Toksoz
August 18th, 2009
You’re right. I’ll be publishing the source code by tomorrow for the test to be repeatable.
Gergely Orosz
August 18th, 2009
Source code would be very much appreciated indeed.
Dani Pfulg
August 19th, 2009
[...] saw the following post where the performance of NHibernate and Entity Framework is compared for a couple of different [...]
Of Course NHibernate Is Slow When You Use It Incorrectly | The Inquisitive Coder – Davy Brion’s Blog
August 19th, 2009
Greg, which versions of EF and NHibernate did you use?
Dmitry Polskoy
August 19th, 2009
[...] la plus utilisée) EF est plus performant que NHibernate. C’est en tout cas mon interprétation des tests effectués par Gergely [...]
EF plus performant que NHibernate :-) , Matthieu MEZIL
August 19th, 2009
I’m agree with Tuna !
minsou
August 19th, 2009
Hello, I’m interested in your code too.
Thomas
August 20th, 2009
Your graphs are all showing the labels incorrectly - shouldn’t time be the vertical axis and no. of operations be the horizontal?
Mike
August 20th, 2009
[...] the launch of the ORMBattle.NET site, a new discussion started on the blogosphere; later on, this post by Gergely Orosz added more ashes to the fire. The subject seems to be, is it possible to [...]
OR/M Performance Comparison | I love .NET!
August 20th, 2009
@Tuna, @Dani, @minsou, @Thomas: the source code is now available for download. Sorry for the delay.
Gergely Orosz
August 20th, 2009
@Mike: thanks for noticing. They were incorrect, I’ve updated them.
Gergely Orosz
August 20th, 2009
@Dmitry: the version I’ve used to evaluate (also in the source code as reference) is NHibernate 2.0.1.4 and Entity Framework 3.5.
Gergely Orosz
August 20th, 2009
I’d assume your NH version of SearchByID would go faster if you didnt do all the unboxing/typeof crap, and simply used generics.
NHEmployee foundEmployee = session.Load(employee.Id);
Travis
August 20th, 2009
WP stripped the type, from the code. But you get the idea…
Travis
August 20th, 2009
It would probably improve performance a bit, you’re right.
Actually @Tuna improved the NH version by modifying the source enabling batching instead of calling Flush() every time thus dramatically improving NH results. I’ll post that updated version with the new results as well. I’ll also get rid of unnecesarry boxing there.
Gergely Orosz
August 21st, 2009
Hi, Gergely. Could you contact ORMBattle.net team and offer them your performance tests. It will be great to have all ORM tests at one place ).
Alexis Kochetov
August 23rd, 2009
@Alex sure… but would be grate to have real performance tests and not MDTD as in ORMBattle.net
Fabio Maulo
August 23rd, 2009
@Fabio, many people talk about real tests, but nobody proposes real scenarios. I would like to have real tests, so feel free to contact me (alexis [dot] kochetov [at] gmail.com) if you have any thoughts about them.
Alexis Kochetov
August 23rd, 2009
waiting for your updated version.
Darius Damalakas
August 23rd, 2009
@Alex If you are interested in something you should study what do, and don’t ask to others to do something for you. If you can’t create a real example it is your problem.
About this post I had published the second part, with the expected behaviour, in my blog:
http://fabiomaulo.blogspot.com/2009/08/nhibernate-perfomance-analisys.html
Fabio Maulo
August 24th, 2009
[...] The blog of Gergely Orosz, Sense/Net 6.0 developer on IT related topics: web, software design, .NET, PHP and the rest « NHibernate vs Entity Framework: a performance test [...]
» Thoughts on ranking ORM tools - Greg Does IT
August 24th, 2009
[...] After Tuna modified the code NHibernate storage speed rocketed up compared to the previous version: [...]
» Nhibernate vs Entity Framework – a revised performance comparison - Greg Does IT
August 24th, 2009
@Alex I’m afraid this test is not a good example for comparing ORMs. See my post http://gregdoesit.com/2009/08/thoughts-on-ranking-orm-tools/ for details on why.
@Darius: published an updated version although Fabio probably wrote an even better blog post on this (http://fabiomaulo.blogspot.com/2009/08/nhibernate-perfomance-analisys.html)
Gergely Orosz
August 24th, 2009
[...] términos de rendimiento entre Entity Framework y NHibernate…y los resultados los podéis ver en este enlace. Y lo mejor para empezar a sacar conclusiones es analizar la tabla comparativa que ha realiza [...]
ADO.NET Entity Framework vs NHibernate: Una primera comparativa! - Blog del CIIN
September 11th, 2009
You are not deleting items in EF properly. To delete an object you only have to create in memory a new instance of the object, then call context.CreateEntityKey(createdObject) and call context.Attach && context.DeleteObject or context.DeleteObject directly, i’m don’t remember the final step but you haven’t got to query the DB for the objects first in order to delete them later. Do you understand my point (sorry for my english)?
Javier Calvarro Nelson
September 11th, 2009
Improper tests due to lack of knowledge of how internally works NHibernate and how to optimize it for certain scenarios.
Ray
September 12th, 2009
[...] you’re interested in the details of the performance measurements, please go to GregDoesIt.com. Categories: Programming Tags: .Net, Entity Framework, NHibernate, ORM Comments (0) [...]
XtraMile, Inc. » NHibernate vs Entity Framework: a performance test
September 21st, 2009
[...] دیدن جزئیات آزمایش [...]
NHibernate در مقابل Entity Framework
September 28th, 2009
the code doesn´t work…
André
October 13th, 2009
Have you looked at the Readme.txt? You have to set up your database in order to run the examples.
Gergely Orosz
October 25th, 2009
The comparison I think is not valid. It all depends on how you have written your nHibernate or EF code to fetch database. NHibernate has lot of performance options like FetchMode->Join, First and Second Level Caching.
At the end it all depends on how you use the tool and not on the tool itself.
Dipen
January 4th, 2010
[...] vs. Entity Framework Sapiando em uns blogs por ai encontrei um post bem interessante. Apesar de não muito recente, ele responde ao que muitos dos que defendem o Entity Framework usam [...]
NHibernate vs. Entity Framework « hesenger
January 7th, 2010