Board index » delphi » Re: Some thoughts re the MM benchmarks

Re: Some thoughts re the MM benchmarks


2005-05-25 03:20:57 PM
delphi277
Hi Anders,
Quote
in absolute numbers in the result columns. Let the reader apply the
weighing...
The weighting is there for uninformed potential users to be able to pick the
best MM for their application. Without the weighting the casual observer
will have no idea which benchmarks are trivial and not really relevant and
which benchmarks are more "real world".
As it is now you can graph specific categories (e.g. only single-threaded
benchmarks or only multi-threaded benchmarks, etc.), so I think all the info
anyone would need is there.
Regards,
Pierre
 
 

Re: Some thoughts re the MM benchmarks

I've had the time today to look a bit into the current benchmarks and
memory managers. Please don't understand the points I raise as critic
on the benchmark or on the people involved.
First thing I saw is the inclusion of a lot of single threaded
benchmarks. That itself is imo not problem of course, except that I
think it is not really real-world like to add them in a way to the
benchmark application so that locking MM's can use the IsMultiThreaded
to completely avoid locking. Most real-world applications these days
will not have IsMultiThreaded set to true anymore especially if they
use databases, internet access (e.g. auto updates), ... I think that
you should either run the tests twice with each setting for
IsMultiThreaded or (a bit drastic i know) not allow the use of it at
all. I think it is a bit misleading for MOST users of the alternative
memory managers.
Secondly, I'd like to see massively threaded tests that are run on
fast multi cpu machines like dual/quad AMD Opterons with full 2gig ram
available. These tests should be designed to provoke heavy contention.
After all these are the machines that are used for applications that
struggle with memory/cpu load often rooting in problems with the stock
MM. It would be very interesting to see how each of the entrants deals
with that.
Lastly i think it would make sense to create two sub scores for the
results table, one for single threaded tests and one for multithreaded
one (again please make sure to be as close to real world). E.g. for
selecting the correct MM for a heavy threaded network service or
database server the scores of single threaded performance play a much
lower role than the singlethreaded ones.
I've no idea where NexusMM will stand in this, I have not even run the
tests here, that is just stuff that came to mind when looking through
some sources
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

Re: Some thoughts re the MM benchmarks

Hi Hannes
I generally agree with you.
A few short comments.
The weigth of benchmarks are not based on the number of benchmarks in a
category. We use an explicit setting.
Do any MM's use the IsMultiThreaded variable?
To all: Please post benchmark and validation results from all the PC's you
have. Multiprocessor runs are expecially welcome.
Regards
Dennis
 

Re: Some thoughts re the MM benchmarks

Hello Hannes,
Thanks for your comments.
1) The benchmark system is based on a 40 % single-thread, 60 % multi-thread
weighting. This is quite reasonable (and see also point 4 below).
2) it is perfectly smart to take advantage of the IsMultiThread variable. If
you know you're running a single-threaded application, you do not need to
lock the memory.
3) In the multi-threaded benchmarks we have *a lot* more contention than you
would find in any real-world situation.
4) Currently the fastest MM in single-thread are also the fastest in
multi-thread.
5) Of course you know where NexusMM stands in this, it is way behind the
current leaders, both in the single-threaded and the multi-threaded
benchmarks.
Take a loook at some of the results Dan Downs posted yesterday on a bi-CPU
machine. Speed-wise NexusMM is at 55.9% of the top performer, memory-wise at
76.7%, combined at 67.5%. that is massive.
Kind regards,
Robert
 

Re: Some thoughts re the MM benchmarks

Quote
1) The benchmark system is based on a 40 % single-thread, 60 %
multi-thread weighting. This is quite reasonable (and see also point
4 below).

2) it is perfectly smart to take advantage of the IsMultiThread
variable. If you know you're running a single-threaded application,
you do not need to lock the memory.
as mentioned, it is fine for "benchmarks for benchmark reasons", but i
think the bench application should have the *option* to disallow it.
"Single threaded" speed IS important but to assume that because a
worker routine is single threaded that there are no other threads
created/active in the application is not real-world like and thus
misleading people.
Quote
3) In the multi-threaded benchmarks we have *a lot* more contention
than you would find in any real-world situation.
i don't agree, sorry. try to run a highly threaded database server with
50+ concurrent users doing queries, batch updates, inserts, updates in
fully transactional environments on a quad cpu machine and you see what
i mean. but of course that is a very specific need. Also look at a
massive size clustered bitmap calculations etc, ...
Quote
4) Currently the fastest MM in single-thread are also the fastest in
multi-thread.
with a good design that is logical isn't it?
Quote
5) Of course you know where NexusMM stands in this, it is way behind
the current leaders, both in the single-threaded and the
multi-threaded benchmarks.
no i don't know where nexusdb WILL be standing with such changes. I've
not said i don't know where we ARE standing, of course i know.
Quote
Take a loook at some of the results Dan Downs posted yesterday on a
bi-CPU machine. Speed-wise NexusMM is at 55.9% of the top performer,
memory-wise at 76.7%, combined at 67.5%. that is massive.
It is, but that was not the point of my post, if you have read the top
few sentences. I don't think you achieve anything really with not
making the benchmark as close to real world as possible. And let's face
it: the small developer running a small app that uses 120 megs ram and
shut down after a day on a single cpu is not really the target audience
is it?
what you are looking at are heavy threaded server application, high
memory usage number crunchers, highly parallel worker threads, etc.
At least the test should have some weighing schemas like: Desktop
application, HighThread server, HighMemoryLoad and so on which the
interested user can apply. I also strongly vote for a forced
IsMultiThreaded setting option.
Again, that has NOTHING whatsoever to do where we stand. I don't think
it will change our position, but it might point out some problems with
each entrant for a certain scenario.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

Re: Some thoughts re the MM benchmarks

Quote
I generally agree with you.
:)
Quote
The weigth of benchmarks are not based on the number of benchmarks in
a category. We use an explicit setting.
sure, I have seen. that is of course very important.
Quote
Do any MM's use the IsMultiThreaded variable?
yes. I have no idea how much influence it has and how they use it, but in
theory it is possible to simply skip any locking in that mode. We had
that in an internal version to skip some code (we don't use locks but
still can skip/replace costly code) and the improvements were for
certain cases quite big.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

Re: Some thoughts re the MM benchmarks

Hello Hannes,
Quote
"Single threaded" speed IS important but to assume that because a
worker routine is single threaded that there are no other threads
created/active in the application is not real-world like and thus
misleading people.
As said, single-thread gets 40% of the over-all weight, multi-thread 60 %.
That's a fair balance and does not mislead any potential users.
Quote
i don't agree, sorry. try to run a highly threaded database server with
50+ concurrent users doing queries, batch updates, inserts, updates in
fully transactional environments on a quad cpu machine and you see what
i mean. but of course that is a very specific need. Also look at a
massive size clustered bitmap calculations etc, ...
Our multi-threaded benchmarks execute *only* memory management operations.
In a normal application the MM would only consume around 10% of the CPU, so
we effectively have a *10 times higher* contention rate than you would get
in a real application.
The eLink benchmarks runs with 16 threads doing nothing else than allocating
mostly blocks smaller than 32 bytes. that is what I call "contention".
Quote
what you are looking at are heavy threaded server application, high
memory usage number crunchers, highly parallel worker threads, etc.
The Link Server benchmark is a replay log from a heavy threaded database
server application. In this benchmark NexusMM is at about 50% of the
performance of the leaders.
Quote
At least the test should have some weighing schemas like: Desktop
application, HighThread server, HighMemoryLoad and so on which the
interested user can apply. I also strongly vote for a forced
IsMultiThreaded setting option.
Your comments suggest that you think you would get different results or
winners by splitting the benchmarks results in these categories.
You're wrong.
Please prove the contrary by entering *any* benchmark on which NexusMM would
perform significantly better than the current benchmark results suggest.
That's a challenge!
Regards,
Robert
 

Re: Some thoughts re the MM benchmarks

Quote
Your comments suggest that you think you would get different results
or winners by splitting the benchmarks results in these categories.
You're wrong.
No. But maybe one or the other might perform better than an all in one
solution for certain tasks?
Quote
Please prove the contrary by entering any benchmark on which NexusMM
would perform significantly better than the current benchmark results
suggest. that is a challenge!
already commented on the purpose of this post, that must do.
--
Hannes Danzl [NexusDB Developer]
Newsgroup archive at www.tamaracka.com/search.htm
 

Re: Some thoughts re the MM benchmarks

Hello Dennis,
Quote
Do any MM's use the IsMultiThreaded variable?
Yes, most do. RTL MM, BucketMM, FastMM, RecyclerMM, WinMem... all use this
feature.
This makes a lot of sense: if you know you're running a single-threaded
application there's no point in having the performance penalty of protecting
the MM against multi-threaded access.
Regards,
Robert
 

Re: Some thoughts re the MM benchmarks

Quote
[...] Most real-world applications these days
will not have IsMultiThreaded set to true anymore
If they don't have it set, they'll experience crashes in the RTL
as Borland relies on it for their default allocator, therefore,
not setting IsMultiThread can be considered a bug (at worst, to test
its influence, it is easy to force it to True at the start of the
becnhmark).
Quote
Secondly, I'd like to see massively threaded tests that are run on
fast multi cpu machines like dual/quad AMD Opterons with full 2gig ram
available.
There are already massively multithreaded tests, though I'd also
like to see more runs from machines with more than 2 cores, there seems
to be an availability issue. The tests on dual CPU and dual CPU+HT
didn't seem to indicate bad behaviour for BucketMem or FastMM though.
Quote
These tests should be designed to provoke heavy contention.
Most are already irrealistically stressful for the MM IMO,
with practically only allocs/reallocs/frees and little access
to the allocated memory.
It might be interesting to have some less deterministic memory
usages (by that I don't mean more random, but with more loops
and code in between, so that the MM's code isn't fully in L1 cache,
or the jump predictions and indirections aren't so predictible
for the CPU).
Quote
Lastly i think it would make sense to create two sub scores for the
results table, one for single threaded tests and one for multithreaded
one (again please make sure to be as close to real world).
Yep.
Eric
 

Re: Some thoughts re the MM benchmarks

Quote
This makes a lot of sense: if you know you're running a single-threaded
application there's no point in having the performance penalty of protecting
the MM against multi-threaded access.
Yes, the 'lock' prefix can have a relatively huge cost, wether or not
there is actually a conflict or not.
The instruction count for most calls to the fast MMs is so low that just
about anything will have a measurable impact. Testing IsMultiThread
means extra instructions, but since it is well predicted, the penalty
is rather puny.
Eric
 

Re: Some thoughts re the MM benchmarks

Hi Both of you especially Robert
This is not a discission about Nexus versus BucketMM ! Hannes emphazised
this.
I think Hannes comments are making sense, we just have considered them quite
well. The B&V is quite realworld and is a very tough test. We just need to
collect a lot of benchmark results on 2 and 4 processor machines as soon as
we reach release 1.0.
We also need to add a few more replays. We have two pending.
It is very important we take these discussions now and are willing to change
the B&V according to the outcome.
I will kindly ask more people to throw in their opinion.
Regards
Dennis
 

Re: Some thoughts re the MM benchmarks

Hi Dennis,
I can run from time to time if needed some tests on a Quad CPU machine.
Would that help?
--
----
michael.moreno.free.fr/
 

Re: Some thoughts re the MM benchmarks

Hi Michael
Quote
Would that help?
Oh yes a lot.
Report your resultsin this NG and I will put them in the spreadsheet.
Do it whenever you will.
Regards
Dennis
 

Re: Some thoughts re the MM benchmarks

Quote
It is, but that was not the point of my post, if you have read the top
few sentences. I don't think you achieve anything really with not
making the benchmark as close to real world as possible. And let's face
it: the small developer running a small app that uses 120 megs ram and
shut down after a day on a single cpu is not really the target audience
is it?
Well actually this is the case for many Delphi developpers !
Changing the MM is actually one of the easiest way to :
- reduce memory usage of your app
- speed it up
So there is really no reasons to actually not change it even for a
simple app.
--
----
michael.moreno.free.fr/