Board index » cppbuilder » Heap Memory Problems

Heap Memory Problems

I have a large multi-threaded application designed to handle real-time
access control and alarm monitoring written in CPP Builder. The threads
pass messages around internally using windows messages and dynamic
memory. On a 32 Meg machine the application hits a wall at around
380,000 transactions. From this point on response times go from ~150ms
to 6 seconds and up and the hard disk sounds as if it's getting
thrashed. I have been able to find no memory leaks and the only way to
correct the problem is to reset the application. I suspect the heap
memory has become to fragmented. This idea is supported by my
investigation of related news groups.  So, how do I fix this show
stopper? I found one company (MicroQuill) which has developed a smart
heap manager but they don't support Borland products (Yet) only
Microsoft. This kind of problem stops my project(s) dead in the water.
It could also explain some of the problems I've seen with another
multi-threaded application I have fielded developed with Delphi. It also
uses dynamic memory for inter-thread communications.  This is a major
BUG in my mind and puts the Borland products back on the shelf for
anything other than single thread simple user apps. My problem is we
have in excess of $50,000 invested in this application alone and we have
to find a solution ASAP.

1. Has anyone else encountered this problem and found a solution?
2. What is it?
3. Does Inprise have any input?

Please respond to both my email and the newsgroup if you can help at
all.  Thanks.

Michael Johnson
SpotLight Software

--
**********************************************
*  With sailing, as in life,                 *
*  It is the voyage that counts,             *
*  not the destination.                      *
*                                            *
*  Michael Johnson                           *
*  Escondido, CA USA                         *
*  S.V. Maggie J, Herreshoff 28              *
**********************************************

 

Re:Heap Memory Problems


One solution is to do your own memory allocation using the naive WIN32 calls
of VirtualAlloc, VirtualFree etcetera. You would have to build your own
memory manager but you could include routines that would attempt to
unfragment areas in order to provide bigger areas for further allocation.
You could even use the C++ "new" placement syntax or develop your own "new"
allocator.

All this is probably more work than you bargained for but if you suspect
that the problem is fragmented memory you might go with this more low level
approach. A good book WIN32 programming that has some excewllent info on
Windows 32 memory management is Jeff Richter's Advanced Windows. In general
I just wanted to point out that you don't have to use the compiler's memory
manager.

Quote
Woody wrote:
> I have a large multi-threaded application designed to handle real-time
> access control and alarm monitoring written in CPP Builder. The threads
> pass messages around internally using windows messages and dynamic
> memory. On a 32 Meg machine the application hits a wall at around
> 380,000 transactions. From this point on response times go from ~150ms
> to 6 seconds and up and the hard disk sounds as if it's getting
> thrashed. I have been able to find no memory leaks and the only way to
> correct the problem is to reset the application. I suspect the heap
> memory has become to fragmented. This idea is supported by my
> investigation of related news groups.  So, how do I fix this show
> stopper? I found one company (MicroQuill) which has developed a smart
> heap manager but they don't support Borland products (Yet) only
> Microsoft. This kind of problem stops my project(s) dead in the water.
> It could also explain some of the problems I've seen with another
> multi-threaded application I have fielded developed with Delphi. It also
> uses dynamic memory for inter-thread communications.  This is a major
> BUG in my mind and puts the Borland products back on the shelf for
> anything other than single thread simple user apps. My problem is we
> have in excess of $50,000 invested in this application alone and we have
> to find a solution ASAP.

> 1. Has anyone else encountered this problem and found a solution?
> 2. What is it?
> 3. Does Inprise have any input?

> Please respond to both my email and the newsgroup if you can help at
> all.  Thanks.

> Michael Johnson
> SpotLight Software

> --
> **********************************************
> *  With sailing, as in life,                 *
> *  It is the voyage that counts,             *
> *  not the destination.                      *
> *                                            *
> *  Michael Johnson                           *
> *  Escondido, CA USA                         *
> *  S.V. Maggie J, Herreshoff 28              *
> **********************************************

Re:Heap Memory Problems


Quote
: Woody <mlj...@connectnet.com> wrote:
>My problem is we
>have in excess of $50,000 invested in this application alone and we have
>to find a solution ASAP.

>1. Has anyone else encountered this problem and found a solution?
>2. What is it?
>3. Does Inprise have any input?

Michael, you provide too little information to diagnose this. Where is
the performance analysis? How can you be sure that it is memory
allocation / management without having analyzed it?

Well, in case it is memory management overhead that you experience:

You can replace the memory manager if you think that suits you -
passing everything through to GlobalAlloc, for instance (why? No idea,
I don't do it).

You can write a class factory that completely prevents heap
fragmentation. I have done that and have guaranteed zero heap
fragmentation plus class construction is about 40 per cent faster and
destruction takes almost no time.

You can use a dynamic memory factory that recycles memory the moment
it is freed (heap fragmentation guaranteed to be zero, too). I have
seen code where the performance jumps by 200 per cent simply due to
that.

All this is working in both C++ Builder and Delphi and there is no
need to jump into MicroQuill's product. Although it sounds nice, IIRC,
its price is a bit excessive.

Quote
>This is a major BUG in my mind

What you are experiencing is not a bug. At most, it is severely
inappropriate use of existing code. IOW, a design issue on your side.

I am off for a vacation now until early January; I will be happy to
discuss this in the newsgroups when I am back.

--
Stefan Hoffmeister       http://www.econos.de/
No private email, please, unless expressly invited.

Re:Heap Memory Problems


Why offer the solution and then leave without me any way to try it? Yes, I
can write my own memory manager but their is a cost-value question that makes
this approach foolish. There is stuff out there already written, tested and
functional as you so nicely suggested and then left without any way to get
it. We are in major crunch mode right now, we need a solution right now, not
in Mid January. Dynamic memory is for exactly what I'm using for (IMO),
variable length, dynamic messages, structures and temporary objects. That the
designers of Borland and Microsoft products have made heap garbage collection
so poor is a BUG and poor design on their part. They tote their products as
"industrial" quality but they fall flat when used that way. The use of the
heap in the manner I am using it is appropriate. Using it this way under
windows 95/98 and with Borland perhaps is not. I come form the industrial
side where this kind of shit does not exist for long. When you pay several
thousand $ for an industrial compiler and OS you expect them to work and if
they don't the companies are not around very long. I made the mistake of
believeing those same standards existed in the general PC market and that's
where I was wrong.

Quote
Stefan Hoffmeister wrote:
> You can write a class factory that completely prevents heap
> fragmentation. I have done that and have guaranteed zero heap
> fragmentation plus class construction is about 40 per cent faster and
> destruction takes almost no time.

> You can use a dynamic memory factory that recycles memory the moment
> it is freed (heap fragmentation guaranteed to be zero, too). I have
> seen code where the performance jumps by 200 per cent simply due to
> that.

> All this is working in both C++ Builder and Delphi and there is no
> need to jump into MicroQuill's product. Although it sounds nice, IIRC,
> its price is a bit excessive.

> I am off for a vacation now until early January; I will be happy to
> discuss this in the newsgroups when I am back.

> --
> Stefan Hoffmeister       http://www.econos.de/
> No private email, please, unless expressly invited.

--
**********************************************
*  With sailing, as in life,                 *
*  It is the voyage that counts,             *
*  not the destination.                      *
*                                            *
*  Michael Johnson                           *
*  Escondido, CA USA                         *
*  S.V. Maggie J, Herreshoff 28              *
**********************************************

Re:Heap Memory Problems


You might try the RogueWave Memory.h++ library.  (www.roguewave.com).
 I'm not sure if they support BCB3 with this product, but it may be
worth a look.

Craig Walters
cwalt...@icon-stl.net

Woody <mlj...@connectnet.com> wrote in article
<367AD413.A9422...@connectnet.com>...

Quote
> I have a large multi-threaded application designed to handle
real-time
> access control and alarm monitoring written in CPP Builder. The
threads
> pass messages around internally using windows messages and dynamic
> memory. On a 32 Meg machine the application hits a wall at around
> 380,000 transactions. From this point on response times go from
~150ms
> to 6 seconds and up and the hard disk sounds as if it's getting
> thrashed. I have been able to find no memory leaks and the only way
to
> correct the problem is to reset the application. I suspect the heap
> memory has become to fragmented. This idea is supported by my
> investigation of related news groups.  So, how do I fix this show
> stopper? I found one company (MicroQuill) which has developed a
smart
> heap manager but they don't support Borland products (Yet) only
> Microsoft. This kind of problem stops my project(s) dead in the
water.
> It could also explain some of the problems I've seen with another
> multi-threaded application I have fielded developed with Delphi. It
also
> uses dynamic memory for inter-thread communications.  This is a
major
> BUG in my mind and puts the Borland products back on the shelf for
> anything other than single thread simple user apps. My problem is
we
> have in excess of $50,000 invested in this application alone and we
have
> to find a solution ASAP.

> 1. Has anyone else encountered this problem and found a solution?
> 2. What is it?
> 3. Does Inprise have any input?

> Please respond to both my email and the newsgroup if you can help
at
> all.  Thanks.

> Michael Johnson
> SpotLight Software

> --
> **********************************************
> *  With sailing, as in life,                 *
> *  It is the voyage that counts,             *
> *  not the destination.                      *
> *                                            *
> *  Michael Johnson                           *
> *  Escondido, CA USA                         *
> *  S.V. Maggie J, Herreshoff 28              *
> **********************************************

Other Threads