Not logged in, Join Here! or Log In Below:  
 
News Articles Search    
 

 Home / General Programming / Malloc issues in Visual Studio 6 and .NET Account Manager
 
Archive Notice: This thread is old and no longer active. It is here for reference purposes. This thread was created on an older version of the flipcode forums, before the site closed in 2005. Please keep that in mind as you view this thread, as many of the topics and opinions may be outdated.
 
Jabbah

February 23, 2005, 05:53 AM

I am having some problems with allocating and deallocating large amounts of memory in Win2K and WinXP.

The problem:

I can allocate, until failing, blocks of a certain size. When it fails I free all the memory. This can be repeated as long as the block size is over about 507kb (520184bytes). If the allocs and frees are performed using a block size under this then I am unable to allocate more than 3 or 4 blocks of over that size. Block sizes under that work fine.

I have tried using a 3rd party malloc library (http://gee.cs.oswego.edu/pub/misc/) which fixes the problem but it is slow when compared to the version with visual studio.

I can only think that there is an error in the malloc routines with VS, or there is something that is causing the allocation tables to be not updated properly at some point.

Any ideas?

This is the test prog I used to highlight the problem:
(output at end)

  1.  
  2. #include "stdafx.h"
  3. #include <list>
  4. #include "malloc.h" //3rd party malloc lib
  5.  
  6. //#define USE_DL
  7.  
  8. void DoRealWorld();
  9. void FreeRealWorld();
  10.  
  11. static int g_nAllocs;
  12. static int g_nBigBlock;
  13.  
  14. static int gnTotal = 0;
  15. static int gnBigBlocks = 0;
  16.  
  17. typedef struct MemorySt
  18. {
  19.         void * p;
  20.         int nSize;
  21. } Memory;
  22.  
  23. static std::list<Memory*> glMemory;
  24.  
  25. int main(int argc, char* argv[])
  26. {
  27.  
  28.         printf("Enter Block Size ");
  29.         scanf("%d", &g_nBigBlock);
  30.  
  31.         while(g_nBigBlock != 0){
  32.                 gnBigBlocks = 0;
  33.                 DoRealWorld();
  34.                 printf("Num Allocs %dn", gnBigBlocks);
  35.                 FreeRealWorld();
  36.                 printf("Num Allocs %dn", gnBigBlocks);
  37.                 printf("Enter Block Size ");
  38.                 scanf("%d", &g_nBigBlock);
  39.         }
  40.  
  41.         return 0;
  42. }
  43.  
  44. void FreeRealWorld()
  45. {
  46.         while( !glMemory.empty() )
  47.         {
  48.                 Memory * pM = (Memory*)glMemory.front();
  49.                 glMemory.pop_front();
  50.                 gnTotal -= pM->nSize;
  51.                 gnTotal -= sizeof(Memory);
  52.                 if( pM->nSize == g_nBigBlock )
  53.                 {
  54.                         gnBigBlocks--;
  55.                 }
  56.  
  57. #ifdef USE_DL
  58.                 dlfree( pM->p );
  59. #else
  60.                 free( pM->p );
  61. #endif //USE_DL
  62.  
  63.                 delete pM;
  64.         }
  65. }
  66.  
  67. void DoRealWorld()
  68. {
  69.         try
  70.         {
  71.                 while( 1 )
  72.                 {
  73.  
  74. #ifdef USE_DL
  75.                         void * pBig1 = dlmalloc( g_nBigBlock );
  76. #else
  77.                         void * pBig1 = malloc( g_nBigBlock );
  78. #endif //USE_DL
  79.  
  80.                         if( !pBig1 )
  81.                                 break;
  82.  
  83.                         g_nAllocs += g_nBigBlock;
  84.  
  85.                         Memory * pMemory = new Memory;
  86.                         pMemory->p = pBig1;
  87.                         pMemory->nSize = g_nBigBlock;
  88.                         glMemory.push_back( pMemory );
  89.                         gnTotal += sizeof(Memory);
  90.                         gnBigBlocks++;
  91.  
  92.                 }
  93.         }
  94.         catch(...)
  95.         {
  96.                 // Done
  97.         }
  98. }
  99.  
  100.  


With VS malloc:
Enter Block Size 525000
Num Allocs 3626
Num Allocs 0
Enter Block Size 625000
Num Allocs 3264
Num Allocs 0
Enter Block Size 425000
Num Allocs 5008
Num Allocs 0
Enter Block Size 525000
Num Allocs 0
Num Allocs 0
Enter Block Size 520177
Num Allocs 0
Num Allocs 0
Enter Block Size 520176
Num Allocs 4103
Num Allocs 0
Enter Block Size 520177
Num Allocs 0
Num Allocs 0

With 3rd party malloc:
Enter Block Size 525000
Num Allocs 3626
Num Allocs 0
Enter Block Size 625000
Num Allocs 3265
Num Allocs 0
Enter Block Size 425000
Num Allocs 4663
Num Allocs 0
Enter Block Size 525000
Num Allocs 3626
Num Allocs 0
Enter Block Size 520177
Num Allocs 4079
Num Allocs 0
Enter Block Size 520176
Num Allocs 4079
Num Allocs 0
Enter Block Size 520177
Num Allocs 4079
Num Allocs 0

 
Fabian 'ryg' Giesen

February 23, 2005, 08:42 AM

Given the allocation patterns, this looks like something that gets allocated between your big blocks is not properly freed. Certain block sizes being a problem practically screams "heap fragmentation".

 
Gerald Knizia (cgk)

February 23, 2005, 11:57 AM

The small things not beeing freed might be the std::list nodes used in the code. Try replacing the std::list of glMemory with a preallocated std::vector or something.

 
Jabbah

February 24, 2005, 03:34 AM

Can you delete posts?

 
Jabbah

February 24, 2005, 03:36 AM

Can you delete posts?

 
Jabbah

February 24, 2005, 03:36 AM

Thanks for the replys guys.

The worry of fragmentation in the main app was why this code was written. After playing with it, there are a couple of reasons that heap fragmentation dont seem to fit:

1) All memory that is been allocated is been freed each time so should not be leaving little bits of allocated memory to fragment the heap. I have also tried calling _heapmin and similar functions to compact the heap etc but no joy.

2) If you stay above a certain block size then there is no problem. If it was heap fragmentation then you would expect to be running into problems whenever you change to a smaller blocksize. If you go below this certain block size then you can use any size blocks below that size with no problems but you cant go back above it.

3) If I use the different malloc library then the problem goes away.

I had thought about the container used for storing the pointers. Originaly it was using a CPtrList to store them, and I have also tried an array, both of which show the same behaviour.

As this problem occurs at a specific block size it seems that a different allocation method is being used for block sizes under that which prevents block sizes larger than that boundry being allocated.

If the other malloc library wasnt so slow I would use that.

Any sugestions for a free 3rd party malloc replacement?

 
This thread contains 6 messages.
 
 
Hosting by Solid Eight Studios, maker of PhotoTangler Collage Maker.