Dynamically Sized Arrays

[Seko1987_MAD]

Hi,

I'm a little confused about how to use dynamic sized arrays with alloc and realloc.

Consider the following code:

     D response        s             10                                   
     D ptr             s               *   inz(*Null)                     
     D array           s              1    dim(32767)                     
     D                                     based(ptr)                     
      *                                                                   
     C                   eval      ptr = %alloc(%size(array) * 10)        
     C                   eval      array(1) = 'A'                         
     C                   eval      array(2) = 'B'                         
     C                   eval      array(3) = 'C'                         
     C                   eval      array(4) = 'D'                         
     C                   eval      array(5) = 'F'                         
     C                   eval      ptr = %realloc(ptr: %size(array) * 5)  
     C                   eval      array(100) = 'Z'                       
     C     array(100)    dsply                   response                 
     C                   dealloc(n)              ptr                      
     C                   eval      *inlr = *on                            
     C                   return

When I do the dsply it print 'Z'. If I reallocate the array for a dimension of 5 why it still prints the value of the position 100? When I debug th program it shows all the array positions.

However I found this site, where it states the following:
"It's important to remember that, while you are dynamically allocating memory for the array, you are not dynamically changing the number of elements in the array. The %ELEM function will still indicate 32767 elements throughout the entire process."

So my question is: does it really allocate/reallocate the memory, or nothing happened? How can I know for sure?

Thanks

[Scott Klement]

Re: Dynamically Sized Arrays

Q: How do I do dynamic arrays.

A: Don't. Someone who isn't familiar with pointer logic and dynamic memory allocation is taking a big risk of creating a program that seems to work, but could fail miserably at unexpected times. Someone who is familiar with dynamic memory and pointers will know better than to create a dynamic array.



Q: But... it's using 32767 bytes of memory! GASP! That's a problem, isn't it?

A: Uhh... it might be if you still have 128k of memory.


Q: My computer still has 128k of memory. I'm still using the same computer I had in the 1980s!

A: Upgrade it. Believe me, the upgrade will make everyone happier, and will probably save you money in the long run.

[Scott Klement]

Re: Dynamically Sized Arrays

Sorry for my previous reply. I'm very frustrated when I see people using techniques like this that. This encourages bugs, makes the program harder to read/maintain, and for what? Almost anything you do will waste more memory than this dynamic array is saving you. The total array size is only 32k -- a TINY amount of memory.

So it frustrates me when people write code like this. (Or much worse, when people write articles demonstrating stuff like this.)

You are doing more harm than good by using this technique, UNLESS your array is MUCH bigger than this. For example, if you are trying to write code to handle more than 16mb of memory -- that's a different story.

[Seko1987_MAD]

Re: Dynamically Sized Arrays

I wanted to try this approach because I have a program with 3 arrays, most of the time these arrays only uses 100/200 elements. However there may be some occasions, although rare, that the array occupation could be huge. So I thought it was a waste of memory, maybe performance, to use a large array when most of the times I required only some array positions.

I think I will define the arrays with the maximum dimension allowed (32767). From your post I assume that large fixed size arrays wouldn't have a signifficant impact on performance/memory, as I first thought.

[Scott Klement]

Re: Dynamically Sized Arrays

When I do the dsply it print 'Z'. If I reallocate the array for a dimension of 5 why it still prints the value of the position 100? When I debug th program it shows all the array positions.

What you have to understand about this is that RPG does not know how much memory you've allocated. It knows that the array stores 32767 elements. It does not know that you've only allocated 5. It assumes that all 32767 are available -- and leaves it entirely up to you to ensure that you do not exceed the allocation that you've provided.

Think of your computer's memory as one big, huge, gigantic string of data. Gigabytes (or maybe even terabytes) long.

You have asked the operating system for 5 bytes of memory. The OS will find a spot in your computer where there are 5 bytes available. It then returns the address (the position in the giant string of memory) where these bytes are available.

For a simplified example, let's say that OS tells you that memory location 10000 (this is the address in memory) is available for you to use. Since you asked for 5 bytes, you can put data in positions 10000-10004.

Now you tell your program "write data at position 10099" (the 100th element of a 1A array is the 100th byte in your allocated memory). What will happen? Neither RPG nor the operating system will stop you from trying to write to places in memory beyond what you've requested. So it really depends on what happens to be there. It's possible that this is outside the limits of what is available to your program. If that's the case, your program will crash. But that will only happen if you're "lucky". Most of the time, this will end up being a place in memory that's either (a) not currently in use by a program, or (b) is currently in use by a program.

If the memory isn't being used, you can store your 'Z' there, and everything will seem to work. UNTIL another program asks for memory, at which point it might overwrite your Z with some other value. Or, a user might run your program in a different job stream that uses the memory differently, and your program might have problems with that 100th element. After all, it doesn't belong to you -- you're just writing data there.

If the memory is being used, you might be changing a variable that belongs to another program. If that program hasn't used the memory recently, it might not even notice. Or, if it does try to use it, it may get "garbage" in it's variable -- and it'll be a complete mystery why that happened, because people will debug that program and find no code that would put the letter "Z" into that variable. It'll be very hard to find that this other program is simply overwriting memory that doesn't belong to it.

In any case, it creates a big mess.

However I found this site, where it states the following:
"It's important to remember that, while you are dynamically allocating memory for the array, you are not dynamically changing the number of elements in the array. The %ELEM function will still indicate 32767 elements throughout the entire process."

That's true... but I'm not sure that you can understand all of the implications just by that quick description.

So my question is: does it really allocate/reallocate the memory, or nothing happened? How can I know for sure?

It will really work. And you can tell it's working by looking at the amount of memory your job is using. But, for data this small (sorry, but as I said before, 32767 is a TINY amount of memory) it'd be very hard to track.

The big issue with working with dynamic memory, pointers, etc in this manner is that there is no "safety net". Neither the OS nor the RPG compiler will be able to discover or tell you about your mistakes. It's entirely up to you to get the code exactly right and do it all perfectly. If anything goes wrong, it'll be up to you to detect this and fix it.

I would strongly discourage using these techniques unless you have a VERY good reason.

[Seko1987_MAD]

Re: Dynamically Sized Arrays

Thank you Scott for the explanation.

I'll follow your advice and declare my arrays with the maximum positions that I think it may need.

If the memory is being used, you might be changing a variable that belongs to another program. If that program hasn't used the memory recently, it might not even notice. Or, if it does try to use it, it may get "garbage" in it's variable -- and it'll be a complete mystery why that happened, because people will debug that program and find no code that would put the letter "Z" into that variable. It'll be very hard to find that this other program is simply overwriting memory that doesn't belong to it.

I didn't know that a program could change variables that belonged to others programs. This would introduce some really nasty bugs.

[tomliotta]

Re: Dynamically Sized Arrays

One minor note... Since you %realloc()'d to a smaller size for the same array, there was no necessary reason for the system to use a different memory address. Always remember that allocating memory is not the same as initializing a variable that is addressed to that memory. And consider what might happen if you %alloc()/%realloc() more than just a single variable/array.

Tom

[RayGilliesJones]

Re: Dynamically Sized Arrays

There are ways to do it conveniently and safely... but that isn't it.