MATLAB's inefficient copy-on-write implementation

MATLAB's copy-on-write memory management seems to have a serious defect, which I think is the reason behind the abysmal performance of subsasgn overloading. (The same problem probably occurs with parenAssign in the new R2021b RedefinesParen class -- I haven't yet experimented with it.) Normally, an array assignment like b = a simply does a pointer copy; the array data is not copied until b is modified (e.g. b(1) = 1). Thereafter, subsequent modification of b (e.g. b(2) = 1) do not copy the full array; they just modify it in place as long as the reference count is 1. For example,

clear, a = zeros(1e8,1);
memory % 2764 MB used by MATLAB
b = a;
memory % 2764 MB
tic, b(1) = 1; toc, memory % 0.329099 seconds, 3540 MB
tic, b(2) = 1; toc, memory % 0.000123 seconds, 3541 MB

However, the benefit of copy-on-write is lost when the variable is changed in a function, e.g.

% test.m
function x = test(x)
x(1) = 1;

In this case, the x reference count is apparently incremented in test before the assignment is made, so this will always result in a full array copy. For example,

clear, a = zeros(1e8,1);
tic, a = test(a); toc % 0.337475 seconds
tic, a = test(a); toc % 0.310373 seconds

To see what's happening with copy-on-write, test.m is modified as follows:

function x = test(x)
memory
x(1) = 1;
memory
return

The array modification inside the function forces a full array copy, even though the original array is immediately discarded:

clear, a = zeros(1e8,1);
memory % 2748 MB
a = test(a); % 2748 MB, 3503 MB
memory % 2740 MB

I would think this problem could be easily avoided by treating any variable that appears as both an input and output argument in a function (e.g. function x = test(x)) as a reference variable, i.e. its reference count is not incremented on entering the function and is not decremented upon exiting. If the function is called with different input and output arguments, e.g. y = test(x), then the interpreter would implement this as y = x; y = test(y).

Is there any particular reason why MATLAB does not or cannot do this? There are many applications such as subasgn overloading that could see a big performance boost if this problem is fixed.

[James Tursa]

Slight point of confusing terms with your description. In the past, MATLAB has passed shared data copies of arguments to functions, not bumping up reference counts. Do you have evidence or know of documentation that shows a change in this behavior, and that now a bumped up reference count method is used for arguments? Why do you write that MATLAB uses this method?

Answer 1

[James Tursa]

See Loren's Blog on this topic. Basically, to write functions that can modify a variable "inplace" you need to call that function from within another function and follow some syntax rules. Then you can avoid the deep data copy.

https://blogs.mathworks.com/loren/2007/03/22/in-place-operations-on-data/?s_tid=srchtitle_inplace_1

There is a subtle caveat to this. If the variable is already shared, then the function will be forced to make a deep copy regardless of how you call it or what syntax you use. And there are no official MATLAB functions that can tell you the sharing status of a variable ahead of time, so it can be hard to predict when a deep copy will be forced and when it will not be forced. E.g.,

X = rand(10); % X will not be shared with anything at this point
Y = 1:10; % Y will be shared with a background variable that is hidden from you

It is not obvious that the simple assignment for Y above should result in shared variables, but that is exactly what happens on later versions of MATLAB for certain sized variables (it will be a reference copy). In this case any attempt to modify Y inplace will result in a deep data copy first.

[Kenneth Johnson]

James, thaks for this explanation and the link to Loren's Blog. The in-place operation works within nested functions but not within a script. Do you know why it doesn't, or couldn't, work with scripts? Test case:

% test.m

function x = test(x)

x(1) = 1;

% junk.m (script)

clear, tic, a = ones(1e8,1); toc % 0.152156 seconds

tic, a = test(a); toc % 0.238830 seconds

tic, a = test(a); toc % 0.233803 seconds

% junk.m

function junk

clear, tic, a = ones(1e8,1); toc % 0.163619 seconds

tic, a = test(a); toc % 0.000082 seconds

tic, a = test(a); toc % 0.000046 seconds

[James Tursa]

It doesn't work in scripts because you are not calling the function from within another function. As to why this is a requirement, I can only tell you "because that is the way MATLAB works".

[Matt J]

Even if the inplace operation is invoked from an intermediate function, it doesn't work.

a=1:1e8;
a=callingFcn(a);

Elapsed time is 0.635266 seconds.

function a=callingFcn(a)
 tic; a = myfuncIP(a) ; toc
end

function x = myfuncIP(x)
x(1)=1;
end

but if you wrap the top level code in a function,

function test

a=1:1e8;

a=callingFcn(a);

end

it does work. The 3 rules seem to be,

(1) The variable must be allocated within a function.

(2) All function calls (and their signatures) involving the variable must be of the form x=func(x,arg1,arg2,...).

(3) The variable must not be shared anywhere in the chain of function calls.

[James Lebak]

This is all correct. We document in-place assignment and its restrictions on the following page:

https://www.mathworks.com/help/matlab/matlab_prog/avoid-unnecessary-copies-of-data.html

in the section "assigning in-place."

In particular, there are a few additional restrictions noted on that page:

"MATLAB cannot apply memory optimization when it is possible to use the variable after the function throws an error. Therefore, this optimization is not applied in scripts, on the command line, in calls to eval, or to code inside try/catch blocks. Also, MATLAB does not apply memory optimization when the original variable is directly accessible during execution of the called function. ... Finally, MATLAB does not apply memory optimization when the assigned variable is declared as global or persistent."

[Kenneth Johnson]

Thanks, James. I will close my Service Request on this question.

[Paul]

@James Lebak

Can you clarify the following statement from the linked doc page in the Copy-on-Write section:

"Because the function does not modify the input values, the local variable X and the variable A in the caller's workspace share the data. After f1 executes, the values assigned to A have not changed. The variable B in the caller's workspace contains the result of the element-wise multiplication. The input is passed by value. However, no copy is made when calling f1." [Emphasis added]

Specifically, according to the first sentence, the A and X "share the data," which is what thought is effectively "pass by reference." But the the second to last sentence says the input is "passed by value," which I thought is synonymous with making a copy of the data. But the final sentence says no copy is made.

What is the definition of "pass by value?"

[James Tursa]

@Paul The phrase "pass by value" written in this documentation page is a bit misleading. To most programmers, the phrase "pass by value" means a deep copy of the variable is passed into the function (ala C/C++/Java). That is not what happens here, at least not right away. In MATLAB, variables are typically passed in as shared-data-copies. There is no deep copy of the data made at the time the function is entered. A new variable header is created that points to the same data memory as the original variable, and that is what is passed into the function. Bottom line is the end behavior is the same as "pass by value" if you alter the variable inside the function, but the behavior is similar to "pass by reference" if you don't alter the variable inside the function.

Caveat: Older versions of MATLAB passed in actual references to mex functions. Also, I haven't checked the latest versions of MATLAB to see if they pass in shared-data-copies or reference-copies to functions. In any event, a shallow copy of some sort gets passed in, not a deep value copy.

If you don't alter this shared-data-copy variable inside the function, then the only thing that happens when the function exits is the shared-data-copy is destroyed (header free'd, data stays intact). But if you alter this shared-data-copy variable inside the function, then a deep copy of the data is made at that time per the normal MATLAB rules for unsharing a variable prior to modification.

Having said all that, this normal behavior can be altered if you follow the "inplace" modification semantics, or you use a class variable that derives from handle.

[Paul]

@James Tursa

The way you explained it is exactly as I understood things to work (for garden variety m-functions, I'm not at all famiilar with mex programming), which is why I'm questioning if TMW is using a different meaning of "pass by value" than that used by most programmers.

I think we agree that the effect of Matlab's implementation on memory allocation is the same as the traditional pass by reference (or pass by const reference in a C++ sense) if the function argument is not changed internal to the function, and pass by value if it is.

Hopefully, @James Lebak can provide some insight into the woding of that section of the doc page.

[Walter Roberson]

Note that when you are using compound variables such as struct or cell array, then if you modify a value and are not doing in-place modifications, then a new data pointer needs to be created to hold the new value, and a new data descriptor needs to be generated for that, and any object that pointed to the old data descriptor needs to be updated, and a new data descriptor for that needs to be generated, and so on. But the pointers for "cousins" can remain the same

Descriptor D1 : cell 1 x 2, pr points to block B1
Block B1: contains pointer to Descriptor D2 and Descriptor D3
Descriptor D2: double 1 x 3, pr points to block B2
Descriptor D3: char 1 x 5, pr points to block B3
Block B2: contains numeric [pi, -2, sqrt(5)]
Block B3: contains 'hello'

In a situation that modifies cell location {1,1}(1,1) and returns the entire cell, the result would look like

Descriptor D4 : cell 1 x 2, pr points to block B4
Block B4: contains pointer to Descriptor D5 and Descriptor D3
Descriptor D4: double 1 x 3, pr points to block B5
Descriptor D3: char 1 x 5, pr points to block B3
Block B3: contains 'hello'
Block B5: contains numeric [7, -2, sqrt(5)]

%and some or all of the below might have been reclaimed
Descriptor D1 : cell 1 x 2, pr points to block B1
Block B1: contains pointer to Descriptor D2 and Descriptor D3
Block B2: contains numeric [pi, -2, sqrt(5)]

This is not a "deep copy". A "deep copy" would require that a new storage area for the 'hello' be generated.

[James Lebak]

@Paul, @James Tursa: I think your understanding for M-functions is correct. I accept the criticism that "pass by value" on the doc page is imprecise and I will pass that on to the appropriate people. Maybe "pass by value with lazy copy" would have been a better description, although as @Walter Roberson points out we do our best to keep the copies shallow for containers.

[Paul]

@James Lebak

Thanks for the response. Frankly, I don't see how "pass by value with lazy copy" adds any clarity to that portion of the doc page, which is specifically explaining how f1() works, where I don't see any kind of pass by value at all.

Regardless of what the specific wording should be, I appreciate your response and initiative to pass along the concern to the doc writers.

Answer 2

[Matt J]

(1) The variable must be allocated within a function.

A workaround to this rule is to wrap the data in a handle object:

a = 1:1e8;

tic,
 obj=refwrap(a); clear a
 testFn(obj); 
 a=obj.data;
toc %Elapsed time is 0.000460 seconds.
 
function testFn(obj)
obj.data(1) = 1;
end