replacing specific values in an array based on logic

This causes an Unable to perform assignment-error because left and right side of the assignment differ in size: You need to use the logical indexing on both sides array1>1.

array1=[0.5 1.3 1.0 0.0 -0.2]
% create logical vector for indexing
lg = array1 > 1
% replace elements
array1(lg) = mod( abs(array1(lg)) ,1)

This should work in MATLAB + Octave. You can also split the different operations :

% ensure positiveness
array1 = abs(array1);
% force to one
lg = array1 > 1;
array1(lg) = mod(array(1),1);

this returns array1 = 0.5000 0.3000 1.0000 0 0.20

If you absolutely want to stick to your approach, you can use a little trick: add +1e-10 to the second input of the mod function to let 1 “survive” the operation 😉

array1 = mod( abs(array1) ,1+1e-10)

This trick will yield slightly different results because the modulus is 1.0000000001 and not 1. The error will be higher, the higher the input number. However, from your example-array I would guess that this risk could be OK.