Swap out Matlab/Runtime folder tree, tar won't ceate symlink

I am on an Ubuntu VM (VirtualBox). I created the VM with limited disk space. I find myself switching between different versions of Matlab and their associated Runtimes. Because of the small virtual disk, I am uninstalling a version of Matlab or a version of its Runtime in order to install something else. The installation takes place over the internet, and unfortunately, I have modest bandwidth meant for home use.

Instead of uninstalling and re-installing, I tried to save time by temporarily swapping out a Runtime or a Matlab version and storing it in the much larger disk of the host Windows OS. If we assume that the host OS user's home folder is mounted as /media/sf_HostOSusr and the Runtime folder tree is rooted at /usr/local/MATLAB/MATLAB_Runtime/v912:

cd /usr/local/MATLAB/MATLAB_Runtime/
tar czf \
/media/sf_HostOSusr/VendorApp/matlab/20230425.1502%usr%local%MATLAB%MATLAB_Runtime%V912.tgz \
v912

I get several dozen errors:

tar: v912/bin/glnxa64/libmwfftw3f_mpi.so.3: Cannot create symlink to ‘libmwfftw3f_mpi.so.3.5.8’: Operation not permitted
tar: v912/bin/glnxa64/libPDFRes.so.7: Cannot create symlink to ‘libPDFRes.so.7.1’: Operation not permitted
tar: v912/bin/glnxa64/libmwfftw3f.so.3: Cannot create symlink to ‘libmwfftw3f.so.3.5.8’: Operation not permitted
<...snip...>
tar: v912/runtime/glnxa64/libmwmclmcrrt.so: Cannot create symlink to ‘libmwmclmcrrt.so.9.12’: Operation not permitted
tar: v912/sys/java/jre/glnxa64/jre/lib/amd64/server/libjsig.so: Cannot create symlink to ‘../libjsig.so’: Operation not permitted
tar: v912/bin/glnxa64/libMatlabDataArray.so: Cannot create symlink to ‘../../extern/bin/glnxa64/libMatlabDataArray.so’: Operation not permitted
tar: Exiting with failure status due to previous errors

Most of these symbolic links are to files existing in the same directory. I didn't think that it is due to incompatibilities with the host Windows 10 (64-bit) file system, since the tar'd up content is encapsulated within a tar ball. **What are some other reasons for such symlink problems in creating a tar ball?**

I tried archiving one symlink and its target from above, which didn't result in any symlink errors:

cd v912/runtime/glnxa64/
tar czf /media/sf_HostOSusr/tmp/Test.zip libmwmclmcrrt.so libmwmclmcrrt.so.9.12cd v912/runtime/glnxa64/

[Walter Roberson]

Have you experimented with cpio? https://gnuwin32.sourceforge.net/packages/cpio.htm

I do not know if it will solve the problem, but in my experience it is more robust and faster than tar.

[FM]

My concern is that the destination is the Host OS (Windows 10) file system. I want to encapsulate the Linux nature of weird files like links within a tar ball. Protect it from Windows's attempt to make sense of them, and failing to.

[Walter Roberson]

Yes, and cpio does the same kind of encapsulation.

cpio by default treats symbolic links by noting them in its internal format and saving the name of the file linked to. But the -L option for cpio instead copies in the file linked to.

[FM]

I'm an idiot. I assumed it as a spiffed up "cp" until I read a bit of Wikipedia. Then I scurried back here to erase my 1st comment. Too late!

Anyway, I erased the Runtime folder already, but I will give cpio a try next time.

Thanks!

[FM]

I may resort to "find" & "cpio" out of desparation. After reading up on this a bit more, and based on a lifetime of learning how to avoid sticky situations with "find", "tar", "zip", and "xargs", I know that large file trees will run me into corner cases, especially with the liberties taken in the use of non-alphanumeric characters in file names these days. It's an interesting alternative to "tar", and one that requires weighing against the hour-long downloads that I face.

[Walter Roberson]

It would, by the way, be common to compress the cpio file afterwards. Some versions of cpio have a built-in compression option.

[FM]

OK, thanks.

Regarding becoming familiar with such find/archive tools, I remember lots of lessons in learned about the subtleties of whether to include leading "." (I forgot the exact use case) which can hamstring the retrieval, the -print0 to ensure that weird filenames don't muck things up, switches to control dereferencing of links (which you've delivered on a platter, thanks), the fact that tar's checking only checks what got archived and not what got missed, etc. I think I'd have to invest in some experimentation to be confident in using cpio. I don't have the space on the VM to unpack a copy to compare with the original file tree. I also don't trust unpacking the archive into the host OS's file space because it won't handle some links.

That's why it isn't quite a shoo-in compared to waiting an hour to re-install a version of Matlab or its runtime. Or even recreating the VM, though I remember a few long nights on that front. Actually, the runtime itself isn't so bad because I can store a copy of the installation zip file. It's Matlab itself that's the killer. I can store the installer, but it is a small fraction of the download. The rest gets downloaded each and every time the installer runs.

There's also the challenge that the client wants the deployed package installer created by Compiler SDK. That installer folds in the runtime installer, so even though I have an already downloaded copy of the runtime installer, the deployed package installer ignores that. If I want to verify glitch-free operation of the package installer, I have to absorbe the time cost of downloading the runtime installer even though I've already downloaded it.

Eventually, we can work with the client to modify the process for setting up the host application so that the deployed package installer is not needed, i.e., the runtime installer is executed separately and they only use the package's *.jar file created by Compiler SDK. But we're not there yet.

[Walter Roberson]

Maybe scp https://linuxize.com/post/how-to-use-scp-command-to-securely-transfer-files/ or rsync https://linux.die.net/man/1/rsync ?

[FM]

Huh! Looks like my attempted response didn't make it. Apologies if it shows up later, as a duplicate post.

I've used scp in the past and just browsed the rsync page now. They are/seem great for file transfer, but I think the key benefit of tar is that it encapsulates the file tree as a black box so that the host OS doesn't mangle it by trying to create a local file system counterpart. So does zip, but I trust tar more bundling up a Linux file system. Too many stealth issues with zip in the past with incompatible file paths, overly long file paths, and oversized zip files. They may be ironed out now, and maybe the Linux port doesn't have some of these issues, but I don't want to tempt fate. Trust is important because the Ubuntu VM doesn't have room for me to unpack the archive snapshot created on the host OS's file system, so I can't compare it with the original. Unpacking it into the host OS file system, which has way more room, will almost surely result in infidelity of the content.

Answer 1

[Nicole Bonfatti]

This is more of a VM issue than a MATLAB issue. It looks like you're extracting to a shared folder from the host, right? In that case symlinks won't work. Instead of a tar file, create a disk image, store it on the host FS, and mount the disk image in the VM.

[FM]

Boy that was a long time ago. That environment no longer exists.

From rereading my description, however, I'm not extracting. I'm bundling into a tarball, which is raw binary. The Windows host OS shouldn't be trying to make sense of the content at all. Based on that, I am assuming that the problem is completely within the Linux environment, and specifically with tar trying to capture those files.

I didn't have LLM assistants back then, but today, Copilot says that tar should be able to bundle up symbolic links as symbolic links even if the targets don't exist. So I'm at a loss as to what is causing the errors.