R2023a Cloudera Hadoop error com.mathwo​rks.toolbo​x.parallel​.hadoop.Ma​tlabStartu​pFailureEx​ception: parallel:m​apreduce:H​adoopTaskO​therFailur​e

On MATLAB R2023a MATLAB Distributed Computing Server (MDCS) running on Cloudera Hadoop we periodically get the below error in a mapreduce job failure but then it will run fine. Any ideas?

2024-04-26 16:04:35,721 ERROR [IPC Server handler 24 on 34933] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1690115930864_20721693_r_000000_0 - exited : com.mathworks.toolbox.parallel.hadoop.MatlabStartupFailureException: parallel:mapreduce:HadoopTaskOtherFailure

Caused by: com.mathworks.util.ShutdownRuntimeException: Java is shutting down

at com.mathworks.mvm.MvmFactory.nativeStartSession(Native Method)

at com.mathworks.mvm.MvmSession.createSession(MvmSession.java:276)

at com.mathworks.mvm.MvmSession.createSession(MvmSession.java:249)

at com.mathworks.toolbox.parallel.hadoop.worker.MvmPool.initializeMvmSession(Unknown Source)

at com.mathworks.toolbox.parallel.hadoop.worker.MvmPool.<init>(Unknown Source)

at com.mathworks.toolbox.parallel.hadoop.link.MatlabWorkerSingleton.initialize(Unknown Source)

at com.mathworks.toolbox.parallel.hadoop.link.MatlabReducer.setup(Unknown Source)

at com.mathworks.toolbox.parallel.hadoop.link.MatlabReducer.run(Unknown Source)

at com.mathworks.toolbox.parallel.hadoop.MatlabReflectionReducer.run(Unknown Source)

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:628)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

Answer 1

[Simar]

Hi Charles,

I understand that you are facing a periodic error in a mapreduce job failure on MATLAB R2023a MATLAB Distributed Computing Server (MDCS) running on Cloudera Hadoop

The error message provided suggests that there is an issue with MATLAB Distributed Computing Server (MDCS) on Cloudera Hadoop, specifically related to initialization and shutdown of MATLAB sessions during a mapreduce job. The key part of error is com.mathworks.util.ShutdownRuntimeException: Java is shutting down, which indicates that the Java Virtual Machine (JVM) is shutting down unexpectedly, causing MATLAB session to fail.

Here are a few workarounds to troubleshoot and potentially resolve this issue:

1. Check Resource Allocation:

Ensure resources allocated to Hadoop cluster are sufficient for running MATLAB jobs. Insufficient memory or CPU resources can cause JVM to shut down unexpectedly.

2. Review Hadoop Configuration:

Verify Hadoop configuration parameters (such as mapreduce.task.timeout) are set appropriately. If tasks are taking too long and timing out, this can cause JVM to shut down.

3. MATLAB Version Compatibility:

Ensure the version of MATLAB Distributed Computing Server (MDCS) being used is fully compatible with the version of Cloudera Hadoop been deployed. Compatibility issues can sometimes lead to unexpected behaviour.

4. Increase Timeout Settings:

Consider increasing the timeout settings for mapreduce tasks. This can sometimes help if the tasks are taking longer than expected to complete.

5. MATLAB Worker Initialization:

Check initialization process of MATLAB workers. The error suggests that the MATLAB session is failing to start properly. Ensure that all necessary environment variables and paths are correctly set for MATLAB to start.

6. Log Files:

Examine log files for more detailed error messages. The Hadoop log files, and MATLAB worker log files can provide additional insights into why the JVM is shutting down.

If the issue persists, consider reaching out to MathWorks support for assistance and keep in handy specific details or logs, so as to share them for more targeted advice.

Hope it helps!

Best Regards,

Simar