Spark RandomForest training StackOverflow 오류

Question

내 모델의 교육을 실행 중이며 12보다 큰 maxDepth를 늘릴 때마다 StackOverflow 오류가 발생합니다. 모든 것이 5,10,11에 대해 올바르게 작동합니다.

spark 2.0.2를 사용하고 있습니다. (앞으로 2 주 동안 업그레이드 할 수 없습니다). 3M 이상의 데이터, 200 개의 피쳐, 2500 개의 트리가 있으며 최대 깊이를 늘려 정확도를 높이고 싶습니다. 이 문제를 극복 할 수있는 방법이 있습니까?

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 92 in stage 789.0 failed 4 times, most recent failure: Lost task 92.3 in stage 789.0 (TID 66903, 10.0.0.11): java.lang.StackOverflowError 
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:3069) 
at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1643) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) 
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) 
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) 
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) 
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) 
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:479) 
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 

Answer 1

는 스파크 제출 옵션을 지정하는 것 같은데 :

--conf spark.executor.extraJavaOptions=-Xss8m --conf spark.driver.extraJavaOptions=-Xss8m 

이 작업을 수행합니다. 이 http://spark.apache.org/docs/latest/configuration.html