GPU가 theano 예측에 관해서 CPU보다 느린가요?

Question

나는 keras + theano를 사용하여 NVidia TK1의 VGG 사전 학습 모델에서 라벨을 예측했습니다.

예측에서 GPU보다 예측 시간이 CPU에서 더 빨라지고 있습니다. 내 기억이 정확하다면 예측은 반복적 인 방식으로 많은 수의 계산을 포함합니다. 나는 왜 CPU가 여기 왜 더 느릴지 이해하지 못한다.

누구든지 좋은 설명이 있습니까?

GPU는 세부 행 : Using gpu device 0: GK20A (CNMeM is enabled with initial size: 75.0% of memory, cuDNN Version is too old. Update to v5, was 2000.)

여기서 예측 프로파일 링 결과를 온다 :

Class 
--- 
<% time> <sum %> <apply time> <time per call> <type> <#call> <#apply> <Class name> 
    39.5% 39.5%  0.019s  6.42e-03s  C  3  3 theano.sandbox.cuda.blas.GpuDot22 
    24.8% 64.3%  0.012s  6.04e-03s  C  2  2 theano.sandbox.cuda.blas.GpuCorrMM 
    16.4% 80.8%  0.008s  1.33e-03s  C  6  6 theano.sandbox.cuda.basic_ops.GpuElemwise 
    7.8% 88.5%  0.004s  1.89e-03s  C  2  2 theano.sandbox.cuda.blas.GpuDownsampleFactorMax 
    4.2% 92.7%  0.002s  2.03e-03s  C  1  1 theano.sandbox.rng_mrg.GPU_mrg_uniform 
    3.8% 96.4%  0.002s  4.57e-04s  C  4  4 theano.sandbox.cuda.basic_ops.GpuContiguous 
    2.3% 98.8%  0.001s  5.66e-04s  C  2  2 theano.sandbox.cuda.basic_ops.GpuFromHost 
    0.5% 99.3%  0.000s  2.51e-04s  C  1  1 theano.sandbox.cuda.nnet.GpuSoftmaxWithBias 
    0.5% 99.8%  0.000s  2.39e-04s  C  1  1 theano.sandbox.cuda.basic_ops.HostFromGpu 
    0.1% 99.8%  0.000s  1.37e-05s  C  3  3 theano.sandbox.cuda.basic_ops.GpuReshape 
    0.0% 99.9%  0.000s  9.54e-06s  C  2  2 theano.sandbox.cuda.basic_ops.GpuSubtensor 
    0.0% 99.9%  0.000s  4.35e-06s  C  4  4 theano.tensor.elemwise.Elemwise 
    0.0% 99.9%  0.000s  5.01e-06s  C  2  2 theano.sandbox.cuda.basic_ops.GpuDimShuffle 
    0.0% 100.0%  0.000s  3.26e-06s  C  3  3 theano.compile.ops.Shape_i 
    0.0% 100.0%  0.000s  4.53e-06s  C  2  2 theano.tensor.opt.MakeVector 
    0.0% 100.0%  0.000s  5.96e-06s  C  1  1 theano.tensor.elemwise.Prod 
    0.0% 100.0%  0.000s  3.10e-06s  C  1  1 theano.tensor.elemwise.DimShuffle 
    ... (remaining 0 Classes account for 0.00%(0.00s) of the runtime) 

Ops 
--- 
<% time> <sum %> <apply time> <time per call> <type> <#call> <#apply> <Op name> 
    39.5% 39.5%  0.019s  6.42e-03s  C  3  3 GpuDot22 
    24.8% 64.3%  0.012s  6.04e-03s  C  2  2 GpuCorrMM{valid, (1, 1)} 
    11.2% 75.5%  0.005s  1.36e-03s  C  4  4 GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)] 
    7.8% 83.3%  0.004s  1.89e-03s  C  2  2 GpuDownsampleFactorMax{(2, 2),True} 
    4.2% 87.4%  0.002s  2.03e-03s  C  1  1 GPU_mrg_uniform{CudaNdarrayType(float32, 4D),inplace} 
    3.8% 91.2%  0.002s  4.57e-04s  C  4  4 GpuContiguous 
    2.9% 94.1%  0.001s  1.43e-03s  C  1  1 GpuElemwise{Composite{Cast{float32}(LT(i0, i1))}}[(0, 0)] 
    2.3% 96.5%  0.001s  5.66e-04s  C  2  2 GpuFromHost 
    2.3% 98.8%  0.001s  1.12e-03s  C  1  1 GpuElemwise{Composite{Switch(i0, (i1 * i2 * i3), i2)}}[(0, 2)] 
    0.5% 99.3%  0.000s  2.51e-04s  C  1  1 GpuSoftmaxWithBias 
    0.5% 99.8%  0.000s  2.39e-04s  C  1  1 HostFromGpu 
    0.1% 99.8%  0.000s  1.60e-05s  C  2  2 GpuReshape{4} 
    0.0% 99.9%  0.000s  9.54e-06s  C  2  2 GpuSubtensor{::, ::, ::int64, ::int64} 
    0.0% 99.9%  0.000s  5.01e-06s  C  2  2 GpuDimShuffle{x,0} 
    0.0% 99.9%  0.000s  4.53e-06s  C  2  2 MakeVector{dtype='int64'} 
    0.0% 99.9%  0.000s  9.06e-06s  C  1  1 GpuReshape{2} 
    0.0% 99.9%  0.000s  4.17e-06s  C  2  2 Elemwise{Composite{((i0 + ((i1 + i2) // i3)) // i3)}}[(0, 2)] 
    0.0% 100.0%  0.000s  5.96e-06s  C  1  1 Prod{acc_dtype=int64} 
    0.0% 100.0%  0.000s  5.96e-06s  C  1  1 Elemwise{Cast{float32}} 
    0.0% 100.0%  0.000s  5.01e-06s  C  1  1 Shape_i{0} 
    ... (remaining 4 Ops account for 0.02%(0.00s) of the runtime) 

Apply 
------ 
<% time> <sum %> <apply time> <time per call> <#call> <id> <Apply name> 
    36.4% 36.4%  0.018s  1.77e-02s  1 33 GpuDot22(GpuReshape{2}.0, dense_5_W) 
    15.7% 52.1%  0.008s  7.64e-03s  1 18 GpuCorrMM{valid, (1, 1)}(GpuContiguous.0, GpuContiguous.0) 
    9.1% 61.2%  0.004s  4.44e-03s  1 28 GpuCorrMM{valid, (1, 1)}(GpuContiguous.0, GpuContiguous.0) 
    5.7% 66.9%  0.003s  2.76e-03s  1 25 GpuDownsampleFactorMax{(2, 2),True}(GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)].0) 
    4.2% 71.0%  0.002s  2.03e-03s  1 20 GPU_mrg_uniform{CudaNdarrayType(float32, 4D),inplace}(<CudaNdarrayType(float32, vector)>, MakeVector{dtype='int64'}.0) 
    3.6% 74.6%  0.002s  1.74e-03s  1 34 GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[ 0.5]]}, GpuDot22.0, GpuDimShuffle{x,0}.0) 
    3.2% 77.8%  0.002s  1.54e-03s  1 22 GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[[[ 0.5]]]]}, GpuCorrMM{valid, (1, 1)}.0, GpuReshape{4}.0) 
    2.9% 80.7%  0.001s  1.43e-03s  1 23 GpuElemwise{Composite{Cast{float32}(LT(i0, i1))}}[(0, 0)](GPU_mrg_uniform{CudaNdarrayType(float32, 4D),inplace}.1, CudaNdarrayConstant{[[[[ 0.80000001]]]]}) 
    2.7% 83.4%  0.001s  1.29e-03s  1 36 GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[ 0.5]]}, GpuDot22.0, GpuDimShuffle{x,0}.0) 
    2.3% 85.7%  0.001s  1.12e-03s  1 31 GpuElemwise{Composite{Switch(i0, (i1 * i2 * i3), i2)}}[(0, 2)](GpuFromHost.0, CudaNdarrayConstant{[[[[ 1.25]]]]}, GpuDownsampleFactorMax{(2, 2),True}.0, GpuElemwise{Composite{Cast{float32}(LT(i0, i1))}}[(0, 0)].0) 
    2.2% 87.8%  0.001s  1.06e-03s  1 14 GpuContiguous(GpuSubtensor{::, ::, ::int64, ::int64}.0) 
    2.2% 90.0%  0.001s  1.06e-03s  1 35 GpuDot22(GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)].0, dense_6_W) 
    2.1% 92.1%  0.001s  1.01e-03s  1 30 GpuDownsampleFactorMax{(2, 2),True}(GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)].0) 
    2.0% 94.1%  0.001s  9.61e-04s  1  3 GpuFromHost(convolution2d_input_1) 
    1.8% 95.9%  0.001s  8.71e-04s  1 29 GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)](CudaNdarrayConstant{[[[[ 0.5]]]]}, GpuCorrMM{valid, (1, 1)}.0, GpuReshape{4}.0) 
    1.6% 97.4%  0.001s  7.58e-04s  1 15 GpuContiguous(GpuSubtensor{::, ::, ::int64, ::int64}.0) 
    1.0% 98.4%  0.000s  4.72e-04s  1 37 GpuDot22(GpuElemwise{Composite{(i0 * ((i1 + i2) + Abs((i1 + i2))))}}[(0, 1)].0, dense_7_W) 
    0.5% 98.9%  0.000s  2.51e-04s  1 38 GpuSoftmaxWithBias(GpuDot22.0, dense_7_b) 
    0.5% 99.4%  0.000s  2.39e-04s  1 39 HostFromGpu(GpuSoftmaxWithBias.0) 
    0.3% 99.8%  0.000s  1.70e-04s  1 19 GpuFromHost(Elemwise{Cast{float32}}.0) 
    ... (remaining 20 Apply instances account for 0.25%(0.00s) of the runtime) 

Answer 1

이 keras + theano이 예측을 수행 할 때, 처음으로 느린 것은 나온다. 모델을 keras.models.load_model으로로드 한 후 모델은 아직 메모리에 완전히 나타나지 않으며 주먹 예측 호출은 나머지 설정을 처리합니다.

첫 번째 예측 이후 나머지 예측은 매우 빠르게 진행됩니다. VGG 모델에 대한 첫 번째 예측은 3 초 정도 걸리지 만 후속 예측에는 15 ~ 2 초가 소요됩니다. 좋아.