Julia가도 커 컨테이너에 큰 데이터를로드하는 데 작동하지 않습니다.

Question

8GB의 데이터를로드하는 Julia 프로그램이 있습니다. 그것은 내 로컬 컴퓨터에서 잘 작동합니다.

하지만 도커 컨테이너에서 데이터를로드 할 때 버스 오류가 발생합니다. 그것은 도커 컨테이너에서 20 MB와 같은 작은 데이터로 잘 작동합니다.

Dockerfile

FROM ubuntu:16.04 

WORKDIR /julia 

RUN apt-get -y update 

RUN apt-get -y install unzip 

RUN apt-get -y install cmake 

RUN apt-get -y install clang 

RUN apt-get -y install wget 

RUN cd /tmp/ 

RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz" 

RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz 

RUN mv julia-3c9d75391c/ ~/julia 

ENV PATH="/root/julia/bin:${PATH}" 

RUN julia --eval 'Pkg.add("JSON")' 

RUN julia --eval 'Pkg.add("HttpServer")' 

RUN julia --eval 'Pkg.add("URIParser")' 

RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")' 

RUN julia --eval 'Pkg.build("AdaGram")' 

CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"] 

고정 표시기 - compose.yml

version: "3.1" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

다음을 내가 뭐하는 거지

sudo docker-compose up 
sudo docker exec -it $(sudo docker-compose ps -q julia) bash 
root@3748d5958f77:/julia# julia 
julia> using AdaGram 
julia> AdaGram.load_model("/julia/full.embed") 

어 도움의 모든 종류의 사전에 ROR

signal (7): Bus error 
while loading no file, in expression starting on line 0 
macro expansion at ./cartesian.jl:62 [inlined] 
macro expansion at ./multidimensional.jl:429 [inlined] 
_unsafe_batchsetindex! at ./multidimensional.jl:421 
_setindex! at ./multidimensional.jl:370 [inlined] 
setindex! at ./abstractarray.jl:832 [inlined] 
#9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64 
#600 at ./multi.jl:1030 
run_work_thunk at ./multi.jl:1001 
run_work_thunk at ./multi.jl:1010 [inlined] 
#597 at ./event.jl:68 
unknown function (ip: 0x7fe1822db16f) 
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined] 
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942 
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined] 
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253 
unknown function (ip: 0xffffffffffffffff) 
Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16 
Bus error (core dumped) 

도커 버전

Client: 
Version:  17.09.0-ce 
API version: 1.32 
Go version: go1.8.3 
Git commit: afdb6d4 
Built:  Tue Sep 26 22:42:18 2017 
OS/Arch:  linux/amd64 

Server: 
Version:  17.09.0-ce 
API version: 1.32 (minimum version 1.12) 
Go version: go1.8.3 
Git commit: afdb6d4 
Built:  Tue Sep 26 22:40:56 2017 
OS/Arch:  linux/amd64 
Experimental: false 

도커 정보는

Containers: 24 
Running: 0 
Paused: 0 
Stopped: 24 
Images: 24 
Server Version: 17.09.0-ce 
Storage Driver: overlay2 
Backing Filesystem: extfs 
Supports d_type: true 
Native Overlay Diff: true 
Logging Driver: json-file 
Cgroup Driver: cgroupfs 
Plugins: 
Volume: local 
Network: bridge host macvlan null overlay 
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog 
Swarm: active 
NodeID: jlkmigmtyjhz6yvi3zuvkobu7 
Is Manager: true 
ClusterID: rqt03ulgvvnym235m1qm8vd17 
Managers: 4 
Nodes: 15 
Orchestration: 
    Task History Retention Limit: 5 
Raft: 
    Snapshot Interval: 10000 
    Number of Old Snapshots to Retain: 0 
    Heartbeat Tick: 1 
    Election Tick: 3 
Dispatcher: 
    Heartbeat Period: 5 seconds 
CA Configuration: 
    Expiry Duration: 3 months 
    Force Rotate: 0 
Autolock Managers: false 
Root Rotation In Progress: false 
Node Address: X.X.X.X 
    Manager Addresses: 
    X.X.X.X:2377 
    X.X.X.X:2377 
    X.X.X.X:2377 
    X.X.X.X:2377 
Runtimes: runc 
Default Runtime: runc 
Init Binary: docker-init 
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0 
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64 
init version: 949e6fa 
Security Options: 
    apparmor 
    seccomp 
    Profile: default 
Kernel Version: 4.10.0-35-generic 
Operating System: Ubuntu 16.04.3 LTS 
OSType: linux 
Architecture: x86_64 
CPUs: 12 
Total Memory: 251.8GiB 
Name: ram 
ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH 
Docker Root Dir: /var/lib/docker 
Debug Mode (client): false 
Debug Mode (server): false 
Username: ramidavalapati 
Registry: https://index.docker.io/v1/ 
Experimental: false 
Insecure Registries: 
    127.0.0.0/8 
Live Restore Enabled: false 

WARNING: No swap limit support 

감사

link to 40MBfull.embed file

Answer 1

문제는 도커 컨테이너에 충분한 공유 메모리가 없음 (기본값은 64MB)입니다.

도커 이미지를 실행하는 동안 --shm-size 옵션을 지정하면 문제가 해결되었습니다. 우리가 swarm mode에서 작업 할 경우

고정 표시기-작성 파일

version: "3.1" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     shm_size: 1g 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

, 우리는 볼륨 섹션에서 공유 메모리를 참조 할 필요가있다.

version: "3.3" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
      - /dev/shm:/dev/shm 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

여기 컨테이너는 실행중인 호스트의 공유 메모리를 사용합니다.