Linux, bash 스크립트에서 메모리 부족, 대화 형이 아닌 ulimit?

Question

은 무엇 리눅스 설정은 C++ 프로그램을 초래하기 때문에 실행할 수 있습니다 :

jellyfish count -m 31 -t 40 -C -s 105 -o k_u_hash_0 pe.cor.fa 

그 명령을 터미널에서 실행하면 잘 작동하지만, bash는 스크립트에서 충돌? 후자의 경우에는 종료하기 전에 411428571480 바이트를 요청합니다. 즉각적으로 종료합니다. 대화식으로 실행하면 위로이 실행되기 시작한 지 몇 초 만에 Virt and Res 메모리의 Gb가 10로 표시되므로 이상합니다. 두 환경에서

ulimit -a 

보여준다 :

core file size   (blocks, -c) 0 
data seg size   (kbytes, -d) unlimited 
scheduling priority    (-e) 0 
file size    (blocks, -f) unlimited 
pending signals     (-i) 2067197 
max locked memory  (kbytes, -l) 64 
max memory size   (kbytes, -m) unlimited 
open files      (-n) 1024 
pipe size   (512 bytes, -p) 8 
POSIX message queues  (bytes, -q) 819200 
real-time priority    (-r) 0 
stack size    (kbytes, -s) 10240 
cpu time    (seconds, -t) unlimited 
max user processes    (-u) 1024 
virtual memory   (kbytes, -v) unlimited 
file locks      (-x) unlimited 

은/proc/meminfo를 보여준다

CommitLimit: 615693660 kB 
Committed_AS: 48320500 kB 

심지어 낯선 단지보다 적 큰 메모리 블록은 calloc의 작은 C 테스트 프로그램을 해파리 직전을 포함하여 어느 환경에서든 실행하면 다음을 수행합니다.

calloc_test 
Testing 4294967296 
Testing 8589934592 
Testing 17179869184 
Testing 34359738368 
Testing 68719476736 
Testing 137438953472 
Testing 274877906944 
Testing 549755813888 
Testing 1099511627776 
FAILED 

즉, 해파리가 요구 한 것보다 큰 549GB 블록을 할당 할 수 있었고 메모리 할당은 두 환경 모두에서 동일하게 작동했습니다.

LD_LIBRARY_PATH이 설정되어 있지 않습니다.

메모리 할당 프로그램의 하위 집합의 동작 차이를 설명하기 위해 두 환경에서 다른 점을 제안 할 수 있습니까? (이 시점에서 하나의 하위 집합입니다.)

감사합니다.

요청으로

,이 스크립트 (만 실패 지점에 이르기까지 플러스 3 개 추가 라인)입니다 :

#!/bin/bash 

# assemble.sh generated by masurca 
CONFIG_PATH="/home/mathog/do_masurca/project.cfg" 
CMD_PATH="/home/mathog/MaSuRCA/bin/masurca" 

# Test that we support <() redirection 
(eval "cat <(echo test) >/dev/null" 2>/dev/null) || { 
    echo >&2 "ERROR: The shell used is missing important features." 
    echo >&2 "  Run the assembly script directly as './$0'" 
    exit 1 
} 

# Parse command line switches 
while getopts ":rc" o; do 
    case "${o}" in 
    c) 
    echo "configuration file is '$CONFIG_PATH'" 
    exit 0 
    ;; 
    r) 
    echo "Rerunning configuration" 
    exec perl "$CMD_PATH" "$CONFIG_PATH" 
    echo "Failed to rerun configuration" 
    exit 1 
    ;; 
    *) 
    echo "Usage: $0 [-r] [-c]" 
    exit 1 
    ;; 
    esac 
done 
set +e 
# Set some paths and prime system to save environment variables 
save() { 
    (echo -n "$1=\""; eval "echo -n \"\$$1\""; echo '"') >> environment.sh 
} 
GC= 
RC= 
NC= 
if tty -s < /dev/fd/1 2> /dev/null; then 
    GC='\e[0;32m' 
    RC='\e[0;31m' 
    NC='\e[0m' 
fi 
log() { 
    d=$(date) 
    echo -e "${GC}[$d]${NC} $@" 
} 
fail() { 
    d=$(date) 
    echo -e "${RC}[$d]${NC} $@" 
    exit 1 
} 
signaled() { 
    fail Interrupted 
} 
trap signaled TERM QUIT INT 
rm -f environment.sh; touch environment.sh 

# To run tasks in parallel 
run_bg() { 
    semaphore -j $NUM_THREADS --id masurca_$$ -- "$@" 
} 
run_wait() { 
    semaphore -j $NUM_THREADS --id masurca_$$ --wait 
} 
export PATH="/home/mathog/MaSuRCA/bin:/home/mathog/MaSuRCA/bin/../CA/Linux-amd64/bin:$PATH" 
save PATH 
export PERL5LIB=/home/mathog/MaSuRCA/bin/../lib/perl${PERL5LIB:+:$PERL5LIB} 
save PERL5LIB 
NUM_THREADS=40 
save NUM_THREADS 
log 'Processing pe library reads' 
rm -rf meanAndStdevByPrefix.pe.txt 
echo 'pe 400 20' >> meanAndStdevByPrefix.pe.txt 
run_bg rename_filter_fastq 'pe' <(exec expand_fastq '/home/mathog/SPUR_datasets/pe_400_R1.fastq' | awk '{if(length($0>200)) print substr($0,1,200); else print $0;}') <(exec expand_fastq '/home/mathog/SPUR_datasets/pe_400_R2.fastq' | awk '{if(length($0>200)) print substr($0,1,200); else print $0;}') > 'pe.renamed.fastq' 
run_wait 

head -q -n 40000 pe.renamed.fastq | grep --text -v '^+' | grep --text -v '^@' > pe_data.tmp 
export PE_AVG_READ_LENGTH=`awk '{if(length($1)>31){n+=length($1);m++;}}END{print int(n/m)}' pe_data.tmp` 
save PE_AVG_READ_LENGTH 
echo "Average PE read length $PE_AVG_READ_LENGTH" 
KMER=`for f in pe.renamed.fastq;do head -n 80000 $f |tail -n 40000;done | perl -e 'while($line=<STDIN>){$line=<STDIN>;chomp($line);push(@lines,$line);$line=<STDIN>;$line=<STDIN>}$min_len=100000;$base_count=0;foreach $l(@lines){$base_count+=length($l);push(@lengths,length($l));@f=split("",$l);foreach $base(@f){if(uc($base) eq "G" || uc($base) eq "C"){$gc_count++}}} @lengths =sort {$b <=> $a} @lengths; $min_len=$lengths[int($#lengths*.75)]; $gc_ratio=$gc_count/$base_count;$kmer=0;if($gc_ratio<0.5){$kmer=int($min_len*.7);}elsif($gc_ratio>=0.5 && $gc_ratio<0.6){$kmer=int($min_len*.5);}else{$kmer=int($min_len*.33);} $kmer++ if($kmer%2==0); $kmer=31 if($kmer<31); $kmer=127 if($kmer>127); print $kmer'` 
save KMER 
echo "choosing kmer size of $KMER for the graph" 
KMER_J=$KMER 
MIN_Q_CHAR=`cat pe.renamed.fastq |head -n 50000 | awk 'BEGIN{flag=0}{if($0 ~ /^\+/){flag=1}else if(flag==1){print $0;flag=0}}' | perl -ne 'BEGIN{$q0_char="@";}{chomp;@f=split "";foreach $v(@f){if(ord($v)<ord($q0_char)){$q0_char=$v;}}}END{$ans=ord($q0_char);if($ans<64){print "33\n"}else{print "64\n"}}'` 
save MIN_Q_CHAR 
echo MIN_Q_CHAR: $MIN_Q_CHAR 
JF_SIZE=`ls -l *.fastq | awk '{n+=$5}END{s=int(n/50); if(s>80000000000)printf "%.0f",s;else print "80000000000";}'` 
save JF_SIZE 
perl -e '{if(int('$JF_SIZE')>80000000000){print "WARNING: JF_SIZE set too low, increasing JF_SIZE to at least '$JF_SIZE', this automatic increase may be not enough!\n"}}' 
log Creating mer database for Quorum. 
quorum_create_database -t 40 -s $JF_SIZE -b 7 -m 24 -q $((MIN_Q_CHAR + 5)) -o quorum_mer_db.jf.tmp pe.renamed.fastq && mv quorum_mer_db.jf.tmp quorum_mer_db.jf 
if [ 0 != 0 ]; then 
    fail Increase JF_SIZE in config file, the recommendation is to set this to genome_size*coverage/2 
fi 

log Error correct PE. 

quorum_error_correct_reads -q $((MIN_Q_CHAR + 40)) --contaminant=/home/mathog/MaSuRCA/bin/../share/adapter.jf -m 1 -s 1 -g 1 -a 3 -t 40 -w 10 -e 3 -M quorum_mer_db.jf pe.renamed.fastq --no-discard -o pe.cor --verbose 1>quorum.err 2>&1 || { 
    mv pe.cor.fa pe.cor.fa.failed && fail Error correction of PE reads failed. Check pe.cor.log. 
} 


log Estimating genome size. 
jellyfish count -m 31 -t 40 -C -s $JF_SIZE -o k_u_hash_0 pe.cor.fa 
export ESTIMATED_GENOME_SIZE=`jellyfish histo -t 40 -h 1 k_u_hash_0 | tail -n 1 |awk '{print $2}'` 
save ESTIMATED_GENOME_SIZE 
echo "Estimated genome size: $ESTIMATED_GENOME_SIZE" 

Answer 1

여기에 스크립트의 (일부) 쓸모없는 메모리와 CPU 사이클 낭비 :

cat pe.renamed.fastq | head -n 50000 | ... 

은

대신

head -n 50000 pe.renamed.fastq | ...