cURL을 통해이 거친 봇 차단기를 우회 할 수 있습니까?

Question

나는 프로그램이 URL의 내용을 읽거나 긁어 할 수 있어야한다 : 나는 사용자 에이전트, 쿠키, 모든 종류의 설정을 시도했습니다

https://www.toysrus.com/product?productId=122872496

. 또한 아래의 컬을 bash는 명령을 제공 개발자 도구 -에서 명령> ​​네트워크 '컬로 복사'크롬을 시도했다 :

curl 'https://www.toysrus.com/product?productId=122872496' -H 'if-none-match: W/"2318d-c7pvMOCypyagcnOWwd+nRwWuY3M"' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-GB,en;q=0.8,en-US;q=0.6' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: max-age=0' -H 'authority: www.toysrus.com' -H 'cookie: GUID=15F299082881F0F; AMCVS_B9241CFE53309B6A0A490D45%40AdobeOrg=1; _abck=CA9FF35985405E426CA113364E2843B95C7A361ECC7F0000CAC6E559CE126315~0~gKgTXIlJpZdnvAbG5IkQwQDNivNRDOZlT2fWOD3JtYs=~-1~-1; entry_domain=toysrus; UrCapture=32c9b0fc-a726-20df-9a6f-62e5b646ae80; Omni_ShopperID=59e5c6d2f0ada2826d8ffee0; invodoSession=qooDReJQgHIivnpF47JvZK; invodoVisitor=A2PNHsy8xb7HkbYFs0WmFv; RES_TRACKINGID=582138174429307; ResonanceSegment=1; ak_bmsc=07593E6331CACE711A54A233912585F35C7A361E8B6000009068E85921E49529~pl51CsTeE5lRXhxDt77+3bSwhUsz0NHEbHDZviNsf5zsoLDiMPES8naaZBbn66eCN80VGsaW+FKJpBRsZmnYSIdqgplNwdRljB7RHos/Mz880QrJOIiIMx8wkCaJ2s5NYFQ3idkdIeql4t4RztiH2oRraA8Bue0r0Ck9wsc8yXmtDZ1hDdGPn4+LqZtrMhlDxZ6cwu1TODBrQSBd5Ni5eGTRvJe1/oauUHr9ztu6ayRaw=; bm_sz=3234D15C4A9BAF5E11789569DCD4FD6C~QAAQHjZ6XO9Ayy5fAQAAt3TYMwFkUQkzB9j2O8d5YvvKQfRZdHumrTmjvdQ1iPQKYi+F0p7HcKT0cVl7JYImlqc9DVfB6C3At4C+9S4b+B4iLA2IAi9fU7gcH6GfDoQxi0iPBbQLk98gkY1cqAPfQbUsjwcUbJ0l+JYY7QjWSET3e054Al8z5jVSTGpFVqlx; AMCV_B9241CFE53309B6A0A490D45%40AdobeOrg=-1330315163%7CMCIDTS%7C17459%7CMCMID%7C52999333410090394193783160145267952168%7CMCAAMLH-1508835659%7C6%7CMCAAMB-1509008146%7CxfQZlSXQfPyHZHNXzhww8jeiMq5K2sHrXyNvM9qY7m_hTGU%7CMCOPTOUT-1508410546s%7CNONE%7CMCAID%7CNONE; TRUSESSIONID=0KYz7VGR4y7wsZpUXcvpdLFQfnRdJxJJf1w6M9JbLJVNqzu1xIPV!-1065198256; mbox=PC#7086875003bc497a93590bb613c34b50.17_21#1509615639|session#888c37d2b3124c40878eb0a1792e3265#1508407899|check#true#1508406099; s_ppvl=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C759%2C1536%2C864%2C1.25%2CL; s_cc=true; __utma=136881205.1239421491.1508230865.1508350392.1508403351.3; __utmc=136881205; __utmz=136881205.1508403351.3.3.utmcsr=lookingforagooddeal.com|utmccn=(referral)|utmcmd=referral|utmcct=/product/test-product/; __pr.7tp=rgrgcsjgwm; utag_main=v_id:015f29909284001f515df5e18dfc04073003306b00978$_sn:3$_ss:0$_st:1508407846067$ses_id:1508403348621%3Bexp-session$_pn:5%3Bexp-session; _cp_dtm_store_name=toysrus; OAS_SC1=1508406046406; IR_PI=1508230866697.bml86prs4m; _br_uid_2=uid%3D6871658334943%3Av%3D12.0%3Ats%3D1508230867182%3Ahc%3D10; PrefID=21-5485403154; s_nr=1508407773155-Repeat; s_previousPageName=en_US%3A%20TRU%3A%20Product%20Detail%3A%20122872496%3A%20LEGO%20City%20Jungle%20Explorers%20Jungle%20Halftrack%20Mission%20%2860159%29; s_cpm=%5B%5B%27ch%27%2C%271508407773158%27%5D%5D; TRUSESSIONID_SET_TIMESTAMP=1508407838524; bm_sv=08D1BE1D2408D1E12F94213DC168DABC~B3fBu73AESdKjl6l/qB2U3aCBKuZw4SCcjmyc+h+CWYxT1NJ0fXXP6GlFba3e+tKujhnRf47tnMJHAn83pK816cqr6bxeL5m0yn1yo9SPOiYNXSZsE35r/9sYgAQhRWDl7Gd4AljS43B80m36ONZMb6q9kL9aT3oJ3sooY69nAE=; s_ppv=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C419%2C1536%2C864%2C1.25%2CL; RT="sl=0&ss=1508406037346&tt=0&obo=0&sh=&dm=toysrus.com&si=d98b4ff0-ddd2-47e6-8a7c-761ce2d21e15&bcn=%2F%2F36e4f0e7.akstat.io%2F&ld=1508406046108&r=https%3A%2F%2Fwww.toysrus.com%2Fproduct%3F328e8990a18940b61934f007b9211205&ul=1508407960823"' -H 'referer: https://foundthru.com/product/lego-city-jungle-explorers/' --compressed 

연결 동수,하지만 그것을 밖으로 단지 시간.

가 나는 또한 시도했다 :

이를 제공하고 (나는 시간 제한을 추가하지 않는 한)를 영원히 중단

wget http://www.toysrus.com 

:

는 또한 노드를 사용하여 시도했습니다. js, phantomjs 및 기수는 다음과 같은 코드를 사용합니다.

horseman 
    .cookies(params.cookies_file_path) 
    .open(params.url) 
    .status() 
    .then(function(status) { 
     horseman.html() 
     .then(function(html) { 
       callback({status: status, data: html}); 
       return horseman.close(); 
     }); 
    }) 
    .catch(function (err) { 
     console.log(err); 
     horseman.close(); 
     callback(false); 
    }); 

무엇이 누락 되었습니까? 이 페이지를 가져 오려면 어떻게해야합니까?

Answer 1

PHP에서 많이 사용합니다. 내가 게시 한 내용을 환경에 맞게 적용 할 수 있습니다.

봇 차단제가 없습니다. 이것은 간단한 HTTPS입니다 : //이
당신은 가능성이 HTTPS로 리디렉션을 수행하지 끊었지고 GET :에서 https //

변경 URL : //
사용하여 HTTP : 리디렉션에 발생 // https : //로
그럼 당신이 원하지 않는 SSL 인증서 문제 :
이 curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

추가이 코드는 테스트 및 작동 :

<?php 
header('Content-Type: text/plain; charset=utf-8'); 
$request = array(); 
$request[] = 'host:www.toysrus.com'; 
$request[] = 'Connection: keep-alive'; 
$request[] = 'Pragma: no-cache'; 
$request[] = 'Cache-Control: no-cache'; 
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'; 
$request[] = 'User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'; 
$request[] = 'DNT: 1'; 
$request[] = 'Accept-Encoding: gzip, deflate'; 
$request[] = 'Accept-Language: en-US,en;q=0.8'; 


$url = 'https://www.toysrus.com/product?productId=122872496'; 
$ch = curl_init($url); 

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
curl_setopt($ch, CURLOPT_POST, false); 
curl_setopt($ch, CURLOPT_HTTPHEADER, $request); 
curl_setopt($ch, CURLOPT_ENCODING,""); 
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); 
curl_setopt($ch, CURLOPT_TIMEOUT,10); 
curl_setopt($ch, CURLOPT_FAILONERROR,true); 
curl_setopt($ch, CURLOPT_ENCODING,""); 
$data = curl_exec($ch); 
if (curl_errno($ch)){ 
    echo 'Retreive Base Page Error: ' . curl_error($ch); 
} 
else { 
    $info = rawurldecode(var_export(curl_getinfo($ch),true)); 
    echo $data; 

} 
?>