php - How to write a bot that does not consume much RAM? -


i have web bot , consumes memory much, after time, memory usage hits 50%, , process gets killed; have no idea why memory usage increasing that, did not include "para.php" library parallel curl requests. want know more things web crawlers, searched lot, not find helpful document or methods can use.

this library obtained para.php.

my code:

require_once "para.php";  class crawling{  public $montent;   public function crawl_page($url){      $m = new mongo();      $muun = $m->howto->en->findone(array("_id" => $url));      if (isset($muun)) {         return;     }      $m->howto->en->save(array("_id" => $url));      echo $url;      echo "\n";      $para = new parallelcurl(10);      $para->startrequest($url, array($this,'on_request_done'));      $para->finishallrequests();      preg_match_all("(<a href=\"(.*)\")siu", $this->montent, $matk);      foreach($matk[1] $longu){         $href = $longu;         if (0 !== strpos($href, 'http')) {             $path = '/' . ltrim($href, '/');             if (extension_loaded('http')) {                 $href = http_build_url($url, array('path' => $path));             } else {                 $parts = parse_url($url);                 $href = $parts['scheme'] . '://';                 if (isset($parts['user']) && isset($parts['pass'])) {                     $href .= $parts['user'] . ':' . $parts['pass'] . '@';                 }                   $href .= $parts['host'];                 if (isset($parts['port'])) {                     $href .= ':' . $parts['port'];                 }                 $href .= $path;             }         }         $this->crawl_page($longu);     } }  public function on_request_done($content) {     $this->montent = $content; }   $moj = new crawling; $moj->crawl_page("http://www.example.com/");

you call crawl_page function on 1 url. it's content fetched ($this->montent) , checked links ($matk).

while these not yet destroyed, go recursive, starting new call crawl_page. $this->moment overwritten new content (that's ok). bit further down, $matk (a new variable) populated links new $this->montent. @ point, there 2 $matk's in memory: 1 links document started processing first, , 1 links document first linked in original document.

i'd suggest find links & save them database (instead of going recursive). clear queue of links in database, 1 1 (with each new document adding new entry database)


Comments

Post a Comment

Popular posts from this blog

jquery - How can I dynamically add a browser tab? -

since using iframes house external pages are, accounts "not preferred method" , disallowed sites (flickr, one), want dynamically generate browser tabs. e.g., when user submits form, want dynamically generate tab on browser in response submitted. e.g., might want add new tab url like: http://www.bigsurgarrapata.com/contact how in jquery? you want form target attribute action attribute. http://www.w3schools.com/tags/att_form_target.asp browsers have ultimate control on result of target attribute open new tab value of _blank. <form action="contact_submit.php" target="_blank" method="post">
Read more

node.js - Getting the socket id,user id pair of a logged in user(s) -

i have web application done using node js , in it,i have authentication system requires users log in before can access pages. have 2 pages,the homepage , updates page. from read docs,a new socket id issued when user opens tab in browser. me being admin,i know socket id of logged in user , more so,the socket id,logged in user id pair logged in users. what method can use socket id,logged in user id pair make sure broadcast realtime info users online?.
Read more

keyboard - C++ GetAsyncKeyState alternative -

i using windows.h library's getasynckeystate function define key pushed on keyboard . here program returns int value of pushed key example. #include <iostream> #include <windows.h> using namespace std; int main (){ char i; for(i=8;i<190;i++){ if(getasynckeystate(i)== -32767){ cout<<int (i)<<endl; } } return 0; } but trying make opengl simple game , function appeared slow . there function not need cycle 180 times , if statement . thanks you're trying make opengl game, you're going use glfw. if that's case can away using glfwgetkey() such: if (glfwgetkey(glfw_key_up) == glfw_press) {...} if (glfwgetkey(glfw_key_down) == glfw_press) {...} that method nicely explained in tutorial 6 opengl-tutorial.org . should start first tutorial of beginners tutorials , later intermediate tutorials . if you're going make win32 window can listen wm_keydown or wm_keyup messages , virtua
Read more