multithreading - C# Multithreaded Mass Parse -


i'm sort of making 'web parser', except 1 website, parsing many different pages @ 1 time.

currently, there may 300,000 pages need parse, in relatively fast manner (i'm grabbing tiny amount of information doesn't take long go through, every page takes ~3 seconds max on network). of course, 900,000 seconds days = 10 days, , terrible performance. reduce couple of hours @ most, i'm reasonable time amount of requests, still needs 'fast'. know can't 300,000 @ 1 time, or website block of requests, there have few seconds delay in between each , every request.

i have processing in single foreach loop, not taking advantage of multithreading ever, know take advantage of it, i'm not sure path should take whether threadpools, or type of threading system or design.

basically, i'm looking point me in right direction of efficiency using multithreading can ease time take parse many pages on end, sort of system or structure threading.

thanks

check out answer this question, sounds might want check out parallel.foreach.

there various other methods of achieving want in multi-threaded fashion. give idea of how works:

  1. download linqpad. (should prerequisite c# developer, imho!)
  2. in "samples", "download/import more samples..." , ensure have "asynchronous functions in c#" downloaded.
  3. work through samples, seeing how fit together.

in fact, here 1 of asynchronous examples works uris:

// await keyword useful when want run in loop. instance:  string[] uris = {     "http://linqpad.net",     "http://linqpad.net/downloadglyph.png",     "http://linqpad.net/linqpadscreen.png",     "http://linqpad.net/linqpadmed.png", };  // try doing following without await keyword!  int totallength = 0; foreach (string uri in uris) {     string html = await (new webclient().downloadstringtaskasync (new uri (uri)));     totallength += html.length; } totallength.dump();  // continuation not 'totallength += html.length', rest of loop! (and final // call 'totallength.dump()' @ end.)  // logically, execution exits method , returns caller upon reaching await statement. rather // 'yield return' (in fact, compiler uses same state-machine engine rewrite asynchronous // functions iterators). // // when task completes, continuation kicks off , execution jumps middle of  // loop - right left off!

Comments

Post a Comment

Popular posts from this blog

jquery - How can I dynamically add a browser tab? -

since using iframes house external pages are, accounts "not preferred method" , disallowed sites (flickr, one), want dynamically generate browser tabs. e.g., when user submits form, want dynamically generate tab on browser in response submitted. e.g., might want add new tab url like: http://www.bigsurgarrapata.com/contact how in jquery? you want form target attribute action attribute. http://www.w3schools.com/tags/att_form_target.asp browsers have ultimate control on result of target attribute open new tab value of _blank. <form action="contact_submit.php" target="_blank" method="post">
Read more

node.js - Getting the socket id,user id pair of a logged in user(s) -

i have web application done using node js , in it,i have authentication system requires users log in before can access pages. have 2 pages,the homepage , updates page. from read docs,a new socket id issued when user opens tab in browser. me being admin,i know socket id of logged in user , more so,the socket id,logged in user id pair logged in users. what method can use socket id,logged in user id pair make sure broadcast realtime info users online?.
Read more

keyboard - C++ GetAsyncKeyState alternative -

i using windows.h library's getasynckeystate function define key pushed on keyboard . here program returns int value of pushed key example. #include <iostream> #include <windows.h> using namespace std; int main (){ char i; for(i=8;i<190;i++){ if(getasynckeystate(i)== -32767){ cout<<int (i)<<endl; } } return 0; } but trying make opengl simple game , function appeared slow . there function not need cycle 180 times , if statement . thanks you're trying make opengl game, you're going use glfw. if that's case can away using glfwgetkey() such: if (glfwgetkey(glfw_key_up) == glfw_press) {...} if (glfwgetkey(glfw_key_down) == glfw_press) {...} that method nicely explained in tutorial 6 opengl-tutorial.org . should start first tutorial of beginners tutorials , later intermediate tutorials . if you're going make win32 window can listen wm_keydown or wm_keyup messages , virtua
Read more