multithreading - Ruby Redis - Thread Performance -

i'm using ruby 1.8.7 , redis-rb gem. tried various approaches write 1 million records redis list.

i noticed performance suffering when i'm using threads. reason that, , how can prevent that? use redis in threaded environment production, i'm concerned.

here results:

1_000_000.times   $r.rpush "test_list", rand(1_000_000) end # took: 74.2986769676208 sec.  threads = [] 1_000.times   threads << thread.new {     1_000.times {       $r.rpush "test_list", rand(1_000_000)     }   } end threads.each { |k| k.join } # took: 391.705540895462 sec.  mutex = mutex.new threads = [] 1_000.times   threads << thread.new {     1_000.times {       mutex.synchronize {         $r.rpush "test_list", rand(1_000_000)       }     }   } end threads.each { |k| k.join } # took: 308.474660873413 sec.  mutex = mutex.new threads = [] 10.times   threads << thread.new {     100_000.times {       mutex.synchronize {         $r.rpush "test_list", rand(1_000_000)       }     }   } end threads.each { |k| k.join } # took: 109.103996992111 sec.  forks = [] 8.times   forks << fork {     125_000.times {       $r.rpush "test_list", rand(1_000_000)     }   } end forks.each { |k| process.wait(k) } # took: 23.7934968471527 sec. 

bump!

this example, shows lots of scenarios. should assume correct number of entries end in "test_list" on redis, because wonder first example... ? (a count useful see after threads joined)

so, interpretation:

1 thread, 1 million writes each

execution time: 74 seconds

single connection redis serialized writes

1000 threads, 100 writes each, no ruby mutex

execution time: 380 seconds

i think have assume redis using it's own mutex writes "overhead" here result of redis's own mutex. may due fact each thread sharing same client object write - did try creating 1000 redis clients? :)

1000 threads, 1000 writes each, ruby mutex

execution time: 308 seconds

it looks writes ("set") redis coming in more orderly fashion due mutex being inside ruby threads. here overhead 999 threads have wait 1 thread write redis...

10 threads, 100,000 writes each, local mutex

execution time: 108 seconds

less threads waiting around mutex each time, cycle of writes quicker.

8 forks, 125,000 writes each

execution time: 23 seconds

there still little overhead here, otherwise 8 different redis connections writing 125,000 keys each. redis laps up... "overhead" in forking of ruby process think (otherwise expect more 10 seconds)

thanks putting test - it's instructional, , going show our developers training exercise!