Terminator - Timeout without Mercy

2008-09-10 17:40:04 +0000

If you have been following my posts on Ruby-Talk and Ruby on Rails and even RSpec mailing list (and who wouldn’t?! I mean, aside from my mother) then you would have noticed I have been banging my head against a brick wall on the subject of System calls not being handled by the Timeout libraries in Ruby…

I’ve done a lot of work with this. I mean a lot.

So I thought I would share.

It all starts with a basic requirement I have, which is to replicate a data set from one side of the planet to the other. This involves logging into a database server remotely using ActiveRecord across some encrypted VPNs, querying some tables, and then inside of a transaction, replicating down certain data on certain rows.

The solution I have works well, I can replicate about 2-5 rows per second across the link which is more than adequate (the dataset changes upto about 100 rows per minute). The problem is that occasionally, ActiveRecord’s link to the remote database will hit a snag, then Ruby will wait, and wait, and wait to Timeout…. This can take hours, and in fact, sometimes, it never does timeout, leaving a zombie process.

To avoid multiple replication problems, I have lock files which prevent further copies of the replicator firing up if it detects that one is already running.

So, in the end, I get stuck with several replicators “running”, all hung, all waiting for a link to come back that never will, and no replication happening. In the mean time, the rows are still changing and I am getting backlogged. Leave it for a day and you are 80,000 rows backlogged and people start asking questions and I start looking for old travel tickets that I had ‘forgotten about’.

But why does ruby do this?

Shouldn’t the Timeout library protect us from such evilness?

Well, yes and no. The problem is because ruby uses Green threads. Green threads mean that there is only one real Ruby process running on the server, and it “internally” makes other “threads” that it schedules within the main ruby process’ kernel time. Green threads are very efficient (they save a lot of context switching) but the problem is that if you get a System call that ‘blocks’, then the kernel will stop servicing your main ruby thread until your system call is complete, and THIS means that the little ‘Timeout.timeout(5) { my code }’ block you put in, will never get called…

And then you get a hang… forever.

I tried a couple of handlings, the first was the SystemTimer library by Philippe Hanrigou. This entry actually has a really good and simple example of what Green threads and blocking system calls mean. If you want to brush up, it is a good read.

But SystemTimer didn’t work. It uses alarm signals which could have some problems. But in my case, it didn’t work and I still got never ending timeouts.

So I asked again on the Ruby-Talk mailing list and Ara T. Howard came back with a quick script that created little homicidal external ruby processes to kill the ruby process I was in if it didn’t “make it” in time.

The approach was actually very clever, and it works!

What he did was go “We can’t be sure that the existing Ruby process will be able to nuke itself, so instead, lets start up another Ruby instance, running on it’s own, that has the PIDs of the ruby instance that spawned it, and if the time runs out, do a system based kill TERM on the pids.”

You could say that our little ruby processes have a license to kill. Literally.

So I implemented his idea, found a couple of problems, gave back some ideas, and he coded, I spec’d, tested and described and we released a brand new gem…

The Terminator.

You can get it with;



gem install terminator

Or grab the source code.

And how do you use it? Simple:

1
2
3
4
5

require 'terminator'
Terminator.terminate 1 do
  sleep 2
  puts "I'll never print"
end

This will never print because the terminator times out after 1 second, which is before the sleep of 2 seconds inside the block. This will raise a Terminator::Error, which you could catch and try again if you want.

This will always work. It is because we are starting a separate process of Ruby (which has some minor overhead) that waits the specified number of seconds and then just simple does a system kernel TERM on our misbehaving process.

Why should you use it?

Well, if you are making ANY calls to external web services, external databases, OpenID, Youtube, Google Maps… anything, then you should have a fail fast policy in place and time out rapidly if these fail. As these are external system calls, they will most likely not be caught by Ruby’s timeout.rb library… and that means that your application will just hang, waiting for the call that never comes back.

It is much better to go “Ok, 2 seconds are up, no response, let’s tell the user to try again in a minute” and render an appropriate message, than have the user frustratingly whack the reload button a few hundred more times wondering why your application is not responding.

So yes, I think you should use Terminator.

Besides, how many other gems do you know that have the method ‘plot_to_kill’ ?

blogLater

Mikel

  1. Mikel Says:
    2008-11-21 21:35:47 +0000

    @Doug and @Phil

    Hmm… are you guys running on Windows?

    Mikel

  2. pistos Says:
    2008-09-11 17:39:37 +0000

    You should maybe link to the source code ( http://codeforpeople.com/lib/ruby/terminator ) for those that want to peruse the code online. :)

  3. Ahsan Says:
    2008-09-16 22:01:06 +0000

    So .. a ruby process will be spawned for every request to a web service .. ?

  4. Mikel Lindsaar Says:
    2008-09-20 08:07:12 +0000

    @Ahsan: That was working here. I just did a clean up on the initial code and added documentation for a 0.4.4 release. Maybe update and try again.

  5. Ahsan Says:
    2008-09-23 00:43:23 +0000

    @Mikel: Ok, I upgraded to 0.4.4, but the problem persists.

  6. Mikel Lindsaar Says:
    2008-09-23 01:39:38 +0000

    @Ahsan, could you pastie your code?

  7. Richy Says:
    2010-01-21 05:26:31 +0000

    I don’t believe “Terminator” supports windows at the moment guys… However if your using “Terminator” with Ruby on Rails via a webhost or apphost. I don’t believe there will be any issues, due to the fact that most of them are using Linux or Unix based OS’s.

  8. Mikel Lindsaar Says:
    2010-01-21 18:44:48 +0000

    No, Terminator will not run on a windows environment, sorry guys and gals.

  9. Hsiu Says:
    2011-10-04 09:33:04 +0000

    I get this error when trying to use this:

    terminator-0.4.4/lib/terminator.rb:127:in `block in terminate’: wrong number of arguments (1 for 0) (ArgumentError)

    Line 127 is of terminator.rb is this:

    trap = getopt :trap, options, lambda{ eval(“raise(::Terminator::Error, ‘Timeout out after #{ seconds }s’)”, block) }
  10. Hsiu Says:
    2011-10-04 10:04:28 +0000

    This seems to cause all sorts of problems when running inside threads, which is generally where I’d want to have potentially long-running things. Any advice on getting it to work?

  11. sfg4rfg Says:
    2012-02-05 20:42:55 +0000

    Terminator isn’t working for me, either. I’m simply trying to connect to a server with WMI, which takes about 15-20 seconds to come back saying that it cannot connect.dprzhlqab hj yblvaghub df tabegam sdf qise

  12. Ahsan Says:
    2008-09-16 23:43:24 +0000

    What if the call finishes within n seconds ? SystemTimer doesn’t seem to work in that case, and Terminator seems to terminate the process completely even if the call (system “ls”) was successfull.

    Any ideas ?

  13. Chuck Bergeron Says:
    2008-09-14 22:00:17 +0000

    This is super helpful, I have noticed some errors when using Ruby’s timeout on a typical XML call.

    Thanks Mikel!

  14. Phil Says:
    2008-10-01 04:41:46 +0000

    The simple example of:

    begin
    Terminator.terminate(1) do
    sleep 3
    end

    rescue Terminator::Error puts “Timed out”

    end

    does not work for me. I’m using rails 1.2.2, ruby 1.8.5, fattr 1.0.3, and terminator 0.4.4.

    Any help is appreciated.

  15. Mike H Says:
    2008-09-10 21:19:23 +0000

    I just gave it a quick test on windows and it doesn’t seem to work. Do you happen to know if it’s supposed to work on windows?

    Thanks!

  16. Mikel Lindsaar Says:
    2008-09-11 06:53:22 +0000

    You know, I thought it might, but I just tested it as well and found the same thing. I’ll look into it.

    Mikel

  17. Mike H Says:
    2008-09-11 22:06:56 +0000

    Thanks!

    I’ve got some code that does a similar thing on Windows, but it’s not nearly as complete as terminator, just a quick hack. If I see anything useful, I’ll let you know.

    Are you guys using git? I have to admit that when I found that the project wasn’t on github (and I couldn’t fork it there), my enthusiasm for possibly contributing a patch dropped off significantly.

  18. Mikel Lindsaar Says:
    2008-09-12 08:49:07 +0000

    @pistos: Good idea, fixed it :)

    @Mike: Let me know how it goes on Windows? I am having mixed reactions. On Git, no it’s a tar file right now. Maybe I’ll get off my butt and go gitify it :)

    Mikel

  19. Justin Says:
    2008-09-16 04:09:44 +0000

    When I do this:

    begin Terminator.terminate :seconds => 2 do content = open(fix_http(uri)) end rescue Terminator.error content = nil end

    it throws ‘Terminator::Error’ and terminates my web server process which is not the behavior I want :-).

    Any ideas?

  20. Mikel Lindsaar Says:
    2008-09-16 10:38:20 +0000

    @Justin, you need to:

    rescue Terminator::Error

    Instead of:

    rescue Terminator.error

    Mikel

  21. Doug Says:
    2008-11-20 20:36:46 +0000

    I’m having a similar problem as Phil. I can rescue Terminator::Error in Rails script/console but not when rails is being run with mongrel. Any

  22. Doug Says:
    2008-11-22 14:48:27 +0000

    @Mikel –
    *Ubuntu (tried on both 7.10 and 8.04)
    *ruby 1.8.6
    *Mongrel 1.1.5
    *Rails 2.1.1

    I tried using a different signal in Terminator and commenting out the TERM trap in Mongrel both with no luck.
    Thanks for the gem, I really like it in theory ;).
    Doug

  23. Adrian Says:
    2009-01-28 12:02:19 +0000

    I’m not sure if this will be helpful to others encountering the rescue problem, but I found that the best solution for me was to simply stop the errors from being thrown altogether:

    custom_trap = lambda { eval(’’) } Terminator.terminate(:seconds => 1, :trap => custom_trap) do … end

    This seems counter-intuitive. However, the inability to rescue these errors was more of a nuisance than simply anticipating the very real possibility that what comes out of that terminate block may be the result of some timed-out requests.

    In any case, this is a wonderful gem. Thank you so much. :)

  24. Sherrod Says:
    2009-05-14 16:51:19 +0000

    Is Terminator still under development? I am deciding between this and SystemTimer, and I’m seeing a lot of problems getting Terminator to work.

    Also, where is it located? I saw the link above, but it points to 0.4.2. However, one of the comments mentions 0.4.4

  25. Pat Says:
    2009-07-15 19:34:18 +0000

    Adrian (comment #18) – thanks so much for that comment. I have a daemon that was hanging with the standard Ruby timeout and Terminator fixed the problem, but introduced a new one – the daemon crashed sometimes after recovering.

    I’m guessing it’s related to this http://snippets.dzone.com/posts/show/6072.

    I’d also like to know if Terminator is still under development.

    Thanks Mikel and Ada!

  26. Confused Says:
    2009-08-23 17:56:41 +0000

    Mikel,

    I recently tried implementing Terminator to help resolve issues we were having with long running processes using system C API’s. Essentially, it was taking (at times) more than 2 minutes for a system call to complete and we wanted to kill it well before that. Standard timeout wasn’t doing the trick.

    After installing terminator, I implemented the following:

    require ‘terminator’

    def invoke(rfc)
    Terminator.terminate 30 do
    rfc.invoke
    end
    end

    When it gets to the point that this call to the rfc occurs, I see the new thread get created (under the process list on linux) with the 30 second timeout. After 30 seconds, the thread goes and nothing more happens. Once the process finishes (about 120 seconds later) the mongrel then restarts.

    I tried the same thing with SystemTimer and get the same results… the process still runs until completion and THEN the timeout action occurs.

    Any ideas as to what I’m doing wrong?

  27. Julio Lima Says:
    2009-10-12 13:40:29 +0000

    I’ve the same issue Pat is reporting.
    Testing under MacOS, with ruby 1.8.7, the following code :

    require ‘rubygems’
    require ‘terminator’

    puts “initing test”
    begin

    Terminator.terminate(5) do
    puts “going to take a nap, just 5 seconds …”
    sleep(10)
    puts “THIS SHOULD NEVER APPEAR!!!!!”
    end

    catch
    puts “wake up! terminator knocking the door”
    end
    puts “ending my test”

    I’m never able to see the “wake up message”, neither the “ending my test”

    Any tip what is wrong?

    Regards

  28. locate a bucket truck Says:
    2013-02-05 10:37:32 +0000

    This is really interesting, You’re a very skilled blogger. I’ve joined your
    rss feed and look forward to seeking more of your excellent post.
    Also, I’ve shared your website in my social networks!

  29. FurbtehelePer Says:
    2013-04-03 19:33:30 +0000

    глащдиаторы римаполныи фильм
    онлайн фильмы в хорошем качестве кз трафик

    КиноГриль представляет собой большой свод кинофильмов, которые можно посмотреть онлайн. Портал сосредоточил значительное количество кинокартин самых разных жанров, производства государств всего мира. Каждый сможет найти и посмотреть кино на любой вкус – от классики мирового кино середины прошлого века и до последних шедевров, которые недавно вышли в кинопрокат.
    Нет нужды ждать полной загрузки кинокартины, тратить время на загрузку – фильм онлайн не забирает пространство в памяти компьютера пользователя. Он может начать просмотр кинофильма сразу при включении плеера.
    При просмотре кинофильмов на КиноГриль не надо регистрироваться, платить, а также заполнять разнообразные анкеты. Также не нужно устанавливать какие-то плагины или приложения. Все кинокартины, размещенные на этом сайте, проигрываются на любом компьютере. Все что нужно для начала просмотра – это наличие современного Интернет-браузера, имеющего поддержку флеш-плеера. Скорость при просмотре кинолент зависит только от качества Интернет-соединения пользователя. Все ссылки на видеофильмы являются рабочими, за этим тщательно следит администрация портала, который делает все для того, чтобы доставить удовольствие от удобного просмотра кинофильмов на этом ресурсе.

    фильм любовь да голуби
    луди икс кино бесплатна

  30. seo company in dallas tx Says:
    2013-04-03 05:29:49 +0000

    Right away I am going away to do my breakfast, after having
    my breakfast coming again to read additional news.

  31. DypepluttyNem Says:
    2013-04-07 01:12:16 +0000

    This free search mechanism searches the free full-text of over 400 online law reviews and law journals, as well as particularize repositories hosting speculative papers and coordinated publications such as Congressional Scrutinization Professional care reports. Particular of the law reviews and lawful journals (such as the Stanford Technology Law Rehash), working papers, and reports are present online only.

    http://biznes-sfera.pl/index?sobi2Task=sobi2Details&catid=45&sobi2Id=296
    http://www.4find.eu/index.php?&rozwin=tak&kategoria=biznes&page=1&articles=100
    http://www.turcja.ambertrail.pl/tag/cel
    http://vesser.edu.pl/interesy/podatki/
    http://burbel.edu.pl/?p=7689

  32. postêpowania s¹dowe Says:
    2013-04-08 06:36:38 +0000

    Fantastic web site. Plenty of helpful information here. I’m sending it to several friends ans additionally sharing in delicious. And of course, thank you on your sweat!

  33. lemiPieftof Says:
    2013-04-15 23:40:57 +0000

    Children’s tooth development begins while the baby is in the womb. Teething usually occurs between the ages of six and nine months. Children usually have their full set of 20 primary teeth (milk teeth, baby teeth or deciduous teeth) by the age of three years. At about the age of six years, the first permanent teeth erupt (push through the gum).

  34. lemiPieftof Says:
    2013-04-15 23:41:02 +0000

    Children’s tooth development begins while the baby is in the womb. Teething usually occurs between the ages of six and nine months. Children usually have their full set of 20 primary teeth (milk teeth, baby teeth or deciduous teeth) by the age of three years. At about the age of six years, the first permanent teeth erupt (push through the gum).

  35. austin Says:
    2013-04-20 04:56:05 +0000

    Hola! I’ve been following your website for a while now and finally got the bravery to go ahead and give you a shout out from Austin Texas! Just wanted to mention keep up the excellent work!

  36. kliknij tu Says:
    2013-04-29 10:46:10 +0000

    Incredible! This blog looks just like my old one! It’s on a entirely different subject but it has pretty much the same page layout and design. Great choice of colors! kliknij tu

  37. AYylFyVc Says:
    2013-04-29 10:59:21 +0000

    ]
    AYylFyVc
    AYylFyVc http://www.hylGi2r13n6BaQj3557Y2J.com/

  38. austin seo company Says:
    2013-05-01 01:46:21 +0000

    It’s going to be finish of mine day, but before ending I am reading this fantastic post to improve my know-how.

  39. http://www.buyandsellacaronline.com/ Says:
    2013-05-01 12:33:23 +0000

    This piece of writing is genuinely a pleasant one it assists new internet users, who are wishing in favor of blogging.

  40. zespół muzyczny Says:
    2013-05-02 18:32:05 +0000

    Thanks , I’ve recently been looking for info about this subject for a long time and yours is the best I’ve found out till now. However, what in regards to the bottom line? Are you certain in regards to the supply? zespół muzyczny

  41. bizhub c35 Says:
    2013-05-02 19:59:31 +0000

    This website is mostly a stroll-through for all the data you wished about this and didn’t know who to ask. Glimpse right here, and also you’ll definitely discover it. bizhub c35

  42. SZWARC & GRUBA - mechanika pojazdowa Gdynia Says:
    2013-05-10 05:25:33 +0000

    You’re my function models. Thanks for the write-up SZWARC & GRUBA – mechanika pojazdowa Gdynia

  43. szwarcgruba.gda.pl Says:
    2013-05-10 05:25:53 +0000

    I like what I see so i am just following you. Look forward to discovering your web webpage yet again. szwarcgruba.gda.pl

  44. Strona firmowa Says:
    2013-05-10 06:08:27 +0000

    It’s actually a great and useful piece of information. I am satisfied that you simply shared this useful information with us. Please stay us up to date like this. Thanks for sharing. Strona firmowa

  45. ATGSW Says:
    2013-05-10 06:07:41 +0000

    Normally I do not read article on blogs, but I wish to say that this write-up very compelled me to check out and do so! Your writing style has been surprised me. Thanks, very great post. ATGSW

  46. ALDO – pompy ciepła Says:
    2013-05-10 06:08:11 +0000

    I like what you guys are up too. Such clever work and reporting! Carry on the excellent works guys I’ve incorporated you guys to my blogroll. I think it will improve the value of my website :) ALDO – pompy ciepła

  47. ALDO TECHNIKA GRZEWCZA SŁAWOMIR WIŚNIEWSKI Says:
    2013-05-10 06:08:19 +0000

    I loved as much as you’ll receive carried out right here. The sketch is attractive, your authored material stylish. nonetheless, you command get got an shakiness over that you wish be delivering the following. unwell unquestionably come more formerly again as exactly the same nearly very often inside case you shield this hike. ALDO TECHNIKA GRZEWCZA SŁAWOMIR WIŚNIEWSKI

  48. Studnie białystok Says:
    2013-05-10 06:08:35 +0000

    Thank you for what you have got. This is the best article I’ve read Studnie białystok

  49. Strona główna Says:
    2013-05-10 06:08:29 +0000

    Wonderful work! This is the kind of information that should be shared across the net. Shame on the search engines for now not positioning this submit higher! Come on over and talk over with my web site . Thank you =) Strona główna

  50. kliknij tu Says:
    2013-05-11 18:18:52 +0000

    Generally I don’t read article on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been amazed me. Thanks, quite nice post. kliknij tu

  51. kliknij tu Says:
    2013-05-20 12:46:43 +0000

    I needed to send you the little bit of note to help give thanks as before just for the beautiful things you have shared on this website. It has been so remarkably open-handed of you to provide publicly what exactly many people would’ve marketed as an e-book in order to make some profit for their own end, most importantly considering the fact that you could possibly have done it in the event you considered necessary. Those smart ideas also served to become a easy way to be aware that other people have similar interest similar to mine to find out great deal more when it comes to this problem. I’m sure there are thousands of more pleasurable situations up front for many who look into your blog. kliknij tu

  52. WENTYLATORY PRZEMYSŁOWE Says:
    2013-05-20 12:47:06 +0000

    This can be the worst post of all, I’ve study WENTYLATORY PRZEMYSŁOWE

  53. WENTYLACJA KARPOL Says:
    2013-05-20 12:47:14 +0000

    Somebody essentially lend a hand to make severely articles I’d state. That is the first time I frequented your website page and so far? I surprised with the research you made to create this actual submit amazing. Wonderful job! WENTYLACJA KARPOL

  54. przenośniki rolkowe Says:
    2013-05-22 01:52:07 +0000

    Hiya, I’m really glad I’ve found this info. Today bloggers publish only about gossips and web and this is really frustrating. A good site with interesting content, this is what I need. Thanks for keeping this site, I’ll be visiting it. Do you do newsletters? Cant find it. przenośniki rolkowe

  55. Tatteanceliny Says:
    2013-05-24 20:27:11 +0000

    Not quite every legacy and new-look IT vendor has its own plagiarize on making the whole details center more programmable via software and less dependent on specialized, proprietary and over the odds hardware.

  56. Tatteanceliny Says:
    2013-05-24 20:27:11 +0000

    Nearly every legacy and new-look IT vendor has its own plagiarize on making the whole data center more programmable via software and less dependent on specialized, proprietary and over the odds hardware.

Leave a Reply