Terminator - Timeout without Mercy
2008-09-10 17:40:04 +0000
If you have been following my posts on Ruby-Talk and Ruby on Rails and even RSpec mailing list (and who wouldn’t?! I mean, aside from my mother) then you would have noticed I have been banging my head against a brick wall on the subject of System calls not being handled by the Timeout libraries in Ruby…
I’ve done a lot of work with this. I mean a lot.
So I thought I would share.
It all starts with a basic requirement I have, which is to replicate a data set from one side of the planet to the other. This involves logging into a database server remotely using ActiveRecord across some encrypted VPNs, querying some tables, and then inside of a transaction, replicating down certain data on certain rows.
The solution I have works well, I can replicate about 2-5 rows per second across the link which is more than adequate (the dataset changes upto about 100 rows per minute). The problem is that occasionally, ActiveRecord’s link to the remote database will hit a snag, then Ruby will wait, and wait, and wait to Timeout…. This can take hours, and in fact, sometimes, it never does timeout, leaving a zombie process.
To avoid multiple replication problems, I have lock files which prevent further copies of the replicator firing up if it detects that one is already running.
So, in the end, I get stuck with several replicators “running”, all hung, all waiting for a link to come back that never will, and no replication happening. In the mean time, the rows are still changing and I am getting backlogged. Leave it for a day and you are 80,000 rows backlogged and people start asking questions and I start looking for old travel tickets that I had ‘forgotten about’.
But why does ruby do this?
Shouldn’t the Timeout library protect us from such evilness?
Well, yes and no. The problem is because ruby uses Green threads. Green threads mean that there is only one real Ruby process running on the server, and it “internally” makes other “threads” that it schedules within the main ruby process’ kernel time. Green threads are very efficient (they save a lot of context switching) but the problem is that if you get a System call that ‘blocks’, then the kernel will stop servicing your main ruby thread until your system call is complete, and THIS means that the little ‘Timeout.timeout(5) { my code }’ block you put in, will never get called…
And then you get a hang… forever.
I tried a couple of handlings, the first was the SystemTimer library by Philippe Hanrigou. This entry actually has a really good and simple example of what Green threads and blocking system calls mean. If you want to brush up, it is a good read.
But SystemTimer didn’t work. It uses alarm signals which could have some problems. But in my case, it didn’t work and I still got never ending timeouts.
So I asked again on the Ruby-Talk mailing list and Ara T. Howard came back with a quick script that created little homicidal external ruby processes to kill the ruby process I was in if it didn’t “make it” in time.
The approach was actually very clever, and it works!
What he did was go “We can’t be sure that the existing Ruby process will be able to nuke itself, so instead, lets start up another Ruby instance, running on it’s own, that has the PIDs of the ruby instance that spawned it, and if the time runs out, do a system based kill TERM on the pids.”
You could say that our little ruby processes have a license to kill. Literally.
So I implemented his idea, found a couple of problems, gave back some ideas, and he coded, I spec’d, tested and described and we released a brand new gem…
The Terminator.
You can get it with;
gem install terminator |
Or grab the source code.
And how do you use it? Simple:
1 2 3 4 5 |
require 'terminator' Terminator.terminate 1 do sleep 2 puts "I'll never print" end |
This will never print because the terminator times out after 1 second, which is before the sleep of 2 seconds inside the block. This will raise a Terminator::Error, which you could catch and try again if you want.
This will always work. It is because we are starting a separate process of Ruby (which has some minor overhead) that waits the specified number of seconds and then just simple does a system kernel TERM on our misbehaving process.
Why should you use it?
Well, if you are making ANY calls to external web services, external databases, OpenID, Youtube, Google Maps… anything, then you should have a fail fast policy in place and time out rapidly if these fail. As these are external system calls, they will most likely not be caught by Ruby’s timeout.rb library… and that means that your application will just hang, waiting for the call that never comes back.
It is much better to go “Ok, 2 seconds are up, no response, let’s tell the user to try again in a minute” and render an appropriate message, than have the user frustratingly whack the reload button a few hundred more times wondering why your application is not responding.
So yes, I think you should use Terminator.
Besides, how many other gems do you know that have the method ‘plot_to_kill’ ?
blogLater
Mikel




2008-11-21 21:35:47 +0000
@Doug and @Phil
Hmm… are you guys running on Windows?
Mikel
2008-09-11 17:39:37 +0000
You should maybe link to the source code ( http://codeforpeople.com/lib/ruby/terminator ) for those that want to peruse the code online. :)
2008-09-16 22:01:06 +0000
So .. a ruby process will be spawned for every request to a web service .. ?
2008-09-20 08:07:12 +0000
@Ahsan: That was working here. I just did a clean up on the initial code and added documentation for a 0.4.4 release. Maybe update and try again.
2008-09-23 00:43:23 +0000
@Mikel: Ok, I upgraded to 0.4.4, but the problem persists.
2008-09-23 01:39:38 +0000
@Ahsan, could you pastie your code?
2010-01-21 05:26:31 +0000
I don’t believe “Terminator” supports windows at the moment guys… However if your using “Terminator” with Ruby on Rails via a webhost or apphost. I don’t believe there will be any issues, due to the fact that most of them are using Linux or Unix based OS’s.
2010-01-21 18:44:48 +0000
No, Terminator will not run on a windows environment, sorry guys and gals.
2011-10-04 09:33:04 +0000
I get this error when trying to use this:
terminator-0.4.4/lib/terminator.rb:127:in `block in terminate’: wrong number of arguments (1 for 0) (ArgumentError)
Line 127 is of terminator.rb is this:
trap = getopt :trap, options, lambda{ eval(“raise(::Terminator::Error, ‘Timeout out after #{ seconds }s’)”, block) }2011-10-04 10:04:28 +0000
This seems to cause all sorts of problems when running inside threads, which is generally where I’d want to have potentially long-running things. Any advice on getting it to work?
2012-02-05 20:42:55 +0000
Terminator isn’t working for me, either. I’m simply trying to connect to a server with WMI, which takes about 15-20 seconds to come back saying that it cannot connect.dprzhlqab hj yblvaghub df tabegam sdf qise
2008-09-16 23:43:24 +0000
What if the call finishes within n seconds ? SystemTimer doesn’t seem to work in that case, and Terminator seems to terminate the process completely even if the call (system “ls”) was successfull.
Any ideas ?
2008-09-14 22:00:17 +0000
This is super helpful, I have noticed some errors when using Ruby’s timeout on a typical XML call.
Thanks Mikel!
2008-10-01 04:41:46 +0000
The simple example of:
begin
rescue Terminator::Error puts “Timed out”Terminator.terminate(1) do
sleep 3
end
end
does not work for me. I’m using rails 1.2.2, ruby 1.8.5, fattr 1.0.3, and terminator 0.4.4.
Any help is appreciated.
2008-09-10 21:19:23 +0000
I just gave it a quick test on windows and it doesn’t seem to work. Do you happen to know if it’s supposed to work on windows?
Thanks!
2008-09-11 06:53:22 +0000
You know, I thought it might, but I just tested it as well and found the same thing. I’ll look into it.
Mikel
2008-09-11 22:06:56 +0000
Thanks!
I’ve got some code that does a similar thing on Windows, but it’s not nearly as complete as terminator, just a quick hack. If I see anything useful, I’ll let you know.
Are you guys using git? I have to admit that when I found that the project wasn’t on github (and I couldn’t fork it there), my enthusiasm for possibly contributing a patch dropped off significantly.
2008-09-12 08:49:07 +0000
@pistos: Good idea, fixed it :)
@Mike: Let me know how it goes on Windows? I am having mixed reactions. On Git, no it’s a tar file right now. Maybe I’ll get off my butt and go gitify it :)
Mikel
2008-09-16 04:09:44 +0000
When I do this:
begin Terminator.terminate :seconds => 2 do content = open(fix_http(uri)) end rescue Terminator.error content = nil endit throws ‘Terminator::Error’ and terminates my web server process which is not the behavior I want :-).
Any ideas?
2008-09-16 10:38:20 +0000
@Justin, you need to:
rescue Terminator::ErrorInstead of:
rescue Terminator.errorMikel
2008-11-20 20:36:46 +0000
I’m having a similar problem as Phil. I can rescue Terminator::Error in Rails script/console but not when rails is being run with mongrel. Any
2008-11-22 14:48:27 +0000
@Mikel –
*Ubuntu (tried on both 7.10 and 8.04)
*ruby 1.8.6
*Mongrel 1.1.5
*Rails 2.1.1
I tried using a different signal in Terminator and commenting out the TERM trap in Mongrel both with no luck.
Thanks for the gem, I really like it in theory ;).
Doug
2009-01-28 12:02:19 +0000
I’m not sure if this will be helpful to others encountering the rescue problem, but I found that the best solution for me was to simply stop the errors from being thrown altogether:
custom_trap = lambda { eval(’’) } Terminator.terminate(:seconds => 1, :trap => custom_trap) do … endThis seems counter-intuitive. However, the inability to rescue these errors was more of a nuisance than simply anticipating the very real possibility that what comes out of that terminate block may be the result of some timed-out requests.
In any case, this is a wonderful gem. Thank you so much. :)
2009-05-14 16:51:19 +0000
Is Terminator still under development? I am deciding between this and SystemTimer, and I’m seeing a lot of problems getting Terminator to work.
Also, where is it located? I saw the link above, but it points to 0.4.2. However, one of the comments mentions 0.4.4
2009-07-15 19:34:18 +0000
Adrian (comment #18) – thanks so much for that comment. I have a daemon that was hanging with the standard Ruby timeout and Terminator fixed the problem, but introduced a new one – the daemon crashed sometimes after recovering.
I’m guessing it’s related to this http://snippets.dzone.com/posts/show/6072.
I’d also like to know if Terminator is still under development.
Thanks Mikel and Ada!
2009-08-23 17:56:41 +0000
Mikel,
I recently tried implementing Terminator to help resolve issues we were having with long running processes using system C API’s. Essentially, it was taking (at times) more than 2 minutes for a system call to complete and we wanted to kill it well before that. Standard timeout wasn’t doing the trick.
After installing terminator, I implemented the following:
require ‘terminator’
def invoke(rfc)
Terminator.terminate 30 do
rfc.invoke
end
end
When it gets to the point that this call to the rfc occurs, I see the new thread get created (under the process list on linux) with the 30 second timeout. After 30 seconds, the thread goes and nothing more happens. Once the process finishes (about 120 seconds later) the mongrel then restarts.
I tried the same thing with SystemTimer and get the same results… the process still runs until completion and THEN the timeout action occurs.
Any ideas as to what I’m doing wrong?
2009-10-12 13:40:29 +0000
I’ve the same issue Pat is reporting.
Testing under MacOS, with ruby 1.8.7, the following code :
require ‘rubygems’
require ‘terminator’
puts “initing test”
begin
Terminator.terminate(5) do
puts “going to take a nap, just 5 seconds …”
sleep(10)
puts “THIS SHOULD NEVER APPEAR!!!!!”
end
catch
puts “wake up! terminator knocking the door”
end
puts “ending my test”
I’m never able to see the “wake up message”, neither the “ending my test”
Any tip what is wrong?
Regards
2013-02-05 10:37:32 +0000
This is really interesting, You’re a very skilled blogger. I’ve joined your
rss feed and look forward to seeking more of your excellent post.
Also, I’ve shared your website in my social networks!
2013-04-03 19:33:30 +0000
глащдиаторы римаполныи фильм
онлайн фильмы в хорошем качестве кз трафик
КиноГриль представляет собой большой свод кинофильмов, которые можно посмотреть онлайн. Портал сосредоточил значительное количество кинокартин самых разных жанров, производства государств всего мира. Каждый сможет найти и посмотреть кино на любой вкус – от классики мирового кино середины прошлого века и до последних шедевров, которые недавно вышли в кинопрокат.
Нет нужды ждать полной загрузки кинокартины, тратить время на загрузку – фильм онлайн не забирает пространство в памяти компьютера пользователя. Он может начать просмотр кинофильма сразу при включении плеера.
При просмотре кинофильмов на КиноГриль не надо регистрироваться, платить, а также заполнять разнообразные анкеты. Также не нужно устанавливать какие-то плагины или приложения. Все кинокартины, размещенные на этом сайте, проигрываются на любом компьютере. Все что нужно для начала просмотра – это наличие современного Интернет-браузера, имеющего поддержку флеш-плеера. Скорость при просмотре кинолент зависит только от качества Интернет-соединения пользователя. Все ссылки на видеофильмы являются рабочими, за этим тщательно следит администрация портала, который делает все для того, чтобы доставить удовольствие от удобного просмотра кинофильмов на этом ресурсе.
фильм любовь да голуби
луди икс кино бесплатна
2013-04-03 05:29:49 +0000
Right away I am going away to do my breakfast, after having
my breakfast coming again to read additional news.
2013-04-07 01:12:16 +0000
This free search mechanism searches the free full-text of over 400 online law reviews and law journals, as well as particularize repositories hosting speculative papers and coordinated publications such as Congressional Scrutinization Professional care reports. Particular of the law reviews and lawful journals (such as the Stanford Technology Law Rehash), working papers, and reports are present online only.
http://biznes-sfera.pl/index?sobi2Task=sobi2Details&catid=45&sobi2Id=296
http://www.4find.eu/index.php?&rozwin=tak&kategoria=biznes&page=1&articles=100
http://www.turcja.ambertrail.pl/tag/cel
http://vesser.edu.pl/interesy/podatki/
http://burbel.edu.pl/?p=7689
2013-04-08 06:36:38 +0000
Fantastic web site. Plenty of helpful information here. I’m sending it to several friends ans additionally sharing in delicious. And of course, thank you on your sweat!
2013-04-15 23:40:57 +0000
Children’s tooth development begins while the baby is in the womb. Teething usually occurs between the ages of six and nine months. Children usually have their full set of 20 primary teeth (milk teeth, baby teeth or deciduous teeth) by the age of three years. At about the age of six years, the first permanent teeth erupt (push through the gum).
2013-04-15 23:41:02 +0000
Children’s tooth development begins while the baby is in the womb. Teething usually occurs between the ages of six and nine months. Children usually have their full set of 20 primary teeth (milk teeth, baby teeth or deciduous teeth) by the age of three years. At about the age of six years, the first permanent teeth erupt (push through the gum).
2013-04-20 04:56:05 +0000
Hola! I’ve been following your website for a while now and finally got the bravery to go ahead and give you a shout out from Austin Texas! Just wanted to mention keep up the excellent work!
2013-04-29 10:46:10 +0000
Incredible! This blog looks just like my old one! It’s on a entirely different subject but it has pretty much the same page layout and design. Great choice of colors! kliknij tu
2013-04-29 10:59:21 +0000
]
AYylFyVc
AYylFyVc http://www.hylGi2r13n6BaQj3557Y2J.com/
2013-05-01 01:46:21 +0000
It’s going to be finish of mine day, but before ending I am reading this fantastic post to improve my know-how.
2013-05-01 12:33:23 +0000
This piece of writing is genuinely a pleasant one it assists new internet users, who are wishing in favor of blogging.
2013-05-02 18:32:05 +0000
Thanks , I’ve recently been looking for info about this subject for a long time and yours is the best I’ve found out till now. However, what in regards to the bottom line? Are you certain in regards to the supply? zespół muzyczny
2013-05-02 19:59:31 +0000
This website is mostly a stroll-through for all the data you wished about this and didn’t know who to ask. Glimpse right here, and also you’ll definitely discover it. bizhub c35
2013-05-10 05:25:33 +0000
You’re my function models. Thanks for the write-up SZWARC & GRUBA – mechanika pojazdowa Gdynia
2013-05-10 05:25:53 +0000
I like what I see so i am just following you. Look forward to discovering your web webpage yet again. szwarcgruba.gda.pl
2013-05-10 06:08:27 +0000
It’s actually a great and useful piece of information. I am satisfied that you simply shared this useful information with us. Please stay us up to date like this. Thanks for sharing. Strona firmowa
2013-05-10 06:07:41 +0000
Normally I do not read article on blogs, but I wish to say that this write-up very compelled me to check out and do so! Your writing style has been surprised me. Thanks, very great post. ATGSW
2013-05-10 06:08:11 +0000
I like what you guys are up too. Such clever work and reporting! Carry on the excellent works guys I’ve incorporated you guys to my blogroll. I think it will improve the value of my website :) ALDO – pompy ciepła
2013-05-10 06:08:19 +0000
I loved as much as you’ll receive carried out right here. The sketch is attractive, your authored material stylish. nonetheless, you command get got an shakiness over that you wish be delivering the following. unwell unquestionably come more formerly again as exactly the same nearly very often inside case you shield this hike. ALDO TECHNIKA GRZEWCZA SŁAWOMIR WIŚNIEWSKI
2013-05-10 06:08:35 +0000
Thank you for what you have got. This is the best article I’ve read Studnie białystok
2013-05-10 06:08:29 +0000
Wonderful work! This is the kind of information that should be shared across the net. Shame on the search engines for now not positioning this submit higher! Come on over and talk over with my web site . Thank you =) Strona główna
2013-05-11 18:18:52 +0000
Generally I don’t read article on blogs, but I would like to say that this write-up very forced me to try and do it! Your writing style has been amazed me. Thanks, quite nice post. kliknij tu
2013-05-20 12:46:43 +0000
I needed to send you the little bit of note to help give thanks as before just for the beautiful things you have shared on this website. It has been so remarkably open-handed of you to provide publicly what exactly many people would’ve marketed as an e-book in order to make some profit for their own end, most importantly considering the fact that you could possibly have done it in the event you considered necessary. Those smart ideas also served to become a easy way to be aware that other people have similar interest similar to mine to find out great deal more when it comes to this problem. I’m sure there are thousands of more pleasurable situations up front for many who look into your blog. kliknij tu
2013-05-20 12:47:06 +0000
This can be the worst post of all, I’ve study WENTYLATORY PRZEMYSŁOWE
2013-05-20 12:47:14 +0000
Somebody essentially lend a hand to make severely articles I’d state. That is the first time I frequented your website page and so far? I surprised with the research you made to create this actual submit amazing. Wonderful job! WENTYLACJA KARPOL
2013-05-22 01:52:07 +0000
Hiya, I’m really glad I’ve found this info. Today bloggers publish only about gossips and web and this is really frustrating. A good site with interesting content, this is what I need. Thanks for keeping this site, I’ll be visiting it. Do you do newsletters? Cant find it. przenośniki rolkowe
2013-05-24 20:27:11 +0000
Not quite every legacy and new-look IT vendor has its own plagiarize on making the whole details center more programmable via software and less dependent on specialized, proprietary and over the odds hardware.
2013-05-24 20:27:11 +0000
Nearly every legacy and new-look IT vendor has its own plagiarize on making the whole data center more programmable via software and less dependent on specialized, proprietary and over the odds hardware.