Page 3 of 4

Re: Frankie 2012

Posted: Mon Oct 22, 2012 10:51 pm
by just brew it!
Just found an excuse to turn up another node for Frankie for the final week. Took on a little water in the basement after some thunderstorms today; there's a patch of carpet that's damp. I've got a fan blowing on it, turned the AC on to dry the air (outside humidity is close to 100%), and figured it couldn't hurt to warm things up a bit more down here to speed the drying. :wink:

According to my monitoring page, that's another ~16K PPD. (It's the "lambic" node, second one down.)

Edit: The glamour shot -- the "lambic" node, in all its glory: :lol:
Image

Re: Frankie 2012

Posted: Mon Oct 22, 2012 11:12 pm
by just brew it!
BIF wrote:
It is heated by the sun, and that would be the problem. 95-100+ F during summertime months.

As long as you've got decent airflow (cheap room fan from the local big box store, eh?) I don't think cooking the components is an issue. As I mentioned, my "folding closet" got up to ~100F back in the day, since it was a rather confined space.

The downside is all that energy isn't serving another purpose (e.g. heating your house in winter). Which is why I shut down in summer now.

Re: Frankie 2012

Posted: Wed Oct 24, 2012 5:28 am
by BIF
just brew it! wrote:
Just found an excuse to turn up another node for Frankie for the final week. Took on a little water in the basement after some thunderstorms today; there's a patch of carpet that's damp...


That's awesome. It's very Frankenbottish!

Re: Frankie 2012

Posted: Wed Oct 24, 2012 6:37 am
by farmpuma
@arpstorm

Your Q9300 system freezing is a puzzling failure. I wonder if it might be caused by a system component going into a power saving sleep mode? Or perhaps a tiny glitch in a memory module or hard drive sector? Good luck with the virtual Ubuntu effort.

My limited experience with the 6.23 SMP client and the A3 work units with bonus points was rock solid stable. I gave up when the majority of work units resulted in 20MB upload files even though the client was configured for nothing over 5MBs. The 20MB files took about four hours to upload and the server often timed out before the upload was complete. I still have my passkey and SMP folding will resume when real internet makes it to my house.

The Stanford stats page for SereneScreen shows they last uploaded a work unit in May of 2009. The last folding posts in their forum were in December of 2010 when johnblommers reported the arpstorm 7,000,000 point milestone. Is john still active in the arpstorm effort?

At his folding peak drfish had upgraded nearly all of his employer's computers to dual or quad core systems and had official permission to fold on them. At home he had the folding basement of doom with systems mounted on plywood sheets and even some hanging from the ceiling beams. After his epic duel with leor the bugs in the 6.22 client drove him to abandon SMP folding completely. To my knowledge he never even tried the 6.23 client.

Re: Frankie 2012

Posted: Wed Oct 24, 2012 8:16 am
by just brew it!
farmpuma wrote:
My limited experience with the 6.23 SMP client and the A3 work units with bonus points was rock solid stable. I gave up when the majority of work units resulted in 20MB upload files even though the client was configured for nothing over 5MBs. The 20MB files took about four hours to upload and the server often timed out before the upload was complete. I still have my passkey and SMP folding will resume when real internet makes it to my house.

Even with "real" Internet (3.0/768 DSL) the uploads were causing horrendous lag for everyone in the house (most likely due to a bad case of bufferbloat). Back when the issue originally came up I couldn't find anything that was both free and simple to set up that looked like it would help, so my solution was to write my own TCP throttling tool (in Python). The throttling script runs as a service on my file server, and connects to the DSL line through a copy of Apache that is configured as an HTTP proxy. All of the folding clients are configured to use the TCP throttling script as their HTTP proxy. It's a pretty horrendous hack, but it actually works quite well.

Edit: For the curious, the TCP throttler:
#!/usr/bin/python

import socket
import select
import threading
import SocketServer
import time

global upCount
global downCount

upCount = 0
downCount = 0

class ThreadedTCPRequestHandler(SocketServer.BaseRequestHandler):

    def handle(self):
        global upCount
        global downCount
        cur_thread = threading.currentThread()
        name = cur_thread.getName()
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.connect(('127.0.0.1', 80))
        clientDone = False
        serverDone = False
        while not (clientDone and serverDone):
            idle = True
            readers, writers, err = select.select([self.request, sock], [], [], 0)
            if self.request in readers:
                idle = False
                data = self.request.recv(4096)
                if len(data) == 0:
                    if clientDone:
                        break
                    clientDone = True
                else:
                    upCount += len(data)
                    while upCount > 0:
                        time.sleep(0.05)
                    sock.send(data)
            if sock in readers:   
                idle = False
                response = sock.recv(4096)
                downCount += len(response)
                if len(response) == 0:
                    if serverDone:
                        break
                    serverDone = True
                    clientDone = True
                else:
                    while downCount > 0:
                        time.sleep(0.05)
                    self.request.send(response)
            if idle:
                time.sleep(0.05)
        sock.close()

class ThreadedTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    pass

def client(ip, port, message):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((ip, port))
    sock.send(message)   
    response = sock.recv(32768)
    sock.close()

if __name__ == "__main__":
    # Port 0 means to select an arbitrary unused port
    HOST, PORT = "0.0.0.0", 12080

    server = ThreadedTCPServer((HOST, PORT), ThreadedTCPRequestHandler)
    ip, port = server.server_address
   
    # Start a thread with the server -- that thread will then start one
    # more thread for each request
    server_thread = threading.Thread(target=server.serve_forever)
    # Exit the server thread when the main thread terminates
    server_thread.setDaemon(True)
    server_thread.start()

    while 1:
        time.sleep(0.05)
        if upCount > 0:
            upCount -= 2000
        if downCount > 0:
            downCount -= 5000

It accepts connections on port 12080, and assumes there's a copy of Apache with proxying enabled running on localhost. The throttling rates (both up and down) are controlled by tuning the sleep duration and/or decrement values for the byte counts at the very bottom of the script.

Re: Frankie 2012

Posted: Wed Oct 24, 2012 9:14 pm
by arpstorm
@farmpuma

Yes indeed, thank you for asking, John is an active member of the arpstorm group of Folders. Indeed he/I is posting this. Speaking of myself in the third person is passing strange :-?

The matter of the SMP Folding client freezing my overclocked Q9300 does deserve closer attention because it suggests a weakness somewhere in the system. IMHO there is no finer stress test than SMP Folding. It hammers the OS (Ubuntu 12.04 64-bit) as well as the hardware. I'm inclined to hink it's heat-related although I don't believe that current temperatures are problematic. The max temp http://www.tomshardware.com/news/intel-dts-specs,6517.html is 100 degrees C.

Image

Those of us Folding on the Playstation 3 are in for a rude surprise. Sony and Stanford are discontinuing the program in November/2012. I'm not going to upgrade to Sony's latest firmware (which in the past removed the guest OS support feature). What kind of company issues updates to remove functionality? Let's see how long I can keep on Folding with it :)

It's good to know you are still an active Folder, farmpuma 8)

Re: Frankie 2012

Posted: Wed Oct 24, 2012 11:18 pm
by Flying Fox
I have a lot of respect for farmpuma since he needs to sneaker-net his WUs, across (almost all?) his computing nodes. That has got to be a major PITA and yet he is still doing it after all these years. If I were in his shoes I would have stopped Folding a long time ago. :o Hats off to him.

Re: Frankie 2012

Posted: Thu Oct 25, 2012 8:43 am
by just brew it!
@arpstorm - Yeah, the temps look OK (assuming they are accurate). Might be worth backing off the OC a notch (especially the RAM) to see what happens.

Re: Frankie 2012

Posted: Thu Oct 25, 2012 2:58 pm
by Flying Fox
Well, we may make 4 million, but to match or even top the 5 million we did last year, we need some serious Folding power increase. :oops:

Too bad the 660 was not out sooner and I was too busy to put it in asap. I am folding it now on a test v7 client, doing somewhere between 12K-17K ppd. The v7 client also seems to dish out bigger SMP WUs, but those things take almost 2 days to complete on my 875K for only 8K ppd (points value with bonus was 14K but it took too long). So there is not much I can do on the SMP front but the GPU client should give some steady points until the end. Damnit they are still biasing towards GPU clients.

Re: Frankie 2012

Posted: Fri Oct 26, 2012 6:58 pm
by arpstorm
TRFrankenbot just climbed to second place in the top 20 list of the Tech Report. W00t!

Re: Frankie 2012

Posted: Fri Oct 26, 2012 8:28 pm
by BIF
Whoohoo!

It's cooler here and I'll be in and out of the house this weekend, so I'm letting my machines run through the weekend.

I'm still putting up between 19,000 and 22,000 ppd depending on the type of WU. At the conclusion of October, I'll total up my points from all my logs.

Re: Frankie 2012

Posted: Sat Oct 27, 2012 5:29 pm
by just brew it!
Keeping the X6 up for Frankie through the end of the month. Due to a home brewing (well actually a home hard cider making) mishap, I've got another patch of wet carpet that needs drying... so keeping it warm down here in the basement is a good thing.

Looks like 4 million is in reach? Yeah, down a little from last year. Oh well.

Re: Frankie 2012

Posted: Sat Oct 27, 2012 7:59 pm
by BIF
How do you have a hard cider mishap? In the basement?

:lol:

Re: Frankie 2012

Posted: Sat Oct 27, 2012 9:47 pm
by just brew it!
BIF wrote:
How do you have a hard cider mishap? In the basement?

:lol:

6 easy steps to a basement cider mishap:

1. Get 5 gallons of unfiltered/unpasteurized Michigan apple cider with the intent of making hard cider.

2. Pile all 5 jugs into the kegerator "just for a couple of days until I can get to this".

3. Get really busy at work (cider sits for 3 weeks and starts to ferment on its own).

4. Pressure buildup causes 2 of the jugs to start leaking.

5. Approximately 3 quarts of cider leak out around the door seal of the kegerator and soak into the carpet.

6. Gee... why does it smell like a bad mix of hard cider and cider vinegar down here? :o

Spent a good chunk of this afternoon cleaning partially fermented apple cider out of the carpet under and around the kegerator. :-?

Re: Frankie 2012

Posted: Sun Oct 28, 2012 5:56 am
by Flying Fox
just brew it! wrote:
Looks like 4 million is in reach? Yeah, down a little from last year. Oh well.
That will still need some increase in production. I am pretty much maxed out now. So nothing more to give. We will need may be ~50K ppd more in order to 4 million. I say we will be lucky if we achieve that.

20% drop in production, oh well... :oops:

Re: Frankie 2012

Posted: Sun Oct 28, 2012 10:05 am
by just brew it!
Just tossed my old Phenom 9550 into the mix for the home stretch... not very good PPD (or production/watt) compared to the newer quad/hex core CPUs, but it'll help some.

Re: Frankie 2012

Posted: Sun Oct 28, 2012 10:41 am
by DancinJack
Alllllllllright, just threw an extra i7 860 on. That's the last CPU I have in the house.

Re: Frankie 2012

Posted: Sun Oct 28, 2012 9:40 pm
by PhilipMcc
I am willing to wager that we make 4 million by a whisker. I put the beta 7.2.9 client on a core 2 duo. I should have done this last weekend.

Re: Frankie 2012

Posted: Mon Oct 29, 2012 12:25 am
by BIF
I got a 3.5-day 6800 point work unit this afternoon, and I saw that the Q6600 wasn't going to finish it up by Halloween night, so I reset the client config to use all 4 cores at 100%. Restarted the client, and then hit "finish" so that the running GPU unit would finish to completion and no more GPU WU's would soak up SMP resources.

At first it didn't look like this helped, but I had stuff to do so I left the house and came back about 4 hours later. By then, the GPU slot was done, and now the SMP slot was hauling ass on that last SMP WU, with 1.5 days estimated to completion. The combination of the ending GPU unit and me removing the processing cap lopped off 2 whole days from the SMP estimate!

It's been running that way for almost 12 hours now and it's only down to 1.41 days ETA. I don't get that, because I would think it would be closer to 1.0 by now. At least the estimated points for this last SMP unit went up from 6,800 to over 9,000. Weird, but I'll take it.

The laptop is cranking away too, but still set to 75% and running both GPU and SMP cores. I'll watch it tomorrow and will time my exit as well as possible to stop accepting units that might finish after Wednesday night.

So, sometime Tuesday morning, I'll log another 9K on the slow machine and by then maybe another 40k on the laptop. Then I have stuff to do with both machines; been delaying other activities so that I can help out with Frankie.

I'll be out of folding until I build my new system. Come on Intel, wake up and do SOMETHING. I want a price cut on Sandy or a 6-core Socket 2011 Ivy for an early Christmas present to myself!

Re: Frankie 2012

Posted: Mon Oct 29, 2012 8:07 am
by BIF
This looks like something from Willy Wonka's chocolate factory. It's the "Incredible Flexible Work Unit!" Forever alive, forever hungry! RAAWWRRR!!

It started off as a 3.5 day, 6800 Point unit.
I removed my CPU caps, it went down to 1.5 days, 9000 points.
Over night, it has magically gone to 1.8 days, 7800 points.

It is project 7034 which is a folding coding research project of some sort.
Timeout is November 30th, a whole month from now.
Expiration is Jan 8th 2013, over 2 months from now!

Long timeline, yes. But even within that, like Frankenstein, it seems to live and it even grows! At this rate, my Q6600 system will still be processing it in time for Frankie 2013. :evil:

I can see it now, 360 days, zero points. Maybe by then it will be negative points to make up for the extra work, hah!

It's 37.89% complete. At least the % complete seems to rise and not fall. Yet. :lol:

Re: Frankie 2012

Posted: Mon Oct 29, 2012 8:18 am
by just brew it!
Didn't realize the script I used to start the client on the 9550 had the -oneunit option set. So of course it was idling when I got up this morning. D'oh! :oops: (Fixed now...)

Re: Frankie 2012

Posted: Mon Oct 29, 2012 5:31 pm
by Flying Fox
We need to keep at near 200K for the next 3 days in order to make 4meeeeiiiilllllliiioooooonnn points. We can make it!

@BIF: that's why PPD estimations are just that, guesses. Until the WU is done and submitted, you don't really get the credit. And when you turn off machines or some background processes start to steal cycles, those numbers are going to change. Remember projected points credits now include the early return bonus, that is why if your projected finish time changes, the points credit will change as well.

Re: Frankie 2012

Posted: Mon Oct 29, 2012 9:06 pm
by BIF
I see what you're saying. It just wasn't obvious until this particular work unit.

After running for 12 hours or so (since my post this morning), that WU says it's ETA is 14 hours. At that point the Q6600 will be offline from folding, only because I have things I need to do on this machine which I have delayed for over a week already. The laptop will continue to run until sometime Wednesday. Giving what I can to Frankie!

Re: Frankie 2012

Posted: Mon Oct 29, 2012 9:08 pm
by BIF
just brew it! wrote:
Didn't realize the script I used to start the client on the 9550 had the -oneunit option set. So of course it was idling when I got up this morning. D'oh! :oops: (Fixed now...)


Whoops! I'm curious, how many PPD can the 9550 produce?

Re: Frankie 2012

Posted: Mon Oct 29, 2012 10:39 pm
by just brew it!
BIF wrote:
just brew it! wrote:
Didn't realize the script I used to start the client on the 9550 had the -oneunit option set. So of course it was idling when I got up this morning. D'oh! :oops: (Fixed now...)

Whoops! I'm curious, how many PPD can the 9550 produce?

Well, as we've been discussing, it fluctuates. But as of right now, my monitoring page says it's producing 4515 PPD (including the estimated SMP bonus given its current rate of progress).

(If you try to hit my monitoring page, don't be surprised if it times out; I've been having some intermittent issues with my Internet connection today.)

Re: Frankie 2012

Posted: Mon Oct 29, 2012 11:54 pm
by BIF
Fascinating! I like the idea of having the current stats all rolled up into one page.

Re: Frankie 2012

Posted: Tue Oct 30, 2012 12:55 am
by just brew it!
BIF wrote:
Fascinating! I like the idea of having the current stats all rolled up into one page.

As do I... that's why I set it up. :wink:

Re: Frankie 2012

Posted: Tue Oct 30, 2012 2:14 am
by arpstorm
So I've now got my MacPro with the SMP client Folding for TRFrankenbot. That will throw about 18,369 more PPD at the monster 8)

Re: Frankie 2012

Posted: Tue Oct 30, 2012 6:38 am
by farmpuma
John .. Thanks for the addition points for the TRFrankenbot effort. I may be thinking of another one of your system, but I seem to recall your MacPro had quite a few cores. How has the Q9300 been doing in Virtual Box?

Flying Fox .. I must decline the sneakernet hats off. While I have recently sneakernetted some drivers and other large files, I have not done it with work units for years. Most of the recent folding clients make it nearly impossible without identical hardware. My bottleneck / chore is that I dial-up and dial-up and dial-up some more. And I directly deal with every single work unit at least twice. Some days it feels like unpaid work.

JBI .. Your TCP throttling program looks to be a very handy utility for in house network tranquility. I've made a mental note to include it when I find the time to reorganize our how-to/history/welcome stickies.

Re: Frankie 2012

Posted: Tue Oct 30, 2012 1:02 pm
by arpstorm
@farmpuma

The MacPro (Early 2008 model) has dual quad-core 64-bit CPUs clocking at 2.8 Ghz 8)

The Intel Q9300 + VirtualBox experiment was a fail. I was using Ubuntu 12.10 64-bit as the guest OS and I have never had such a sluggish virtualization experience. When I started up the SMP Folding client it froze up the host OS (Ubuntu 12.04 64-bit). So meanwhile it's back to folding natively on that machine, at least until I find some quality time to probe the matter.

I'm being stubborn, in that I really do want to keep this Q9300 overclocked at 3Ghz (from its native 2.5Ghz) for performance reasons. Lots of good people have noted that this invites instability and that could be the root cause of the lockups during overnight Folding as well as the VirtualBox issue :(