Personal computing discussed

Moderators: renee, farmpuma, just brew it!

 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Fri Feb 01, 2008 10:54 pm

As theMASS spotted, I spent this evening finishing off merging in the benchmark stuff, I've also added the new URL and a few more tweaks. I didn't think I would get it done tonight, so it is actually labelled with tomorrow's date 8)

Download it at the usual place - http://reilly.homeip.net/folding/
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Sat Feb 02, 2008 12:10 am

notfred wrote:
As theMASS spotted, I spent this evening finishing off merging in the benchmark stuff, I've also added the new URL and a few more tweaks. I didn't think I would get it done tonight, so it is actually labelled with tomorrow's date 8)

Download it at the usual place - http://reilly.homeip.net/folding/


You may not be fred... but you da man ;)

I'm gonna grab it now.

You also should have a PM from me.
Image
 
jmfox
Gerbil In Training
Posts: 1
Joined: Sat Feb 02, 2008 5:31 am

Re: New release of diskless folding suite

Sat Feb 02, 2008 5:36 am

Notfred,

Awesome package... I have the diskless folding CD running on a board (MSI K9AGM2-L w/a Athlon 64x2 BE & a few G of RAM)

every couple of WU's it locks up. This last time I copied out the backup tarball and the entire work directory.

How might I go about restoring this after rebootin? or is it better to just dump it and let it run?
 
RAH
Gerbil
Posts: 10
Joined: Tue Jan 15, 2008 10:08 pm

Re: New release of diskless folding suite

Sun Feb 03, 2008 9:27 pm

Does this update fix the hang upon completion on the USB stick too?

I'm having that problem. Needs to be restarted to send.
Other then that, works great.
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Sun Feb 03, 2008 10:47 pm

RAH wrote:
Does this update fix the hang upon completion on the USB stick too?

I'm having that problem. Needs to be restarted to send.
Other then that, works great.


The hang script should work in any configuration. Do you have a log that I can look at? It can take up to 10 mins before the hang is detected. There was an issue when the autosend ran between hang and the script but that has been fixed.

Are you running the version dated Feb 2 2008? Or asking before updating?

It doesn't actually "fix" the hang issue, it's a workaround but should work. Some older releases didn't always work but the last two haven't had any reported issues.
Image
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Mon Feb 04, 2008 9:16 am

jmfox wrote:
every couple of WU's it locks up.

Locks up at the end or locks in the middle? My hang check will only handle locks at the end of WU at the moment, locks in the middle can be due to bad WU, but if you are locking up that frequently then I'm suspicious that you don't have a stable PC.

jmfox wrote:
This last time I copied out the backup tarball and the entire work directory. How might I go about restoring this after rebootin? or is it better to just dump it and let it run?
Easiest way is to make sure you haven't disabled backups (i.e. "Backup Interval" is set to something other than 0) and have a USB stick plugged in. It should back up to the USB stick regularly and if it hangs you can reboot and it should restore from the USB stick.
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Tue Feb 05, 2008 5:58 pm

Notfred, I too am also getting a hang on systems. Using new beta 2 cd / client.
I have 5 systems running this. All very stable. All have have this happen at least twice in the last two months, but not in a row. System will run 5 great .. then this .. Others will do 10 then this ..
[18:52:25] Completed 500000 out of 500000 steps  (100 percent)
[18:52:25] Writing final coordinates.
[18:52:26] Past main M.D. loop
[18:52:26] Will end MPI now
[18:53:26]
[18:53:26] Finished Work Unit:
[18:53:26] - Reading up to 3728544 from "work/wudata_03.arc": Read 3728544
[18:53:26] - Reading up to 1785264 from "work/wudata_03.xtc": Read 1785264
[18:53:26] goefile size: 0
[18:53:26] logfile size: 19063
[18:53:26] Leaving Run
[18:53:27] - Writing 5537271 bytes of core data to disk...
[18:53:27]   ... Done.
[18:53:27] - Shutting down core
[18:53:27]
[18:53:27] Folding@home Core Shutdown: FINISHED_UNIT
[22:03:09] - Autosending finished units...
[22:03:09] Trying to send all finished work units
[22:03:09] + No unsent completed units remaining.
[22:03:09] - Autosend completed



Backup + This Issue = If you catch this early you can manually restart and hope that the last save was 95%-99%. If it reaches 100% on the backup, it will never send or complete. After restart, it will download new WU and overwrite on backup.

I was lucky on this WU, I did the manual restart and here is the result:
[22:56:18] Project: 2605 (Run 10, Clone 155, Gen 25)
[22:56:18]
[22:56:18] Assembly optimizations on if available.
[22:56:18] Entering M.D.
[22:56:24] Calling FAH init
[22:56:24] Read topology
[22:56:24] (Starting from checkpoint)
[22:56:25] 000 out of 500000 steps  (99 percent)
[22:56:25] Extra SSE boost OK.
[22:56:25] es
[22:56:25] Completed 495000 out of 500000 steps  (99 percent)
[22:56:25] Extra SSE boost OK.
[23:13:43] Writing local files
[23:13:43] Completed 500000 out of 500000 steps  (100 percent)
[23:13:43] Writing final coordinates.
[23:13:43] Past main M.D. loop
[23:13:43] Will end MPI now
[23:14:43]
[23:14:43] Finished Work Unit:
[23:14:43] - Reading up to 3728544 from "work/wudata_03.arc": Read 3728544
[23:14:43] - Reading up to 1785296 from "work/wudata_03.xtc": Read 1785296
[23:14:43] goefile size: 0
[23:14:43] logfile size: 27075
[23:14:43] Leaving Run
[23:14:48] - Writing 5545315 bytes of core data to disk...
[23:14:48]   ... Done.
[23:14:48] - Shutting down core
[23:14:48]
[23:14:48] Folding@home Core Shutdown: FINISHED_UNIT
[23:14:52] CoreStatus = 64 (100)
[23:14:52] Unit 3 finished with 65 percent of time to deadline remaining.
[23:14:52] Updated performance fraction: 0.682033
[23:14:52] Sending work to server
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Tue Feb 05, 2008 9:50 pm

Hmm, that should get picked up by the auto hang check in that it says FINISHED_UNIT but then doesn't say CoreStatus.

I've got family staying in my computer room this week, will take a look at it when I get a chance.
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Tue Feb 05, 2008 10:54 pm

notfred wrote:
Hmm, that should get picked up by the auto hang check in that it says FINISHED_UNIT but then doesn't say CoreStatus.


Thanks NotFred for Looking into that.

Also, I know you have a ticked submitted on it for a while, but where do you stand on the 2 x SMP for the quads?

Would this be a major rewrite to run SMP per 2 Cores found .... or just a simple config change .. and are you willing to explore doing this?
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Wed Feb 06, 2008 10:21 am

EvilAlchemist wrote:
Also, I know you have a ticked submitted on it for a while, but where do you stand on the 2 x SMP for the quads?

Would this be a major rewrite to run SMP per 2 Cores found .... or just a simple config change .. and are you willing to explore doing this?

It's a simple config change, I just have "4" in a few places in the init script, just need to replace that with a variable like the others and set the webpage to allow selecting "2" or "4". Probably only an hours work including testing - just need to get time to do that hour!
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Thu Feb 07, 2008 2:55 am

Trying to do a little reading on my own to see what the problem might be because it seems i am one of the few to have this keep happening.

All I can find is http://foldingforum.org/viewtopic.php?f=12&t=777

This might just be a problem with the beta .... but many have said your hang check fixed the problem .... just not for me ..

Was not sure if you had seen this post.

This is a weird glitch.
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Thu Feb 07, 2008 2:15 pm

Thanks for a pointer to that, I am running a relatively old version of glibC, may be worth an upgrade...
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Fri Feb 08, 2008 1:13 am

notfred wrote:
Thanks for a pointer to that, I am running a relatively old version of glibC, may be worth an upgrade...


Since no one has added anything to that thread in over 15 days... that might be part of the problem ....i really hope it is ....... if it solved his, maybe it will work here as well..

but i have no idea , notfred is the expert here ....
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Sat Feb 09, 2008 11:24 pm

Notfred - I got a pm from the user and the updated glibC.

He stated that it was still hanging some times .. just not as much ...


Update #2 :Had one hang ... but it was the first one in five days .....

From what I can tell ... tons of users are having this happen. Linux and Mac both.. Guess we will just have to wait till the final version comes out to fix this completely!
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Tue Feb 19, 2008 9:47 pm

EvilAlchemist wrote:
Notfred - I got a pm from the user and the updated glibC.

He stated that it was still hanging some times .. just not as much ...


Update #2 :Had one hang ... but it was the first one in five days .....

From what I can tell ... tons of users are having this happen. Linux and Mac both.. Guess we will just have to wait till the final version comes out to fix this completely!


The Hang Check seems to have been broken in the latest release. I've been beyond busy the last few weeks so I only updated one box and it hangs regularly. (It was fine before the upgrade) The 6 machines I run the previous version on still work perfectly.
Image
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Wed Feb 20, 2008 5:23 am

theMASS wrote:
The Hang Check seems to have been broken in the latest release. I've been beyond busy the last few weeks so I only updated one box and it hangs regularly. (It was fine before the upgrade) The 6 machines I run the previous version on still work perfectly.


Good information to have. Glad someone else is having the same thing happen.

I hope Notfred has some time soon to look at this. He is probley tired of seeing me on here .. so "theMass" , it is your turn ...


PS ... Notfred ... thanks for all your hard work. I know you have been busy with family at your place latley!
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Wed Feb 20, 2008 5:55 am

EvilAlchemist wrote:
theMASS wrote:
The Hang Check seems to have been broken in the latest release. I've been beyond busy the last few weeks so I only updated one box and it hangs regularly. (It was fine before the upgrade) The 6 machines I run the previous version on still work perfectly.


Good information to have. Glad someone else is having the same thing happen.

I hope Notfred has some time soon to look at this. He is probley tired of seeing me on here .. so "theMass" , it is your turn ...


PS ... Notfred ... thanks for all your hard work. I know you have been busy with family at your place latley!


Just for the "fun of it" ;) I updated another box after my previous post and it "hung" too.

notfred maybe the hangcheck code in the latest release is using an old version of the code?

...and yes I am gonna get some online docs put together hopefully by the end of the week ;)
Image
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Wed Feb 20, 2008 12:32 pm

I've got a concert tonight and a rehearsal tomorrow, will try to look at the hang check detection on Friday night.
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Fri Feb 22, 2008 10:07 pm

Didn't get much time to look at this tonight, but at least the basic hangcheck stuff is working as before (not tried it with extra text after the FINISHED_UNIT line yet, will try that next) in that it kill -9's the cores. There may be something different in the beta2 client that means the old workaround of kill -9 the cores no longer fixes the hang. Are there any reports of that?
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Fri Feb 22, 2008 10:53 pm

notfred wrote:
Didn't get much time to look at this tonight, but at least the basic hangcheck stuff is working as before (not tried it with extra text after the FINISHED_UNIT line yet, will try that next) in that it kill -9's the cores. There may be something different in the beta2 client that means the old workaround of kill -9 the cores no longer fixes the hang. Are there any reports of that?


I've been pretty sick the last 2 days and haven't really been monitoring my boxes. OTOH nothing seems to have "hung" in the last 2 days.

On an unrelated note... A few of my boxes that I monitor with both FahMon and FahSpy have disappeared from both apps, but continue to run?? One of the "missing" boxes is a laptop and I did switch it from wired to wireless mid WU so I'm guessing that would explain that one, but the others are kind of a mystery right now. They don't show up in the diskless workgroup but can be seen via their IP address.
Image
 
EvilAlchemist
Gerbil
Posts: 28
Joined: Tue Jan 29, 2008 1:54 am

Re: New release of diskless folding suite

Sat Feb 23, 2008 3:39 am

I am still having a few hanging issued. I let it sit there for 6 hours and the hang check never kicked in!

One thing i did do today is bind all my Mac Adresses in the router to specific ip's. I know the SMP client dosn't like network changes and it seems like i would get a hang every time to DNS refreshed.

I just changed it today so will see how that works.
 
Ragnar Dan
Gerbil Elder
Posts: 5380
Joined: Sun Jan 20, 2002 7:00 pm

Re: New release of diskless folding suite

Mon Feb 25, 2008 12:18 am

I found it sitting for several hours after reaching FINISHED_UNIT on at least 2 occasions last week on the one machine I was using the newer ISO, so I reverted to the November 28 version. I'm probably going to set up the tftp server or something to protect myself from this thing, but that will take some time to set up since it's in a VM and I'll also have to make it do DHCP instead of the letting the gateway do it.
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Mon Feb 25, 2008 1:34 am

Ragnar Dan wrote:
I found it sitting for several hours after reaching FINISHED_UNIT on at least 2 occasions last week on the one machine I was using the newer ISO, so I reverted to the November 28 version. I'm probably going to set up the tftp server or something to protect myself from this thing, but that will take some time to set up since it's in a VM and I'll also have to make it do DHCP instead of the letting the gateway do it.


Last week saw it several times on the box (and then on the "test" box) with the latest ISO. Since I posted a few days ago no hangs, but also no hang recoveries either. I've been getting different WUs, 305x's instead of the steady stream of 3060's that I've been getting for the last month and they finish much faster than the 3060s so I've completed about 10 over the weekend. My VM'd clients have always been hang free even before the hang check was added. I'm even running wireless on a new laptop and VMWare's networking seems to be a pretty good barrier against DHCP/wireless issues.

Nothing wrong with setting up a tftp server but I don't think it will have any effect on whether or not your machines hang. USB sticks are my preference for backups, and booting for that matter.

What host OS are you running your VM under?
Image
 
Ragnar Dan
Gerbil Elder
Posts: 5380
Joined: Sun Jan 20, 2002 7:00 pm

Re: New release of diskless folding suite

Mon Feb 25, 2008 3:30 pm

The purpose of the tftp server is just to get the backups so when I have to reboot, I have the backups to start over again rather than having to lose all the work. The problem with that is that I'm not sure if the hanged WU will be in good shape or not, but that's a worry for another day.

The VM is running in Windows XP-32, and I've had excellent reliability with it. I think getting it to do DHCP will probably be quite an effort, but maybe I'll be surprised.
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Wed Feb 27, 2008 5:59 am

Well had another hang earlier with the latest software... Seems odd that it would be a beta2 issue as the only machine that the hang check isn't working on is the one with Feb 2 2008 version running. The other 6 have hung but the script kicks in and does its thing.

On another note I've got a very rough Diskless Folding Guide...well more of an outline or alpha version of what I hope will become "THE" Diskless Folding Guide, posted at http://foldingcd.com ...don't worry more sections, content, and a slew of screenshots on the way.
Image
 
notfred
Maximum Gerbil
Topic Author
Posts: 4610
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: New release of diskless folding suite

Wed Feb 27, 2008 12:11 pm

Thanks for the update. I started doing some debugging but I'm just swamped at work and home right now. I'm adding a log file (max length 1000lines so should cover about the last day) for the auto hang check, I'm thinking of adding links to manual per-instance hang checks plus a monster killall link as well. I think that plus the SMP per 2 procs option should be enough for this drop of code.
 
runlinux
Gerbil
Posts: 11
Joined: Wed Feb 27, 2008 1:29 pm

Re: New release of diskless folding suite

Wed Feb 27, 2008 1:34 pm

hey notfred, i love you setup as it is the easiest way for me to get my farm working.

but, like others around here, i have issues with the hang check. i have lost maybe 10+ units the past few weeks as the checks dont seem to be working for me. I have my linux server working as DHCP and TFTP so i get my backups done eveyr 15 minutes. i have three rigs going head and diskless, and im they alone are worth 8.5k ppd.

i know how to work my way around linux, and i took a look at the source, but i dont know how i would go about making changes and then applying them to my systems.

i think it would be nice if you could add a Kill link on the node so if one notices they are acting up, we can just kill the cores and have it restart and hopefully not loose any work.

i have posted about your work over at http://www.forums.extremeoverclocking.com and http://www.xtremesystems.org and people there seem to like it a whole lot, so i stand behind you on this project!
Image
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Wed Feb 27, 2008 3:38 pm

runlinux wrote:
hey notfred, i love you setup as it is the easiest way for me to get my farm working.

but, like others around here, i have issues with the hang check. i have lost maybe 10+ units the past few weeks as the checks dont seem to be working for me. I have my linux server working as DHCP and TFTP so i get my backups done eveyr 15 minutes. i have three rigs going head and diskless, and im they alone are worth 8.5k ppd.

i know how to work my way around linux, and i took a look at the source, but i dont know how i would go about making changes and then applying them to my systems.

i think it would be nice if you could add a Kill link on the node so if one notices they are acting up, we can just kill the cores and have it restart and hopefully not loose any work.

i have posted about your work over at http://www.forums.extremeoverclocking.com and http://www.xtremesystems.org and people there seem to like it a whole lot, so i stand behind you on this project!


Does using the "Remote Reboot" result in lost WUs for you? I run my boxes with USB sticks and a "Remote Reboot" starts me back up at 99%. I don't know what it is but there is something about $10 worth of solid state memory that I just can't resist ;)
Image
 
runlinux
Gerbil
Posts: 11
Joined: Wed Feb 27, 2008 1:29 pm

Re: New release of diskless folding suite

Wed Feb 27, 2008 5:47 pm

digital_exhaust wrote:
[23:41:28] Completed 500000 out of 500000 steps  (100 percent)
[23:41:28] Writing final coordinates.
[23:41:29] Past main M.D. loop
[23:41:29] Will end MPI now
[23:42:29]
[23:42:29] Finished Work Unit:
[23:42:29] - Reading up to 3722208 from "work/wudata_02.arc": Read 3722208
[23:42:29] - Reading up to 1775176 from "work/wudata_02.xtc": Read 1775176
[23:42:29] goefile size: 0
[23:42:29] logfile size: 16912
[23:42:29] Leaving Run
[23:42:32] - Writing 5518696 bytes of core data to disk...
[23:42:32]   ... Done.
[23:42:32] - Shutting down core
[23:42:32]
[23:42:32] Folding@home Core Shutdown: FINISHED_UNIT
[23:57:48] - Autosending finished units...
[23:57:48] Trying to send all finished work units
[23:57:48] + No unsent completed units remaining.
[23:57:48] - Autosend completed



im having this issue when i do a remote reboot. anytime during a unit it is fine as they are backed up on my server, but when they finish, it just hangs and upon reboot they are dead saying that they cant open the local file. i believe i have the latest vesrsion too. i'll try the usb drive as i have 3 im not using right now.

upon a finshed unit and the hang, the reboot spits this out:

[15:51:01] Folding@Home Gromacs SMP Core
[15:51:01] Version 1.74 (November 27, 2006)
[15:51:01]
[15:51:01] Preparing to commence simulation
[15:51:01] - Ensuring status. Please wait.
[15:51:01]
[15:51:01] Error: Could not write local file.  Exiting.
[15:51:06] - Shutting down core
[15:51:18]   Exiting.
[15:51:18] - Shutting down core
[15:53:10] CoreStatus = 12 (18)
[15:53:10] Client-core communications error: ERROR 0x12
[15:53:10] Deleting current work unit & continuing...
[15:57:31] - Warning: Could not delete all work unit files (7): Core returned invalid code
Image
 
theMASS
Gerbil First Class
Posts: 132
Joined: Thu Sep 27, 2007 3:24 am

Re: New release of diskless folding suite

Wed Feb 27, 2008 8:23 pm

runlinux wrote:
digital_exhaust wrote:
[23:41:28] Completed 500000 out of 500000 steps  (100 percent)
[23:41:28] Writing final coordinates.
[23:41:29] Past main M.D. loop
[23:41:29] Will end MPI now
[23:42:29]
[23:42:29] Finished Work Unit:
[23:42:29] - Reading up to 3722208 from "work/wudata_02.arc": Read 3722208
[23:42:29] - Reading up to 1775176 from "work/wudata_02.xtc": Read 1775176
[23:42:29] goefile size: 0
[23:42:29] logfile size: 16912
[23:42:29] Leaving Run
[23:42:32] - Writing 5518696 bytes of core data to disk...
[23:42:32]   ... Done.
[23:42:32] - Shutting down core
[23:42:32]
[23:42:32] Folding@home Core Shutdown: FINISHED_UNIT
[23:57:48] - Autosending finished units...
[23:57:48] Trying to send all finished work units
[23:57:48] + No unsent completed units remaining.
[23:57:48] - Autosend completed



im having this issue when i do a remote reboot. anytime during a unit it is fine as they are backed up on my server, but when they finish, it just hangs and upon reboot they are dead saying that they cant open the local file. i believe i have the latest vesrsion too. i'll try the usb drive as i have 3 im not using right now.

upon a finshed unit and the hang, the reboot spits this out:

[15:51:01] Folding@Home Gromacs SMP Core
[15:51:01] Version 1.74 (November 27, 2006)
[15:51:01]
[15:51:01] Preparing to commence simulation
[15:51:01] - Ensuring status. Please wait.
[15:51:01]
[15:51:01] Error: Could not write local file.  Exiting.
[15:51:06] - Shutting down core
[15:51:18]   Exiting.
[15:51:18] - Shutting down core
[15:53:10] CoreStatus = 12 (18)
[15:53:10] Client-core communications error: ERROR 0x12
[15:53:10] Deleting current work unit & continuing...
[15:57:31] - Warning: Could not delete all work unit files (7): Core returned invalid code


Are you running the latest version of nf's software? Feb 2, 2008? Did you have the issue with the previous version? ...And I'm very curious to know if the USB sticks help.
Image

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On