Robocopy help

From Visual Basic to GNU C, this is the place to talk programming.

Moderators: SecretSquirrel, just brew it!

Robocopy help

Postposted on Fri Jan 11, 2013 4:32 pm

I need help with this script I'm working on!

I've got a script with this line:
Code: Select all
robocopy \\server\share E:\backup /mir /maxage:1 /mt


Here's the rub: because of the combination of /mir and /maxage:1, I end up with a lot of empty directories.

Is there any way to get robocopy not to copy the directory if there are no new files in that directory that need to be copied?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 11, 2013 5:03 pm

Try changing to the following:
Code: Select all
robocopy \\server\share E:\backup /s /purge /maxage:1 /mt


/mir is equivalent to /e /purge so /s /purge should be equivalent minus copying empty dirs. I usually add /z too just in case the the script gets interrupted.
Flatland_Spider
Gerbil Elite
 
Posts: 852
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Robocopy help

Postposted on Fri Jan 11, 2013 5:03 pm

Looking at the TechNet KB article, it looks like /mir is the same as "/e /purge". You can probably use "/s /purge" instead.
2600K @ 4.8GHz; XSPC Rasa/RX240/RX120 Phobya Xtreme 200; Asus P8Z68-V Pro; 16GB Corsair Vengeance 1333 C9; 2x7970 OC w/ Razor 7970; Force GT 120GB; 3x F3 1TB; Corsair HX750; X-Fi Titanium; Corsair Obsidian 650D; Dell 2408WFP Rev. A01; 2x Dell U2412m
mortifiedPenguin
Gerbil Elite
 
Posts: 812
Joined: Mon Oct 08, 2007 7:46 pm

Re: Robocopy help

Postposted on Fri Jan 11, 2013 6:43 pm

Ah, yes. It's quite silly for me not to have noticed that!
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Tue Jan 15, 2013 10:07 am

OK, so here's another, but related question:

If using /maxage=1 and no files in a directory have changed, the directory itself still gets copied. Any way to prevent that?
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 1:31 pm

Can anyone verify:

Robocopy makes copies of empty subdirectories if I am not the directory's owner. So when I robocopy from a NAS, it copies all subdirectories, regardless of whether using /s or not. Conversely, if I create a bunch of empty directories on my local drive, robocopy with the /s switch will not copy empty subdirectories.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 1:44 pm

All of these recursive mirroring tools seem to have their shortcomings. Rsync (the organically complex equivalent of Robocopy on the *NIX side of the house) certainly has its share of "Wait, I didn't mean to do THAT!" moments. (Which is why I'm a big fan of rsync's "--dry-run" option... :lol:)
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37831
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Robocopy help

Postposted on Fri Jan 18, 2013 1:50 pm

just brew it! wrote:All of these recursive mirroring tools seem to have their shortcomings. Rsync (the organically complex equivalent of Robocopy on the *NIX side of the house) certainly has its share of "Wait, I didn't mean to do THAT!" moments. (Which is why I'm a big fan of rsync's "--dry-run" option... :lol:)


Robocopy has what might be a cousin of that: /l

I haven't done any searching yet, but if anyone has a recommendation on how to get rid of empty directories, I'm listening.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 2:00 pm

flip-mode wrote:I haven't done any searching yet, but if anyone has a recommendation on how to get rid of empty directories, I'm listening.

Well, in *NIX, there'd be a shell script for that. Kind of the old school/uber-geek equivalent of "There's an app for that!" :lol:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37831
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:16 pm

Did a search:
http://downloadsquad.switched.com/2008/ ... -batch-fi/

for /f "usebackq" %%d in ("dir /ad/b/s | sort /R") do rd "%%d"

Raymond Chen, the venerable Microsoft coder, explains how this works. Basically, it uses a trick to enumerate the directories in reverse order. Since you can't delete a directory that contains data (including other directories) the only way to automate deleting directories is to start at the very bottom of the tree and work your way up.


Haven't tried it yet.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:38 pm

OK, being the scripting noob that I am, I'm getting errors executing that command. I'm assuming that I have to change my working directory to the root level folder that I want to purge empty subdirs from. Executing the command from there, I get:

%%d was unexpected at this time

I'll google that right quick ... aaaand google doesn't like searching for % characters :(

____________________________________
Edit:

So I was getting that error because %% is for batch scripts, while from the command line only one % is needed, so I tried one % and I get the next error.

Did some more google searching ... found this:
http://simonwai.com/developments/empty_folder_nuker/
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:47 pm

If memory serves, %% is only for inside of .bat files. So if you copy and paste that to the command line, error. Dump it in "test.bat" and then run the .bat and it should work. Or, take out the double %% and replace it with a single % for running directly on the command line.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:50 pm

So the next error was...

The system cannot find the file dir /ad/b/s | sort /R.

Ya?
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:55 pm

Scrotos wrote:So the next error was...

The system cannot find the file dir /ad/b/s | sort /R.

Ya?

Yes, that was the next error.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 3:58 pm

The article you linked did a copy-paste wrong.

http://blogs.msdn.com/b/oldnewthing/arc ... 99914.aspx

Missed single quotesbackquote, that stupid thing on the ~ key:

Code: Select all
for /f "usebackq" %%d in (`"dir /ad/b/s | sort /R"`) do rd "%%d"


I just did a test of this:

Code: Select all
for /f "usebackq" %%d in (`"dir /ad/b/s | sort /R"`) do echo rd "%%d"


Seemed to work fine. And just to clarify, it seems like it lists ALL subdirectories and relies on "rd" being unable to remove a directory if there's a file in it. You can probably "rd /q" to suppress any complaints about directories not being empty but that might only be useful in conjuction with "rd /s" which is basically DELTREE and NOT what you want to use.

Actually, any of THESE will work better as the original one doesn't handle spaces in the names. These three do, tested.

Code: Select all
for /f "usebackq delims=" %%d in (`"dir /ad/b/s | sort /R"`) do rd "%%d"
for /f "tokens=*" %%d in ('dir /ad/b/s ^| sort /R') do rd "%%d"
for /f "delims=" %%i in ('dir /s /b /ad ^| sort /r') do rd "%%i">NUL
Last edited by Scrotos on Thu Jan 24, 2013 10:31 am, edited 2 times in total.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 4:12 pm

Thanks Scrotos. I have some robocopy scripts that I may plug those into once I get a chance. I'm dealing with thousands of empty directories in some cases.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Fri Jan 18, 2013 4:31 pm

Ya I love arcane crazy batch file stuff. I squirrel this junk away from when I need to automate this stuff at work for things like migrating recorded phone calls to a backup server (using robocopy, no less!) a month at a time. I must have a dozen "test.bat" and "test2.bat" and "trythis.bat" scattered around with various one-liners like this. Ugh, I got stuff to check current admin rights and rotate .mdb on a per-day basis depending on if the rotation script had already been run once on the current day or not. Batch files. What an adventure!

[cries]
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 4:39 pm

@just brew it!
Indeed, find piped to xargs executing rmdir would do exactly what he wants.

@flip-mode
You might try the /m option to only copy files with the archive bit set then clear the bit.

Looking into something that supports differential backups might be a good idea. Robocopy is a good bulk copy tool, but I'd look at something else for more fine grained work.
Flatland_Spider
Gerbil Elite
 
Posts: 852
Joined: Mon Sep 13, 2004 8:33 pm
Location: The 918/539

Re: Robocopy help

Postposted on Fri Jan 18, 2013 4:52 pm

Funny you mention xargs, on the MS blog I linked some people started arguing about the right way to do that in Unix, too.

So you see a problem on a Windows machine and your first thought is to install some Unix tools. Now you have two problems...
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 5:15 pm

Scrotos wrote:Funny you mention xargs, on the MS blog I linked some people started arguing about the right way to do that in Unix, too.

So you see a problem on a Windows machine and your first thought is to install some Unix tools. Now you have two problems...

Calm down, nobody seriously suggested that he install *NIX (or even Cygwin).

This whole tangent started with me pointing out that the *NIX tools for doing this are similarly arcane. I didn't expect it to devolve into an OS religious battle. :oops:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37831
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Robocopy help

Postposted on Fri Jan 18, 2013 5:58 pm

You seriously don't know the famous quote about "...now you have two problems?" Been floating around *nix circles for years in regards to sed, awk, reg ex, that kinda stuff. I'm not hatin' or bent outta shape. It was a joke!
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.

Re: Robocopy help

Postposted on Fri Jan 18, 2013 6:04 pm

Scrotos wrote:You seriously don't know the famous quote about "...now you have two problems?" Been floating around *nix circles for years in regards to sed, awk, reg ex, that kinda stuff. I'm not hatin' or bent outta shape. It was a joke!

Heh. Guess not. Color me embarrassed. :oops:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 37831
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Robocopy help

Postposted on Fri Jan 18, 2013 6:25 pm

I've got 99 problems, but a useful Linux command line script ain't one?

p.s. I love CL robocopy, I read this thread just to see if I could help but you guys use it in more advanced ways than I do...I just felt obligated to reply after reading :D
MadManOriginal
Graphmaster Gerbil
 
Posts: 1438
Joined: Wed Jan 30, 2002 7:00 pm
Location: In my head...

Re: Robocopy help

Postposted on Sat Jan 19, 2013 1:24 am

I just use 'WinRoboCopy'. It's just a front end program. Select the options you want and hit run. Shows you the command/s it uses also so you can learn to do it by yourself.
Has scheduling and the ability to save as batch files too.

Sure beats having to remember all the commands.
Slinky
Gerbil
 
Posts: 12
Joined: Thu Feb 24, 2011 2:36 am
Location: Australia

Re: Robocopy help

Postposted on Sat Jan 19, 2013 4:23 am

I just pull up this page for reference whenever I need a robocopy command that I'm not already sure of: http://ss64.com/nt/robocopy.html
MadManOriginal
Graphmaster Gerbil
 
Posts: 1438
Joined: Wed Jan 30, 2002 7:00 pm
Location: In my head...

Re: Robocopy help

Postposted on Wed Jan 23, 2013 9:16 am

Slinky wrote:I just use 'WinRoboCopy'. It's just a front end program. Select the options you want and hit run. Shows you the command/s it uses also so you can learn to do it by yourself.
Has scheduling and the ability to save as batch files too.

Sure beats having to remember all the commands.


Does WinRoboCopy magically prevent empty dirs from being copied? That's the issue for me at this point. Anyway, Scrotos, I'm about to try implementing some of your suggestions for removing the empty directories.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Wed Jan 23, 2013 9:48 am

Yikes, the "for loop" stuff is kinda steep. Not going to understand this in the 60 minute I scheduled for it.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Wed Jan 23, 2013 1:03 pm

It's a bit late now but I used to find the SS64 page really useful.
<insert large, flashing, epileptic-fit-inducing signature (based on the latest internet-meme) here>
Chrispy_
Gerbil Jedi
Gold subscriber
 
 
Posts: 1955
Joined: Fri Apr 09, 2004 3:49 pm

Re: Robocopy help

Postposted on Wed Jan 23, 2013 4:07 pm

Chrispy_ wrote:It's a bit late now but I used to find the SS64 page really useful.


Yes, it's useful. I have that bookmarked. In fact, what I find most useful is being able to read the same explanation written two different ways - if my mind doesn't grok the first way it might grok the second. This is a total fail if the two sources aren't trying to explain exactly the same thing two different ways.
flip-mode
Gerbil Khan
Silver subscriber
 
 
Posts: 9084
Joined: Thu May 08, 2003 12:42 pm
Location: Cincinnati, OH

Re: Robocopy help

Postposted on Thu Jan 24, 2013 10:21 am

flip-mode wrote:Yikes, the "for loop" stuff is kinda steep. Not going to understand this in the 60 minute I scheduled for it.


Ya, no kiddin'. Make some empty test folders and some that aren't empty, pop one of those lines in a test.bat, and run it from the root of the test folders. Confirm whichever one you like works and just roll with it. I always have to sit there and try deconstructing the FOR crap whenever I run across this because I don't use it myself on a regular basis. I assume you're already hitting up http://ss64.com/nt/for_f.html and http://ss64.com/nt/for.html ?

Code: Select all
for /f "usebackq delims=" %%d in (`"dir /ad/b/s | sort /R"`) do rd "%%d"
for /f "tokens=*" %%d in ('dir /ad/b/s ^| sort /R') do rd "%%d"
for /f "delims=" %%i in ('dir /s /b /ad ^| sort /r') do rd "%%i">NUL


The last one just dumps the output into NUL so it's "quiet", you probably figured that one out. The %%d and %%i can be any letter. The DIR and SORT stuff you can easily figure out. In the top one it's setting "usebackq" to enclose `"dir /ad/b/s | sort /R"` but as you can see in the 3rd example, that's not needed. I'm guessing it was an attempt to handle folders with spaces in the name but it didn't work because it's only if you were specifying a file to read/process from with a space in the path. The only benefit I see is that you don't have to escape the |. It's the tokenizing/delimiting part that's probably jackin' up your brain.

You've probably figured out that the output of DIR and SORT are being "piped" back to FOR to process one line at a time. You could add an additional step before the FOR statement to dump that output into a file and instead have FOR read that file to process; conceptually the same. Each line is being stored in %%d based on the delims or tokens. For the second example, it processes ALL tokens on the line and stores it in %%d. This is why folder names with spaces work on that one. For the others where delims=[null], you explicitly say there are no delimiters on the line and to store the entire thing into %%d. If you don't specify that, well, If you don’t specify delims it will default to "delims=[tab][space]". This is why the original example that was missing the ` didn't work with spaces in the folder names since it found a space and stored it into %%d. And, based on the explanation I recently read, the other parts of the line were automatically stored in %%e, %%f, %%g, etc.

And actually, I updated my "examples for the future" based on what I just read up.

Code: Select all
for /f "usebackq delims=" %%g in (`"dir /ad/b/s | sort /r"`) do rd "%%g"
for /f "tokens=*" %%g in ('dir /ad/b/s ^| sort /r') do rd "%%g"
for /f "delims=" %%g in ('dir /ad/b/s ^| sort /r') do rd "%%g">NUL


I'd go with either the 2nd or 3rd one if I were using this in a batch file. I don't know if I'd redirect output to NUL or maybe a .log or maybe not care and let it scroll in the console.

If the token/delim stuff was giving you trouble, I hope my blah blah blah was useful.
Scrotos
Graphmaster Gerbil
 
Posts: 1035
Joined: Tue Oct 02, 2007 12:57 pm
Location: Denver, CO.


Return to Developer's Den

Who is online

Users browsing this forum: No registered users and 4 guests