Best solution for automated Windows file processing?

From Visual Basic to GNU C, this is the place to talk programming.

Moderators: SecretSquirrel, just brew it!

Best solution for automated Windows file processing?

Postposted on Fri Jun 12, 2009 3:46 pm

Hello everyone,

In a nutshell, I need to be able to automate the processing of some text files - searching for specific strings, piping the output to a new file, etc. On Linux I would've done this most likely with shell scripting and sed, but for Windows (XP) I'm at a loss. Could anyone recommend a good solution (language, etc.) for manipulating and processing text/data automatically on a windows system?

Thanks!
Luke
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Fri Jun 12, 2009 4:01 pm

Maybe you can do this in Windows Powershell? It looks like there is a version for XP
Evaders99
Gerbil First Class
 
Posts: 147
Joined: Fri May 16, 2008 10:48 am

Re: Best solution for automated Windows file processing?

Postposted on Fri Jun 12, 2009 4:16 pm

If you know what you'd do with linux CLI, just install cygwin and do it on windows. No need to reinvent the wheel if you're already comfortable with those.
...
mattsteg
Gerbil God
 
Posts: 15761
Joined: Thu Dec 27, 2001 7:00 pm
Location: Applauding the new/old variable width forums

Re: Best solution for automated Windows file processing?

Postposted on Fri Jun 12, 2009 4:44 pm

Well, it's been about ten years since I did that kind of stuff in Linux so either way I'll be starting from scratch. That being the case I thought I might as well use a windows specific tool, but yeah - not a bad idea. That way if I ever need to do something in Linux, I'm good to go.

Thanks!
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Fri Jun 12, 2009 5:18 pm

There really are a number of ways. These are some of them:
- mattsteg's suggestion: *nix tools on cygwin
- Ruby
- VBScript
- JScript
- findstr command (XP and up)
- etc.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24576
Joined: Mon May 24, 2004 2:19 am

Re: Best solution for automated Windows file processing?

Postposted on Sat Jun 13, 2009 5:48 pm

Let me walk through what I hope to do and see if anyone thinks a particular method really stands out as being the simplest.

1. Grab a name from text file (it will always be on the same line and in the same format)
2. Use windows "Grep" to search a different text file for that name, if it finds any results, save grep's results to a new text file.

That's it really. So other than having the batch file automatically run grep filling in the name as one of the command line options (which I currently don't know how to do), the only other thing I need to know how to do it read that name from the other text file and pass it to grep.

Thanks again,
Luke
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Sat Jun 13, 2009 6:03 pm

My vote would be to install either Cygwin or Python, and use that.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38085
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Best solution for automated Windows file processing?

Postposted on Sat Jun 13, 2009 6:53 pm

I'll try it - thanks!
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Sat Jun 13, 2009 7:13 pm

To clarify the Python recommendation in my previous post: I'd only suggest that if you already know Python, or want to learn it. Otherwise go the Cygwin route (since you've worked with *NIX shell tools before).

I do think that Python is a very useful skill to have these days... but I don't know if you're in the mood to learn a new scripting language right now (only you can answer that). :wink:
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38085
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Best solution for automated Windows file processing?

Postposted on Mon Jun 15, 2009 9:12 pm

Do you know any programming languages with Regular Expressions (e.g., Python, Perl, Java, .NET...)?
wibeasley
Gerbil Elite
Gold subscriber
 
 
Posts: 952
Joined: Sat Mar 29, 2008 3:19 pm
Location: Norman OK

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 8:18 am

wibeasley wrote:programming languages with Regular Expressions (e.g., Python, Perl, Java, .NET...)?
Whoah, what happened to grep, sed, awk and friends? What's with all this new fangled stuff? Now get off my lawn! :P
notfred
Grand Gerbil Poohbah
 
Posts: 3772
Joined: Tue Aug 10, 2004 10:10 am
Location: Ottawa, Canada

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 8:41 am

notfred wrote:
wibeasley wrote:programming languages with Regular Expressions (e.g., Python, Perl, Java, .NET...)?
Whoah, what happened to grep, sed, awk and friends? What's with all this new fangled stuff? Now get off my lawn! :P

Some of those require cygwin to be installed on Windows. The solution can of course include those but the grep'ed results need to be used for further searching, so something on the "outside" would need to be used to coordinate the searches. Just a search tool is not quite enough.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24576
Joined: Mon May 24, 2004 2:19 am

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 11:12 am

Well, it really has been at least 10 years since I've done any of this stuff, so I don't have anything in my 'toolbox' as of now. Having said that, to the best of my recollection, In the past on linux/unix I probably would've used a combination of grep, sed and some shell scripting to control it all (and of course a cron job to run it). So it sounds like cygwin w/ grep and sed for windows will pretty much allow me the same functionality. Yes?
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 11:57 am

mrlament wrote:Well, it really has been at least 10 years since I've done any of this stuff, so I don't have anything in my 'toolbox' as of now. Having said that, to the best of my recollection, In the past on linux/unix I probably would've used a combination of grep, sed and some shell scripting to control it all (and of course a cron job to run it). So it sounds like cygwin w/ grep and sed for windows will pretty much allow me the same functionality. Yes?

You'll need to learn a bit on the Windows Task Scheduler for the cron part, but pretty much.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24576
Joined: Mon May 24, 2004 2:19 am

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 6:24 pm

Cool. Thanks everyone - time to put the ol' nose to the grind stone.
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 7:06 pm

Flying Fox wrote:You'll need to learn a bit on the Windows Task Scheduler for the cron part, but pretty much.

Actually, Cygwin includes a ported version of cron as well.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38085
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Best solution for automated Windows file processing?

Postposted on Tue Jun 16, 2009 8:09 pm

Sweet! I'm sold!
mrlament
Gerbil In Training
 
Posts: 7
Joined: Fri Jun 12, 2009 3:42 pm

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 12:01 pm

just brew it! wrote:To clarify the Python recommendation in my previous post: I'd only suggest that if you already know Python, or want to learn it. Otherwise go the Cygwin route (since you've worked with *NIX shell tools before).

I do think that Python is a very useful skill to have these days... but I don't know if you're in the mood to learn a new scripting language right now (only you can answer that). :wink:
Out of curiosity, is there any specific python tutorial or online learning reference that you recommend? It's looking like python is probably going to be the way to go for something I've been ruminating on.
...
mattsteg
Gerbil God
 
Posts: 15761
Joined: Thu Dec 27, 2001 7:00 pm
Location: Applauding the new/old variable width forums

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 12:07 pm

mattsteg wrote:
just brew it! wrote:To clarify the Python recommendation in my previous post: I'd only suggest that if you already know Python, or want to learn it. Otherwise go the Cygwin route (since you've worked with *NIX shell tools before).

I do think that Python is a very useful skill to have these days... but I don't know if you're in the mood to learn a new scripting language right now (only you can answer that). :wink:
Out of curiosity, is there any specific python tutorial or online learning reference that you recommend? It's looking like python is probably going to be the way to go for something I've been ruminating on.


Coming from a programming background the python docs should be a good start
http://docs.python.org/tutorial/
Fernando!
Your mother ate my dog!
cheesyking
Minister of Gerbil Affairs
 
Posts: 2283
Joined: Sun Jan 25, 2004 7:52 am
Location: That London (or so I'm told)

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 12:37 pm

cheesyking wrote:
mattsteg wrote:Out of curiosity, is there any specific python tutorial or online learning reference that you recommend? It's looking like python is probably going to be the way to go for something I've been ruminating on.

Coming from a programming background the python docs should be a good start
http://docs.python.org/tutorial/

Agreed.

The top-level index (http://docs.python.org/) also has some material you will want to look at, like the Library Reference. If you can give a little more detail regarding the project you're thinking of using Python for, I may be able to direct you towards more specific information.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38085
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 4:41 pm

just brew it! wrote:If you can give a little more detail regarding the project you're thinking of using Python for, I may be able to direct you towards more specific information.

Glorified shell scripting that I might throw a GUI on top of. Lots of calls to external programs that I need to generate programatically and execute. Hackjob glue to hold together 2-3 other hackjobs :).

Currently using OGR2OGR to merge and combine shapefiles, along with a build of gpsbabel that I've modified to have more functionality (and fixed broken stuff) with Delorme .an1 files while actually working. I'm also probably going to hack on shp2txt to get a command-line program that does what I current'y only have someone else's GUI for (write text files in Delorme's text format for areas/polygons). Overall, it's a huge mishmash of working with 2-3 different applications that either do what I want done or can be easily modified to do so, because I don't want to go through the effort of developing a full application. The easiest way forward seems to be modifing multiple programs to take advantage of the "low hanging fruit" that each offers, and then glue them together with some bubble gum. Python seems like a good option for this convoluted cluster of a task, plus it ships with the normal way of obtaining binaries of the ogrtools so if I get a useful tool and distribute it to others, using python doesn't add an additional requirement. As a hackjob, functionality and implementation is probably going to bounce around - I need something that works, not necessarily something that works elegantly.

The end goal is to develop an easy workflow to move specific data between usgs large scale DLGs and Canadian 50k vector topo maps and Delorme software (and thus to my GPSr). I'd also probably throw in conversion from jp2 to geotiff using GDAL, as Delorme software doesn't speak jp2 and some data that I get is in that format. The volume of this sort of data in file size and/or numbers can add up fast, and managing it can be a challenge.
...
mattsteg
Gerbil God
 
Posts: 15761
Joined: Thu Dec 27, 2001 7:00 pm
Location: Applauding the new/old variable width forums

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 5:12 pm

I've used Perl for this before. Works great. Scheduling is done with the task scheduler, there are also third-party ones.
StuffMaster
Gerbil
 
Posts: 49
Joined: Thu Sep 20, 2007 8:41 am

Re: Best solution for automated Windows file processing?

Postposted on Wed Jun 24, 2009 9:09 pm

Sounds like Python should work fairly well for what you're doing (but you already knew that, or you wouldn't be asking about it).

The standard GUI toolkit for Python is Tkinter, which is actually a set of Python bindings to Tcl's Tk toolkit. It isn't elegant or pretty (it uses its own widgets instead of the ones native to the OS it is running on), but it does get the job done for simple GUI tasks, and is completely cross-platform. The wxWidgets GUI toolkit has been gaining popularity lately, and may be the heir to Tkinter; I've been meaning to check wxWidgets out but haven't had time yet.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 38085
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Best solution for automated Windows file processing?

Postposted on Thu Jun 25, 2009 8:37 am

I don't use Python (although it may be interesting to learn since I can move that skill to coding S60 apps), but Ruby should be able to do the same thing too. At work we have settled on using Ruby for general scripting.
Image
The Model M is not for the faint of heart. You either like them or hate them.

Gerbils unite! Fold for UnitedGerbilNation, team 2630.
Flying Fox
Gerbil God
 
Posts: 24576
Joined: Mon May 24, 2004 2:19 am


Return to Developer's Den

Who is online

Users browsing this forum: No registered users and 3 guests