Personal computing discussed

Moderators: renee, SecretSquirrel, just brew it!

 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 11:09 am

Now that I have a Macbook Air (yay) my immediate concern is to encode selected albums with FLAC files into AAC format so as not to fill up my SSD. To that end I've spent half the day writing up a Python script (note: the Manager class might not run). The point was to not be limited to one encode at a time, but there's a problem. I don't know if it's possible/people have thought of creating objects dynamically. As in, I say I have num CPUs, where num = 4, and so it'll create 4 instances of Workers at runtime. I have no idea how to tell Python that, nor do I know how to reference the created objects. This is what I've got so far, I wonder if I'm doing it wrong? Maybe everybody else is doing concurrent encoding a different way.
#!/usr/bin/python
import sys
import glob
import subprocess
import string
import re

class Worker:
   def __init__(self,id):
      self.tags = {}
      self.id = id
      boss.reportfree(id)
   def setfilename(self,filename):
      self.filename=filename
      return
   def analyze(self):
      self.tags.clear()
      required=["ARTIST","ALBUM","GENRE","DATE","TITLE","TRACKNUMBER"]
      for r in required:
         temporary = subprocess.check_output('/opt/local/bin/metaflac --show-tag='+r+" \""+self.filename+"\"",shell=True)
         temporary = re.sub(r'^[^=]*=', '', temporary)
         temporary = re.sub(r'[\n]', '', temporary)
         self.tags[r] = temporary
      print self.tags
      return
   def encode(self):
      justthename=re.sub('.flac','',self.filename)
      print justthename
      subprocess.check_output("flac -cd \""+self.filename+"\" | faac \"-\" -o "+"\""+justthename+".m4a\"",shell=True)
      boss.reportfree(id)
      return
class Manager:
   def __init__(self,filelist):
      work = filelist
      jobnumber=0
      Worker1 = Worker(1)
      Worker2 = Worker(2)
      Worker3 = Worker(3)
      Worker4 = Worker(4)
      return
   def reportfree(id):
      if(jobnumber<len(work)):
         Worker+eval(id).setfilename(work[jobnumber])
         jobnumber+=1
      else:
         print "I think we're done here..."
      return
   
for arg in sys.argv:
   filelist = glob.glob(arg+"*.flac")
boss = Manager(filelist)
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
Wyglif
Gerbil
Posts: 13
Joined: Tue Jul 13, 2010 7:58 pm

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 12:29 pm

You could pass in the number of workers to the Manager constructor and use a loop to create workers in an array.
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 12:31 pm

I'm a little unclear as to what your issue is. Can you instantiate objects dynamically in Python? Yes * 10000. Practically everything in Python is dynamic to the point where it gets a little bit silly.

If you know that you have 4 CPUs (either passed as an argument or your program figures it out by querying the OS) then you simply would instantiate 4 different objects. Then *inside* each of the 4 objects I would spawn the child process using the standard stuff in the subprocess module (refer to the Python docs for more information).

Looking at your code right now, you seem to be hard-coding for 4 worker processes. You could simply store references to each Worker in an array and dynamically spawn the workers using something similar to this pseudo-code:

=================
# trivial code for initializing maxWorkerCount worker objects. You are responsible for deciding
# what maxWorkerCount should be
# Note: a list comprehension would be a much slicker way to write this but I'm keeping it simple on purpose
workers = []
for workerNum in range (0, maxWorkerCount):
    workers.append (Worker (workerNum))
=================


Now whenever you want to access your workers (could be any number of them) you simply say, for example:
workers [0].sefilename (<your filename here>)



As I said before, Python is an *incredibly* dynamic language. The examples I gave above are very simple, but keep reading the references at python.org and looking at online examples. Python gives you facilities for quickly and efficiently handling objects that are much much nicer than most other languages if you put the time in to use Python's facilities to their greatest extent.
Last edited by chuckula on Thu Sep 06, 2012 12:33 pm, edited 1 time in total.
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
ChronoReverse
Gerbil Elite
Posts: 757
Joined: Wed Dec 12, 2007 4:20 pm

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 12:32 pm

I couldn't help but post this: http://xkcd.com/353/
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 12:43 pm

Thanks, I never knew Python/C++ allowed dynamic object creation before, I just figured it would be really useful, but I didn't know how to access the objects once they were created. And yes, I'm loving Python!
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 1:21 pm

If you want to try stuff out, you can run stuff directly from the python shell. Just open a Terminal and run python, or python3. It's pretty fun.
 
ermo
Gerbil
Posts: 55
Joined: Tue Jan 25, 2011 7:35 pm

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 2:22 pm

Flatland_Spider wrote:
If you want to try stuff out, you can run stuff directly from the python shell. Just open a Terminal and run python, or python3. It's pretty fun.


@Crayon Shin Chan:
I generally tend to use python's integrated development environment -- called IDLE -- across Linux and Windows. It's got nice syntax highlighting and code completion and running the code is a simple 'F5' away.

Try it, you might like it. 8)

And yeah, using a list to manage the workers as they are created sounds like a neat approach.

EDIT: In your situation, wouldn't something like Wavpack in hybrid mode be useful? Or is wavpack not in vogue around here?
Linux: 2x FX-8350, 16/32GB, GTX970/HD7970, Solus+KDE/Exherbo+Xfce
Hackintosh: i7-2600K, 32GB, HD7970, High Sierra
HTPC: Q9400, 8GB, GTS 450, Solus/GNOME
Server: FX-8350, 16GB ECC, iGP, F30/Srv
Wintendo: R7 2700X, 32GB, 2x RX Vega 64 (air), W10Pro
 
Flatland_Spider
Graphmaster Gerbil
Posts: 1324
Joined: Mon Sep 13, 2004 8:33 pm

Re: Creating objects dynamically in Python: possible?

Thu Sep 06, 2012 7:35 pm

I like using Dreampie (http://dreampie.sourceforge.net/) when I'm messing around with python.
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Sun Sep 09, 2012 10:10 am

Well, I'm stuck. I need a way for the workers to communicate with the manager, so that they can tell the manager when they're free and new tasks can be assigned, but I don't think it's proper practice to define the Manager class, declare an instance of it, and tell the workers to communicate to that instance (probably because I come from a C++ world). Is it good form? If I create an intermediate "event layer" class, it'll still need to be instanced before the definitions of Worker anyway, so I don't think it'll be any different.

So I've been searching a bit, and I found this:
http://www.doughellmann.com/PyMOTW/mult ... ation.html
Is this applicable? I think it might be overkill, since I just want to make objects that communicate with each other (and they just happen to be spawning different threads due to the nature of their work).
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Creating objects dynamically in Python: possible?

Sun Sep 09, 2012 11:02 am

The article you found is in the right direction but may be more than you need.
When you spawn a new process (in Python or any other language for that matter) you are basically "cloning" the parent process and then proceeding to do new things in the child process. The parent process still exists and needs to monitor the child processes so that you don't get the dreaded "zombie" processes once a child process exits.

Basically, you need some form of inter-process communication (IPC)* The simplest form of IPC that could be good enough for your purposes is to look at "signals". The signal is a very simple numeric return code that you get when your child process exits. You can wait on a child process and listen for the signal to indicate that the encoding program has exited. The numeric return value can sometimes tell you if the program succeeded or failed too. Generally speaking, a signal return of "0" means success, and other numbers indicate some sort of error or that the program was aborted.

It looks like you are spawning a bunch of processes to do FLAAC --> AAC conversions in this program. I would recommend a "fire and forget" approach where each child process only ever runs a single encoding program on a single file. The parent program (Manager) is simply responsible for controlling how many children are spawned at once. When once child program exits, you get a return code and the parent program spawns another child to handle the next file. You don't need to do complicated IPC this way, just fire off a new child process after an earlier child process gets finished. Since you have multiple child processes, I would recommend using Python's asynchronous waiting functions like the "poll" function from the subprocess module (see a more in-depth explanation here: http://stackoverflow.com/questions/6365 ... rom-python) The advantage to using asynchronous calls is that they are non-blocking, meaning you can check the status of a child process without forcing the parent to wait until the child is closed. You can use a monitoring loop and go down the full list of child processes every so often to see which ones are running and which ones have exited, and spawn new child processes accordingly.

* Without using some form of IPC you can't have two processes talk to each other. Using a trivial example, say we have a parent process with a variable A = 10. Next, we spawn a child process that gets its own copy of the variable, A = 10. Now say the parent changes A: A += 1. The child process still sees A = 10 because the child is now a completely different process, even if both the parent and child have the same "A" variable. This is a difference between multi-process and multi-threaded programming, because in a multi-threaded program if "A" has the correct visibility, then both a parent thread and child thread see a single value of "A" in a single memory space. Multi-process programming (like what you are doing) means that the child has its own memory space that is completely separate from the parent process, just like the memory space that the OS is showing to my web browser is kept separate from the one it is showing to my command terminal. You can read up more in "virtual memory" in modern operating systems for how this works.
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 9:20 am

Help! Suddenly glob doesn't work for me anymore! Simply putting this code in a separate test script:
import glob
glob.glob('*.flac')

gives me nothing! But when I use the python interpreter, started from the same directory that the script was started in, it works perfectly!
>>> import glob
>>> glob.glob('*flac')
['01 - The Girl from Ipanema.flac', '02 - Doralice.flac', '03 - Para Manuchar Meu Corac\xcc\xa7ao.flac', '04 - Desafinado (Off Key).flac', '05 - Corcovado (Quiet Nights of Quiet Stars).flac', '06 - So Danc\xcc\xa7o Samba.flac', '07 - O Grande Amor.flac', '08 - Vivo Sonhando (Dreamer).flac', '09 - Garota de Ipanema - 45 rpm issue.flac', '10 - Corcovado (Quiet Nights of Quiet Stars) - 45 rpm issue.flac']
>>>



Even os.listdir doesn't work.
asuka:Getz-Gilberto shinichi$ python
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir('.')
['01 - The Girl from Ipanema.flac', '02 - Doralice.flac', '03 - Para Manuchar Meu Corac\xcc\xa7ao.flac', '04 - Desafinado (Off Key).flac', '05 - Corcovado (Quiet Nights of Quiet Stars).flac', '06 - So Danc\xcc\xa7o Samba.flac', '07 - O Grande Amor.flac', '08 - Vivo Sonhando (Dreamer).flac', '09 - Garota de Ipanema - 45 rpm issue.flac', '10 - Corcovado (Quiet Nights of Quiet Stars) - 45 rpm issue.flac', 'covers', 'audiolocalconvert.py', '._audiolocalconvert.py', '.DS_Store']


.....
for arg in sys.argv:"""
os.listdir('.')
#filelist = glob.glob(arg+"*.flac")
asuka:Getz-Gilberto shinichi$ ./audiolocalconvert.py
asuka:Getz-Gilberto shinichi$
 

Absolutely nothing.
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 10:06 am

Check your current-working directory from inside the python script. For some reason, it may be changing to another directory in a non-obvious way. You may need to use absolute file paths or chance the working directory before using the glob and listdir calls.
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 10:15 am

It seems that the output of glob is not being saved into the variable "filelist". I don't know why, it works perfectly in the Python shell.

import os
import glob
print "Current working directory: %s " %os.getcwd()
print "%s" %glob.glob('*')
filelist = glob.glob('*.flac')
print "I globbed, now I spit"
filelist


Current working directory: /Volumes/Warehouse/Getz-Gilberto 
['01 - The Girl from Ipanema.flac', '02 - Doralice.flac', '03 - Para Manuchar Meu Corac\xcc\xa7ao.flac', '04 - Desafinado (Off Key).flac', '05 - Corcovado (Quiet Nights of Quiet Stars).flac', '06 - So Danc\xcc\xa7o Samba.flac', '07 - O Grande Amor.flac', '08 - Vivo Sonhando (Dreamer).flac', '09 - Garota de Ipanema - 45 rpm issue.flac', '10 - Corcovado (Quiet Nights of Quiet Stars) - 45 rpm issue.flac', 'covers', 'audiolocalconvert.py', 'test.py']
I globbed, now I spit
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 10:28 am

Oh wait.. facepalm time!
Look at this line:
#filelist = glob.glob(arg+"*.flac")


Notice the "#" in front where you comment out the glob operation? Won't work too well with the code commented out :wink:
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 10:34 am

Thanks, but I did that to ensure that I would only see what os.listdir() would output, and not confuse that with glob's output (if it suddenly chose to output anything :( )

In my latest post, the one before your reply, the same line is not commented out before print "I globbed, now I spit". Yet it seems nothing is stored in filelist after all.
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
chuckula
Minister of Gerbil Affairs
Posts: 2109
Joined: Wed Jan 23, 2008 9:18 pm
Location: Probably where I don't belong.

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 11:11 am

OK, good point, there are different code snippets floating around.

I have another crazy idea: When you simply type "filelist" and press ENTER from the interactive interpreter, the interpreter dumps the contents of filelist to the screen. HOWEVER, if you simply put the term "filelist" in a non-interactive python script, the interpreter basically does nothing. Putting "filelist" on a line by itself doesn't raise an error because you have declared it, but you aren't telling Python to actually do anything with filelist.

Try this code instead (new code after the "I globbed, now I spit" line):

import os
import glob
print "Current working directory: %s " %os.getcwd()
print "%s" %glob.glob('*')
filelist = glob.glob('*.flac')
print "I globbed, now I spit"
for nextGlobEntry in filelist:
   print "Found this *.flac file: %s" % (nextGlobEntry)
4770K @ 4.7 GHz; 32GB DDR3-2133; Officially RX-560... that's right AMD you shills!; 512GB 840 Pro (2x); Fractal Define XL-R2; NZXT Kraken-X60
--Many thanks to the TR Forum for advice in getting it built.
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Creating objects dynamically in Python: possible?

Mon Sep 10, 2012 11:16 am

That's IT! Thanks so much!

asuka:Getz-Gilberto shinichi$ python test.py
Current working directory: /Volumes/Warehouse/Getz-Gilberto
['01 - The Girl from Ipanema.flac', '02 - Doralice.flac', '03 - Para Manuchar Meu Corac\xcc\xa7ao.flac', '04 - Desafinado (Off Key).flac', '05 - Corcovado (Quiet Nights of Quiet Stars).flac', '06 - So Danc\xcc\xa7o Samba.flac', '07 - O Grande Amor.flac', '08 - Vivo Sonhando (Dreamer).flac', '09 - Garota de Ipanema - 45 rpm issue.flac', '10 - Corcovado (Quiet Nights of Quiet Stars) - 45 rpm issue.flac', 'covers', 'audiolocalconvert.py', 'test.py']
I globbed, now I spit
Found this *.flac file: 01 - The Girl from Ipanema.flac
Found this *.flac file: 02 - Doralice.flac
Found this *.flac file: 03 - Para Manuchar Meu Coraçao.flac
Found this *.flac file: 04 - Desafinado (Off Key).flac
Found this *.flac file: 05 - Corcovado (Quiet Nights of Quiet Stars).flac
Found this *.flac file: 06 - So Danço Samba.flac
Found this *.flac file: 07 - O Grande Amor.flac
Found this *.flac file: 08 - Vivo Sonhando (Dreamer).flac
Found this *.flac file: 09 - Garota de Ipanema - 45 rpm issue.flac
Found this *.flac file: 10 - Corcovado (Quiet Nights of Quiet Stars) - 45 rpm issue.flac
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On