Personal computing discussed

Moderators: SecretSquirrel, notfred

 
cheesyking
Minister of Gerbil Affairs
Topic Author
Posts: 2664
Joined: Sun Jan 25, 2004 7:52 am
Location: That London (or so I'm told)
Contact:

Cron and gui applications

Mon Feb 06, 2017 6:49 am

I've got a python script that uses selenium to control a chromium browser window and I'd like to have it run on a schedule on a dedicated VM. The problem I have is that if I use cron to do this I'd have to add something like:
export DISPLAY=:0 && /path/to/command
but this only works if the user is logged in and I'd really like it to run without the user being logged in. Since it's a dedicated VM I suppose I could just run it with a user set to automatically log in but I'd like to at least have looked at doing it some other way.

I've been reading about using xinitrc but this seems a little heavy handed.

Any suggestions for another way of doing this?

(ubuntu 16.04)
Fernando!
Your mother ate my dog!
 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2297
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)
Contact:

Re: Cron and gui applications

Mon Feb 06, 2017 7:27 am

What part only works if the user is logged in?

One of the environments I'm responsible for at work has a Chromium session that runs in kiosk mode. It runs at system boot as a service and runs as a non-root user. The user isn't logged in and everything works fine. Haven't tried using selenium to interact with it, but interacting with the X display works just fine. You do have to set the DISPLAY environment variable as you note.

--SS
 
just brew it!
Gold subscriber
Administrator
Posts: 49718
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cron and gui applications

Mon Feb 06, 2017 7:28 am

What's the application here? Does the window need to be visible or are you just trying to automate some HTTP actions?

If it is the latter (i.e. the window does not need to be visible), you might be better off using a pure CLI solution like wget or cURL instead of automating a browser.
Nostalgia isn't what it used to be.
 
cheesyking
Minister of Gerbil Affairs
Topic Author
Posts: 2664
Joined: Sun Jan 25, 2004 7:52 am
Location: That London (or so I'm told)
Contact:

Re: Cron and gui applications

Mon Feb 06, 2017 8:37 am

SecretSquirrel wrote:
What part only works if the user is logged in?  

Well I tried it from the command line while logged in with ssh. If I have the user logged in to a desktop session then chrome starts and everything works fine. If I don't have the user logged in to the desktop then the python script throws an error (after a long pause) about "chrome failed to start".

just brew it! wrote:
What's the application here? Does the window need to be visible or are you just trying to automate some HTTP actions?

If it is the latter (i.e. the window does not need to be visible), you might be better off using a pure CLI solution like wget or cURL instead of automating a browser.


The window doesn't need to be visible as I'm just trying to automate some http however I spent quite a lot of time looking at pure CLI solutions and the webapp I'm automating seems to be actively hostile towards this. It heavily relies on javascript and everything short of a full browser like I get with selenium seemed completely unworkable. TBH even using selenium was difficult as the names/IDs of many page elements change between each page load.

Just for reference this is downloading call recordings from a voip supplier who doesn't offer automatic ftp transfers or have any kind of API, just a very fiddly website that's often very slow.
Fernando!

Your mother ate my dog!
 
vishalsuvagia
Gerbil In Training
Posts: 3
Joined: Mon Feb 06, 2017 8:18 am

Re: Cron and gui applications

Mon Feb 06, 2017 8:45 am

If you are trying to achieve automation (selemium), which also needs to be run on a regular / periodic basis, It may help with to use build tools like jenkins (jenkins.io) and hudson (hudson-ci.org) .
Using the tools it will not only automatically build the task periodically, you can also view the details of each task, the command being run and the outputs.
This is my TR sign.
 
vishalsuvagia
Gerbil In Training
Posts: 3
Joined: Mon Feb 06, 2017 8:18 am

Re: Cron and gui applications

Mon Feb 06, 2017 9:00 am

cheesyking wrote:
The window doesn't need to be visible as I'm just trying to automate some http however I spent quite a lot of time looking at pure CLI solutions and the webapp I'm automating seems to be actively hostile towards this. It heavily relies on javascript and everything short of a full browser like I get with selenium seemed completely unworkable. TBH even using selenium was difficult as the names/IDs of many page elements change between each page load.

You might not need to login at-all.If you are running selenium tests, you can do the same headlessly ( i.e on a linux without a display ), you will need to set the Display buffer using something like 
export Xvfb :99,DISPLAY:99 and some additional parameters like screen width and height


You can do a search for linux run browser on headless instance ( as I am a new user, yet unable to post links.)
This is my TR sign.
 
just brew it!
Gold subscriber
Administrator
Posts: 49718
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cron and gui applications

Mon Feb 06, 2017 9:04 am

Running in a vncserver session is another possibility. A full VM (while it'll get the job done) is overkill.
Nostalgia isn't what it used to be.
 
cheesyking
Minister of Gerbil Affairs
Topic Author
Posts: 2664
Joined: Sun Jan 25, 2004 7:52 am
Location: That London (or so I'm told)
Contact:

Re: Cron and gui applications

Mon Feb 06, 2017 10:46 am

Knew you guys would have some good suggestions. Thanks all I've got it working with xvfb. I'll have to have a look at using something like jenkins in the future and getting vnc working with xvfb so I've got the option of checking what selenium is up to.

xvfb-run --auto-servernum -s "-screen 0 1366x768x24" python /callRecordings/callscraper.py

Spent ages trying to figure out why it wouldn't work with ...1366x768x32... but it turns out that's a known bug.

Thanks again :D
Fernando!

Your mother ate my dog!
 
SecretSquirrel
Minister of Gerbil Affairs
Posts: 2297
Joined: Tue Jan 01, 2002 7:00 pm
Location: The Colony, TX (Dallas suburb)
Contact:

Re: Cron and gui applications

Mon Feb 06, 2017 1:51 pm

cheesyking wrote:
Knew you guys would have some good suggestions. Thanks all I've got it working with xvfb. I'll have to have a look at using something like jenkins in the future and getting vnc working with xvfb so I've got the option of checking what selenium is up to.

xvfb-run --auto-servernum -s "-screen 0 1366x768x24" python /callRecordings/callscraper.py

Spent ages trying to figure out why it wouldn't work with ...1366x768x32... but it turns out that's a known bug.

Thanks again :D


For a bit of learning.... The reason it wasn't working for you with the normal X server is a permissions thing. The X server started at boot is owned by root. By default, you don't have permission to access the display. Part of the graphical login process grants your account access to the display provided by the system X server. If you did need to see the activity, you could have done the same thing as above with a real X server. The key is that you need to start that X server as the account you want to run the tests. That way you don't have permissions issues.

Glad you got it working.
--SS
 
cheesyking
Minister of Gerbil Affairs
Topic Author
Posts: 2664
Joined: Sun Jan 25, 2004 7:52 am
Location: That London (or so I'm told)
Contact:

Re: Cron and gui applications

Mon Feb 06, 2017 3:07 pm

Thanks again, the graphical side of things in Linux is something I know very little about.
Fernando!

Your mother ate my dog!
 
just brew it!
Gold subscriber
Administrator
Posts: 49718
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cron and gui applications

Mon Feb 06, 2017 4:05 pm

cheesyking wrote:
Thanks again, the graphical side of things in Linux is something I know very little about.

The X Window System has an awful lot of legacy baggage; it has been around for more than 3 decades, and was designed while modern GUI concepts were still being developed. Even the basic terminology is confusing at times (e.g. the display being referred to as the "server" and the application being referred to as the "client"). It was designed around the idea that the display was a separate device (an "X terminal") connected over a network to whatever was running the application code; this inevitably results in inefficiencies for the (now more common) use case where the application and display are all running on a single device/system.

The proliferation of DEs and toolkits just adds to the confusion...

There are replacements for X being developed, but it has over 3 decades of inertia behind it at this point, and (even assuming one of the alternatives gets enough traction) de-X-ifying the Linux desktop will likely be a long process.
Nostalgia isn't what it used to be.
 
Redocbew
Gold subscriber
Graphmaster Gerbil
Posts: 1350
Joined: Sat Mar 15, 2014 11:44 am

Re: Cron and gui applications

Mon Feb 06, 2017 4:17 pm

It's a bit self-sustaining now also, because for many people new to the system X is just totally weird, and that just adds to the up-front cost of working on a replacement for it.  I personally never really made it a point of studying too deeply beyond the basics of how X works just because I found many other parts of the system so much more accessible.
Do not meddle in the affairs of archers, for they are subtle and you won't hear them coming.
 
just brew it!
Gold subscriber
Administrator
Posts: 49718
Joined: Tue Aug 20, 2002 10:51 pm
Location: Somewhere, having a beer

Re: Cron and gui applications

Mon Feb 06, 2017 4:49 pm

Another way to look at it: The X Window System was invented much closer in time to the invention of MS-DOS than to the present day. :wink:
Nostalgia isn't what it used to be.
 
Captain Ned
Gold subscriber
Global Moderator
Posts: 26471
Joined: Wed Jan 16, 2002 7:00 pm
Location: Vermont, USA

Re: Cron and gui applications

Mon Feb 06, 2017 5:49 pm

just brew it! wrote:
Another way to look at it: The X Window System was invented much closer in time to the invention of MS-DOS than to the present day. :wink:

MS-DOS: August 1981
X: May 1984
If the Earth were flat, cats would have pushed everything off of it by now.

Who is online

Users browsing this forum: No registered users and 1 guest