Crontab Curl Once a Day Less Work Tutorial

Crontab Curl Once a Day Less Work Tutorial

We arrange on this RJM Programming domain, via crontab/curl means, many once a day changes to the website. But we want to stop repeating some of the work, whereby MySql queries happen too often in our Australian Eastern Standard Time mornings. First cab off the rank to change this “rush and load” is our use of our inhouse changed tutorial_options.php, coming at it from two different approaches …

• change the timings of crontab to not have jobs too close together
• change the code of tutorial_options.php to not unnecessarily repeat work …
  
$huhfind='MyNew-' . date('Ymd');
$huhvs="";
$index = "index";
if (isset($_GET['index'])) {
$index = $_GET['index'];
} else if (isset($_POST['index'])) {
$index = $_POST['index'];
}

$isuffix = "";
if (isset($_GET['mobile'])) {
$isuffix = $_GET['mobile'];
if (file_exists("../indexmobile.html")) {
$huhvs=file_get_contents("../indexmobile.html");
if (strpos($huhvs, $huhfind) === false) {
$huhvs=" ";
} else {
exit;
}
}
} else if ($index == "slideshow") {
if (file_exists("../slideshow.html")) {
$huhvs=file_get_contents("../slideshow.html");
if (strpos($huhvs, $huhfind) === false) {
$huhvs=" ";
} else {
exit;
}
}
} else {
if (file_exists("../index.htm")) {
$huhvs=file_get_contents("../index.htm");
if (strpos($huhvs, $huhfind) === false) {
$huhvs=" ";
} else {
exit;
}
}
}


Previous relevant SSL on Linux Crontab Curl Tutorial is shown below.

SSL on Linux Crontab Curl Tutorial

Do you remember when we used PHP and the dynamic duo of Crontab and Curl on a Linux web server for the www.rjmprogramming.com.au domain to continually update its landing page to populate a dropdown of our blog’s postings? Back then, as you can see with More on Linux Crontab Curl Tutorial, and into all the incarnations of tutorial_options.php we never considered SSL and the thought that one day we’d want to be using the web address URL …


https://www.rjmprogramming.com.au
... rather than just plain old ...
http://www.rjmprogramming.com.au


…


What a difference a day made an "s" makes
Twenty-four little hours
Brought the sun and the flowers
Where there used to be rain


… added security by encryption of your web page contents, that is.

You can view what amounted to a one line (added) change as per …


// ... other code above ...
function guidedextras($ing, $intitle, $xid, $pexcerpt) {
global $prevdod, $prevint, $prevret, $retvalp, $prevxid, $and, $ptg, $index;
$ing=str_replace("https:","",str_replace("http:","",$ing));
$thisint = explode(" ", str_replace("_", " ", str_replace(" ", "++", str_replace('"', '%22', str_replace("'", "%27", $intitle)))));
// ... other code below ...


… this way in this tutorial_options.php code.

This is our first of a series of blog posts outlining our journey through SSL www.rjmprogramming.com.au issues. They often revolve around changing absolute URLs starting with http:// to just // syntax. If this interests, hope you tag along as we go into more.

Previous relevant More on Linux Crontab Curl Tutorial is shown below.

More on Linux Crontab Curl Tutorial

Sometimes for a website you only want to do a web server task once or a few times a day, and you, ideally, wouldn’t want to run into the remote possibility that two people could perform the task at precisely the same time. This scenario happens often when using server-side languages like ASP.Net and PHP. If it didn’t matter that two people might do the same thing at once then you could do the job within you web-based PHP, and you may consider using a date and time test in your web-based PHP to do your “once or a few times” task in your PHP. This latter idea has the advantage that file ownership issues will probably not crop up because the web user will be manipulating any media or HTML files you may be dealing with, rather than the administrator user that will probably be overseeing your “once or a few times” cron job that we show you here today.

So the task today is to keep the rjmprogramming.com.au’s landing page “Tutorials” dropdown up to date with the latest blog postings, rather than having to manually attend to this with interactive (HTML code) edits and (s)ftp uploads. You may be wondering why not just write the landing page in PHP, and this is a fair point, but the strategy here at this domain is that the landing page and its “fellow” webpages should be independent of MySql, which, under big loads, can go down, and here we use a watchdog to fix that, should it happen. Previously we had a very similar task you can read about with the Linux Crontab Curl Primer Tutorial below. It is so similar, let’s use curl, PHP and crontab to do this task like with that previous task. So this is command line, and this is batch work, but that worry in other types of batch work where you need to worry about where you are as you do any file management, is handled quite well by the wrapping of the job in curl, which takes as its argument a URL, so, implicitly, the place to work with file management, is handled by curl and the PHP we write (specifically where the code uses dirname(__FILE__)).

Here is the downloadable PHP programming source code you could call tutorial_options.php and you will see, if you examine the code that it is very specific to conditions here at this rjmprogramming.com.au domain, so please be aware of that, but maybe its concepts can help you out with something else you are tackling. A list of the specifics you would want to address (should you adapt it for your own purposes) include:

• the PHP explode methodology (in the code above) assumes there are two select dropdowns (spelt in lowercase) in the underlying HTML code (for rjmprogramming.com.au’s landing page) we are rewriting, and that it is the second one we update with information gleaned (via SQL query) from the WordPress MySql database the PHP reads (where we piece together a $retval string of the HTML option tags belonging to the tutorials HTML select tag)
• the MySql database access hostname, username, password, database name, database table name
• the Linux web server file owner used during the (commented out) chown command you may need to consider to allow the crontab’s user be able to leave the files owned by the website user

PHP commands of interest in today’s tutorial include:

• file_get_contents
• file_put_contents
• dirname
• chown
• explode
• str_replace

Maybe it is just me, but liking “Onion of the 4th dimension” the kind of thinking I gravitate towards is:

• supervisory command line arrangements
• all “replace” type functionality … to get the problem into a shape suitable for the bits down deeper into the “core” of your “onion” … Linux-speak would say “kernel” of your “onion”
• parameters and arguments … continuing on that theme of slotting something (the user is specifically interested in, data-wise) in dynamically for something that “represents” it in the code … sort of, like an “interpretive” approach (often people like to say “scripting”)

Programmers can think in entirely different ways, and if other methods work then that is great (woh!) but if they do not work for any platform (sometimes you don’t intend to cater for every platform known to humans and mangoes … damn … was doing so well … persongoes), and these ideas may really irk many programmers, but you need to consider styles of work that suit your way of thinking best.

So, to do this task we wrote new PHP code and cloned the crontab code used in the tutorial below but change the minute of the hour the task is run, and what is run, in crontab on the Linux web server for the domain at rjmprogramming.com.au … happy studies!

Previous relevant Linux Crontab Curl Primer Tutorial is shown below.

Linux Crontab Curl Primer Tutorial

Linux (or Unix) is a command line environment that supports interactive and non-interactive task management. Often non-interactive tasks are known as batch processes, often intended not to rely on any user intervention, unless there is a problem.

Today, in this tutorial (which builds on Linux Background Primer Tutorial), we see the power of combining Curl under the auspices of crontab to run a webpage piece of functionality once a day at a specified time.

Some CMS systems like Drupal use crontab to perform actions on the database at regular intervals.

For Windows systems you might want to look at Windows Task Schedular for similar functionality.

Linux has several approaches to activating a task, some (not mentioning all the ways one process can fork or launch another process) of them being:

• interactively (from the command line)
• kick off a process from command line, and place in in the background, without supervision, necessarily, via the suffix &
• kick off a process without supervision, necessarily, via crontab
• kick off a process in the background (without supervision, necessarily) via nohup
• kick off a process from command line, and place it in the background, without supervision, necessarily (and optionally change its state via fg and bg)

The previous Linux Background Primer Tutorial is well worth reading below.

Linux Background Primer Tutorial

Linux (or Unix) is a command line environment that supports interactive and non-interactive task management. Often non-interactive tasks are known as batch processes, often intended not to rely on any user intervention, unless there is a problem.

In the pre-GUI days batch processes were very prominent, but they still are today, as far as actually getting things done. It is just that the more glamorous activities pass our notice more, but there are servers out there churning away with batch processes, dreaming of the day when they’ll work out how they can make that cup of coffee for you in the morning. And do we ever offer them even a cup’o’tea … it’s a disgrace, so it is.

Linux has several approaches to activating a task, some (not mentioning all the ways one process can fork or launch another process) of them being:

• interactively (from the command line)
• kick off a process from command line, and place in in the background, without supervision, necessarily, via the suffix &
• kick off a process without supervision, necessarily, via crontab
• kick off a process in the background (without supervision, necessarily) via nohup
• kick off a process from command line, and place it in the background, without supervision, necessarily (and optionally change its state via fg and bg)

The last four above are often best performed on a shell script, whether that be a Bash, Korn, Bourne or Cshell script. There are some good general tips for batch process scripts:

• never assume where you are, rather specify the directory to be in, via “cd”
• never assume that the environment variables will be exactly the same as for an interactive session
• be more thorough to log errors and log activity to files to see, later, what happened
• it is not good to proceed on errors (unless you have set up independent means of checking) because there will be no human to interactively decide that the problem should not stop the job proceeding
• pretty obviously, don’t expect anybody to answer an interactive entry requirement (but you can simulate lots of interactive input via redirected standard input eg. < [filename])

Take a look at some example Linux background processing at this tutorial. In it you will see some crontab work where the parameters are (as explained here … thanks):

MIN = Minute 0-60

HOUR = Hour [24-hour clock] 0-23

MDAY = Day of Month 1-31

MON = Month 1-12 OR jan,feb,mar,apr …

DOW = Day of Week 0-6 OR sun,mon,tue,wed,thu,fri,sat

COMMAND = Command to be run Any valid command-line

The */6 * * * * /Applications/MAMP/htdocs/pdftoimage.sh of our usage runs /Applications/MAMP/htdocs/pdftoimage.sh every six minutes.

If this was interesting you may be interested in this too.

If this was interesting you may be interested in this too.

If this was interesting you may be interested in this too.

If this was interesting you may be interested in this too.

If this was interesting you may be interested in this too.