Terminal Mark Primer Tutorial
XML data often comes in files with one record that is a very long string, so think long and hard about tools you rely on to find strings in such XML data. We’re biassed, but we like PHP on the command line or on the web for this, as we find its …
… to be the bee’s knees here.
So we want to tell you about a job with XML like this to identify when, in a group of files, we find amongst them, multiple instances of XML identifiers found in …
- section tags
- string occurring after ?id= … and …
- before “>
This is (almost) right down the alley of the work of Spreadsheet and XML Global Substitution Genericization Tutorial as shown below, and which we recommend you read, if this topic interests you.
So, how about we fill in the usual bits (of interest to us was …
$startfind='<section ';
$endfind='">';
$findthis='Detailed.jsp?id=';
) … in code of that tutorial above called tr_mapping.php where just adding the bold bits in code snippet below …
foreach (glob($filespec) as $filename) {
$cont=@file_get_contents($filename);
$sections=explode($startfind, $cont);
for ($i=1; $i<sizeof($sections); $i++) {
$thisbits=explode($endfind, $sections[$i]);
$thebit=" " . $thisbits[0];
if (strpos($thisbits[0], $findthis) !== false) {
echo explode($findthis, $thisbits[0])[1] . "\n";
}
if (strpos($thebit, $oldlabelfindstart) !== false && strpos($thebit, $newlabelfindstart) !== false && ($findthis == '' || strpos($thebit, $findthis) !== false) && ($findthistwo == '' || strpos($thebit, $findthistwo) !== false) && ($findthisthree == '' || strpos($thebit, $findthisthree) === false)) {
if (str_replace("]","\\]",explode($oldlabelfindend,explode($oldlabelfindstart, $thebit)[1])[0]) != "") {
$csvout.=$delim . "," . explode($oldlabelfindend,explode($oldlabelfindstart, $thebit)[1])[0] . ",," . explode($newlabelfindend,explode($newlabelfindstart, $thebit)[1])[0];
$kshout.=$delim . "cat " . $indexfilecalled . " | sed '/\\" . str_replace("]","\\]",explode($oldlabelfindend,explode($oldlabelfindstart, $thebit)[1])[0]) . "/s//\\" . str_replace("]","\\]",explode($newlabelfindend,explode($newlabelfindstart, $thebit)[1])[0]) . "/g' > " . str_replace($uext,$ualt,str_replace($lext,$lalt,$indexfilecalled)) . " ; rm -f " . $indexfilecalled . " ; cat " . str_replace($uext,$ualt,str_replace($lext,$lalt,$indexfilecalled)) . " > " . $indexfilecalled . " ; ";
$delim="\n";
}
}
}
}
… and save to some PHP we’ll call corp_file.php then this is the missing ingredient for our job solution below, featuring the Mac OS X desktop application called Terminal as the interface to (you may as well think of it as) a Linux command line environment where we like to work with Korn Shell scripting, and today’s star functionality …
Edit -> Select Between Marks
…
- get all the XML files over to a new directory over on the MacBook Pro on Mac OS X we’ll call /Applications/MAMP/htdocs/CorpFiles/ (which if you use the local MAMP Apache/PHP/MySql web server, would equate to http://localhost:8888/CorpFiles/) … we used a memory stick with Mac OS X Finder desktop application to do this … then …
- on MacBook Pro open the Terminal desktop application
- cd /Applications/MAMP/htdocs/CorpFiles/
- construct /Applications/MAMP/htdocs/CorpFiles/corp_file.php via tr_mapping.php as outlined above … then …
- back at Terminal type these commands
php corp_file.php > x.x # create list of XML identifiers as per the criteria we talked about above
cat x.x | sort > x.xx # alphabetically sorts list
uniq -d x.xx > x.xxx # only outputs duplicate findings ... what we are after for today's job
cat x.xxx | sed '/^/s//fgrep -c "/g' | sed '/$/s//" *.xml | grep -v ":0"/g' > x.ksh # outputs some useful Korn Shell script
# Here you might expect us to have put ksh -x x.ksh > report.txt or even ksh -x x.ksh 2> report.txt > report.txt or even ksh -x x.ksh | tee -a report.txt 2> report.txt but none of this does the job ... but ...
- in Terminal menu where you are now on command line use Edit -> Marks -> Mark
- at Terminal type this command
ksh -x x.ksh # wait until it finishes and follow this up with ...
- in Terminal menu use Edit -> Marks -> Mark
- in Terminal menu use Edit -> Select Between Marks
- in Terminal menu use Edit -> Copy
- back at your TextWrangler session File -> New -> Text Document
- Edit -> Paste
- an optional nicety is to highlight that great long list of files, Command-C Search -> Find -> Find: (Command-V) Replace: *.xml -> Replace All
- File -> Save As… report.txt
- Email report.txt as an attachment in an email with explanation to complete the job, as you can see with today’s tutorial picture
This is the command line meeting copy and paste GUI thoughts midway perhaps. The Mark concept has been around command line work for many years on many platforms, and kind of wished I’d thought more laterally about its use before now. Never think that whatever is presented as the steps in a job are the only ways to achieve things, but we hope these ideas are either directly or indirectly of use for you in an XML job you’re tackling.
Previous relevant Spreadsheet and XML Global Substitution Genericization Tutorial is shown below.
Spreadsheet and XML Global Substitution Genericization Tutorial
It’s one thing to write a useful one off web application with quite a few hard codings, but what about an attempt to genericize it, and by so doing, oftentimes you are improving its documentation aspects, so that, if the code is revisited years later …
- its generic qualities will be plain to all … and at the same time …
- it will be far easier to imagine as far as inputs are concerned …
- the user can (still) break the job up
- the user has less to worry about as far as a backup of data goes
- it does not feel like a one off any more
- it is less likely to be ill used for an inapplicable application
… so that, all in all, we feel much more confident such code can last the test of time and usefulness into the future than yesterday’s (albeit useful) one off feeling version of the code you can see at Spreadsheet and XML Global Substitution CSV Tutorial as shown below.
What’s the main driver of genericization, in our book (but not our pamphlettes) for small jobs?
- turn all hardcodings you can into parameterizable variables, and today that is via PHP $_GET[] variables off the web browser address bar
- allow the user to change these, as the hardcodings just become defaults, and are presented in a submittable HTML form whose action is to recall the same piece of PHP software (code)
… simple, huh?!
Again, you can see the various aspects of this, in play, with today’s tutorial picture, and though it is not much use to run the PHP code live, its style is far more generic now, so we want to share tr_mapping.php (changed this way) with you for your perusal … just in case (it is of use for you).
Previous relevant Spreadsheet and XML Global Substitution CSV Tutorial is shown below.
Spreadsheet and XML Global Substitution CSV Tutorial
Programmatically, we came in half way with the programming when we presented Spreadsheet and XML Global Substitution Primer Tutorial as shown below. The programming, then, had two inputs, namely …
- input spreadsheet’s CSV file manually created
- index XML file
… but that CSV file can be programmatically created rather than manually created. And while we’re at programmatically creating the CSV we could also programmatically create the Korn Shell (ksh) easier there too, with the same program, rather than using TextWrangler’s Grep (RegEx) talents … not that we’re ungrateful or anything … but it is good to mix things up to improve procedures sometimes.
And what programming language can we use, and what environment for that programming code? We think …
- coding wise, we’ll use PHP (starring PHP’s glob() method) … and the environment for that will be that …
- we’ll use a (local Apache/PHP/MySql web server) MAMP subfolder (ie. how desktop application “can meet” web application) off its Document Root (/Applications/MAMP/htdocs/) … /Applications/MAMP/htdocs/tr_mapping/ … to store the XML data files (no CSV needed as input this way, as it will be programmatically created in part 1 of 2 parts to the whole job) … which becomes accessible in two ways …
- http://localhost:8888/tr_mapping/tr_mapping.php#in_a_web_browser
- at Mac OS X Terminal desktop application command line via …
cd /Applications/MAMP/htdocs/tr_mapping
ksh -x tr_mapping.ksh
Again, email is the conduit for both sides of …
- input in
- output out
… to complete proceedings. You can see the various aspects of this, in play, with today’s tutorial picture, and though it is not much use to run the PHP code live, its style is leaning towards the generic side enough for us to want to share tr_mapping.php with you for your perusal … just in case (it is of use for you).
Previous relevant Spreadsheet and XML Global Substitution Primer Tutorial is shown below.
Spreadsheet and XML Global Substitution Primer Tutorial
Yesterday when we were discussing Worldbank API World Country Reporting Regex Tutorial we mentioned …
… and we use
it(ie. Regex) with serverside PHP today, under the auspices of the preg_match function, though we most often use RegEx thinking with the Javascript replace function, as the way to make substitutions for more than one occurrence, (the one occurrence design being) a default “curiosity” (but can be useful too) about Javascript’s version of substitution. You may know this RegEx usage of the Javascript replace function as “global substitution”.
… and that term “global substitution”. Many editing jobs, especially text file based ones, require or benefit from “global substitution” carefully applied, that is. It is common to see an editor who shies away from “global substitution” methods, and in many cases that is wise, but “global substitution” gets good results when you …
- substitute things you know exist in the precise form you intend to search for, and only there, where you want to replace … to
- replacements should not feed back into the substitution list … doh … or you will end up with a confused unintended result
In real life, it is often the case that the conditions above are easy to obey, because you are mapping an old numbering and/or naming system to a completely new and dissimilar numbering and/or naming system. That’s the case in a little job we drilled down into, to show you what we did, that involved RegEx thoughts, to solve a problem.
So, with our job we had …
- Aim: Change some XML in one file to have the text in one column of a Spreadsheet be mapped to the contents of another column of that same Spreadsheet
- Inputs: Excel Spreadsheet with those two columns as mentioned above and the one input XML file
- From the User: Asked for the user to send the Excel Spreadsheet … Saved As Comma Separated Values (CSV) in MS-DOS format and the one input XML file as two attached files in an Email
- Processing:
- Opened Email with Gmail web application in Safari web browser desktop application, on a MacBook Pro laptop
- Downloaded the two Attachments and copied over to where we like to work … the home of MAMP local Apache/PHP/MySql web server … on a Mac OS X system is /Applications/MAMP/htdocs (which we’ll access later with the Mac OS X Terminal desktop application later via “cd /Applications/MAMP/htdocs”)
- Opened our favourite Text Editor desktop application, called TextWrangler, whose “Find and Replace” “Grep” suboption will be a feature of today’s solution
- File -> Open the Spreadsheet CSV file
- Search -> Find… … Matching Mode: Grep … Find: ^ Replace: # … Replace All … remember our “RegEx” “cheat sheet” discussion (lots of which is relevant to TextWrangler Matching Mode: Grep as well) at that aforesaid mentioned tutorial …
- ^ can mean “start of”
- $ can mean “end of”
- . can sometimes mean “one existant character wildcard” … or sometimes it is % or ? for this in other “systems”
- * can often mean “zero or more of preceding character wildcard”
- [] and () bracketing rules are pretty crucial for the more esoteric usages … also study | usage
? … well, we want to start out mapping all lines to non-acting Korn Shell command lines
- Typed as the new top line #!/bin/ksh … just for completeness sake … is optional step
- Search -> Find… … No Matching Mode … Find: #,[ Replace: cat COMM.MIL~INDEX.xml | sed ‘/\[ … Replace All
- Search -> Find… … Matching Mode: Grep … Find: ]$ Replace: \\]/g’ > x.xxx ; cat x.xxx > COMM.MIL~INDEX.xml ; rm -f x.xxx … Replace All
- File -> Save As… fix_csv.ksh (to /Applications/MAMP/htdocs directory)
- Opened Terminal desktop application that has a default Bash environment (a lot like Linux, but is (giving you access to) a Mac OS X BSD operating system, really)
- Typed in: cd /Applications/MAMP/htdocs # to get to data
- Typed in: cp COMM.MIL~INDEX.xml COMM.MIL~INDEX_original.xml # to backup data ahead of processing, as well as to compare file sizes with later, as a sanity check
- Typed in: ksh -x fix_csv.ksh # access Korn Shell interpreter and run the TextWrangler created Korn Shell Script (and the -x switch tells the interpreter to be verbose with output reporting)
- Typed in: ls -l COMM.MIL~INDEX*.xml # first sanity check verified files different, and not disastrously so … good first sign
- Typed in: fgrep -c ‘[S1.12.4.20]’ COMM.MIL~INDEX*.xml ; fgrep -c ‘[CCR.28E.20]’ COMM.MIL~INDEX*.xml # second sanity check to prove old/new parts of first/last relevant Spreadsheet CSV file records were correctly mapped … and they were … so
- Opened Email with Gmail web application in Safari web browser desktop application (and used “Forward” option, attaching that new XML file), on a MacBook Pro laptop … so that …
- Output: One XML file with the global substitutions expressed in the Excel Spreadsheet performed, returned to User via Email “Forward” option, attaching that new XML file
We hope you can see the good use you can make with Email and a good Text Editor and Linux type shell scripting, influenced by RegEx pattern matching regarding …
Which leaves us with today’s PDF slideshow of snapshots of making this job work, here.
If this was interesting you may be interested in this too.
If this was interesting you may be interested in this too.
If this was interesting you may be interested in this too.
If this was interesting you may be interested in this too.
