Using grep, curl, and tail to scrape data from a Web page

Posted on: Sunday, Feb 04, 2018

Linux has cool utilities!

Another post on this blog provides a Bash script that automates the installation of the most recent version of Firefox Developer Edition (FFDE). The original version of that script required the manual input of FFDE's latest version number. Looking up that number was a hassle to the say the least--and added lots of friction to a process that should simple and fast.

Rather than require you to look up the most recent version number and then provide that value as argument to the Bash script, the script now uses three of the really handy utilities that lurk within Linux. curl, grep, and tail work together to fetch the most recent version number from the FFDE downloads page. This post goes into the detail of how that script uses these Linux utilities to get the latest FFDE version number. With these in place, running that script is quite simple now.

You can read more about curl, grep, and tail here:

  • curl - transfer the contents of a URL
  • grep - find lines matching a pattern
  • tail - output the last part of files

Scraping data from a Web page

Mozilla provides a "releases" download page that shows the versions of FFDE available. The most recent version number is the last number in the list. Visit the releases page to see it. There isn't much to it, it's mostly just a list of version numbers.

Follow each of these steps by clicking the clipboard icon to copy a line to your clipboard then pasting it in a terminal session to run it.

In an open a terminal session pull down the FFDE release page's HTML with curl:

curl https://download-installer.cdn.mozilla.net/pub/devedition/releases/

Use curl to see the release page's HTML.

This script needs that HTML in a text file, so it uses curl's -o flag to specify an output file:

curl -o releases.txt https://download-installer.cdn.mozilla.net/pub/devedition/releases/

Use curl to save the release page's HTML in the releases.txt file.

With the releases.txt file available, we'll run grep against that file to extract the version numbers from it. To do so, grep uses a simple regular expression that matches a FFDE version number (59.0b6. for example), where [0-9] specifies a single numeric digit, \. looks for a single period (unescaped, the . means any character to regex), and [a-z] specifies a letter from between a and z.

grep -o '[0-9][0-9]\.[0-9][a-z][0-9]' releases.txt    

Use grep to see the version numbers in a terminal.

This list is all of the version numbers (with each one repeated twice), but we only need the last number (the most recent version) in the list. To get the last number, grep pipes its output into the tail command.

grep -o '[0-9][0-9]\.[0-9][a-z][0-9]' releases.txt | tail -1

Pipe grep's output into the tail command to get the last number (the last line of the file).

The last bit of this step is get the most recent version number into a Bash variable for use in a Bash script. This is done with Bash's substitution operator, $(...).

VERSION=$(grep -o '[0-9][0-9]\.[0-9][a-z][0-9]' releases.txt | tail -1) && echo $VERSION

Capture the most recent version number in a Bash variable and show it.

While that was a long explanation it distills down to three lines (including a line to delete the releases.txt file).

curl -o releases.txt https://download-installer.cdn.mozilla.net/pub/devedition/releases/
VERSION=$(grep -o '[0-9][0-9]\.[0-9][a-z][0-9]' releases.txt | tail -1)
rm releases.txt

The general technique here of pulling down from a page with curl and then parsing it with grep, tail (and whatever other Linux utilities you need to use) is very handy. Please let me know in the comments what tasks you're using Linux utilities for.


Comments

  • Hi, here on the forum guys advised a cool Dating site, be sure to register - you will not REGRET it [url=https://bit.ly/3mmOmi0]https://bit.ly/3mmOmi0[/url]


    Submitted by Haroldfep
    10 months ago
  • Hi, here on the forum guys advised a cool Dating site, be sure to register - you will not REGRET it [url=https://bit.ly/2RA7I5l]https://bit.ly/2RA7I5l[/url]


    Submitted by Douglaslit
    10 months ago
  • excellent points altogether, you just gained a new reader. What would you suggest in regards to your post that you made a few days ago? Any positive? <A HREF="https://liveone9.com/baccaratsite/" TARGET='_blank'>모바일바카라</A>


    Submitted by 모바일바카라
    10 months ago
  • Respect to website author , some good selective information . <A HREF="https://www.nolza2000.com" TARGET='_blank'>바카라사이트</A>


    Submitted by 바카라사이트
    10 months ago
  • Very good blog post.Really looking forward to read more. Much obliged.<A HREF="https://vfv79.com/theking/" TARGET='_blank'>더킹카지노</A>


    Submitted by 더킹카지노
    10 months ago
  • https://bit.ly/34GSPFv - Sex without obligation - https://bit.ly/34RxuK7


    Submitted by Gabrieldek
    8 months ago
  • п»їhttps://bit.ly/34GSPFv - Sex without obligation in your city https://bit.ly/3pn28mx - Sex without obligation @@@==


    Submitted by Gabrieldek
    8 months ago
  • https://t.me/Dating_Flirting - Flirting in your city https://hot-desire.com/T1kMpvjB - Sex without obligation in your city @@@==


    Submitted by Gabrieldek
    8 months ago
  • https://hot-desire.com/T1kMpvjB - Sex without obligation in your city https://bit.ly/2TBGPyA - Meet, be inspired, communicate and continue flirting! Follow the link @@@==


    Submitted by Gabrieldek
    8 months ago
  • Online casino, first deposit bonus of $100 [url=https://bit.ly/2LG4A7Z]https://bit.ly/2LG4A7Z[/url]


    Submitted by Hymanhiply
    6 months ago
  • [url=https://forum.mesign.com/index.php?action=profile;u=354994]https://forum.mesign.com/index.php?action=profile;u=354994[/url]


    Submitted by Randallric
    6 months ago
  • Hi, here on the forum guys advised a cool Dating site, be sure to register - you will not REGRET it [url=https://bit.ly/2MpL94b]https://bit.ly/2MpL94b[/url]


    Submitted by Ronaldfroth
    5 months ago
  • Great site. Chic design. I found a lot of interesting things here. Check out my theme site - and rate it http://scunt.xyz AVN NEWS VIDEO FOR ADULTS ^^XxX=+


    Submitted by Gabrieldek
    5 months ago
  • Great site. Chic design. I found a lot of interesting things here. Check out my theme site - and rate it http://didlo.xyz Wonderful Video ^^XxX=+


    Submitted by Gabrieldek
    5 months ago
  • https://bbw-xxx.info - biseksual as$#$%f!&


    Submitted by FrancisSpups
    4 months ago
  • https://milf-xxx.xyz - sex xxx as$#$%f!&


    Submitted by FrancisSpups
    4 months ago
  • <a href=https://vsviagrav.com/>bob viagra[/url]


    Submitted by engepophy
    2 months ago
  • Hi, look what a cool site I found, there you can win from $ 50 to $ 3000 , I managed to win $ 540, good luck to you [url=https://bit.ly/2SmXf0A]https://bit.ly/2SmXf0A[/url]


    Submitted by CharlesAlope
    2 months ago
  • Hi, look what a cool site I found, there you can win from $ 50 to $ 3000 , I managed to win $ 540, good luck to you [url=https://bit.ly/34jKEhC]https://bit.ly/34jKEhC[/url]


    Submitted by CharlesEvome
    2 months ago



Add your comment
You email is never shared with anyone else.

© Copyright 2017 by Roger Pence. All rights reserved.