Fetching multiple files from an internet site as a batch job

Sometimes one encounters a website that displays a book or manuscript page-by-page as individual jpeg files.  But what you need for your research is to have a single PDF of the item, so that you can move about it easily, and consult it offline.

There are several quick ways of getting these images as a batch job: here's one.   

• First you have to identify the URL of one of the images.  I use Firefox, so I 
  • first bring up a page that displays the first folio of the MS. 
  • Then I press ctrl+I to get the "page info" (or Firefox menu Tools/Page Info).  
  • Then I select "Media" on the top line of the Info window.  
  • Then scroll down to the graphics file of the whole page, right click and ctrl+c to copy the URL.
    
    You now have a URL that looks like this:
    
    http://awebsite.net/uploads/manuscripts/miscellaneous/sometext/001.jpg

There may be a more direct way of getting this URL, but this is good enough for me.

The next bit is the nice bit.  Drop to the command line and use the utility "curl".  Here's the syntax ($ is my command prompt):

$ curl -O http://awebsite.net/uploads/manuscripts/miscellaneous/sometext/[001-268].jpg

• Hit "enter" and several hundred jpeg files will be transferred to your directory.  It takes a couple of minutes, depending on your bandwidth.
  The bit in square brackets, "[001-268]" is curl's syntax for "please fetch 001.jpg, 002.jpg, ... 267.jpg, and 268.jpg".  Curl is one of the few tools with this simple ability to fetch lots of different files with a single simple command.

To convert them to a single PDF, I use ImageMagick:

$ convert *.jpg Hayanaratna.pdf

and wait for ten seconds.

(I was taught about curl by Patrick McAllister - thanks Patrick!)

A quite different approach is to use wget to fetch a whole website in a single gulp.   That's what I use for GRETIL, for example, so that I have the whole archive on my hard drive.

From Gnome to Cinnamon

Gnome 2 and 3

Ubuntu with Gnome 2

Ubuntu with Unity

After moving to Ubuntu GNU/Linux for all my work, in 2009, I used the default interface, Gnome 2, for a while.  Later, with version 3, Gnome moved to a completely new concept, but Ubuntu forked the development and moved to Unity, so I did that too.  

Gnome 3

Gnome 2 and Unity both had their virtues and their flaws.  The six-monthly upgrade cycle ("cadence") was never as smooth as it should be, so there have often been niggles that lasted a few weeks or months.  I didn't like Unity's two different search boxes.

Ubuntu with Gnome 3

I started using the Gnome Shell, ver. 3, on Linux (Ubuntu) after seeing my friends and colleagues using it at the TeX conference in Trivandrum in 2011.  I really liked Gnome 3, but with the update from 3.6 to 3.8 and 3.10, they did some major, major things wrong, and I've finally dumped it, in favour of Cinnamon.

The biggest boo-boo in the development of Gnome from version 3.8, was fooling with the default file-manager, Nautilus.  Many people have complained about the stripping out of function, like split-screen, and that was bad enough.  So was the nonsense about shifting the menus to the panel bar (or not!).  But what hasn't got mentioned so much (at all?) is that the new Nautilus changed all the keyboard shortcuts and rearranged the shortcuts relating to the menu system.  So Alt-F didn't bring up a "File" menu any more, for  example.  Right mouse-click+R didn't begin renaming a file.  If one uses computers all day, then one's fingers get trained, and no interface designer should mess with that stuff without expecting backlash.  With Nautilus 3.8, it was like being a beginning typist again, looking at my fingers, chicken-pecking for keys.

I liked the general design model of Gnome 3, with the corner switch to the meta level for choosing programs, desktops, and so on.  Searching for lesser-used programs with a few keystrokes rather than poking hopelessly through nested menus.  Much better.  A genuine and valuable contribution to the vision of how a computer should work.

Thanks to Webupd8, I was able to work around the Nautilus problem by uninstalling it and using Nemo instead.

But things just kept going wrong.  The shell crashed too often.  On two of my machines it stopped coming up at login, and had to be started manually.  Only after a couple of weeks did I track this down to a bad file in ~/.config/gnome-session (and I'm still not 100% sure).  Frequent crashes of the gnome-control-panel and other utilities.  More and more extraordinary tweaking to make it comfortable and useable.  Finally, I've had enough.

 

Cinnamon

Ubuntu with Cinnamon

I'm in my first few days of using Cinnamon, and so far things are okay.  I'm running Cinnamon on top of Ubuntu.  It's like stepping back in time, a bit, all those menus.  But one doesn't have to use them, and with a bit of tweaking one can set things up so that actual shell behaviour is very similar to Gnome 3.6.  Nemo is there - what a relief.  Alacarte actually works, but I've dumped it in favour of Menulibre in any case.  Configuration and tweaking is much nicer.  Many useful add-ons, and although I liked the http://extensions.gnome.org system, Cinnamon handles the add-on business in a much more integrated way.  Ibus+m17n work as expected again.  In general, it's an update from Gnome 2 in the direction of Gnome 3 but not the whole way.  And it seems more stable, which is critical to getting work done.

Human interfaces

“Because I believe interfaces to government can be simple, beautiful, and easy to use.”

-- Jen Pahlka, via Tim O'Reilly's blog.

Up The Junction

Under Windows, Dropbox wants everything it backs up to live in its own directory on your hard drive. Not soft/sym-links, but actual files.

However, if you're running Windows and have an NTFS drive, then you can in fact make "hard" soft links to your Dropbox directory without having to actually copy everything physically into the that directory. The tool is Junction, which handles hard links under NTFS. NTFS supports this natively, but MS in their wisdom never distributed a public tool to handle this feature.