http://www.earthpoint.us/ is a nice site that aggregates Google Earth lookups. Although, if you are a programmer of moderate skill, you’re better off doing it directly (which is free for low volume of requests), probably with https://developers.google.com/places/
So, I converted the GPS stamp in one of my photos into a map location. Now, assuming that you can trust Google Maps (I think that’s fair), and that with the photo in question I know where I was to within about 10 feet, the GPS stamp in the picture is about 4000 feet away.
From Google maps, my latitude and longitude at the time of the photo was 33.6588 -117.7668. But the GPS stamp in the photo says
GPS Latitude Ref : North GPS Latitude : 33 deg 38' 59.40" GPS Longitude Ref : West GPS Longitude : 117 deg 45' 60.00" GPS Time Stamp : 13:32:53.72 GPS Img Direction Ref : True North GPS Img Direction : 171.6590909
which, if I do the math, is 33.649833 -117.76666.
Is Google Maps that far off? Or did I do the math wrong? I think I did the math wrong. Also, other people think that GPS locators in cellphones can be drastically off some of the time.
http://mashable.com/2011/03/09/smartphone-gps-accuracy/
is an article about Shopkick’s experience in correlating GPS location from the cellphone with a known location.
This is strange, considering that GPS itself is pretty darn accurate.
http://www.gps.gov/support/faq/ and http://www.gps.gov/systems/gps/performance/accuracy/
Given the time I took the photo, this would have been with my iPhone 1, purchased the first day iPhones were available. Maybe the GPS has gotten better in newer models? I’ll need to test.
Ah, maybe it’s an indoors issue? Apparently, GPS signals are partially blocked by building walls, so the accuracy indoors is degraded. That sucks, considering that a lot of my photos are taken indoors. http://stackoverflow.com/questions/4424387/how-accurate-is-the-gps-on-the-iphone-4
Also see this: http://academics.skidmore.edu/blogs/onlocation/2012/03/smartphone_accuracy/. For outdoors, an iPhone is ok to several meters, outdoors.
And apparently the iPhone 4 has a vastly improved GPS receiver. Time to upgrade.
FUSE isn’t really a stackable filesystem, but it can be used as one.
http://dazuko.dnsalias.org/wiki/index.php/Main_Page
http://www.redirfs.org/tiki-index.php
Also, maybe fanotify could be used for this, which is more limited but faster (?) than inotify. The underlying system of all of these is fsnotify.
Of course, these are all for Linux.
Watchdog, in Python, is a cross-platform system used to monitor filesystem events. https://github.com/gorakhargosh/watchdog. It uses inotify on Linux, FSEvents and kqueues on Mac, ReadDirectoryChanges on Windows).
Article on IBM DeveloperWorks about Linux VFS: http://www.ibm.com/developerworks/linux/library/l-virtual-filesystem-switch/
Most JPEG files have EXIF metadata in them.
For Perl, there is the handy-dandy Image::ExifTool library that encapsulates ExifTool.
With this, one can whip up quick programs to do all kinds of things. For example, I have lots of photos scattered across hard disks and iPhoto libraries. Lots of these are duplicates, and some have broken metadata. Reorganizing by hand is painful, but reorganizing with a program is only painful for the 15 minutes it takes to write one.
Some cameras and most smartphones even have GPS nowadays, so if you want to find all the pictures you took in England, you could do a quick search across the metadata.
Also, since actually scanning hard disks is slow, another thing would be to extract all the metadata and put it in one place. For this, someday I hope we’ll have FUSE-level capabilities for Windows and Mac OS X, so I could write a FUSE filesystem to automatically maintain the metadata database. If you’re unfamiliar with FUSE, here’s a teaser article: http://pramode.net/articles/lfy/fuse/pramode.html.
Here’s an example of what you have access to. This is the metadata that ExifTool pulled out of one of my photos.
---- ExifTool ---- ExifTool Version Number : 9.01 ---- File ---- File Name : IMG_0890.JPG Directory : Z:\Media\Images\bfitz\Pictures\iPhoto Library/Modified/2008/Jul 20, 2008 File Size : 742 kB File Modification Date/Time : 2008:07:30 12:01:32-07:00 File Permissions : rw-rw-rw- File Type : JPEG MIME Type : image/jpeg Exif Byte Order : Big-endian (Motorola, MM) Comment : AppleMark\x0A Image Width : 1200 Image Height : 1600 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3 Y Cb Cr Sub Sampling : YCbCr4:2:2 (2 1) ---- EXIF ---- Make : Apple Camera Model Name : iPhone Orientation : Horizontal (normal) Orientation : Horizontal (normal) X Resolution : 72 Y Resolution : 72 Resolution Unit : inches Software : QuickTime 7.5 Modify Date : 2008:07:30 12:01:32 Host Computer : Mac OS X 10.4.9 Y Cb Cr Positioning : Centered F Number : 2.8 Exif Version : 0220 Date/Time Original : 2008:07:20 14:51:42 Create Date : 2008:07:20 14:51:42 Color Space : sRGB Compression : JPEG (old-style) X Resolution : 72 Y Resolution : 72 Resolution Unit : inches Thumbnail Offset : 470 Thumbnail Length : 5639 Y Cb Cr Positioning : Centered ---- ICC_Profile ---- Profile CMM Type : appl Profile Version : 2.2.0 Profile Class : Input Device Profile Color Space Data : RGB Profile Connection Space : XYZ Profile Date Time : 2003:07:01 00:00:00 Profile File Signature : acsp Primary Platform : Apple Computer Inc. CMM Flags : Not Embedded, Independent Device Manufacturer : appl Device Model : Device Attributes : Reflective, Glossy, Positive, Color Rendering Intent : Perceptual Connection Space Illuminant : 0.9642 1 0.82491 Profile Creator : appl Profile ID : 0 Red Matrix Column : 0.45427 0.24263 0.01482 Green Matrix Column : 0.35332 0.67441 0.09042 Blue Matrix Column : 0.15662 0.08336 0.71953 Media White Point : 0.95047 1 1.0891 Chromatic Adaptation : 1.04788 0.02292 -0.0502 0.02957 0.99049 -0.01706 -0.00923 0.01508 0.75165 Red Tone Reproduction Curve : curv\x00\x00\x00\x00\x00\x00\x00\x01\x023 Green Tone Reproduction Curve : curv\x00\x00\x00\x00\x00\x00\x00\x01\x023 Blue Tone Reproduction Curve : curv\x00\x00\x00\x00\x00\x00\x00\x01\x023 Profile Description : Camera RGB Profile Profile Copyright : Copyright 2003 Apple Computer Inc., all rights reserved. Profile Description ML : Camera RGB Profile Profile Description ML (es-ES) : Perfil RGB para Cámara Profile Description ML (da-DK) : RGB-beskrivelse til Kamera Profile Description ML (de-DE) : RGB-Profil für Kameras Profile Description ML (fi-FI) : Kameran RGB-profiili Profile Description ML (fr-FU) : Profil RVB de l’appareil-photo Profile Description ML (it-IT) : Profilo RGB Fotocamera Profile Description ML (nl-NL) : RGB-profiel Camera Profile Description ML (no-NO) : RGB-kameraprofil Profile Description ML (pt-BR) : Perfil RGB de Câmera Profile Description ML (sv-SE) : RGB-profil för Kamera Profile Description ML (ja-JP) : カメラ RGB プãƒãƒ•ã‚¡ã‚¤ãƒ« Profile Description ML (ko-KR) : ì¹´ë©”ë¼ RGB í”„ë¡œíŒŒì¼ Profile Description ML (zh-TW) : 數ä½ç›¸æ©Ÿ RGB 色彩æè¿° Profile Description ML (zh-CN) : 相机 RGB æ述文件 ---- Composite ---- Aperture : 2.8 Image Size : 1200x1600 Thumbnail Image : <thumbnail image data>
It’s pretty daunting. But you can see I took this with an iPhone, the orientation is horizontal, it’s a 1200×1600 image at 72 DPI. Also note that I modified this photo, and you can see when it was modified, and that it was likely modified on a Mac (the “Host Computer” key).
What are some things you would want to fix?
Sometimes a photo orientation is recorded incorrectly. Rather than mess with rotating it, you could just fix the metadata. Obviously, it’s an AI task for a program to look at a photo and say “hey, the orientation is wrong”, but once you’ve determined this, my personal preference would be to fix the original data. Another thing you can do is fix dates – sometimes my photos claim to have been taken in 2000, or 2015, or some other crazy date. So, I would prefer to fix the metadata in the photo.
Also note that JPEGs can have embedded thumbnails. You might want to strip those out. Easy enough to do. Or you can insert new thumbnails. You can even add your own metadata. Or you could fix the GPS location if you have better information than what was stamped in the photo (my iPhone 3G has been spectacularly off for indoor pictures, sometimes a mile off).
Here is the start of a program I wrote to find and manipulate metadata in photos. I guess I should start putting this on Github rather that pasted into blog postings.
#!/usr/bin/perl
use 5.014; # implies use strict
use warnings;
use feature ':5.14';
use Digest::MD5;
use Data::Dumper;
use File::Find;
use Getopt::Long qw();
use Image::ExifTool;
use Time::HiRes qw();
Fix->new()->run();
# =======================================================================================
# =======================================================================================
package Fix;
use parent -norequire, 'Object';
sub run
{
my ($self) = @_;
$self->{'option'} = Option->new()->read_options();
$self->{'diag'} = Diag->new()->init($self->{'option'}->{'verbose'});
$self->do_commands();
}
sub do_commands
{
my ($self) = @_;
Scan->new()->scan($self->{'option'}, $self->{'diag'});
}
# =======================================================================================
# =======================================================================================
package Scan;
use parent -norequire, 'Object';
sub scan
{
my $self = shift;
$self->{'option'} = shift;
$self->{'diag'} = shift;
$self->{'start_time'} = Time::HiRes::time();
$self->{'file_count'} = 0;
$self->{'bytes_processed'} = 0;
$self->{'exif_tool'} = new Image::ExifTool;
$self->{'exif_tool'}->Options(Unknown => 1);
my $process = sub { $self->process($_); };
no warnings 'File::Find'; # don't want noise about dirs we can't iterate
&File::Find::find({wanted => $process}, $self->{'option'}->{'base'});
}
sub process
{
my ($self, $file) = @_;
@{$self->{'stat'}} = stat($file);
return unless -f _;
return unless $file =~ /\.(JPG|JPEG|jpg|jpeg)$/;
return if $file =~ /\.\_/; # skip ._ AppleDouble files
$self->{'file'} = $file;
$self->{'path'} = $File::Find::name;
$self->{'file_size'} = -s _;
return if $self->{'path'} =~ /\.AppleDouble/; # skip fake JPG inside AppleDouble folders
$self->{'file_count'} += 1;
$self->{'bytes_processed'} += $self->{'file_size'};
my $info = $self->{'exif_tool'}->ImageInfo($self->{'path'});
print sprintf("%10s %s\n", main::commify($self->{'file_size'}), $self->{'path'});
my $group = "";
foreach my $tag ($self->{'exif_tool'}->GetFoundTags('Group0'))
{
if ($group ne $self->{'exif_tool'}->GetGroup($tag))
{
$group = $self->{'exif_tool'}->GetGroup($tag);
print "---- $group ----\n";
}
my $value = $info->{$tag};
if (ref($value) eq 'SCALAR')
{
$value = $$value;
}
$value =~ s/([\x00-\x1F])/sprintf("\\x%02X", unpack("C", $1))/eg;
print sprintf("%-32s : %s\n", $self->{'exif_tool'}->GetDescription($tag), $value);
}
print "\n";
$self->{'diag'}->progress(
sub {
my $dur = Time::HiRes::time() - $self->{'start_time'};
$dur = 1 if $dur {'file_count'});
my $procMB = main::bytes_str($self->{'bytes_processed'});
my $mbsec = main::bytes_str($self->{'bytes_processed'} / $dur);
sprintf("%s files, %s (%s/sec) processed",
$proc, $procMB, $mbsec);
},
$self);
}
# =======================================================================================
# Diag - diagnostic output
#
# char, line: print if verbose >= 3 (-v -v)
# status: print if verbose >= 2 (-v)
# progress: print if verbose > 0 (no option)
# =======================================================================================
package Diag;
use parent -norequire, 'Object';
sub init
{
my $self = shift;
$self->{'diag'} = $_[0] // 0;
$self->{'need_newline'} = 0;
$self->{'last_progress'} = 0;
return $self;
}
sub progress
{
my ($self, $callback, $param) = @_;
return unless $self->{'diag'} > 0;
return if time() == $self->{'last_progress'};
$self->{'last_progress'} = time();
print STDERR "\n" and $self->{'need_newline'} = 0 if $self->{'need_newline'};
print STDERR sprintf("%-78s\r", $callback->($param));
}
sub progress_clear
{
my ($self) = @_;
return unless $self->{'diag'} > 0;
print STDERR sprintf("%-78s\r", "");
}
# =======================================================================================
# Option - parse command-line options
# =======================================================================================
package Option;
use parent -norequire, 'Object';
sub read_options
{
my ($self) = @_;
my @opts = ('verbose|v+', 'quiet|q', 'base=s');
my $status = Getopt::Long::GetOptions($self, @opts);
$self->{'verbose'} = 0 if $self->{'quiet'};
# get from ARGV once I add separate commands
return $self;
}
# =======================================================================================
# Object - base class
# =======================================================================================
package Object;
sub new
{
my $self = shift;
my ($class) = ref($self) || $self; # allow both virtual (member) and static (class)
$self = {};
bless $self, $class;
return $self;
}
# =======================================================================================
# Not in a class yet
# =======================================================================================
package main;
sub bytes_str
{
my $val = shift;
if ($val < 10_000) { return sprintf("%.2f KB", $val / 1_000); }
elsif ($val < 100_000) { return sprintf("%.1f KB", $val / 1_000); }
elsif ($val < 1_000_000) { return sprintf("%d KB", int($val / 1_000)); }
elsif ($val < 10_000_000) { return sprintf("%.2f MB", $val / 1_000_000); }
elsif ($val < 100_000_000) { return sprintf("%.1f MB", $val / 1_000_000); }
elsif ($val < 1_000_000_000) { return sprintf("%d MB", int($val / 1_000_000)); }
elsif ($val < 10_000_000_000) { return sprintf("%.3f GB", $val / 1_000_000_000); }
elsif ($val < 100_000_000_000) { return sprintf("%.2f GB", $val / 1_000_000_000); }
elsif ($val < 1_000_000_000_000) { return sprintf("%.1f GB", $val / 1_000_000_000); }
else { return sprintf("%.3f TB", $val / 1_000_000_000_000); }
}
sub commify
{
local $_ = shift;
1 while s/^([-+]?\d+)(\d{3})/$1,$2/;
return $_;
}
# =======================================================================================
=head1 NAME
fix-jpeg: Clean up JPEG files and EXIF information
=head1 SYNOPSIS
fix-jpeg [options] [args]
fix-jpeg --base BASEDIR -v
=head1 DESCRIPTION
Fix file dates to match EXIF dates, or EXIF dates to match file dates. Rename files.
Find duplicates. Fix iPhoto libraries. Other magic stuff.
=cut
NTFS Streams were introduced in Windows NT 3.1 to enable Services for Macintosh (SFM) to store Macintosh resource forks and finder information. At a technical level, the implementation was cool, because it was a generic solution that could be expanded. Microsoft did very little with it over the years, until very recently. It’s been a source of annoyance and even used as a vector for viruses. You can write an executable to an alternative data stream of any file, and then even execute it, and the stock Windows file system tools don’t really acknowledge the existence of alternative data streams: explorer.exe and others just show the size of the default stream.
Sadly, Microsoft has abandoned SFM as of Windows Server 2008, but third parties such as ExtremeZ-IP still offer support, by using alternate data streams. Mac literature for this refers to it just as “named streams”.
Mac OS X v10.5 and up writes Mac metadata and resource forks to named streams now, instead of using AppleDouble ._<filename> files. You can enable or disable this as a default, or per mount point.
There is one bug I’ve seen so far – if you’re copying a file with no data fork, this confuses Windows, because it doesn’t try to create the default stream before attaching the alternative data stream. The one case where I’ve seen this happen is in aliases copied to netatalk servers. The netatalk server stores its data using .AppleDouble folders, because it expects to run on file systems that lack support for multiple streams.
Resource forks
Streams
NTFS Alternate data streams
AppleSingle and AppleDouble are methods to save the extra Mac metadata on systems that otherwise could not (SMB, NFS, NTFS, ext2, etc). AppleSingle and AppleDouble are the same file format; the only difference is that the data fork is stored as a separate file in AppleDouble and included in the AppleSingle file. As such, AppleSingle is more intended for archive or transmission, whereas AppleDouble allows for non-Mac applications to easily access the data fork, ignoring the metadata. Since most applications store their file’s data in the data fork, especially for anything considered to be cross-platform, this means AppleDouble is a fairly useful system.
There are two conventions for AppleDouble files’ naming. The data fork is generally stored with the name of the file itself. The metadata is stored either in a folder named .AppleDouble parallel with the file (and the metadata itself named the same as the file inside that folder) or the metadata is stored in a file name prefixed with “._” next to the file itself. In the .AppleDouble folder case, the .AppleDouble folders also generally have the .DS_Store file (contains directory metadata unique to a Mac) and a .Parent file, which is also a AppleSingle-format file containing metadata about the parent directory itself (name, dates, icon, finder info). Interestingly, the .DS_Store file is also a AppleSingle-format file, and I’m not sure why there are both. Time to look in the source?
Here’s the appledouble.h header file from opensource.apple.com (pretty sure I can reproduce this in full since it’s under Apple’s open-source license):
/* Information pulled from:
* "AppleSingle/AppleDouble Formats for Foreign Files Developer's Note"
* (c) Apple Computer 1990
* File assembled by Rob Braun (bbraun@synack.net)
*/
#ifndef __APPLEDOUBLE__
#define __APPLEDOUBLE__
#include <sys/types.h>
#include
/* Structure of an AppleSingle file:
* ----------------------
* | AppleSingleHeader |
* |--------------------|
* | ASH.entries # of |
* | AppleSingleEntry |
* | Descriptors |
* | 1 |
* | . |
* | . |
* | n |
* |--------------------|
* | Datablock 1 |
* |--------------------|
* | Datablock 2 |
* |--------------------|
* | Datablock n |
* ----------------------
*/
struct AppleSingleHeader {
uint32_t magic; /* Magic Number (0x00051600 for AS) */
uint32_t version; /* Version #. 0x00020000 */
char filler[16]; /* All zeros */
uint16_t entries; /* Number of entries in the file */
};
#define XAR_ASH_SIZE 26 /* sizeof(struct AppleSingleHeader) will be wrong
* due to padding. */
#define APPLESINGLE_MAGIC 0x00051600
#define APPLEDOUBLE_MAGIC 0x00051607
#define APPLESINGLE_VERSION 0x00020000
#define APPLEDOUBLE_VERSION 0x00020000
struct AppleSingleEntry {
uint32_t entry_id; /* What the entry is. See defines below */
uint32_t offset; /* offset of data, offset beginning of file */
uint32_t length; /* length of data. can be 0 */
};
/* Valid entry_id values */
/* Entries 1, 3, and 8 are typically created for all files.
* Macintosh Icon entries are rare, since those are typically in the resource
* fork.
*/
#define AS_ID_DATA 1 /* Data fork */
#define AS_ID_RESOURCE 2 /* Resource fork */
#define AS_ID_NAME 3 /* Name of the file */
#define AS_ID_COMMENT 4 /* Standard Macintosh comment */
#define AS_ID_BWICON 5 /* Standard Macintosh B&W icon */
#define AS_ID_COLORICON 6 /* Standard Macintosh Color icon */
/* There is no 7 */
#define AS_ID_DATES 8 /* File creation date, modification date, etc. */
#define AS_ID_FINDER 9 /* Finder Information */
#define AS_ID_MAC 10 /* Macintosh File information, attributes, etc. */
#define AS_ID_PRODOS 11 /* ProDOS file information */
#define AS_ID_MSDOS 12 /* MS-DOS file information */
#define AS_ID_SHORTNAME 13 /* AFP short name */
#define AS_ID_AFPINFO 14 /* AFP file information */
#define AS_ID_AFPDIR 15 /* AFP directory id */
/* 1-0x7FFFFFFF are reserved by Apple */
/* File Dates are stored as the # of seconds before or after
* 12am Jan 1, 2000 GMT. The default value is 0x80000000.
*/
struct MacTimes {
uint32_t creation;
uint32_t modification;
uint32_t backup;
uint32_t access;
};
/* Finder Information is two 16 byte quantities.
* Newly created files have all 0's in both entries.
*/
/* Macintosh File Info entry (10) a 32 bit bitmask. */
/* Entries can be placed in any order, although Apple recommends:
* Place the data block (1) last.
* Finder Info, File Dates Info, and Macintosh File Info first.
* Allocate resource for entries in 4K blocks.
*/
/* AppleDouble files are simply AppleSingle files without the data fork.
* The magic number is different as a read optimization.
*/
#endif /* __APPLEDOUBLE__ */
The old tech note describing the AppleSingle and AppleDouble file formats is AppleSingle/AppleDouble Formats: Developer’s Note, but the URL has moved several times, so I’ll try to not rely on it. And this should all maybe go in Wikipedia?
I’m going to write a quick AppleSingle disassembler. Here it is
#!/usr/bin/perl
# =======================================================================================
# dump-applesingle.pl
#
# disassemble an AppleSingle file to text.
#
# Maybe I should call this dump-appledouble since that's the more likely file to
# encounter?
# =======================================================================================
use 5.014; # also does strict
use warnings;
use feature ':5.14'; # I think this is redundant?
use Getopt::Long qw();
Dump->new->run();
# =======================================================================================
# =======================================================================================
package Dump;
use parent -norequire, 'Object';
# ---------------------------------------------------------------------------------------
sub run
{
my ($self) = @_;
$self->{'option'} = Option->new()->read_options();
$self->{'diag'} = Diag->new()->init($self->{'option'}->{'verbose'});
$self->dump();
}
sub dump
{
my ($self) = @_;
foreach my $file (@{$self->{'option'}->{'files'}})
{
$self->{'file'} = $file;
$self->{'diag'}->status("Parsing $file\n");
local $/ = undef;
open FILE, "{'blob'} = ;
close FILE;
$self->dump_blob();
}
}
sub dump_blob
{
my ($self) = @_;
my $header = substr($self->{'blob'}, 0, 26);
my @header = unpack("L> L> x[16] S>", $header);
my $magic = $header[0] == 0x51600 ? "APPLESINGLE_MAGIC"
: $header[0] == 0x51607 ? "APPLEDOUBLE_MAGIC"
: "** ERROR **";
my $version = $header[1] == 0x20000 ? "APPLESINGLE_VERSION" : "** ERROR **";
my $blob_length = length($self->{'blob'});
print "\n$self->{'file'}\nlength=$blob_length\n";
print " AppleSingleHeader\n";
print " {\n";
print sprintf("%04X: uint32_t magic = 0x%08X $magic\n", 0, $header[0]);
print sprintf("%04X: uint32_t version = 0x%08X $version\n", 4, $header[1]);
print sprintf("%04X: char filler[16] = {0}\n", 8);
print sprintf("%04X: uint16_t entries = %d\n", 24, $header[2]);
print " }\n";
my $count = $header[2];
my @descriptors = unpack("(L> L> L>)[$count]", substr($self->{'blob'}, 26, $count * 12));
# some of these are from the netatalk source, their adouble.h file
my @annotated_offsets;
print "\n";
foreach my $i (0 .. $count-1)
{
print " entry[$i] =\n";
print " {\n";
my $addr = 26 + 12 * $i;
my $entry_id = $descriptors[3*$i + 0];
my $entry_id_str = ($entry_id == 1) ? " (AS_ID_DATA)"
: ($entry_id == 2) ? " (AS_ID_RESOURCE)"
: ($entry_id == 3) ? " (AS_ID_NAME)"
: ($entry_id == 4) ? " (AS_ID_COMMENT)"
: ($entry_id == 5) ? " (AS_ID_BWICON)"
: ($entry_id == 6) ? " (AS_ID_COLORICON)"
: ($entry_id == 8) ? " (AS_ID_DATES)"
: ($entry_id == 9) ? " (AS_ID_FINDER)"
: ($entry_id == 10) ? " (AS_ID_MAC)"
: ($entry_id == 11) ? " (AS_ID_PRODOS)"
: ($entry_id == 12) ? " (AS_ID_MSDOS)"
: ($entry_id == 13) ? " (AS_ID_SHORTNAME)"
: ($entry_id == 14) ? " (AS_ID_AFPINFO)"
: ($entry_id == 15) ? " (AS_ID_AFPDIR)"
: ($entry_id == 0x80444556) ? " (AD_DEV - netatalk)"
: ($entry_id == 0x80494E4F) ? " (AD_INO - netatalk)"
: ($entry_id == 0x8053594E) ? " (AD_SYN - netatalk)"
: ($entry_id == 0x8053567E) ? " (AD_ID - netatalk)" : "";
my $offset = $descriptors[3*$i + 1];
my $length = $descriptors[3*$i + 2];
my $entry_val_and_label = ($entry_id < 16) ? sprintf("%d$entry_id_str", $entry_id) : sprintf("0x%08X$entry_id_str", $entry_id); push @annotated_offsets, [$offset, $length, $entry_val_and_label]; print sprintf("%04X: uint32_t entry_id = $entry_val_and_label\n", $addr + 0); print sprintf("%04X: uint32_t offset = 0x%08X\n", $addr + 4, $offset); print sprintf("%04X: uint32_t length = %d\n", $addr + 8, $length); print " }\n"; } my @sorted_offsets = sort { $a->[0] $b->[0]} @annotated_offsets;
my $pos = 26 + $count * 12;
my $bytes = 0;
my $need_newline = 0;
while ($pos < $blob_length) { # if we have run into one of the objects, print its name if ($pos == $sorted_offsets[0]->[0])
{
print "\n" if $need_newline;
$need_newline = 0;
$bytes = 0;
print sprintf("\n $sorted_offsets[0]->[2] length = %d\n", $sorted_offsets[0]->[1]);
shift @sorted_offsets;
}
if ($bytes == 16) { print "\n"; $need_newline = 0; $bytes = 0; }
print sprintf("%04X:", $pos) unless $need_newline;
print sprintf(" %02X", unpack("C", substr($self->{'blob'}, $pos, 1)));
$need_newline = 1;
$bytes += 1;
$pos += 1;
}
}
# =======================================================================================
# Diag - diagnostic output
#
# char, line: print if verbose >= 3 (-v -v)
# status: print if verbose >= 2 (-v)
# progress: print if verbose > 0 (no option)
# =======================================================================================
package Diag;
use parent -norequire, 'Object';
sub init
{
my $self = shift;
$self->{'diag'} = $_[0] // 0;
$self->{'need_newline'} = 0;
$self->{'last_progress'} = 0;
return $self;
}
sub char
{
my ($self, $char) = @_;
return unless $self->{'diag'} > 2;
print STDERR $char;
}
sub line
{
my ($self, $line) = @_;
return unless $self->{'diag'} > 2;
print STDERR "\n" and $self->{'need_newline'} = 0 if $self->{'need_newline'};
print STDERR $line;
$self->{'need_newline'} = 1 unless substr($line, -1) eq "\n";
}
sub status
{
my ($self, $line) = @_;
return unless $self->{'diag'} > 1;
print STDERR "\n" and $self->{'need_newline'} = 0 if $self->{'need_newline'};
print STDERR $line;
$self->{'need_newline'} = 1 unless substr($line, -1) eq "\n";
}
# =======================================================================================
# Option - parse command-line options
# =======================================================================================
package Option;
use parent -norequire, 'Object';
sub read_options
{
my ($self) = @_;
my @opts = ('quiet|q', 'verbose|v+');
$self->{'verbose'} = 1;
my $status = Getopt::Long::GetOptions($self, @opts);
@{$self->{'files'}} = @ARGV; # anything left is a file operand
$self->{'verbose'} = 0 if $self->{'quiet'};
if ($self->{'verbose'} > 2)
{
print STDERR "options:\n";
map { print STDERR " $_ = $self->{$_}\n"; } sort keys %$self;
}
return $self;
}
# =======================================================================================
# Object - base class
# =======================================================================================
package Object;
sub new
{
my $self = shift;
my ($class) = ref($self) || $self; # allow both virtual (member) and static (class)
$self = {};
bless $self, $class;
return $self;
}
# =======================================================================================
=head1 NAME
dump-applesingle: disassemble AppleSingle/AppleDouble files
=head1 SYNOPSIS
dump-applesingle [options] [args]
dump-applesingle -v ._appledoublefile
=head1 DESCRIPTION
dump-applesingle disassembles AppleSingle and AppleDouble files into a textual
representation. Currently that's all it does, but it will probably evolve into
something to do bulk operations on them (find, change, delete, create etc)
=cut
I guess I need to (1) get some WordPress plugins to format code better and (2) change my theme so that people can view code better (narrow margins are good for text but not quiet as good for code).
More links
The MyBook Live series is based around Linux, so you can fix some of WD’s design decisions and improve performance. You can do just about anything, but there are some simple performance improvements you can do.
Assuming you are blocking port 22 from outside your network, enable SSH by logging in to the web dashboard, then switch to the page UI/SSH, and enable from there. If your NAS had the address 192.168.1.101, you would first go to
http://192.168.1.101
log in, then go to
http://192.168.1.101/UI/ssh
to enable SSH. At that point, you can ssh to your box; ssh is built into Mac and Linux boxes, but you’ll need to use Putty or some other client on Windows.
I got this from What are the steps to enable SSH from the WD Community forums.
WD has a monitor script as a daemon. That in itself is fine, but it periodically does an ls of the entire drive to build up some simple statistics. Once your drive has a large number of files on it, this will cripple performance. Since I don’t ever look at their stats, I prefer to disable this part of monitorio.sh. I want to repeat this: if you do this, then the web front end will no longer show accurate disk space usage (it will show whatever you had up to the point where you nerf the file_tally function).
I suggest the following: ssh to your NAS, edit /usr/local/sbin/monitorio.sh, rename file_tally to file_tally_old, and insert an empty file_tally function. Your file will look like this:
file_tally() {}
file_tally_old() {
if [ ! -p $TALLY_PIPE]; then
You’ll want to restart the monitorio daemon (or reboot the NAS, which is more dramatic).
/etc/init.d/monitorio restart
You can also use ps and kill to stop any existing ls process if you’re impatient.
I didn’t figure this out, I got it from Solving the MyBook Live insane load.
Some people think that performance is better if you switch to the CFQ scheduler. This will definitely depend on how you use the NAS. WD has defaulted to the Anticipatory scheduler. You can check to see what yours is set at.
cd /sys/block/sda/queue cat scheduler
and if you see
noop [anticipatory] deadline cfq
then your NAS is currently using the Anticipatory scheduler. You can switch by writing CFQ to the scheduler file like this
echo cfq >scheduler
And, if you have multiple drives in your WD Live box (for example, a Live Duo), you’ll need to do this for sdb as well.
I got this from Performance problems? Read this first.
You can read more about the various scheduling algorithms in this Red Hat page: Choosing an I/O scheduler for Red Hat Enterprise Linux 4 and the 2.6 Kernel. At some point, this will be out of date, but it was valid as of late 2012.
Good presentation by Robert C. Martin (Uncle Bob) at Rub Midwest 2011: Keynote: Architecture the Lost Years
defaults write com.apple.finder AppleShowAllFiles TRUE killall Finder
But this only controls showing “dot” files (like .git). Finder Info might also exist and have the hidden bit set (FinderInfo came from the classic Mac OS system). Or, an ACL might exist that doesn’t allow browsing for the current user. Files with a ‘@’ appended to the file mode means “this file has extended attributes”, and if there’s a ‘+’ appended, the file has an ACL. You can show the size and kind of the extended attributes with a ‘@’ parameter to ls, and you can show specific FinderInfo attributes in a friendlier fashion with a ‘O’ parameter.
bfitzair:~ ls -l@Oed /Users/bfitz
...
drwx------@ 41 bfitz staff hidden 1394 Sep 23 21:03 Library
com.apple.FinderInfo 32
0: group:everyone deny delete
...
You can manipulate some flags with chflags
chflags nohidden /path/to/file
or you can remove the FinderInfo from extended attributes with
xattr -d com.apple.FinderInfo /path/to/file
You can manipulate ACLs with chmod. It’s moderately complicated to edit them (see the man page for chmod), but you can nuke one by using the -N
sudo chmod -N /path/to/file
These are files (or files in folders) that preserve Mac resource forks and extended attributes that cannot be otherwise preserved on the file system. For example, if you copy Mac files to a NFS or SMB file server volume, the Mac will by default create these for files with resource forks or extended attributes.
If you don’t ever want this, you can disable this
defaults write com.apple.desktopservices DSDontWriteNetworkStores true
This might be the drastic fix you need, but you’ll be unable to save classic Mac files (which generally have resource forks) or you’ll lose file attributes that you might actually want.
http://en.wikipedia.org/wiki/AppleSingle_and_AppleDouble_formats
AppleSingle combined everything into one file, in a way similar to the older MacBinary. This has an advantage (single file) but a big disadvantage in that the data fork (usually the one most applications care about) is now shrouded inside the AppleSingle file.
AppleDouble separates the data fork and resource fork + finderinfo into two files. In some instances, the second file is stored as ._{filename}. In other cases, it’s stored in .AppleDouble/{filename}.
In times past, I actually wrote code different platforms to work on their file formats (e.g. converting Apple IIgs resource forks to and from Mac resource forks, or to even do the same between Windows and Mac). It’s a lot easier when you have AppleDouble-style layout.
Windows has a slightly more strict view of what characters can be in a file name. Unix is pretty permissive, and only bans \0 (nul) and ‘/’ (and even the directory separator can be stored in the low-level file system, it’s just a pain to manipulate it if you do that). However, Windows disallows the following characters from being in a leaf name (file or directory)
\ / : * ? " < > |
(the explorer shell also disallows the typing of any “control” character, e.g a character <= \x1F, but these can be used in filenames). The Mac is a lot more permissive; for example, ‘|’ is a legal character in leaf names.
It gets slightly weird due to a compatibility decision in the transition from Mac OS 9 (“Classic Mac”) to Mac OS X. Before Mac OS X, Apple used ‘:’ as the path component separator character. However, Unix uses ‘/’. Apple decided that the file system would continue to use ‘/’, but the Finder would display these as ‘:’. And conversely, any files with ‘/’ characters in them would be stored in the underlying filesystem as ‘:’ characters.
Now, both characters are not legal for Windows. But SMB will accept the full path and store it, which the Mac side can use. Windows, however, doesn’t like seeing a file name with ‘:’ in it, so you get the synthesized short name when you view the file on a Windows machine. This is better than not being able to store files (from the Mac) or work on them (from Windows), but it’s awkward.
The only solution, if you want to work on files conveniently from both operating systems, is to rename files and directories to the legal subset of characters. In most cases, this means using the Windows set of legal characters as the set of allowed characters for file names. Or, you could write your own explorer replacement?
There are further restrictions in Windows using the Win32 API – files can’t be named CON, PRN, AUX, CLOCK$, NUL, COM[0-9], LPT[0-9].
Of course, there are a few files that you should not rename. For example, a file named “Icon\r” (yes, there is a carriage-return character in the filename) is used by the Mac Finder to display a custom icon for a file. For me, at least, the short name for “Icon\r” is “I7CIPB~N”, so they are easy to recognize (the short name algorithm is deterministic, I hope).
And finally, copying these short-name-aliased files with a Windows machine tends to mutate filenames, because many Windows clients don’t use the underlying NT APIs, and so end up garbling things (e.g. copying an Icon\r file to an SMB share with Mac using SMB and then duplicating it to a different SMB server using the Windows explorer changes it to a non-hidden file named “Icon”.
And more finally, we won’t even talk about real file names that have characters outside the limited ASCII/ISO-646 range. These tend to be garbled very easily, for reasons that may or many not be obvious to you (read up on encoding).
Assuming you mean “Mac Finder lock bit” and not Unix permissions (which are set with chmod), you use chflags. For example, to unlock recursively (and see what you unlocked), you would do
chflags -v -R nouchg <folder>
and all files rooted at <folder> would be examined and unlocked if needed. The -v flag will print a line for each file/folder that gets unlocked.
This is actually a FreeBSD command and changes “file flags”, which on the Mac has been interpreted to be finder flags. In fact, the man page on Mac OS 10.7.4 and the man page in FreeBSD look very similar (perhaps even identical). A slightly more Mac-specific page can be found at http://ss64.com/osx/chflags.html.