Category Archives: Operating systems

Interprocess Communication – Mac, Windows, Linux

This is a work in progress.

IPC on Mac

Here are the options

  • Sockets
  • Mach ports
  • Pipes
  • FIFOs
  • Shared memory

There is also XPC, but this requires you to refactor your code into XPC services, so this isn’t really IPC as generally talked about.

<more here>

IPC on Windows

Here are the options

  • Data Copy (WM_COPYDATA)
  • File mapping
  • Named shared memory
  • Pipes
  • RPC
  • Windows Sockets

This ignores several older techniques such as Clipboard, COM, or DDE. Mailslots are technically IPC, but generally used for broadcast and best-effort instead of point-to-point guaranteed communication.

Data Copy

This only works between processes on the same machine.

http://msdn.microsoft.com/en-us/library/windows/desktop/ms648710(v=vs.85).aspx

File Mapping

This only works between processes on the same machine.

Uses CreateFileMapping and MapViewOfFile to map a file to memory, then read and write to memory to perform the IPC operations. This can be fairly efficient, although the processes will need to coordinate reads and writes. The downside is that this will result in file writes, reducing efficiency. Generally, you would avoid doing this in favor of the named shared memory variant.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366878(v=vs.85).aspx

Named Shared Memory

This only works between processes on the same machine.

This is a special variant of File Mapping that creates file mapping objects referencing the system paging file. The advantage here is that no actual file is needed. This is the closest equivalent on Windows to shared memory as found on Linux or Mac.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366551(v=vs.85).aspx

Pipes

Anonymous pipes can only be between processes running on the same machine, and typically between a parent process and a child process. However, it is possible to send the pipe handles to another process by a separate IPC mechanism (such as shared memory).

Named pipes can be used between arbitrary processes on the same or different machines.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa365780(v=vs.85).aspx

IPC on Linux

<coming>

Reference

https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemTechnology/SystemTechnology.html

http://www.puredarwin.org/curious/ipc

http://stackoverflow.com/questions/2846337/best-way-to-do-interprocess-communication-on-mac-os-x

http://afewguyscoding.com/2012/07/ipc-easy-introducing-xpc-nsxpcconnection/

https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingXPCServices.html#//apple_ref/doc/uid/10000172i-SW6

https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemTechnology/SystemTechnology.html

http://www.cocoawithlove.com/2009/02/interprocess-communication-snooping.html

http://my.safaribooksonline.com/book/operating-systems-and-server-administration/mac-os-x/0321278542/interprocess-communication/ch09lev1sec3

 

File system madness

It’s interesting trying to read disk catalog entries in some cases. For example, I copied some files from a Mac to a PC, indirectly through an SMB file share (Windows server). When I tried to access them with a Python program that iterated directories with os.walk, I got an error:

WindowsError: [Error 123] The filename, directory name, or volume label syntax 
is incorrect: 'D:/Xcode/usr/share/man/man3\\Git??I18N.3pm'

On the original Mac machine, I see this

drwxr-xr-x  27 bfitz  admin    918 Jun  9  2013 .
drwxr-xr-x   7 bfitz  admin    238 Jun  9  2013 ..
-rw-r--r--   1 bfitz  admin  25681 Feb  6  2013 Git.3pm
-rw-r--r--   1 bfitz  admin   4804 Feb  6  2013 Git::I18N.3pm
-rw-r--r--   1 bfitz  admin   5514 Feb  6  2013 Git::SVN::Editor.3pm
-rw-r--r--   1 bfitz  admin   6966 Feb  6  2013 Git::SVN::Fetcher.3pm
-rw-r--r--   1 bfitz  admin   5018 Feb  6  2013 Git::SVN::Memoize::YAML.3pm
-rw-r--r--   1 bfitz  admin   5622 Feb  6  2013 Git::SVN::Prompt.3pm
-rw-r--r--   1 bfitz  admin   4787 Feb  6  2013 Git::SVN::Ra.3pm
-rw-r--r--   1 bfitz  admin   4617 Feb  6  2013 Git::SVN::Utils.3pm
-rw-r--r--   1 bfitz  admin   6355 Feb  6  2013 SVN::Base.3pm
-rw-r--r--   1 bfitz  admin  63700 Jun  9  2013 SVN::Client.3pm
-rw-r--r--   1 bfitz  admin  22950 Feb  6  2013 SVN::Core.3pm
-rw-r--r--   1 bfitz  admin   7468 Feb  6  2013 SVN::Delta.3pm
-rw-r--r--   1 bfitz  admin  21923 Feb  6  2013 SVN::Fs.3pm
-rw-r--r--   1 bfitz  admin  30291 Feb  6  2013 SVN::Ra.3pm
-rw-r--r--   1 bfitz  admin  15618 Feb  6  2013 SVN::Repos.3pm
-rw-r--r--   1 bfitz  admin  18271 Feb  6  2013 SVN::Wc.3pm
-rw-r--r--   1 bfitz  admin   5867 Feb  6  2013 arch.3
-rw-r--r--   1 bfitz  admin     19 Feb  6  2013 end.3
-rw-r--r--   1 bfitz  admin   2154 Feb  6  2013 get_end.3
-rw-r--r--   1 bfitz  admin   2522 Feb  6  2013 getsectbyname.3
-rw-r--r--   1 bfitz  admin     25 Feb  6  2013 getsectbynamefromheader.3
-rw-r--r--   1 bfitz  admin     25 Feb  6  2013 getsectdata.3
-rw-r--r--   1 bfitz  admin     25 Feb  6  2013 getsectdatafromheader.3
-rw-r--r--   1 bfitz  admin    940 Feb  6  2013 getsegbyname.3
-rw-r--r--   1 bfitz  admin  10232 Feb  6  2013 kld.3

On the Windows box, I see this

D:\projects\git\brian\disk>dir d:\Xcode\usr\share\man\man3
 Volume in drive D is SSD
 Volume Serial Number is B48A-050D

 Directory of d:\Xcode\usr\share\man\man3

09/29/2013  06:58 PM    <DIR>          .
09/29/2013  06:58 PM    <DIR>          ..
02/06/2013  02:11 PM             5,867 arch.3
02/06/2013  02:11 PM                19 end.3
02/06/2013  02:11 PM             2,522 getsectbyname.3
02/06/2013  02:11 PM                25 getsectbynamefromheader.3
02/06/2013  02:11 PM                25 getsectdata.3
02/06/2013  02:11 PM                25 getsectdatafromheader.3
02/06/2013  02:11 PM               940 getsegbyname.3
02/06/2013  02:11 PM             2,154 get_end.3
02/06/2013  02:42 PM            25,681 Git.3pm
02/06/2013  02:42 PM             4,804 Git??I18N.3pm
02/06/2013  02:42 PM             5,514 Git??SVN??Editor.3pm
02/06/2013  02:42 PM             6,966 Git??SVN??Fetcher.3pm
02/06/2013  02:42 PM             5,018 Git??SVN??Memoize??YAML.3pm
02/06/2013  02:42 PM             5,622 Git??SVN??Prompt.3pm
02/06/2013  02:42 PM             4,787 Git??SVN??Ra.3pm
02/06/2013  02:42 PM             4,617 Git??SVN??Utils.3pm
02/06/2013  02:11 PM            10,232 kld.3
02/06/2013  02:43 PM             6,355 SVN??Base.3pm
06/09/2013  10:09 AM            63,700 SVN??Client.3pm
02/06/2013  02:43 PM            22,950 SVN??Core.3pm
02/06/2013  02:43 PM             7,468 SVN??Delta.3pm
02/06/2013  02:43 PM            21,923 SVN??Fs.3pm
02/06/2013  02:43 PM            30,291 SVN??Ra.3pm
02/06/2013  02:43 PM            15,618 SVN??Repos.3pm
02/06/2013  02:43 PM            18,271 SVN??Wc.3pm
              25 File(s)        271,394 bytes
               2 Dir(s)  459,978,698,752 bytes free

D:\projects\git\brian\disk>

The file names are not actually “Git??I18N.3pm” on the Windows box, because that path isn’t actually a legal path. The actual path has some characters that aren’t legal on Windows, and the Mac SMB compatibility layer preserves them so that the files can roundtrip from Mac to Mac. But not Mac to anything else.

In this particular case, the Mac copied file names that have ‘:’ characters in them. In Unix, any character is legal in a filename except ‘/’. Windows is a lot more restrictive, and ‘:’ characters are not legal. Now, interestingly enough, these can be copied on the Windows box by Explorer. I also suspect that some other character than ‘:’ is being stored.

This is a variant of the character encoding issue; if you inadvertently change character encodings when iterating directories, you’ll be unable to manipulate the files, because you aren’t using the actual name.

In this particular case, it’s not just that the Python code is scrambling the name. It’s very unlikely that the name is stored with ?? characters. Either the name has :: in it (which isn’t legal at the high-level, but perhaps legal at the lowest NT level), or there is some other characters being substituted. I suspect the latter, and I think that Samba is doing this. I copied this with a Mac OS 10.8 box which is still using Samba and not Apple’s replacement.

I’ll have to write some Windows code to iterate the directory to see what the names come back as, and then figure out what I want to do with my Python code.

Also interesting in that explorer.exe can copy this directory. I suspect it is either falling back to short names, or doing other heroic actions.

Where is my 10Gb Ethernet!

It’s been a very long time since 1Gb Ethernet was cheap, and file sizes have kept growing. I really want 10Gb Ethernet for both home and work.

Alas, this is just an ad for Intel, but at least it TALKS about 10Gb as something people would actually have:

http://techpageone.dell.com/technology/modernize-network-10-gigabit-ethernet

Stuff is still expensive

https://www.google.com/search?q=10gb+ethernet+nic&espvd=210&es_sm=93&source=univ&tbm=shop&tbo=u&sa=X&ei=YDO_UqCFDcPnoATYw4GIBw&ved=0CCsQsxg&biw=1228&bih=987

Adapters are $200 to $500, and switches are $1000 to $5000.

There is a motherboard with 10GB on board – http://www.supermicro.com/products/motherboard/QPI/5500/X8DTU_.cfm?TYP=SAS&LAN=10 – about $900.

What you want to buy is 10GBASE-T as it uses Cat6 or Cat6a twisted pair cables.

Netgear has an 8-port 10GBASE-T switche – http://www.netgear.com/landing/10gigabit/ – $850 on Amazon – http://www.amazon.com/Netgear-8-Port-10-Gigabit-ProSafe-Switch/dp/B00B46AEE6.

Intel 10GBASE-T NICs are about $350 for single-port – http://www.amazon.com/Intel-X540T1-10GBase-T-Ethernet-Coverged/dp/B008NJPL7C/ref=pd_sim_pc_3 – and $500 for dual port – http://www.amazon.com/Intel-Ethernet-Converged-Network-X540T2/dp/B0077CS9UM/ref=pd_bxgy_pc_text_y

So a small network with 4 computers on 10Gb would be around $2500 when you add cables, or about $600 a port. For 8 computers, it still works out to about $450 a port.

Unfortunate that bonding isn’t an answer.

 

Virtualization

Some words to search with:

  • Xen
  • VMWare
  • VirtualBox
  • Parallels Desktop, Parallels Workstation
  • Microsoft Virtual PC (previously from Connectix)
  • SoftPC
  • KVM (RedHat)
  • hypervisor
  • paravirtualization
  • Hyper-V
  • VHD

There is hardware assistance for virtualization

  • Intel VT-x
  • AMD-V

There’s also emulation, which often goes hand-in-hand with virtualization

  • Bochs
  • QEMU
  • DOSBox

Some vendor-specific pages about virtualization

Rapid setup and deployment

Linux and Mac

https://www.docker.io/

http://docs.docker.io/en/latest/installation/vagrant/

https://flynn.io/

http://blog.scoutapp.com/articles/2013/08/28/docker-git-for-deployment

Windows

http://boxstarter.org/

http://chocolatey.org/

http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx

http://www.hanselman.com/blog/LessVirtualMoreMachineWindows7AndTheMagicOfBootToVHD.aspx

http://www.hanselman.com/blog/StepByStepTurningAWindows7DVDOrISOIntoABootableVHDVirtualMachine.aspx

http://blogs.technet.com/b/haroldwong/archive/2012/08/18/how-to-create-windows-8-vhd-for-boot-to-vhd-using-simple-easy-to-follow-steps.aspx

http://social.technet.microsoft.com/wiki/contents/articles/516.how-to-boot-from-a-vhd.aspx

http://technet.microsoft.com/en-us/library/hh825691.aspx

http://technet.microsoft.com/en-us/library/hh825709.aspx