Rafael Baptista

Thursday, February 22, 2007 :: How to Play Sounds on Os X

Apple has this way of wanting to do everything differently. Sometimes its better... sometimes not so much. In a recent project for Os X we wanted the program to periodically play a sound from a .wav file that we specify. If you are trying to do this under Os X, you probably have already realized its annoyingly complex. If you don't care about exposition - the answer is to use quicktime. A complete example is given at the bottom of this post.

Windows

Playing a sound in Windows is pretty straightforward:

#include <windows.h>
PlaySound( "MySoundFile.wav", NULL, SND_SYNC | SND_NODEFAULT ); 

Cocoa

To play a sound on Os X is not so simple. There are three main ways for code to play sounds on Os X. One is to write your program using Cocoa.

- (IBAction) playSound:(id)sender
{
   if (loaded && ![sound isPlaying]) 
   {
      [sound play];
      [infoTextField setStringValue:@"Playback in progress"];
   }
}

The problem with this method is that you have to use Cocoa, a non-portable little Apple programming ghetto. Working within one of these closed vendor specific programming environments is generally a very bad idea. Think about how many there have been in recent years - COM, activeX, .net. Blech. Once you commit to one of these technologies you are trapped - like your email on AOL.

Core Audio

The other way is to use Apple's CoreAudio api. This is a very nice, powerful and extensible sound programming API in C++. It is predicated on the concept of "audio units" that can read audio files, and then play and transform audio streams. After going through some trouble to set up the CoreAudio infrastructure you can quickly create powerful audio programs. The problem here is that if all you want to do is play a wav file once in awhile you still have to end up writing and maintaining a complex body of code. If you want to go this route, the XCode dev kit includes a 'simple example' of playing back a sound file using CoreAudio- which is still 5 pages long and then requires you to complile in a giant library of helper functions. If you need fine grained control of audio, Core Audio is the way to go. If not - then it is entirely too much trouble.

QuickTime

The last way is to use Apple's legacy API's from Os 9, and use QuickTime to play audio files. This method can get you a sound play function that is only a page long. Writing it is a little tricky because the legacy API functions don't use normal posix paths - like all modern operating systems. They use a crusty old Os9 FSSpec. The FSSpec is initialized with a string in 'Pascal Format', a 256 byte buffer where the first byte is the length of the string. And the path itself must be expressed in HFS format - the path format used in Os 9 and before. If you haven't done this before is can be an unfun bunch of hours reading though old mac manuals - like I did. Lets save you some time!

You translate regular paths into Os 9 paths using CFURL. You have to pass in the output buffer as well. It only has to be 256 bytes long, because that's the longest path that QuickTime can deal with anyway.

#include <CoreFoundation/CFURL.h>

void macifyPosixPath( const char* posixPath, char* macPath )
{
   // CFURL objects can be used to translate paths into different formats.
   CFURLRef url = CFURLCreateWithBytes( kCFAllocatorDefault, 
                                        (u8*) posixPath, 
                                        strlen( posixPath ), 
                                        GetApplicationTextEncoding(), 
                                        NULL );
                                        
   // ask for the path in goofy old Os 9 style, and extract the data
   CFStringRef string = CFURLCopyFileSystemPath( url, kCFURLHFSPathStyle );
   CFStringGetCString( string, macPath, 255, GetApplicationTextEncoding() );
   
   // release the CF objects. You don't need these any more.
   CFRelease( string );
   CFRelease( url );
}

To play a sound, first translate the path into HFS style, then pack it into a Pascal String, and then create your FSSpec. Now you are ready to use the FSSpec to ask Quicktime to play you a 'movie', except that your movie just happens to have no actual video, just sound.

#include <QuickTime/QuickTime.h> 

int playSound( const char* inputFileName ) 
{
   OSErr  err;

   // translate the path into Os9 or HFS path style
   char macPath[256];
   macifyPosixPath( inputFileName, macPath );
   
   // turn it into a 'Pascal String'
   Str255 pascalPath;
   CopyCStringToPascal( macPath, pascalPath );

   // create your FSSpec
   FSSpec inputFile;
   err = FSMakeFSSpec( 0, 0, pascalPath, &inputFile );
   if ( err )
   {
      return -1;  
   }

   // Initialize quicktime. You probably can do this just once in your program.
   EnterMovies();
   
   // use QuickTime routines to play the sound
   short  myRefNum;
   Movie  myMovie;
   err = OpenMovieFile( &inputFile, &myRefNum, fsRdPerm );
   if ( err )
   {
      return -1;  
   }

   NewMovieFromFile( &myMovie, myRefNum, NULL, NULL, 0, NULL );
   if ( myRefNum != 0 )
     CloseMovieFile( myRefNum );
   StartMovie( myMovie );
   while ( !IsMovieDone( myMovie ))
      MoviesTask( myMovie, 0 );
   DisposeMovie( myMovie );
   return 0;
}

SDL

Another thing that you might also consider is using SDL. This is a cross platform media library. Its designed to give you a portable API for accessing audio, video, and input. If you need a cross platform method for all of these things it worth considering SDL. If all you need is to trigger a sound once in awhile - just use QuickTime.

Monday, January 29, 2007 :: Finding your Computer's Mac Address

All ethernet cards have a hardware address, a six byte sequence called the mac address. Your computer will have one mac address per network interface. Just about all modern workstations will have at least one. The mac address is a good way of uniquely identifying a computer. Be aware that it is possible on most network hardware to change the mac address, though this isn't something most users do. So while its a good way of identifying a computer, its not completely reliable.

Getting the mac address on various operating systems is not difficult.

On Windows you do it like this:

#include <windows.h>
#include <intrin.h>
#include <iphlpapi.h>

void printMacAddress()
{
   IP_ADAPTER_INFO AdapterInfo[32];
   DWORD dwBufLen = sizeof( AdapterInfo );

   DWORD dwStatus = GetAdaptersInfo( AdapterInfo, &dwBufLen );
   if ( dwStatus != ERROR_SUCCESS )
      return; // no adapters.

   PIP_ADAPTER_INFO pAdapterInfo = AdapterInfo;
   while ( pAdapterInfo )
   {
      u8* addr = pAdapterInfo->Address;
      
      printf( "%d-%d-%d-%d-%d-%d\n", 
              addr[0], addr[1], addr[2], addr[3], addr[4], addr[5] ); 
              
      pAdapterInfo = pAdapterInfo->Next;
   }
}

On Linux you do it like this:

// some of these headers may be redundant
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <netinet/in.h>
#include <netinet/in_systm.h>
#include <netinet/ip.h>
#include <netinet/ip_icmp.h>

#include <sys/types.h>
#include <sys/ioctl.h>
#include <linux/if.h>
#include <linux/sockios.h>

void printMacAddress()
{
   int sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP );
   if ( sock < 0 ) return;
   
   // enumerate all IP addresses of the system
   struct ifconf conf;
   char ifconfbuf[ 128 * sizeof(struct ifreq)  ];
   memset( ifconfbuf, 0, sizeof( ifconfbuf ));
   conf.ifc_buf = ifconfbuf;
   conf.ifc_len = sizeof( ifconfbuf );
   if ( ioctl( sock, SIOCGIFCONF, &conf ))
   {
      assert(0);
      return;
   }
   
   // get MAC address
   struct ifreq* ifr;
   for ( ifr = conf.ifc_req; (s8*)ifr %lt; (s8*)conf.ifc_req + conf.ifc_len; ifr++ ) 
   {
      if ( ifr->ifr_addr.sa_data == (ifr+1)->ifr_addr.sa_data ) 
         continue;  // duplicate, skip it
     
      if ( ioctl( sock, SIOCGIFFLAGS, ifr )) 
         continue;  // failed to get flags, skip it
      
      if ( ioctl( sock, SIOCGIFHWADDR, ifr ) == 0 ) 
      {
         u8* addr = &(ifr->ifr_addr.sa_data);
         printf( "%d-%d-%d-%d-%d-%d\n", 
                 addr[0], addr[1], addr[2], addr[3], addr[4], addr[5] ); 
      }
   }
   
   close( sock );
}

I found several code examples of how to get the mac address on BSD based systems, like Os X, here. Their versions require you to name the interface you are interested in. To actually enumerate all the ethernet interfaces yourself, do it like this:

#include <sys/types.h>
#include <sys/socket.h>
#include <net/if_dl.h>
#include <ifaddrs.h>
#include <net/if_types.h>

void printMacAddress()
{
   struct ifaddrs* ifaphead;
   if ( getifaddrs( &ifaphead ) != 0 )
      return;

   // iterate over the net interfaces
   bool foundMac1 = false;
   struct ifaddrs* ifap;
   for ( ifap = ifaphead; ifap; ifap = ifap->ifa_next )
   {
      struct sockaddr_dl* sdl = (struct sockaddr_dl*)ifap->ifa_addr;
      if ( sdl && ( sdl->sdl_family == AF_LINK ) && ( sdl->sdl_type == IFT_ETHER ))
      {
         u8* addr == (u8*)(LLADDR(sdl));
         printf( "%d-%d-%d-%d-%d-%d\n", 
                 addr[0], addr[1], addr[2], addr[3], addr[4], addr[5] ); 
      }
   }
  
   freeifaddrs( ifaphead );
}

Tuesday, January 23, 2007 :: Finding your Process' Executable Name

Running processes are usually started by the execution of a program file. Finding the name of the program is easy. Most operating systems pass the the name of the calling program as the first element of the argument array. In C and C++ this is the first element of the argv array:

int main( int argc, char** argv, char** env )
{
   printf( "%s\n", argv[0] );
}

This will print only the name of the program file itself, but not its complete path. Getting the complete path is different on various operating systems.

On Windows there are two ways. This is the simpler and preferred method:

const char* getExecutableName()
{
   static char buf[MAX_PATH] = { '\0' };
   if ( buf[0] == '\0')
     if ( !GetModuleFileName( NULL, buf, MAX_PATH ))
     {
          // Error!
     }
   return buf;
}

#include <Tlhelp32.h>
#include <psapi.h>

const char* getExecutableName()
{
   HANDLE hSnapshot = ::CreateToolhelp32Snapshot( TH32CS_SNAPMODULE, GetCurrentProcessId() );

   MODULEENTRY32 me32        = {0}; 
   me32.dwSize = sizeof(MODULEENTRY32); 

   Module32First( hSnapshot, &me32 );
   CloseHandle(hSnapshot);
   return me32.szExePath;
}

On Linux, and many Unixes, the best way is to use the /proc file system, or procfs. This is a simulated file system where the files contain information about the state of the operating system itself, including all the running processes. Usually there is a symlink from /proc/self/exe to the appropriate executable:

const char* getExecutableName()
{
   static char buf[1024] = { '\0' };
   if ( !buf[0] )
   {
      s32 len;
      if (( len = readlink( "/proc/self/exe", buf, sizeof( buf ) - 1 )) == -1 )
      {
         // Error!
      }
      buf[len] = '\0';
   }
   return buf;
}

Some versions of /proc don't have /proc/self/exe and you have to get the process id and then try something like:

// Instead of passing in /proc/self/exe you may need to do:
sprintf( buf, "/proc/$d/exe", getpid() );

// or...
sprintf( buf, "/proc/$d/file", getpid() );
sprintf( buf, "/proc/$d/path/a.out", getpid() );

On Mac Os X ( Darwin ), and many BSD Unixes, and many Mach Unixes there is no /proc. On those you can usually use _NSGetExecutablePath like this:

#include <sys/param.h>
#include <mach-o/dyld.h>

const char* getExecutableName()
{
   static char* str = NULL;
   if ( !str )
   {
      u32 size;
      _NSGetExecutablePath( NULL, &size );
      str = new char[size+1];
      if ( _NSGetExecutablePath( str, &size ))
      {
         // Error!
      }
   }
   return str;
}

I hope this little article saved you some time.

Friday, November 10, 2006 :: Woot! We won!

The nation is saved. Future blog posts will be about programming again. I need to upgrade this to decent blog software.

Sunday, October 29, 2006 :: The Upcoming Congressional Elections

Normally I avoid blogging about politics, and stick to nerdy technology issues, but this upcoming election is too important.

Our federal government is out of control. It doesn't matter what your party affiliation is, it should be clear to anyone that we are badly off course, away from any rational course.

Iraq

We are stuck in a quagmire in Iraq. We knew things were going wrong when the reasons for going into Iraq started changing. First because they had WMD and were supposedly an imminent threat. Then when none were found it was to get rid of Saddan Hussein and bring democracy to Iraq. Now its so we can prevent a civil war, which would never have happened if we hadn't toppled their government in the first place.

Despite what our officials tell us we see daily a situation that is more and more out of control. US military casualties keep rising. The civilian death toll is now over 650,000. The costs keep rising, to over a trillion dollars, and increasing by more than than 200 million dollars per day. What could your local town government do with the money spent in just one day on this Iraqi misaventure? I know that here in Arlington where I live the money spent in just 2 hours of this war would renovate two elementary schools that he town has put off renovating for lack of money.

Our leaders keep giving us the same naive and over optimistic description of the situation on the ground. We heard the mission was accomplished, that the insurgency was in its last throes, that our foes were a few stubborn hold out - the so called "dead enders". At every juncture we are told that this time, finally we are getting close to having the problem solved. Incredibly, our President challenges the enemy to 'bring it on'.

How many times were we told that the situation was going to turn around? After we take Baghdad, after we capture Saddam Hussein, after we form a new government, after we kill Zarqawi, after after elections, after they pass a new constitution. At each juncture things have only gotten worse. Each time the insurgents increase their attacks on our troops we are told this is a good sign. A sign that they are getting desperate because they see that the US is achieving its objectives. We have achieved no objectives. As bad as life may have been under Saddam, its clearly worse now.

Despite an perfect record of failure in Iraq our leaders have never admitted to a mistake. Never have we heard them say, that they were wrong and that we need to change the strategy. Year after year they do exactly that same thing which has already failed every single year.

A trillion dollars has been spent in Iraq, and there is no accounting for where it went. Most of it was handed out to military contractors - in bundles of cash - mob style - without any accounting controls at all. None. We have no idea where this money has gone. Take just one example, $900 million dollars was given to the Iraqi government to fight the insurgency. This money has simply disappeared. The best we can hope for is that it was simply stolen. It is not unlikely that much of it was funneled to the insurgents.

And our congress asks no difficult question. They don't demand answers. They sheepishly accept this complete failure as if it is the best this country can do.

Civil Rights

The chance that an American will be killed by a terrorist is about one in 3 million . About the odds that you will be killed by a shark. You are 300 times more likely to be killed in a car accident. The government has used this irrational fear to justify an attack on our civil liberties.

The fourth amendment of the Constitution makes it illegal for the government to search our us without a warrant. The constitution is very specific that we can only be searched with a warrant describing the specific person or place to be searched. It is clearly prohibited for the government to search people's bags. Yet today we accept that the government may search us anywhere. Here in Boston they now do random searches on the subway.

The government now spies on our phone calls, and They can search our phone records, credit card records, and library records without a warrant. When the government conducts these searches, the parties who hold these records who normally have a responsibility to protect our privacy, are now required by law to tell no one that these unconstitutional searches are happening.

A new law has been passed that allows the government to declare any person, even a US Citizen, to be an enemy combatant. This does not require a trial, or a judge. The federal government just decides it. They can then arrest this person, in secret, and take them to a secret prison, in the US or more likely abroad. They have no requirement to provide you with access to a lawyer, to present evidence against you, or even to notify your family that you have been taken. They can hold you as long as they want. And while they claim they will not torture anyone, they decide what the definition of torture is.

These are the tools of a dictatorship. They promise never to use these tools against the people. The question is, how much do you trust the government?

Other Reasons

Of course there are many other things wrong.

The current congress is the most corrupt I have ever seen. Corporations bribe our officials to provide them with no-bid contracts, which prevent the government from negotiating with drug companies for lower prices, and which make it illegal for citizens to buy these same drugs for lower prices from overseas etc.

This is congress is the most financially irresponsible one in history. Cutting taxes for the wealthy, at the same time that they have grown the budget deficit to half of a trillion dollar record. All while underfunding education, health care, and scientific research.

And then there is the constant lying - about the lobbyists, about the war, about everything.

We need change the Congress

This is not about party. This is about changing control of the Congress so that we can put the brakes on an out of control political situation. We need have the Congress and the Presidency in opposite parties. This way we will restore the natural adversarial nature of the relationship between the presidency and the congress. Fewer laws will be passed. There will be more scrutiny of the ones which do pass. And when things go wrong there the congress, instead of covering for the Whitehouse will ask questions.

This means tipping the balance of control from Republican to Democrat. If all the congressional seats were being contested in this election there is no doubt that there would be a change of control in congress. But unfortunately there are relatively few Republican seats being contested this time around, and they are mostly in heavily Republican districts. Each seat will require a great deal of effort to capture.

Fortunately the majority of this country sees the problem and is ready for a change. The result is that there are very few seats which are a sure bet for being overturned, but there are a very large number of congressional seats which are on the edge, and can be turned to the Democrats with some effort.

This time it is too important, and too close for people to stay on the sidelines. We need everyone to participate in the political process. Voting is just the beginning.

There are many things you can do, but the most effective thing you can do is to call people in states with close elections to make sure that we turn out the vote. You can do this easily from home. All you need is a phone and a computer. You register for a service which gives you phone numbers of people to call, and a script that you use to encourage them to vote.

One such initiative is at MoveOn.org. But if you don't like them I'm sure there must be others. You don't have to agree with any of MoveOn's politics. If you believe that the current congress is out of control and needs to change to the opposite party, then use MoveOn's calling service to do that.

You should also consider skipping work on November 7th, election day, and spending the day calling people to make sure they voted.

This is our chance to change control of Congress. We kicking ourselves for the next two years if we didn't do everything we can to change course.

Sunday, October 29, 2006 :: More Information about Specific Congressional Races

Thursday, October 26, 2006 :: Using Gmail as a Spam Filter and SMTP Gateway

The ISP which hosts my domain provides me with basic email service. There were two things about it that I was unsatisfied with, the lack of spam filtering, and that the SMTP server is on the standard SMTP port and so is blocked on some networks. This is important to me because I work in many different offices and sometimes from home, and on each network I was having to use a different SMTP server.

I looked at Google Gmail and there is a lot that I like about it. It has great spam filtering. You can access their SMTP server from anywhere. Its on a non-standard port which is never blocked. Any network that blocks GMail will hear angry complaints from their users pretty fast. I also the fast that GMail lets you download your email through the standard POP4 protocol, so you can use whatever email client you like.

But ultimately GMail has a limitation which is a deal breaker for me - you have to have a GMail.com address. I want my email address to continue to be hosted from my own domain, oroboro.com. With my own domain I can have any email address I want. I like to have different addresses for different projects for example. And with GMail all the obvious addresses are taken, and you have to chose long and unweildy addresses. It is also nice to be in control of your email address. Ultimately GMail owns your Gmail address. You can never decide to take your gmail address to some other hosting company. And finally a GMail email address just doesn't seem savvy enough

With a little research and some creativity it is possible to extract the features I want from GMail and continue to have my email hosted on my own domain.

Using GMail as an SMTP Gateway

When you send email your email client connects to an SMTP server. This server forwards your email messages onto the global network of email servers. Every email installation needs access to an SMTP server to send email. Normally when you get an email account from an ISP that hosts your web site they provide you with access to their SMTP server. Many ISP's that provice internet connectivity also have their own SMTP servers. To cut down on SPAM origination from their network some ISP's require you to use their SMTP server when you are on their network. This is very annoying when you move around because you have to keep reconfiguring our SMTP settings. Some networks go so far as to block the default SMTP port ( 25 ) completely. GMail is never blocked and works the same on every network - so there are many advantages to using GMail.

Using Gmail as your SMTP server is simple to do. First register for any Gmail address. If you can get a reasonably short one which is meaningful to you great, if not, it doesn't matter - no one will ever see it anyway. Once you get it log in, go to "settings", then "accounts". Select "add another email address". In the pop-up that opens up type in the email address you want to use - the one hosted on your own domain. This is the address people will see as the "from" address on your email. Go through the dialog on that pop-up. GMail will require that you verify the address by sending you mail on that address. When you are done go back to the "accounts" screen. Select "make default" for that address. And then select the radio button that says "Always reply from my default address". Now you are done configuring GMail.

All you need to do now is configure your email client. This is different for every client. I use Mozilla Thunderbird, so I can walk you though that. Go to "tools", then "account settings". In the account settings dialog, in a box on the left choose "Outgoing Server (SMTP)". Remove your existing SMTP server entries. If you are just experimenting, may want to write down the existing settings before you remove them. Now select "add". In the dialog that opens up, under "settings" enter the following:

Description: gmail smtp
Server Name: smtp.gmail.com
Port: 465

Under "Security and Authentication", select "Use name and password". For "User Name" enter your new gmail account name. Under "Use Secure Connection" select "SSL". Now press ok to get out of both dialogs and you are done.

Try sending an email to yourself. If everything worked correctlyit will go through GMail. You should receive an email that looks like it came from your own domail. If you open up the email headers, by selecting for example "view" then "message source" in Thunderbird, you should see something like this:

Return-Path: 
Received: from ?192.168.1.100? ( [72.72.19.185])
        by mx.google.com with ESMTP id g7sm6483071wra.2006.10.26.20.37.06;
        Thu, 26 Oct 2006 20:37:07 -0700 (PDT)
Message-ID: <45417EE0.8070406@oroboro.com>
Date: Thu, 26 Oct 2006 23:37:04 -0400
From: Rafael Baptista 
User-Agent: Thunderbird 1.5.0.7 (Windows/20060909)
MIME-Version: 1.0
To: Rafael Baptista 
Subject: test
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Rafael Baptista 

You can see that the from address is your hosted address, but the sender is your gmail address. The people you correspond with will only see the "from" address unless they like to read email headers. Those that like to read email headers will be impressed by your nerdiness.

Using GMail as a Spam Filter

GMail has an excellent spam filter. Google is in the business of classifying huge amounts of public information, and presenting what is most useful to users. So it makes sense that they would have a great spam filter. To use GMail as a spam server you need to have your host ISP forward all your email to your GMail account, where it will be filtered by Google's massive spam filters. Then you configure your email client to download your scrubbed email from your gmail account.

First some background on POP servers. Just as you use an SMTP server to send email, the standard for receiving email is a POP server. Your email client connects to a POP server and downloads any email which may be waiting for you.

The first step is to configure GMail to allow you to connect to your GMail account through POP. Just follow Google's own directions here. Then configure your email client to use GMail as the POP server. Using Thunderbird as an example, select "tools", then "account settings". In the bar on the left choose the email account that you want to point to GMail. Enter the following:

Server Settings: pop.gmail.com
Port: 995
User Name: your gmail email address
Use Secure Connection: SSL

Now your POP server is GMail. Try sending email directly to your GMail account. A few minutes after sending it you should receive it. If all went well, you need have your domain hosting company shut down your POP4 of IMAP email account, and to turn all your hosted email addresses into aliases that point to your GMail account. Most well run ISP's have a control panel application that lets you do this yourself. Or you can call them and ask them to do it. It should take them just a few minutes do do this for you.

This setup is nearly perfect. You will have your email scrubbed clean by GMail's leading edge spam filters. You will have access to an SMTP server which is available everywhere you go, on just about every public network. Yet will still be in control of your own email address on your own domain. You will be able to have any address you want on that domain. The people you correspond with will never know that the mail is being routed through GMail ( unless they like to read raw email headers ). And if you ever become disatisfied with GMail, you can easily go back to another ISP without disrupting your email communications. The only drawback is that all your email is now going through one more company, with the attendant loss of privacy, and risk of government snooping.

Thursday, October 12, 2006 :: We Need a Better Email System

Everyone who works with email can tell that the current email system is broken. We are inundated with SPAM, phishing attacks, and viruses. Our email communications are insecure, we know that corporations and governments spy and archive our emails.

Every time I delete spam from my inbox, or update all the filters I run, or call my ISP for help with my email, or read an article about email system abuse by the government, I start thinking about how the email system could be designed in a better way.

I have started to put my thoughts into a document for the design of an improved email system. Once I have the design fleshed out, and I have time I may program this up as a proof of concept.

Monday, September 18, 2006 :: Mutex Deadlocks and fork()

The interaction between thread synchronization methods such as mutexes, and spawning new processes using fork() and exec() can cause unexpected mutex deadlocks.

Suppose you have a multi-threaded program which controls access to a data structure using mutexes. When each thread accesses the shared resource, it locks a mutex, then uses the data structure and finally releases the mutex. If a second thread needs to use the same data structure, the mutex will cause that thread to stall waiting for the first thread to release the mutex. Very standard.

On linux the way to spawn new processes is to use fork() and exec(). Fork< spawns a new process by cloning the current process. Logically, the new process looks just like the old process, including copying the state of all the variables. Each thread can detect if it is the original thread or the cloned copy. Normally the copy will then call exec() to start new program using the copied thread. In multi-threaded programs, fork() only clones the current thread, the one that called it. The cloned process will not have clones of the other threads.

And this is the source of the problem - if your original process had a mutex locked when the new process was spawned, the new process will start with a locked mutex. But since the new process doesn't have a copy of the thread that locked the mutex, the copy of that mutex will never get unlocked in the new process. If the new process attempts to lock this mutex - it will deadlock.

How do you avoid this? One way to make sure that no mutex gets locked between the call to fork() and the call to exec(). This is not always possible. For example, if you have a mutex protected memory manager, it is not always possible to make sure that no memory will be allocated between calls to fork() and exec(). An alternative is to make sure that no mutexes are locked when you call fork(). You can do this by locking such mutexes yourself right before the call to fork(), and then unlocking them right after. Remember to unlock them both in the case where fork returns a new process id ( the child process ), and when it returns an invalid id ( the parent process ).

One possible solution that normally does not work is to turn off mutexes completely in the child process. This seems like it would work because the child process has only one thread, and so appears not to need mutexes. What you must consider is that when the thread was spawned another thread may have been in a critical section and the protected data structure may be in an unstable state.

Monday, September 18, 2006 :: Stl Iterators Considered Harmful

The iterator concept, as used in container classes like STL, seems powerful and useful at first. The iterator class encapsulates the concept of iterating over a series of objects in a container. The programmer does not have to concern himself with the details of how the container is organized. It could be a vector, a list or a map. The iteration code is mostly the same. In addition concepts like the first, next and last elements of the container are defined by the iterator class. The programmer does not have to define these things himself. Finally, the iterator itself can return the contained object. The programmer does not have to worry about how to get the object out of the container.

Another benefit is that iterators should make it less error prone to iterate over objects in a container when that container is changing. For example iterating over a container as elements are added or removed.

In practice, STL iterators are cumbersome and result in code which is needlessly complex and obscure.

Consider iterating over a vector. Imagine a simple vector container class:

To iterate over the elements of this vector is very simple, and straightforward to understand:

To do the same thing with an stl vector and an iterator is a little more complex:

What is worse about this code is that it has subtle complexities in it. For example, because the iterator i is a class, and the increment operator is function, doing i++ will work, but is slower than ++i. The first one forces the compiler to create a temporary object while the second does not. To use the iterator efficiently the programmer must remember to do this. Accessing the object requires dereferencing the iterator. The iterator object itself points to internal data structures in the vector. So while it looks like a self standing object, it really isn't. If the vector is modified or deleted, the iterator will be invalidated. Finally in order to use the iterator the programmer must remind the compiler the name of the class that is being iterated over, foo, to construct the correct type of constructor. The first case doesn't need to mention the name of the class being iterated over. The compiler already knows.

Problems with STL iterators become worse as the operations become more complex. Consider iterating over a vector looking for elements that meet a certain condition and removing them. In the simple vector case you would do:

The first is simple and easy to understand. The mental concept for a programmer remains very simple, an array of objects referred to using a numerical index. Removing one, shortens the array, and so the index must be decremented by one. The STL case is more complex and less readable:

In addition to the problems already mentioned in the previous case, the programmer must know to overwrite the iterator i with the return value of the erase operation, otherwise the iterator will be stale. The very standard and easy to inderstan for loop syntax is now broken. The increment operator is moved to its conditional branch inside the loop, making the loop less readable. It is now no longer immediately obvious that we are iterating over all the elements sequentially. One must read the loop code itself to make that determination. The new code is also much wordier.

Debugging code with iterators is also more complex. Instead of simply inspecting an index variable as an offset to an array, a programmer must examine the contents of two complex inter-related c++ objects.

When you look at these stripped down examples if may seem that while the STL version is a little wordier, it is not inordinately complex. Any decent programmer should be able to understand both versions. What you should consider is that these loops do not exist in isolation but are usually buried in complex code, often in nests of multiple loops with many conditional tests. Making code like this simple and easy to read makes it possible for programmers to focus their attention more on their programming task, and less on these structural issues.

Container classes like STL should make management of object containers simple and natural. STL would be better if iteration was handled directly with indexes instead of with iterators.