Sunday, November 30, 2014

Online Drum Practice Log

I recently attended a weekend drum camp with Thomas Lang. where I learned a lot of things about drumming in general, but one thing very specific:  Practice time is everything.   This is also a motto of the amazing drum teacher Mike Johnston, who recently posted this simple truth video about practice.

My own modest experience as a struggling drum student certainly mirrors Thomas' and Mike's teachings and underscores the importance of persistent and consistent practice.   I think that I have been practicing a good amount, but I have been relying on my own memory for the number of hours practiced rather than a real log of my time.     At camp, Thomas spent a bunch of time teaching the importance of a detailed log and gave some great advice on what should be in the log and why.   I will attempt to summarize the teachings and then give details about an online tool for keeping a drum log practice log.     If you are not a drummer, keep reading, this tool could easily be adapted as a practice log for any instrument, or any skill that could benefit from a practice log.

What should be in a practice log?

  • Duration - This one seems pretty obvious.   How much time was spent practicing.   I choose to log each time that I sit down at my kit rather than summarizing at the end of the day.
  • Mood - Did the practice feel great or awful?  Some days are better than others, and for many people the time of day effects their mood and productivity.    By keeping track of the mood or general productivity in a detailed format, the log can help identify the best times or days to practice in the future.
  • Time of day - The time of day is recorded so that the mood can be correlated.     With enough data it should be possible to see what time of day is the most productive time to practice so that future practice sessions can be optimized.
  • Date - This allows for the correlation of mood as well as knowing how many minutes are practiced each day.
  • What - What was practiced in general.    If consistent categories are used, the tool can tell how many minutes are practiced on a given category over time.
  • BPM - "Beats Per Minute" track the speed of a given exercise so that progress can be tracked on rudiments, grooves and other exercises where building speed matters.   This may not apply to all practice sessions but is useful for many.  
  • More Details - A place to record specific details about what was practiced, problems, successes and any other notes that might be useful

There is a lot of data to record and I am notoriously bad about keeping paper records, so I immediately started to look for digital versions of a practice log.

My wish list for a digital practice log are:
  • All of the data points listed above should be easy to capture
  • The tool must work on a variety of platforms including: PC, MAC, Android and IOS
  • The data needs to be centrally hosted or synced among all the application instances so that I can record practice time from anywhere.    It definitely cannot be trapped on my phone only.
  • The data should be exportable so that I can put it into a spreadsheet or another program to get better insight into the data.
  • It shouldn't cost too much. :)
I was not able to find anything that met all of my criteria, so I started to investigate how difficult it would be to make my own.      One of my goals is that the data should be exportable to a spreadsheet, so I started my process by looking at my favorite collaborative spreadsheet program, Google Sheets. Google Sheets is part of Google Docs/Drive and is similar to MS Excel.     I knew that if I was able to get all the data into Google Sheets I would be able to meet most of my criteria since it is a cross platform centrally hosted program.   I knew that I would not be happy manually entering the data, so I started looking for ways to make data entry easy.       It turns out that Google Sheets has  a programming API that allows for mobile or web apps to query and enter data, but as I started working on a web app I realized that it was going to be a fairly involved effort.    I started researching some more and found that Google sheets has a very cool "forms" mode that I had never used before.    The more I played with it, the more I realized that it would be the simplest and most effective method to build the log tool with.  

Try out my solution

I ended up writing a new solution myself that works well for me.  Would you like to try it out?     The data entry form is here and the back end spread sheet is here

When you go to the data entry form you will notice that you do not need to enter the date or time of day, because that information will be automatically captured at the time you submit the form.    If you need to enter data for a prior day, simply log the information with the form and then go to the spreadsheet and change the date.    

The spreadsheet provides all the tools needed to graph practice time over days, months and years.    And since the data is in a standard spreadsheet program it is easy to export to any other program.    Since Google sheets is cross platform, the form and spreadsheet can be accessed from any device, mostly through a web browser, but also through the Google drive mobile app.

Set it up yourself

If you want to set this up for yourself, follow these steps.
  • Make sure that you already have a Google Drive account set up.
  • Go to the shared spreadsheet here
  • In the "File" menu choose "Make a Copy..."
  • Give the file a new name and save it.   It will now be in your Google Dive account.
  • Select the sample data by clicking and dragging and use "edit->Delete Values" to clear it.
  • Under the "form" menu, choose "Go to Live Form".   This is your entry sheet and can be used from any web browser on a computer, phone or tablet.    Save a bookmark to it.
  • Start practicing and log each practice session using the "Live Form" to make it easy to use.
  • Visit the spreadsheet to see all your sessions and graphs of your practice over time
  • You may also want to edit the input form to customize it for your name.   Go to the "form" menu in the spreadsheet and choose "edit form".   Change "Music Practice Log" to "YOUR NAME's Practice Log"
Within the spreadsheet are several tabs along the bottom that contain summary graphs for different time periods.    They are made using pivot tables and summarize the data based on the following:
  • Last 60 Days - Time summarized by day
  • Days   (All days for the entire practice log)
  • Months - Time summarized by month
  • Years - Time summarized by year
All of the graphs are embedable in web pages in case you want to put up a page that shows your progress.

Advanced notes

If you are interested in using this tool for anything other than a drum log, or to expand the functionality, this section is for you.     You can easily change the names or headings of the data collected by using the form editor in Google Sheets.   Open the spreadsheet and look under the "form" menu.   The form editor will allow you to change the name of the items being collected and will allow you to add new items to the form.    If new items are added, a new column will be created in the spreadsheet automatically.    If you know how to use Excel or another similar spreadsheet program, you will be able to add to or analyze the data in almost any way you can imagine.    The benefit to the data being in a spreadsheet is that it is easy to manipulate and graph.


Comments

Feel free to post a comment below if you have any problems getting the spreadsheet to work, or if you have other questions about changing the spreadsheet to work for your needs.   I will try to reply as soon as I can.

Tuesday, October 7, 2014

Moto 360 Review

As I write this I have had the Moto 360 watch for 4 days, so I am not yet a seasoned pro, but I'm now starting to get used to it.

(Update after 2 weeks is below)

What is it?

In case you have not seen it, the Moto 360 is a smart watch that is paired with your phone.   It is far from the first smart watch, but it is the first watch that I thought was good enough to buy.   A big part of "good enough" is that it looks like a real watch, being round, and it looks great.

How I use it

Mostly I use it as a second display for my phone.   It is much easier to see a notification on my watch than it is to pull out my phone and see what just came in.   The most useful alerts are text messages.   I have also gotten a bunch of Facebook messages and call notifications.       There is a setting to ignore messages from certain apps that I have configured for games, LinkedIn and WiFi notifications.  
You can reject or answer calls from the watch.   If you answer the call and are on a Bluetooth headset you can start talking right away.   If your phone is needed, you still have to pull it out of your pocket. I have a Bluetooth speaker and mic in my ski helmet, so I think this will be a very useful feature while skiing.

Google Now provides alerts for many of my interests.    It will display the SF Giants score as it is happening (configurable for spoilers), tell me when I need to leave to drive to an appointment and other interesting news and event.

Other great uses

I have set up my watch to alert me if I leave my phone somewhere.     Whenever I walk more than 20-30 feet from my phone, my watch alerts me to it.    I'm using Wear Aware.  

My phone unlocks automatically when I am near it.   I set up my watch as a trusted device so that I can skip the lock screen or fingerprint reader.    The phone locks automatically if I leave it somewhere.     Try this or this.

Not so great

Technically, you can initiate actions using the watch.   You can say "Call Fred Phelps" and it should make a call.   Or you can make Google searches or a bunch of other things.   All of this must be done by speaking to the watch, which I find aggravating and error prone.  I find it to be much slower than just pulling out my phone.    I may try to make the call initiation work when I'm skiing, but I'm not really optimistic that it will work in a noisy outdoor environment.

Battery Life

This was the one area I was most concerned about, but fortunately I have not had a problem.    I charge it every night on the included wireless charging stand and I usually have between 40-50% battery life left at the end of the evening.    This is in the default configuration.    There is an "ambient mode" which keeps the screen on all the time that I would prefer to use, but I heard it will drain the battery to fast so I have not tried it yet.    I will try it sometime in the future and report back.

Summary

Overall I impressed and am liking the watch even more than I thought I would.    The display is beautiful, the UI works and the battery life is fine.    I have lots of great mechanical watches, so I'm not sure if the 360 can take over completely, but I should know in about a month if it is a "must keep" device or just a toy.

2 Week Update

I have had the watch for two weeks now and I have found the battery life to be way better than expected.   I was able to put the watch into "ambient" mode and never run out of battery.    Ambient mode makes the watch behave much more like a real watch, without any delay in updates or glance capability.    Lots of people are asking about the watch and really like it.

Wednesday, June 19, 2013

Another housing bubble in the Bay Area

I moved to the Bay Area in 1994.
Source: Zillow.com

I bought my first house in 1996.

I sold that house in 2000 for more than twice what I paid for it, and moved up to Truckee, CA.

I moved back to the Bay Area in 2006 during the last housing bubble and felt that the market was overvalued, so I decided to rent.    I found a high end rental house and have been renting for the last 7 years.  I have saved roughly $600K in housing costs during that time vs owning the same house.

Record growth this year
The market this year is red hot.    Year over year home prices are up about 31%.    Across the street from me a developer built a 4,000 sqft house on a 8,000 sqft lot and sold it for $4 million.    The most expensive home on the street previously had been only $2.5 million.   The owner of the house I have been leasing noticed and has wisely decided to cash in on the current market, so now I need to find a new place to live.   I have been making the tour of houses in Menlo Park for the last two months and I'm exasperated.  Very average houses are going for well above asking price and land values are crazy.    Complete tear downs are selling for close to $2 million.  How does this make sense?    I created a spread sheet to try and analyze the market and see if it makes any sense.

According to Zillow the average house in Menlo Park is currently $1.46 million dollars, that is up from $1.04 million in May of 2011.   Keep in mind that these are averages and include condos and distressed properties and houses literally right next to highway 101.    The real truth is that you can't buy a "normal" house for less than $1.9 million and that "normal" house will be about 1,600 sqft.   The average cost per square foot is at or above $1000 sq/ft.   Any house that you might consider "fancy" is 3 million or more.

The recommended guidelines for housing costs are about 28% of after tax income.   In California, people are used to spending more income on housing so doubling the figure is about 55%.   I built a spread sheet that calculates the full cost of owning a home and will calculate the income required for 28%, 55% and 75% of income going towards housing.   The spreadsheet takes into account the mortgage interest tax deduction and the property tax deductions.

If I run the numbers for $1.46 million and $1.9 million here are incomes necessary to have 28%, 55% and 75% of income going to housing

Necessary income for $1.4 Million house
28.00%55.00%75.00%
$688,198.86$322,812.16 $221,766.95

Necessary income for $1.9 Million house
28.00%55.00%75.00%
$885,066.96$421,091.92 $292,782.77

Now lets look at the average income in Menlo Park.    Average income can be misleading since it also includes people who are not home owners.   The published average household income for Menlo Park is about $110,000.    If I dig a little deeper and take just the most affluent area of Menlo Park we can find that the household income for West Menlo is about $220,000.  (The data source is city-data.com)

So we can see that at $1.4 million the average household would be spending 75% of income on housing.   The real reality of $1.9 million would find that average household would have to spend 94.5% of their income on housing.

I think that it is pretty clear from the data that the average person cannot afford the average house right now.    But what does that mean?

What is causing the spike in prices
If the average person living in an area could not afford to re-buy the house that they are living in, what does that mean?   
  • The neighborhood could have a recent influx of new buyers that are more wealthy than the existing base.  (This is usually called gentrification)
  • Supply is constrained and this is temporarily pushing up prices.
  • Interest rates are about to rise and people are buying before they go up.
  • The Facebook effect?
  • The tulip craze is back.
Is it really Gentrification?
Gentrification is a possibility, but is it really happening?  How many people really make $885K a year?     Let's me more reasonable, lets allow for 55% of income for affordability, so the real question is how many people really make over $420K.    According to the IRS in 2010, 800,000 people in the US make $500K or more.   How many of them live in California?   I can't find a direct source for this, but we can estimate it.    According to the IRS in 2009 about 20% of Millionaires live in California.   If I generously estimate that 75% of them live in the Bay Area and not LA then we have about 15% of all the Millionaires living in the Bay Area.    That's 120,000 people (very approximately) in the Bay Area with household income above $500,000 per year.     So I guess this is  a possibility.  

Similar Neighborhoods in the penninsula:  (number of households)
  • Menlo Park: (12,000)
  • Palo Alto:   (25,000)
  • Atherton:   (2,500)
  • Los Altos:  (10,000)
  • Hillsborough: (3,600)
  • Los Gatos:  (12,000)
  • Woodside:   (2,000)
  • Total: (67,100)  peninsula only
Constrained Supply
This does seem to be a serious problem.   There just are not enough houses for all the people moving here.    Additionally, Prop 13 encourages people to stay in their homes and never move, so there is very little turn over in the market.   High prices combined with taxes hurt turnover as well.    If you are living in a house that is suddenly worth 2 million dollars it is very hard to sell because taxes will take half the proceeds.   Then you need someplace else to live and everything costs more than a million dollars.

Basic economics says that prices will go up if there is more demand than supply.    But how high and why now?    Supply has been a problem for the last 20 years.   Why is the market up 30% in one year?

Interest rates on the rise
I think the recent spike may be related to interest rates.   As interest rates climb the monthly cost of a home goes up substantially.    I did all my affordability calculations at 4% interest.   Here is a table of annual cost for a 1.9 million dollar house at different interest rates: 

4%5%6%7%
$146,000$160,000$173,000 $188,000

That equates to more than a thousand dollars a month in difference for each percentage point.    At 7% you would spend $42,000 more per year on the same house than at 4%.    If interest rates return to normal levels of 7% over the next few years, it should cool the market considerably.


Facebook?
There is more than 75 Billion dollars worth of new value in the Menlo Park area.   The vast majority of that is owned by a small number of people, but there are at least a few hundred new Millionaires in the last year.

Tulip craze?
Are we in another temporary bubble?    I really don't know, but I hope so.    Nobody benefits from sky high real estate prices, except the real estate industry itself.    Surrounding neighborhoods are also experiencing record setting prices.    The Zillow Palo Alto index is over $1.7 Million and even towns like Sunnyvale and Mountain View, which are traditionally far less expensive, are approaching $1 million.     Local companies will not be able to attract talent to the area if people can't find an affordable place to live.   Keep in mind that the national average for a house is $158,300.    I would never expect the Bay Area market to be average, but I would expect it to be closer to 5x the norm rather than 10x.

What am I going to do?
I love living in the Bay Area and I love Menlo Park most of all.    The combination of an amazing climate, proximity to fantastic natural wonders like Tahoe, Marin and Napa and last but certainly not least the amazing people who live here.     If I could convince myself that the market won't fall again in the near future I would be happy to buy an overpriced home here.   Unfortunately I'm convinced of the opposite.   I think the market will correct downward at least 25% in the future, but I don't know how long it will take.

Some people called me crazy back in 2006 when I predicted a correction.   They said that real estate in California never goes down and even bet me that it wouldn't.  I won money on those bets, but I was polite enough not to collect since the folks who said it lost a lot in the down turn.    I was complacent in 2009-2010 and didn't buy then, so now I'm in a difficult spot, my mistake.

Barring a happy accident falling into my lap with an under priced gem, I will be renting again.    The rental market is just recently starting to correct closer to ownership prices, but it is still a much better value than owning, it also carries no risk of a massive loss if the market corrects back to normal levels again.

Here is my spreadsheet that has the calculations

Please leave a comment with your favorite theory and prediction for housing prices in the future.

Tuesday, June 18, 2013

Why great software artists steal rather than borrow

"Good artists borrow, great artists steal", Pablo Piccasso

It's not a perfect analogy for software, but it sure gets your attention.   A different way to say this is "Don't borrow the ideas of other products, steal the building blocks themselves instead of rebuilding them."

Software teams are able to get new products and updates to market much faster than they were 10 and 20 years ago.   Much of the productivity gains have come from "stolen" building blocks that have been made open source or integrated into the development stacks that we use.

When I helped build Netscape Navigator nearly 20 years ago we had to start at a very low level and build everything up.    The networking libraries started with TCP sockets and even then there was not a consistent API.   The UI was built up from fonts, pixels, primitive menu API's and drawing commands.    When we wanted to display a GIF or JPEG we wrote all the code ourselves.   Everything was built using plain old "C" and we had to build all of our support libraries from scratch.    Simple list structures, hashing functions, time functions, threads, better memory allocation to deal with 16 bit machines.  We eventually built a cross platform support infrastructure called the Netscape Portable Runtime (NSPR) that encapsulated all the support code into one library and let anyone build tools on top of it.   We also gave it away, so anyone could use it.     Thousands of people over the last 20 years have been building similar libraries and making them available through open source for others to build upon.

Today when we build an application we can code the UI in HTML and let the browser figure out how to render it.   We can build the business logic in a high level scripting language that handles threading, memory management and a whole host of other complex low level problems for us.   We have access to thousands of open source libraries that can do incredibly complex tasks.    On top of all that we can even host our applications in the cloud and let someone else build, maintain and monitor all our hardware for us.

New code is still hard
UI's are easier, infrastructure is easier, memory management is easier, but coding is still pretty much the same.     We have lots of new buzzwords for software development these days:   Scrum, Agile, Extreme, Waterfall.   It all sounds very exciting, but I can tell you that team programming is still pretty much the same.    I have been doing versions of Agile programming for more than 20 years and we have now codified the methodology significantly, but the nature of the game has not changed very much.    New code is still hard.   It takes longer that we estimate and getting it bug free is very hard.   Keeping it bug free while extending it is even harder.

Don't build anything you don't have to
Every single thing you build will take about 10x the time in maintenance than the time it takes to build.   The things we build tend not to be as fully featured or as well thought out as a successful open source library and likely won't have half the features of the components you could have "stolen".   

So when do you decide to build something?
Deciding when to build something new versus using something that exists is really tricky.   One of the hardest parts is knowing what is out there.    There are literally tens of thousands of open source libraries that we could grab that do virtually anything.    If you want to make smart decisions about when to "steal" a component you need to educate yourself and get to know what is available.     It may seem like a lot of time spent researching and reading, but imagine how much time you are going to save by not having to build a major component for your development project.    And how much time you will have later on by not have to fix the bugs and add features because the component you found is already feature complete and stable.    Get on the internet and ask people if searching doesn't yield results.   

Spend all your time wisely
In a start-up, and in any company, but especially a start-up, you should spend 100% of your time building unique value.    Time spent reinventing the wheel does nothing to generate unique value.    Investors won't care that you built the coolest new version of crontab to control your back end processes, they care how many users you have and how much money you can make.  

Use your components wisely
Don't try to use components in ways they were not designed to be used.   First, you will likely find bugs that no one else has encountered because they have never used that component in that particular way.   Second, it probably won't scale and perform the way you expect.   

Don't try to second guess the design unless you are an expert
Understand and follow the best practices for each component.   You might not understand all the reasons for the best practices, but they are there.    If you want to use the Zend PHP framework you are going to have to use a model-view-controller paradigm.   You may think that it's too complicated for the simple website that you are trying to get done right now and you don't want to spend the time to learn how Zend's M-V-C model works, but you need to do it.    There are really good reasons why you should do these things, but they really don't manifest themselves until you start maintaining the code or decide to change the UI later in the product life cycle.   Or you might decide that SQL is way too complicated and design an intermediate language to sit on top of it.   If you think that's a good idea and your not a bona fide SQL and relational DB expert you are going to get in trouble.     SQL got complicated for a reason, trying to second guess 30+ years of iteration is very dangerous.

Don't use components that suck
I have a whole separate blog article on this one.  "For software development tools it's the ecosystem that matters most".   Make sure all the components and tools you use are bulletproof.    Time spent fixing someone else's code takes away from your product.

Give back
On a final note, I would like to encourage you to give back when you can.   All of the code that we "steal" comes from somewhere and wouldn't be there without the hard work and generosity of those who put it there.   Recognize that and give back to the open source community by contributing new libraries and enhancements and bug fixes.    I have open sourced almost all the code I have written and it has been a highly rewarding experience.

Wednesday, June 12, 2013

For software development tools it's the ecosystem that matters most

In this article I will be exploring the different types of software development tools and how to choose the right ones for your next start-up or project.

In my career I have had the pleasure of working with several successful software teams on a variety of projects.  I have also had the opportunity to mentor or review more than a hundred software development teams. I have written programs for Windows, Mac and Unix, Android and a bunch of Web delivered apps. At Netscape I was part of a small cross platform development team that wrote Netscape Navigator which at that time was the most used application in the world.

In this article I would like to concentrate on platforms and tools. Platforms and tools are one of the least understood or appreciated since they technically focused and most engineers don't get the opportunity to choose their tools.  Whenever a new software project is created we make important choices about what platform and tools we are going to build with.  I'm defining the platform to be the hardware and operating system and the tools to be everything else above the operating system.

Platform
Platform choices used to be pretty easy:  Windows, Mac or Unix. When the world was 90% Windows, most software developers knew right away that they would be using Windows as their platform.  Today we have many choices. Win, Mac, Linux, Android, iOS, Blackberry, Windows Mobile or HTML.  Platform choices are usually driven by business reasons, but there are still technical choices to be made.  For instance should you have a native windows or Mac app?  Web applications are preferable for most since they solve the distribution and update problem, but they are not able to integrate seamlessly with the computer and can't handle long running background tasks.   Music players are a great example of this.    Lots of music companies have cloud players that run in the browser, but most choose to do a native app to support offline play and to have a fancy native UI.  Mobile app companies also have an important choice about what platforms to support. Many mobile start-ups I see are just concentrating on their mobile apps, and not creating a web based experience. Unless they are a gaming company this seems extremely short sighted.  Most of us live and work in a heterogeneous environment, so a web based experience allows us to work with a service no matter where we are. GoodReads and Google Maps are good examples of this.

Tools
I find that tools are the most difficult choices that we make at the beginning of a project. There are many different classes of development tools. These are just a few:
  • Software language:  C++, Java, PHP, Ruby, ...
  • Framework: LAMP, Zend, Rails, Tomcat, ...
  • Compilers & IDEs: G++, MSVC, Eclipse, XCode, ...
  • Debuggers: GDB
  • Databases: Oracle, MySQL, Postgres, Mongo, Cassandra
  • Bug tracking: Bugzilla, Jira, FogBugz, ...
  • Revision Control: SVN, Git, CVS, Perforce, ...
  • Integration: Jenkins, Bamboo, ...
  • Unit Testing: CppUnit, Google Test, JUnit
  • Static Analysis: Lint, Klocwork, Coverity, ...
  • Hosting: Equinix, Amazon, RackSpace, Google, ...

Many software tools are chosen on the basis of what was used on the last project.    Familiarity is an important consideration, but should not be the only one.    If we simply go with what we know we are missing opportunities to choose a tool that has advanced or is simply better suited to the current project.  Here are some important criteria to consider:
  • How suited is the tool to what I am building?
  • Will the tool handle the scalability that I need? 
  • Maturity
  • Supporting tools
  • Community

How suited is the tool to what I am building?
To a hammer the whole world looks like a nail. Perhaps you really need a wrench or a screwdriver?   If you are building a financial application, don't try to use a fancy new noSQL database.   Use a good old fashioned conservative SQL database.  If the type of app you are writing has a large portion of users using one tools set, you should probably choose that.   Why try and build an Android app with MSVC when everyone else does it with Eclipse? 

Scalability
If you are building a web application for a small group, performance probably doesn't matter.   If you are trying to be the next Facebook, you might want to think about how the tools you are using will scale up to millions of transactions and a big design team.  Facebook is experiencing this problem in spades with their choice of PHP as a language.  Their improvements to PHP are very impressive, but the effort could have been avoided by choosing a different language.  Scalability also applies to team programming.    Dynamically typed languages, such as PHP, Ruby and Python can be challenging for very large software projects since the functions are not explicitly typed.  This makes it hard to detect software problems at link time and requires engineers to document their code in ways that are not enforced by a compiler.

Maturity
Tool maturity is too often overlooked.  I think this one should be easy, but I see start-ups making the wrong choices over and over again.  If a tool isn't 99.999% stable, you probably should not use it.  All of your time should spent solving technical and business challenges.  If a start-up needs to spend time debugging their tools they are wasting valuable time and resources solving a problem that they didn't need to solve.    How should you evaluate maturity?   Past personal experience or a referral by someone you know and trust is best.   Otherwise, look to see who else is using it.    If you can't find a large list of other companies that are using the tool in a production environment, stay away.    Even the most promising tool can turn into a nightmare if you make it part of your product and it starts to crash or otherwise misbehave.  Building tools is hard, not just because they are hard to build, but because they are very hard to perfect. You need tools that work every time, not just most of the time.

Supporting tools
This is probably the most important tool trait. Every tool class listed above comes with an ecosystem of tools that support it.  With a database you need backup tools, profiling tools, monitoring tools, API libraries and documentation.  New tools often look promising until you look under the covers and find out that it is lacking basic support infrastructure.  Do you really want to find out after you go into production that your tool needs to be taken offline just to back it up?

Profiling and debugging tools are critically important and are often the last thing on a product road map.  If you have ever experienced a critical performance bottleneck and have been without a real profiling tool you will understand how important a profiler is.  How about finding a thread deadlock with a debugging tool that doesn't properly deal with threads?  Memory corruption detection without a dedicated memory debugger?  These are support tools that usually take many years if not decades to get right in a major tool class, but they are absolutely necessary.   

Community
Last but not least is the community of people that can support you for questions and problems.  The absence of a strong community should be a huge red flag. Poke around and ask questions, make sure that there is still active development of the tool that you are choosing and lots of active users.  Look for example code and places where people congregate to ask and answer questions.   You will save a lot of time if you can draw upon the community to figure out problems for you.  Open source libraries built for your tool can also make your job a lot easier.  Plugins to support your other tool choices and layer in great product features.

Looking at the whole stack
Now that you have a framework for evaluating individual tools, you now need to go up a level and look at your entire tool stack.  Almost all of the development tools you use will interact with each other so they need to be compatible.  Want to use a Java based messaging framework but you use C++ everywhere else?   Think again.  Love that C++ static analysis tool, but you use Python?  Sorry no dice.  Your choices on one tool effect every other tool you will use.   Make sure that you evaluate the whole stack before you commit to any one tool.  Otherwise your one choice will likely dictate and limit all your other tool choices.    

Current Experience Levels
Last but not least please be realistic about your team's experience relative to the tools you are choosing.  It might seem like a great idea to use Ruby on Rails for your next major web project, but if there is nobody on your team that has any experience with Ruby you are probably going to run into trouble.  If you already know PHP the benefits to moving to Ruby are probably overshadowed by all the mistakes that you will make learning a whole new framework.  Remember that the end product is more important than the language used to express it and the user isn't going to care if you are using the latest and coolest technology to pull it off.  Conversely, using a technology that very few people know, even if your team is very proficient with it, will limit your ability to hire new people in the future.  

Apply this same methodology to support libraries as well
All of the points above also apply to support libraries that you will use in your product.   Here are a few examples.
  • Logging
  • Messaging frameworks
  • General Utility libraries such as boost for C++
  • Caching and proxy libraries and servers
  • XML & JSON libraries

Summary
Look at all your tools carefully before you start using them.   Make sure that they are all of the highest quality and maturity and that they come with the supporting infrastructure to handle the whole development process:    Coding - Testing - Debugging - Profiling - Monitoring
Doing this will save you and your team countless hours of frustration relative to less mature tools.

Monday, June 10, 2013

An HTML5 fancy user interface project

Visualizing disk space with a sunburst graph
Sunburt graph
I recently finished a new project that uses HTML5 to display and graph file system data and I thought I would write a blog post about it.
 
History
HTML has come a long way from the early days when I was working on Lynx. The latest and greatest features are pushing graphics and user interfaces that are approaching native code levels of complexity and performance. I'm particularly fascinated by these features because they represent the realization of a vision that we had in the early days of Netscape where HTML could take over as a complete application programing interface.

WWW applications?
Using the Web just like a computer program seems pretty obvious these days, we use it that way on many if not most web sites, but in the beginning it was really just a document publishing platform. Hypertext, the basis for the web, was designed as a way to make documents more powerful by linking them together with references that could be accessed incredibly easily.  The idea of putting powerful application logic behind a hypertext interface was pretty far fetched in the early '90s.  I happened to have a need to do just that due to a need at my university. I came up with some ideas and put forth an idea to the other developers I knew that were working on the web. This idea soon sparked the addition of form elements to the web. Later while at Netscape, we took the next step by adding a scripting language, Javascript, to add even more client side capabilities and responsiveness.

Full GUI support
One of the visions we had at Netscape was to create a platform that had a fully complete and functional set of graphical elements so that other developers could create any application imaginable. The foundational elements are reasonably well known and are included in all modern operating systems and major graphics libraries.  The difficulty in delivering GUI primitives to the Web was figuring out a way to efficiently add them to HTML and to make them massively cross platform.    When I say "massively cross platform" I don't mean just Windows and Mac. We were thinking about operating systems, screen sizes, resolution, and a whole bunch of other things.  The web was designed from the ground up to try to be device neutral and that makes the implementation very hard.  CSS is one technology that supports multiple displays by providing different layout and styling for different displays as required.  Netscape as a company ran out of life before it was able to complete its roadmap for a full complement of GUI primitives, but it did make a good effort.  A technology called XUL (pronounced Zool) was introduced that allowed a large variety of GUI to be expressed, unfortunately, XUL didn't move forward fast enough and was essentially dead on arrival.     Simultaneously Microsoft smothered the market with Internet Explorer and tried its best to keep the Web from moving forward with HTML.
Visualizing disk space with a tree graph
Tree graph

Enter HTML5
Fast forward several years from the Netscape era and the world has changed substantially.    Firefox, Chrome and a variety of Webkit based browsers have wrestled away the strangle hold that IE6 held on the market and began to make substantial improvements to the abilities of HTML.  HTML5 was soon born and with it a whole bunch of new graphical technologies that enable many new types of GUI applications.    SVG, advanced CSS, Canvas and Javascript now combine to create an ability to create very rich interfaces that are entirely driven within the Web browser.

My experiment  
A while back I came across a group that was creating a set of tools that harnessed the new power of HTML5.  The project is called D3 for Data Driven Documents.  I saw what D3 could do graphically and wanted to try it out and do something personally useful with the technology.   Since my current company Zetta.net is in the cloud storage space I decided to write a tool to visualize the size of the data on a computer hard drive.  One of the difficult issues with a tool to analyze disk space is that it needs access to all of the data on a drive and that data is sensitive and private.  I needed to figure out a way to scan the data and not violate a user's privacy. Standard HTML applications run in the cloud and couldn't do something like this without sending data to a server, but the new world of HTML5 can make everything happen locally in the browser.

Some complications
At this point in my project I had an idea of what I wanted to do, and a fancy new graphical library for graphing, but I still needed a way to collect the disk space data to be graphed.  Javascript has chosen to stay away from local file access to simplify the security model, but Java long ago added the ability to ask the user for local disk access.  I ended up writing a Java beans integrated module that spawns a Java application to scan the local disk and return the data to Javascript running in the browser for analysis.  The beauty of the Java/Javascript integration is that none of the data ever leaves the local computer that it is running on. The downside is that Java is often broken on user machines, or is disabled due to security concerns.  I hope at some point Javascript figures out a workable security model to allow some forms of local disk access.
Visualizing disk space with a tree map graph
Tree map graph

The finished project
Please check out the finished project if you are interested. I have made a video demo and a walk through to explain the use and features.   There is also sample data to play around with that skips the scanning process.  The code has been published as open source with a BSD license.    This project just might come in handy for you the next time you run out of disk space and need to know what is using it all up.

http://wheresmydiskspace.com/


Wednesday, June 5, 2013

A phone for every feature. Where is Apple?

6/12/2013: Update:  The Galaxy Zoom was announced today.   This is a pretty great looking device.    If you are serious about photography and have been carrying around a Canon s100 plus a phone like I have, this could potentially make your pockets lighter.  (in two ways)

Today Samsung announced the Galaxy S4 active, a dust proof, water resistant, rugged phone.    Motorola is talking about the new X phone, a phone with a battery big enough to last for days and a host of new sensors to know when it's in your pocket, traveling in a car or just hanging out.    The latest generation of phones from Samsung and HTC include infrared transmitters and NFC and a screen sizes for every hand size and preference.  I'm overjoyed with the choices and features in phones these days, but in all of this I have to wonder where is Apple?  The former leader in phone innovation which introduced and dominated the smart phone market seems to have quietly slipped into an underdog role.   


A bet on the future
A few years back I had a thoughtful debate with a VC friend of mine about the future of the phone market.  I had just gotten a shiny new HTC Android phone and was a member or a small minority.    I was looking at the fairly unpolished features of Android 2.3 and seeing all the untapped potential.   My friend was basking in the glow of the polished iPhone experience and confident that Apple would continue its dominance in the smart phone marketplace.  Being an open source advocate certainly influenced my thinking, but I was confident that the open model would eventually produce a phone that would outclass the iPhone.    I was so confident in my new Android phone I bet one whole dollar that Android would eventually rule the market.  (wow!)
 
Android turns the tables
New Android phones today ship with a version of Android (4.2) that is significantly more polished than a few years ago.   Android is now a legitimately beautiful and elegant solution that still offers the flexibility and power to customize your phone endlessly to your own whims.
Android phones currently outsell the iPhone 3 to 1 in total units, but the iPhone is still a dominant player in the premium market.  Here in Silicon Valley most of the affluent people I know are still iPhone users and Apple still has a fantastic brand and an ability to move the entire market, but how long will that last?   If Apple continues to sell only one form factor and one premium phone can it really compete in a marketplace full of specialized choices?   Should I have to wrap my phone in a big case if I want to make it rugged?  Should I have to add an external battery case if I want the battery to last?  I want choices and I want my phone to fit my lifestyle.


What could Apple do?
I would like to see Apple learn from its previous mistakes and open up its ecosystem with other hardware vendors.    I want iOS to be set free.   I believe that if Apple had started licensing iOS several years ago and satisfied the demands of hardware manufacturers it would never have lost market share to Android and would control nearly the entire mobile market.    As a consumer I'm quite happy that there isn't one dominant closed source vendor, but from Apple's perspective iOS is really missing the market.    Waterproof rugged iPhones,  iPhones with 6 inch screens and a whole host of other features would be fantastic, and I would expect that at the top of the food chain Apple would still be selling the highest quality, highest margin iPhone.

Now that I've said it, forget it.
Apple will never do this.   They will argue that it would make it impossible to have a seamless experience, that they tried it before with Mac clones and it failed then, and that they are a hardware company.    Too bad, opportunity lost.    My prediction is that Apple's mobile market a few years from now will look a lot like their desktop market.   10-20% market share on the high end and no real influence in the true direction of the market.   They will still command high prices with reasonably high margins, but they will not have a dominate position in anything.

Is Apple losing market share a good thing?
Yes!   A market controlled by Apple or any other closed ecosystem vendor would be very scary to me.   Apple has a history of telling it's users what they want and ignoring market data.    Apple also is trying to extract high margins from its products and benefits by keeping prices high.    Fortunately for us the Android alternative is here and has fantastic fundamentals.    Android is, at its core, a true open source and truly open technology.   Android is built upon Linux, and most of the code is released under the Apache license.  That means that we can take the code and use it even if Google doesn't want us to.    There are parts of Android that are closed and proprietary, but the meaty core is open to us and essentially free forever.   So lets say Google goes all evil on us in the future?   Will they be able to pry my phone from my cold dead hands?   No, we the consumers, and any other company that wants to, can take the core of Android and release something even better.    

OK I'm biased, I admit it
Yes, I admit it, I'm shamelessly writing a blog article about a subject that people just love to argue about.   I'm also a huge open source bigot so I'm naturally inclined to like Android.     I really just wanted to write about how much I love the progress that Android has made and the fact that for the first time in history an open source operating system has dominate market share.   It really is about time!


Flame on!
Feel free to start a debate in the comment section, but please try and put forth reasonable arguments.   Nobody benefits from name calling and bickering.    I'm sure lots of people will disagree with my predictions, if you have your own, call them out and tell us why you think they are right.