In the coming years, there will be a file format war that will be reminiscent of the DVD format wars of the mid 2000′s. To most people, this war will seem like a tempest in a teapot, a bunch of computer weenies arguing agrily over minutiae. However rather than being minutiae, the differences between the file formats are significant and can have serious financial consequences for your business.

File formats are critical in that you may want to open a document 10 or 20 years down the road and discover that your software won’t open the documents.  Hiring consultants to convert the files into a usable format can be expensive.  Worse, a conversion effort and can introduce errors.  Sticking with one software platform, Microsoft Office for example, is not the way to avoid this problem; later versions of Word have been known to have trouble opening documents created in earlier versions of the application.

To meet the need for long term reliable data formats, there have been several attempts by various organizations to develop file formats that would remain consistent over years.  These include Adobe’s Portable Document Format (PDF), Microsoft’s Office OpenXML Format (OOXML), or OASIS’ Open Document Format (ODF).

The Controversy

The most capable specifications are OOXML and ODF; they can store dynamic information such as the contents of a spreadsheet, unlike PDF documents. Unlike the PDF specification which describes how to render the document, the other two formats are nominally XML formats, which describe the document structure while allowing the software to decide the best way to render or display the data.  While the ODF format is technically superior to OOXML, Microsoft’s support for ODF is very poor.

The PDF Format

The chief benefit of he PDF format is that it has been around for a while, and there is no chance that you will be unable to open a PDF document.  However, the PDF format is based on a printing format; it is not designed to store spreadsheets with calculated cells, for example.  Moreover, as a publishing format, it is really optimized for generating printed documents.  It is not conducive for producing well rendered documents when read on displays with non-standard displays such as cell-phones.

OOXML

The chief benefit of OOXML is that it is supported by Microsoft Office.  The problem with it is that nobody else supports it, and because OOXML is suffering from the same creep that plagued older versions of Office files.  The OOXML files produced by Office 2007, ofr example, are not standard OOXML files.  Moreover, the OOXML format is deeply flawed;  in an effort to maintain backward compatibility with office components that can’t read XML documents.

The end result is that software developers who do not work for Microsoft are struggling to produce software that can read OOXML files.  This lack of a broad adoption means that Microsoft will face little opposition if they should unilaterally alter the file format in future versions of Office.

ODF Format

The ODF format was originally invented by engineers working at Sun Microsystems.  It is a truly open specification in that the published specification is complete and fully describes how to write and read documents.  Moreover OASIS has a good system for inviting people to submit recommendations for improving the standard and for reviewing the changes.  As  a result it has supplanted the original native formats of many non-Microsoft office suites such as OpenOffice or KOffice.

The benefit of saving files in the ODF format is that once a file is saved in this format, you will be able to open the file at any arbitrary date in the future.

The major problem with the format is Microsoft’s hostility to it.  Microsoft’s business model is dependent on people routinely upgrading Microsoft Office and Microsoft Windows.  Unfortunately for Microsoft, OpenOffice is a full featured and no-cost to use office suite.  In order to limit the adoption of this free rival, Microsoft has set upon a strategy of non-interoperability.  They reason that so long as alternate programs are incapable of reading Microsoft Office files while Microsoft Office programs are incapable of opening files produced by rival programs, there will be large disincentives associated with abandoning Microsoft Office. Their native support for ODF is intentionally very  poor.  I strongly expect that they are going to attempt a version of their famous Embrace, Extend, Extinguish strategy.  However, this strategy is probably doomed to failure, because there exist several plug-ins that allow a person to save ODF files from within Microsoft Office.

My Recommendations

In my experience, Microsoft file formats are a poor long-term storage option.  Microsoft cannot resist modifying their formats in unpredictable ways.  The ODF format is what a industry standard should be.

I recommend that people either use OpenOffice, or download and install and use Sun’s ODF Plugin for Microsoft Office.

• The Problem
• Microsoft Office Add Ons and Customization
• Database Programming
• Data Analysis

The Problem

A teacher wants to assign letter grades to his students, but does not want to do it by hand. Nor does he want to use a difficult to decipher, complex formula built out of the formulas that are built into Excel.

The Data

The roster of students is:

Back to Top

The school has a policy regarding letter grades:

Back to Top

Writing a Custom Function Using Visual Basic for Applications (VBA)

There are several tricky ways to use the built in excel formulas to generate a letter grade when input a numerical grade, but they are all complicated and easily broken. The most elegant way to handle this is to write a custom function.

We open the Visual Basic editor,

Tools -> Macros -> Visual Basic Editor

We select

Insert -> Module

In the module we type the following:

Back to Top

Using the Function

At this point, we can type in the name of the function, LetterGrade in the formula of any cell. For example, if we wanted cell C2 to contain a letter grade based on a numerical grade stored in cell B2, we would set C2‘s formula equal to ‘=LetterGrade(B2)‘.

Back to Top

Downloads

Click here to download an Excel spreadsheet demonstrating this function. Please feel free to adapt this code to your needs.

Back to Top

The conventional wisdom for a firm entering troubled times is to cut back on spending and to focus on making it through the down-turn, like  a turtle withdrawing its head and limbs into its shell until the downturn has passed.  When most businesses apply this principle to their IT systems, they usually decide to stop all spending on new software, or software upgrades, and to focus instead on spending the minimum necessary to keep existing systems going.

The justification of such a policy is simple;  if it ain’t broke, why fix it?  Most people reason that since they are turning a profit now, there is no need to spend any additional money on their systems. Unlike a delivery van which will break down if its engine isn’t routinely serviced,  a computer program will work indefinitely.

The Cost of Shoddy I.T.

If builders built houses the way programmers built programs, the first woodpecker to come along would destroy civilization.
Gerald Weinberg

The phrase I would use to describe most I.T. systems is jury-rigged.  Most businesses figure out some way to use their computers to support their business on their own, and generally the software packages they use, and the procedures they use to operate them are poorly thought out and inefficient.  This is not surprising, if most businessmen designed and constructed  their facilities from the ground up, bypassing those pesky, expensive architects, engineers and foremen, the result would be rickety buildings that tended to fall down alot.   Because the collapse of a badly designed building is a great deal more devastating than the failures of a poorly designed software set up, most businessmen recognize the need to pay for the facilities work, while remaining oblivious to the money that their inefficient I.T. setup is costing them.

Hiring professionals to design and set up a business’ I.T. systems is not panacea either;  most computer programmers are oblivious to the harm and inefficiencies as well.  An engineer who designs an unreliable electrical system that occasionally catches fire risks the near certainty of being fired and on the street with an unemployable reputation.  A computer programmer who produces a unreliable computer program which crashes frequently – on the other hand – often finds that to be the best guarantor of a regular pay-check.  This phenomenon is widespread enough to provide fuel for one of the funnier I.T. related sites on the Internet, The Daily WTF.

The Benefits of Good I.T.

When done well, however, good I.T. can make a business far more agile than all its competitors.  It can take a lackluster business model or process, and make it the subject of admiring reviews in business journals.  The reason is quite straightforward: just as an atl-atl gave the stone age hunter the ability to fell wholly mammoth, a well designed I.T. set up can ensure that the people executing the busienss processes are extremely productive.  A good I.T. system can ensure that a worker is earning $60.00 an hour, while his identical twin struggling with a poorly designed or integrated system stuggles to produce $45.00.

The Difficulties in Reliably Implementing Good I.T.

There is no sure-fire way to reliably develop a good I.T. system.  A bewildering variety of methods and schemes, described in hundreds of bestselling books, claim to describe a sequence of steps, which if followed slavishly, will result in a cost-effective, usable and productive system for your business.  These claims are to put it politely exagerated.   Complex systems, of course, do benefit from a formal, rigorous approach to allow people to design the system without being overwhelmed by its complexity.  For simple systems the benefits of the formal methodology are outweighed by its cost, making such efforts unjustifiable from a profit/loss perspective.

Rationally Examining Whether to Upgrade or Not

The decision to upgrade depends on two factors:

• How much money you can save, or extra money you can earn…
• The likelihood that the project will produce the desired savings or extra revenues

Calculating Savings and Costs

In a downturn, the firm which cuts its costs the most effectively will probably thrive.  It  will be the most capable of reducing its prices while maintaining a profit margin.  Then as the downturn drives out the marginal suppliers, and prices start picking up again, it will be well positioned to expand and grab marketshare.

The question that the decision-maker  should be cosnidering, therefore, is how can I quantify the costs and benefits of maintaining the status quo to improving your systems.  Some measures to consider are:

• Costs
  • Man hours consumed using the system.
  • Lost business or production due to shortcomings of the system
  • Implementation costs
  • Lost business or production due to training
• Benefits
  • Improved business or production.
  • Savings due to better reliability

Using the numbers generated by such an exercise, you can then calculate an upper ceiling to how much a better system can improve your bottom line.  If this number is sufficiently large, then one can consider the likelihood of success.

The Probability That the Project Will Meet Expectations

Typically software projects are oversold; it’s tempting for a developer to promise everything that he thinks it will be feasible for the software to accomplish.  Almost inevitably, these promises prove to be too ambitious, and the buyer is left trying to use a lemon that they wish they had never purcahsed in the first place.

Just as there is no surefire process to produce good software, there is no surefire process to quantify this risk.  For a company that is experienceing consistently high profits, taking on an unquantifiable risk is not necessarily a problem.  For a company struggling to maintain profitability, the failure of a risky project can be the kiss of death.

However, we can roughly gauge the risk of engaging in a project by examining the following criteria:

• Is the project an incremental improvement or a radical “great leap forward”.
• How much of the project is dependent on custom programming and how much of it is dependent on use off-the-shelf components?
• How radically will the work-flow of users be altered?

Risk Matrix Graphic

Generally, any project should be structured to minimize its riskiness.    Radical changes should be avoided in favor of incremental ones.  Instead of writing a custom application from scratch, take an existing application, and if needed customize it.  Rather than changing the work-flow dramatically, make improve the current workflow.

If I were to guess which of these risk factors is the most underappreciated one, it is the latter one, user worflows.  Often an idea that seems wonderful to management and to the developers will turn out to be a real turkey, because it does not meet the needs of the people on the shop floor. If the users can’t use it, then the project has failed.

Analyzing the Worst Case Scenario

One helpful exercise when considering a proposal is to lay out the worst-case scenario.  By qunatifying the losses the firm will suffer should the project turn out to be a failure and comparing them to the status-quo,  a decision-maker can make a more informed decision concerning the risk involved. Moreover, the decision-maker can take steps to minimize the harm of the worst-case scenario.

For example, our customers’ worst case scenario typically consists solely of the loss of money paid for us to do the project.   Typically we structure our contracts so that our customers pay no money until they judge a milestone to be completed fully to their satisfaction, with no payment required until a working protoype has been demonstrated.  These terms transfer the costs of failure to us, the people performing the work.

Typically, vendors are quite willing to accept this sort of risk during a down-turn.  After all, they are suffering from a slow-down too.

Making the Decision

A down-turn can be the a good time to make improvements.  Vendors are competing for your custom.  A succesful project can ensure that your firm will be one of the ones that best weather the storm.   By realistically and critically examining the costs and possible benefits of a project using the criteria discussed here,  you can identify opportunities to improve your firm’s bottom line, even in the worst of times.

Of course, you don’t have to make the decision yourself:  being customer focused, we aren’t afraid to examine a project proposal and honestly to tell a prospective client that the project is not worth their time, even if it would be very lucrative for us.

In this week’s newsletter, we will show how to analyze and graph some lab data that might be collected in a typical high school physics lab.  We will give an overview of several useful features built into OpenOffice Calc.

Entering the raw data:

As ever we start with our raw data.  While allowing a glider to slide down a nearly frictionless air-track, a student collects the following observations of its speed:

The first step is to create a new spreadsheet and enter our data into it. We rename Sheet 1 to ‘Raw Data’, and type in the data into columns A and B.  If you wish to skip this step, you can download a spreadsheet containing the unformatted raw data here here.

Formatting the Data

Initially, the data is hard to read.  The numbers show different number of digits  after the decimal point.  Nor are digits aligned properly.  We fix this by editing the formatting of the cells.

We want the time values to be listed to tenths of a second.  We want the velocity values to also be listed in tenths of a cm/sec.  We alter the size of the font to make the data more readable. We then adjust the column widths to be appropriate for the data.  Finally, we add borders around the data to make it more readable.

Inserting the Chart

We wish to produce an v vs t chart.  We highlight all the data, including the labels, and select Insert → Chart.  We use the Chart Wizard to define the chart we want:

1. ‘Chart Type’ dialog: we select ‘XY(scatter)’, specifying that we are interested in a scatter plot containing ‘Points Only’
2. ‘Data Range’ dialog: we select $’Raw Data’.$A$1:$B$12 as our data range. We specify that the data series are in columns and that the first row of each column is the label of that data series.
3. ‘Data Series’ dialog: we specify the data ranges that contain coordinates and labels for the data series we wish to plot.  fortunately, the wizard will do all the work for us, and we will not need to alter the settings on this screen.
   The wizard will detect that we only have two columns of data, and will assume, correctly, that the leftmost column, our time values, is the independent variable, which we will use as the x-coordinates in our plot.  It will assume that all the columns to the right of the first one are the data series providing the y-coordinates for the plot.  In this case, since we only have two columns, so the wizard  only generates one data series for our graph.
4. ‘Chart Elements’ dialog: we specify the title, subtitle, and axis labels of the chart.  We also specify that we want grid-lines for  both the x and y axes.

Once we have finished with the wizard, we have a chart, but it is not very presentable. Cut and paste the plot onto a new worksheet. Drag the edges of the plot to fill up the visible region of the worksheet. Rename the worksheet ‘Plot’.
We then double click on the chart to edit it.

We double click on chart elements to adjust their appearance. We alter the appearance of the data series, making the data points smaller. We alter the ranges of the axes to improve the appearance of the display.

We can see the results here. I encourage you to download and experiment with the enclosed spreadsheet.

Analyzing the data

From the plot, we see an apparent linear relationship between the velocity and time. In order to identify the parameters of the linear relationship, we perform a linear regression. You can observe the work we did by looking at the enclosed spreadsheet.

We create a new worksheet to hold the data, and duplicate our raw data by building a table of cells that are set equal to the corresponding cells in the raw data.

We may not want to include all the data in our analysis. After all, one or more of the data points may have been taken in error. So we will add a column that specifies whether or not the data will be used in the analysis. Since we will want to eventually add a trend-line to our plot, we skip column C and store “Y” values in column D.

We then create a conditional table, which has the original raw data if the ‘include’ value is set to “Y”, and leaves the cell blank if the value is set to anything else.

Finally, we perform our linear regression analysis. We make use of the OpenOffice functions SLOPE and INTERCEPT to find the parameters. We also use the function CORREL to find the correlation coefficient for the linear regression (R).

We use the slope and intercept to calculate the values that are predicted by the parameters. This is added as a data series to the chart.

To add the data series, we double click on the chart to edit it, and then right-click on the plot area, and select ‘Data Ranges’ and add the range as shown in the video.

We edit the range to add a line between the data points, and remove any data point markers.

From the experimental data, it is clear that the last observation was made at a point where the glider was no longer traveling freely. We therefore decide to discard it from the regression analysis.

We are then done. We have experimentally found the line of best fit through our data, and have a nice plot showing our data.  Thie final spreadsheet may be found here.

I’d like to take a little time to review the various programs available.

Text editors

Text files are the simplest way to store a list. The benefits of storing data in text files are manifold:

1. The size of the file is minimal. In the modern era, where megabytes of disk space are priced in terms of cents, this is no longer much of a concern. However, if you are paying per kilobyte to store or transfer data, then using text files can save you money.
2. The data can be read by any program. Once every few months, I speak to someone who has a problem; a critical document is stored in an old version of some obsolete file format, and they need to convert it to a modern document. The conversion is usually difficult and expensive. While use of open source programs like Open Office, which have a pretty good library of converters can mitigate the problem, but why take the risk? Why go through the panic and headeaches and lost productivity when a little forethought can avoid it?
3. The data is human readable. This can be very handy if, for example, you need to examine the raw data for whatever reason.

A text editor will produce text files. It requires minimal memory, won’t crash, and is very fast. Modern text editors have some pretty amazing features, they will highlight text according to rules you specify, and can perform some very dramatic sophisticated operations on the text, such as sorting or complicated search and replace operations.

The drawback with a text editor is that it cannot, generally, be used to perform sophisticated operations like filtering and performing calculations on data. So if you need to calculate totals, or want to perform queries on your data, a text editor is not a good idea. Moreover if you want to format your text, the

Word Processors

Word processors are a step up from text editors in that they permit you to format your list.  Most word processors now support macros which allow you to write and execute custom programs that manipulate the list or perform actions based on the contents of the list, such as sending out emails to everyone in a list.  However, all of these extra features are also found in spreadsheet programs.  As a result, a word processor is generally not the best tool for maintaining a list.

Spreadsheet Programs

Programs like Microsoft Excel, Openoffice Calc, and Quattro Pro are generally the best option for working with lists.  Spreadsheets are designed to store lists of data and to perform calculations on elements of the list.  All modern spreadsheet programs allow you to format the entries, use formulas to calculate values, embed macros that automate the lists, to perform sophisticated sorting and filtering of the data.

The major drawback of these programs is the amount of data they can store.  Microsoft Excel, famously limits a list to 256 columns and 65,000 rows.  Furthermore, when a great deal of data is loaded into them, operations like searches can be very slow.

For most business tasks, though, these limitations are not an issue – and spreadsheet programs are, as a result, the workhorses with respect to storing lists.

Desktop Database Programs

The next step up in power or feature sets may be found in desktop database programs like Microsoft Access.  These programs permit you to store vast quantities of data for long periods of time and to perform any imaginable type of analysis on it.

The price that one pays for this power is in ease of setup.  Unlike a list stored in a spreadsheet or in a text file, format changes can be very difficult.  If one does not invest thought into designing the database, the result can be comething that is less usable than a spreadsheet.

Another major drawback of desktop programs are that they don’t easily allow multiple people to share the same data.  Additionally, most desktop based programs suffer from size limits.  For example, Microsoft Access 2003 has a 2.4 GB limit on the amount of data it can store.

Server Based Databases

Server based databases are the most powerful, most advanced class of programs used to store lists.  These programs permit you to store unlimited amounts of data, allow multiple people to access the data simultaneously, and allow you to perform any conceivable form of analysis on the data.

As with desktop based databases, a significant amount of thought must go into designing the database.  In fact, unless you have previous experience with database design, you should hire a professional to design the database.  Preparing a well designed and properly tuned database on a server can easily cost $10,000 (in 2009 dollars).

Choosing the Right Program

In general, you should use a spreadsheet program like Microsoft Excel.  If there is a substantial amount of data to store, or if you want to have multiple people to work with the data simultaneously, or if you need sophisticated operations to be performed every time new data is entered, then you should use a database program.

If multiple people need to have access to the data, then you should use a server based program.

If you are unsure, start storing your data in a spreadsheet.  The spreadsheet will allow you to make changes easily as you work ou the most useful scheme for storing and using the information.  When the data becomes too large for the spreadsheet, or your demands for reports or data analysis exceeds the capabilities of the spreadsheet program, you can build or comission the construction of  database to store the data.  It will be a very simple task to import the contents of the spreadsheet into the database.

Skype allows you to easily communicate with other team members that are online, either by turning your computer’s microphone and headphones into a telephone, or through the ‘chat’ feature.  It also allows you to transfer files from computer to computer.  It is free, works on all major computer operating systems, and for a small fee can even be integrated with your cell phone. It is reasonably secure;  the data transmitted from your computer to the computers of the people you are communicating with is ecrypted, and then broken up and sent via different paths.  There are some experts, however, who have strong concerns about its security.  However, these concerns can be mitigated – soemthing I shall discuss in a future post.

It is such a good program that many professional podcasters and radio announcers have started using it as their preferred method for having guests phone into their shows. This has happened with little fanfare; if you listen closely, you can hear the audible event notifications that Skype makes in the background. There are also a wide variety of third-party applications that enhance Skype, allowing you to use it as a white-board, or to publish your desktop.

The simplest way to set up Skype for your business is to establish a business account with them.  A business account allows you to purchase credits from Skype and distribute them to your staff.

I strongly encourage people to examine the program and consider whether they can use it to improve their productivity.

Introduction

Microsoft Office 2007 comes with a radical redesign of the User Interface. People who are used to older versions of Microsoft Office often panic when confronted with the new look and feel of their applications.

Today, I’d like to provide people who are using Word 2007 and who are used to older versions of Word with a map of where various common commands are located.

Using Help

The first step is to master the help system that is built into Word. You can do this by clicking the help button.

Figure 1: Finding the Help Button;

This brings up a basic screen. You can select from the list of the most commonly requested items, or you can type in what you are looking for. For example, if you wanted to convert some tab delimited text to a table, you could type “text to table” in the search field, and would receive the following results:

Figure 2:Help System Search Results;

As you can see, the first result is what you are looking for. This is the simplest way to find the obscure menu commands you used to use. It can also lead to wonderful discoveries of features and tools you didn’t know were available. So I highly recommend that you use the help feature as frequently as you can as you get to know the new Microsoft Word.

Mapping Old Commands to New Ones

However, the help system can be cumbersome, especially on a “slow” computer. So let us look at the most frequently used commands.

The first thing you might notice is that the file menu seems to have disappeared. This is quite a shock to people because they don’t know how to open or save a document. It hasn’t disappeared. It is now contained in the Microsoft Office Button. Clicking it reveals many familiar commands:

Figure 3: Microsoft Office Button

Note how everything is categorized. Commands that are used to publish a document are found in the “Publish” sub-menu, for example. The “Save As” sub-menu allows you to directly save the file in the most common formats, or to look for plug ins that allow you to export to more exotic file formats. etc.  The commands are:

• New:   This option allows you to start a document from scratch.  Clicking it will open a window containing links to your template library.
• Open:    This option allows you to open a preexisting document. Clicking it opens the Open File window.
• Save:  This option saves the document.
• Save As:  This option provides you with a list of popular file types that you can convert the word document, including formats that are readable by Word 2003.   It also allows you the option of searching for plug ins that convert Word Documents to more exotic formats.
• Print:  Allows you to see a preview of the document, or to send it to the printer.  The main difference between Print and Quick Print is that Quick Print will use default of previously selected printer settings for the document.
• Prepare:  If you are intending to send your document to outside organizations or to make them available to the general public, you will want to use the commands in this group.  Here you can edit document properties, inspect the document for private meta data, digitally sign a document, finalize a document, and examine it for compatibility with earlier versions of Word.
• Send: Allows you to send a document via email or via internet fax.
• Publish: Allows you to upload the document to a blog, document management server or to a document workspace.  This is, of course, very useful if many people are collaborating to produce the document.
• Close:  Closes the document.