Going forward with PHP

I run a number of websites which provide a daily update of a shareprice together with the value of the shareprice converted to a loical currency (e.g. £, $, Aus$). When I started, I selected Perl as my scripting language which provided me the necessary facilities (the LWP package) to retrieve a shareprice from the Yahoo finance site. I used the URL

http://finance.yahoo.com/d/quotes.csv?f=l1d2c1p2&s='.$stock_symbol

as my data page which returned the share's closing price and other useful information.

I also generated a graph which showed the historical share price, relative to one of the major indices (e.g. FTSE, CAC40).

As I couldn't host my scripts on my web site, I was required to run a number of Perl scripts to generate updated files and then upload the files to the webhost. I wrapped the scripts in a DOS batch file and set up a scheduled task (under Windows) to run the scripts every weekday after the markets had closed.

This worked very well, most of the time. However it suffered from a number of problems. If I wasn't around, and didn't turn my PC on, then the scripts didn't run and the shareprice wasn't updated. I also suffered from some problems with my web connection which meant that the scripts didn't run as expected and I obtained a zero shareprice.

I had been wondering how to make the system better and less dependent on my PC being turned on. Most ISPs that I have experienced don't allow the LWP package to be installed as part of the standard Perl installation, presumably based on security concerns. I therefore considered alternative scripting languages. I had already seen Python at SPA2008 and was keen to try and see Python was a suitable replacement. However, my web host didn't offer Python in its standard configuration, but did offer the other P language, PHP. 

PHP treats a URL as a filename and does not appear (in the standard configurations installed on webhosts) to suffer from the security constaints that Perl installations suffer. A simple function retrieves the values for a stock symbol returning the parameters in an array.

function get_share_price($stock_symbol)

{

$url="http://finance.yahoo.com/d/quotes.csv?f=l1t1d1c1p&s=".$stock_symbol;

$filesize = 2000;

$handle = fopen($url, "r");

$arr = fgetcsv($handle , $filesize , ',');

fclose($handle);

return $arr;

}

This resolved the problem of updating a daily share price, and also offered the opportunity for offering live share price updates (subject to the market delays which free services such as Yahoo suffer - at  least 15 minutes delay).

To resolve the historic share price required PHP again but this time with a different Yahoo feed which I had only recently discovered, the ichart interface which returns the data as a stream of  CSV separated values. 

function plot($numdays, $stock_symbol)

{

// Get current date

list($e,$d,$f)=split('-',date("j-n-Y"));

$d--; // Months are 0-11

$yahoostr="http://ichart.yahoo.com/table.csv?s=".$stock_symbol."&a=0&b=1&c=2003&d=".$d."&e=".$e."&f=".$f."&g=d&ignore=.csv";

$arrResult = array();

$min=1000;

$max=0;

$handle = fopen($yahoostr, "r");

if( $handle ) 

{

$n=0;

while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) 

{

if ($n == 0)

{

$n=1; // Ignore header row

}

else

{

$arrResult[] = $data;

}

}

fclose($handle);

// Select data subset

$arr1=array_slice($arrResult,0,$numdays);

// Determine chart scale

foreach ($arr1 as $data ) 

{ 

$min = min($data[4], $min); 

$max = max($data[4], $max); 

} 

// Data is currently stored Newest-> Oldest.

$arr = array_reverse($arr1);

// Now plot data (date in element 0, value in element 4).

}

}

In processing the data, I discovered a number of additional PHP features, including

• array_reverse -to reverse an array (required because the Yahoo data is returned with the most recent data first)
• array_slice -to take a subset of an array
• min, max - to determine the minimum and maximum values (how many times has code for this suimple function been written!)
  

This meant that I had the data in a format which I could then process to produce a graph (dynamically). Now the next question was to select a suitable graphing package which offered a good PHP interface. But that is another story.

The Power of the Door

I have just read an interesting presentation on the rules of productivity. It presents 8 rules to determine the most productive number of hours per week (40), the type of office environment (team rooms) and how a team should be located (non-siloed - multi-disciplined). In most cases the results are possible counter-intuitive to most management but each of the results are backed up by sound evidence.

It made me think about my experiences on numerous development projects and I would tend to agree with the recommendations outlined in the presentation. I remember when I was working long hours AND studying for a post-graduate degree trying to write some very simple Eiffel code and failing to get it work after 2 hours of staring at a simple logic problem. I simply couldn't solve it at the end of a 15 hour working day. I went home and came back a few days later, refreshed. I solved the problem in 5 minutes. The lesson was clear to me then - you need sleep, not heroes. This was admirably demonstrated in an overnight session to get a demo working and after 8 hours through the night of being in a no better situation than we were when we started!.

As organisations have changed over the years, office space has become at a premium. The 'power of the door' was demonstrated when I was a young engineer doing a major retargetting exercise. I remember we had to convince our manager of the benefit of having our own server to work on this exercise - he agreed provided we could reduce the schedule by 6 months. This we did easily, not just because of having our own server but because we had a large wooden door on our small team office. If it was shut (which it was normally) people just walked past so you didn't get disturbed. I would guess that if we reverted back to small offices, the software industry would be much more productive than it is today. I wonder how many project  managers have the power or insight to challenge the office environment and make the necessary changes that will increase the chance of project success and increase team morale. 

The role of Project Managers in Agile Projects

I recently attended Allan Kelly's presentation on 'Why and How to Become Agile', an event organised by the BCS Project Management Specialist Group. As you would expect from an experienced agile practitioner, Allan provided a good overview of what agile was and why he considered agile to be better.

However, given that the audience was mainly project managers of one sort or another, I was interested in his statements on the role of project managers in agile developments. Mainstream agile methods such as XP, SCRUM or Crystal are very silent about the role of the Project Manager. This doesn't mean that they can be dispensed with, it is just that the various agile developments concentrate on the approach to improving business value with a development rather than the associated management tasks. I know agile promotes self-organising teams (in my experience this is removing a hierarchy of developers, architects and testers) but I know of few organisations that don't allocate a project manager to a development project however small. I therefore advocate that the project manager role remains as critical as ever although the scope of some of his tasks may change.

Regardless of the development approach, all projects need to manage risk, budgets, communications and resources (physical and people). The classic management approach is for these tasks to be allocated to a (often dedicated) project manager so that the workers can get on with serious development work. I don't think there is any need for this approach to significantly change. One task which Allan indicated would change for the project manager was planning as the emphaisis on planning changes in agile developments to be much less formal. This may be appropriate for (small) purely software development projects but the vast majority of projects are multi-disciplined in which the dependencies between the various activities need to be accepted and understood by all parties regardless of the formality of capturing this information. The project manager's role still remains an important factor in the eventual success of the project and the choice of project manager is probably more important than ever. In my experience the best project managers for developments adopting some agile practices are those who are hands-on, are developers who have experienced agile first-hand, understand the project and are empowered to make decisions. It is the last point which mustn't be under-estimated; delays in decision making processes are classic signs of a development that is struggling and agile developments can't afford unnecessary delays.

I note that the latest DSDM version (Atern) now explicitly includes the role of a Project Leader (you can even get a qualification). Does this now recognise that the classic project manager now needs to more formally recognised in agile developments? I would probably say no; it is probably more a reflection on the type of organisations using DSDM who feel comfortable with an explicit role being defined rather a comment on the management of agile developments in general.

Now that agile development approaches have become more accepted as a 'normal' way of performing software developments, it is probably true to say that the role of the project manager has survived relatively intact. However the project manager now has a key role in being much more involved with the development rather than performing a purely managerial overview role; but this is what the best project managers have always done.

Agility for Complex Systems

At SPA2008, there was much debate amount the use of agile practices in developing software.

A comment that I heard was 'that for all the noise that the agile community makes us believe that agile is mainstream and the most common approach in developing software today, the reality is somewhat different'. Certainly there are many good examples of where agile approaches have been successfully applied but I guess that many of these are in 'young' and 'smallish' companies and not in large, corporate companies which have many decades of history.

It is not easy to convince these large and well-established companies that there are alternative ways of developing software. One reason, is often the 'process' police. Large companies, in my experience, are very keen on following a tried and tested approach, with some limited tailoring to cater for different sizes of developments.

So what are the challenges in applying agile techniques to complex systems?

The Relentless March of the MicroChip

I have just returned from the inaugural Kilburn Lecture which concluded a day celebrating 60 years since the first stored program computer (the Manchester 'baby'). The excellent lecture given by Professor Steve Furber gave a historical perspective of the major innovations which have originated from Manchester University over the last 60 years as the technology used in computing has developed together with a personal view of the developments which he has been personally involved.

Over the last 60 years, the technology has changed from the vacuum tubes used in the Manchester 'baby' and the Ferranti Mk1, the first commercial computer, through the transistor, which although invented in Bell labs in 1947 wasn't adopted for computers until the 1960's when the Atlas computer was developed, the fastest computer at the time, to the integrated circuit used in MU5 (a forerunner to the ICL 2900 series), Dataflow and Amulet systems. The Atlas computer also introduced the concept of the single level store, which is more commonly referred as virtual memory which one of the attendees at the lecture, Professor Dai Edwards, remarked that he still received payments for this invention. Each decade has seen the level of complexity increase as the number of transistors has increased which shows no sign of slowing down.

Steve also provided a personal perspective of his involvement in microprocessor design starting with the BBC Micro of 1982 whilst at Acorn. While the BBC Micro was primarily built from off the shelf components including the 6502 microprocessor, Steve designed two simple bespoke chips which helped reduce the chip count by 40. The success of the BBC Micro and its design led to Acorn developing further bespoke chips resulting in the Acorn Risc Machine (ARM) being released in 1985. This is probably one of the most significant microprocessor developments over the last 25 years as derivatives of this processor have now become a significant component in mobile phone technology. The ARM machine had a simple design, was small and had low power consumption and was really a System on a Chip (SoC). By 2008, there have been over 10 billion ARM processors delivered, making it the most numerous processor in the world. When Steve moved to Manchester University in 1990, he continued to use ARM technology in the AMULET system. Over various generations of AMULET, the size of the chips didn't change but the chips became more complex as the transistor spacing reduced from 1 micron in 1994, to 0.18 microns in 2003, which is smaller than the wavelength of visible light.

There was an interesting comparison of the changing energy requirements over the last 60 years. in 1948, the Manchester 'baby' required 3.5 KW of power in order to execute 700 instructions per second which represents 5 Joules/instruction. Contrast this with the ARM968 processor on 2008, which requires 20 mW of power and can execute 200 million instructions per second which would represents 0.0000000001 Joules/instruction. This represents a 50,000,000,000 times improvement! - there are few examples of such a dramatic improvement in energy efficiency. Steve did give a warning though as he stated that more efficient computing can lead to increasing power requirements.

No lecture on the development of processors can ignore Gordon Moore's seminal article (colloquially known as Moore's Law), and this lecture was no different. The original paper published in 1965 predicted the exponential increase in the number of transistors per chip only until 1975. However, this prediction is still true today and has become a self-professing policy and has become a key input into planning next generation microprocessor designs (see International Technology Roadmap for Semiconductors). The reduction has been achieved by shrinkage in transistor size, cheaper components and reduced power consumption. Steve cited that the current generation of microSD cards, which provide a flash memory of 12 GB contain over 50 million transistors in the size of a fingernail, demonstrates how much progress has been made since the original use of transistors in computers in such systems as Atlas. However, exponential progress cannot go on indefinitely and there are now some physical limits which will constrain progress. As components have increased in complexity, their reliability and lifetime (in many cases less than 12 months for some items) has reduced as the tolerances on such a great number of components are very fine. There is also a recognition that the cost of design to achieve advances in technology is becoming uneconomic as it is increasing at 37% per year. Steve considers that Moore's Law may survive for another 5-10 years with current technology but there is will need to be advances in alternative technology for it to continue beyond this (and there is no signs of this happening at the moment).

The current generation of microprocessors, dual core/multi-core, have tried to address the increasing constraints by selling more processors per microprocessor rather than a single faster processor. Moors' Law can also apply to multi-core systems however there is an increasing problem with the use of such processors by application software. As single processors have increased in performance, there has been little change to the way software has been developed. However, with the advance of multi-core technology, general purpose parallelism is required in order to maximise the available processing capacity. This is one of the 'holy grails' of computing and it is becoming an increasingly important problem to solve. The use of such multi-core processors also needs to be carefully considered. It would appear that is preferable to have lots of cheap (and simple) cores rather than a small number of faster (and complex) cores due to the significant power efficiency differences.

Steve finished by giving us a glimpse into the future as hew showed how microprocessor design was converging with biology. The human brain has many attributes which are similar to the requirements of a complex network of computers e.g. tolerant to component failure (e.g. loss of a neuron), adaptive, massively parallel, good connectivity and power efficient. Steve's current project SpiNNaker is trying to build a system which can perform a real-time simulation of biological transactions mapped onto a computer architecture. The project is using 1000's of ARM processors and is trying to meet one of the UKCRC's Grand Challenges in which the architecture of the mind and brain are modelled by computer.

Assessing code quality

How do you assess a software modules's quality? It is a question I have been struggling with for some time as I try to perform a peer review of a large code base.

Over time, a software module evolves from its intended form to something less than beautiful as bugs are discovered (and fixed) and enhancements over the original requirements are implemented. This is particularly true for code which is developed on a multi-person project, where personnel can change and often a module gets changed by different engineers. Although I adhere to the rule, that the code structure should reflect the original author's style (and how many people change the comment at the top of the file to identify that they have been one of the author's? This assumes that this information isn't automatically added by the configuration management system.), it can become increasingly difficult to make changes.

So what is the best way to assess code quality throughout it's development?

It's good to be free

I have just returned from the BCS Manchester event on The Free Software Movement in Ethics and Practice presented by Richard Stallman, the founder and president of the Free Software Foundation. Richard, speaking for over 2 hours without notes or any visual aids, captivated the audience of over 250 with his passion about establishing a world in which a user's freedom is respected.

Traditional software was compared on it's functionality, cost and practicality and never considered 'what does the software do to my freedom?'. Conventional software does not respect a user's freedom as the user is utterly helpless as he can't change it (because he hasn't the source code) and the user has to use the software in the way that the developer deems appropriate.

To set the context, Richard identified four freedoms which are required for software to be free:

• Freedom 0 - Being able to run the program as you wish


• Freedom 1 - Being able to study and change the source code to do what you want


• Freedom 2 - Being able to help your neighbour by distributing the software


• Freedom 3 - Being able to contribute to the community by sharing and distributing the modified/evolved code to neighbours.

Any software which does not meet all of these freedoms, should be considered proprietary and could be considered unethical as it isn't contributing to society.

Freedom 0 is essential so that you have control of your computing. There are many applications which restrict who (named users), where (specific computers), when (whilst valid licence held) you can run your (purchased) program. The control is often not via a license but via specific code which determines what can and can't be done.

Freedom 1 allows to determine what the application is doing to you. There are many instances of applications which spy on you; this is a growing trend particularly with the always connected society connected to the Internet where you are offered (or in some cases not) updates to your software. Failure to comply with this freedom demonstrates that the applications are actually controlling you and not working for you. Examples such as Digital Rights Management (DRM) and Microsoft Vista are tremendous examples of advances in restricting users! As there are millions of users who can never use this freedom, as they don't have the necessary skills (i.e. they aren't programmers) and so much software, Freedom 3 was defined to recognise the need to work together as part of a community.

Freedom 2 is a basic moral right which proprietary software prevents. There is much propaganda with regarding distributing copies of software by calling this piracy. Therefore helping a neighbour is equated with attacking a ship; Richard thought that pirates used guns to ply their trade rather than software, so it is a very strange use of the word pirate!

Freedom 3 is all about working together and to allow co-operation as changes evolve. This results in co-operative progress regardless of the original intent of the software.

With software conforming to the 4 freedoms, everyone gets the benefits and society reaps the rewards. The software is developed under the control if its users with no single user having all the power. Free software cannot have any malicious features; if such a feature were added, someone else will take it out (and the originator will get a bad reputation).

After extolling the virtues of the 4 freedoms, Richard moved on to explore the Business View of Free Software and how businesses can benefit. An instinctive view is that free software could be bad for business, particularly software developers. This is not true, because whilst there are a small fraction of businesses developing software who may be affected by the availability of free software, there are many more businesses who use software which is vital for their survival who can benefit hugely and take advantage of the 4 freedoms. One area where businesses can benefit is in the area of support and services. With proprietary software there is an essential monopoly with little incentive to fix problems etc (typical response - we have a new upgrade in a few months time, see if that has fixed your problem (and introduced a few more)); for free software there is the option for free market support as well as a choice of paid for support from multiple suppliers.

Free Software does not remove the need for custom software development as software development (specifically programming) is a small proportion of the overall software development cost. Free software could actually generate jobs in the adaption and extension of existing software, which may be commissioned and paid for, even though the resulting software is to be subject to the 4 freedoms.

A clear distinction was made between Free Software and Open Source Software (OSS) as there is often the perception that they are the same. OSS was introduced as a distinct phrase in 1998 to distance some software from the user's freedom. OSS has different philosophical values to FSF and does not offer anything specific to respect a user's freedom.

Richard's presentation was extremely compelling and thought provoking, and whilst noting that user's freedom isn't easy to respect, there are approaches which can lead to a better society.