Web Security - an eyeopener

An impromptu Birds of a Feather (BoF) session given by Dave Misell at the SPA Conference admirally demonstrated what happens behind your back when you visit a website. Key to the demonstration was showing (very simply) how much data is passed between sites using cookies and how much data is retained (for considerable length of time in some cases) between site visits.

The session used the Paros proxy server with Firefox which easily showed the web transactions when entering a simple URL (I won't disclose the URLs used in the demonstration, suffice to say that the sites were well-known). Using Paros, it is easy to see how much information is passed via cookies. Stopping cookies results in the same information being passed via URL (although this is more obvious since the data appears in some form in the address bar). In many cases the same data is passed to a number if websites regardless of whether the data is appropriate or useful to the receiving website.

Although it is possible to see the information being sent to a website, it is not possible to determine what is done with the data by the receiving site. In most cases the data is used to trace a journey through website (so that the 'user experience' can be improved) which is totally transparent to the user since these are performed server side.

So is it possible to stop the information being transferred? Not easily, but choose your sites carefully. There is some legislation in the EU which has tightened up the exchange of information through websites, particulalry to third parties, without the express permission of the user. Unfortunatley this legislation is dependent on where the website is hosted, which isn't always obvious from a simple URL. There is always 'education' which applies to both the end user and also to the developers of the site, and there are some good courses now which can help (e.g. ethical hacking and MSc in Information Security at Royal Holloway College in London) increase awareness.

Can you scrum on your own?

An interesting question posed by Peter Bell at this year's SPA conference. His premise was that some of the lean and agile approaches can be equally applicable when you are THE team. He called his approach Solo Scrum.

Peter identified a couple of scenarios where he felt that 'Solo Scrum' could be appropriate:

• A full time developer doing open source development part-time
• A solo consultant developing applications for non-technical clients
• A 'work at home' contractor managing multiple clients

Clearly working on your own brings a number of challenges (note that I didn't say problems!) which need to be considered when working on your own. I think that probably one of the biggest challenges to overcome, (and this is based on personal experience), is to try and avoid too many distractions within the 'home' work area. The second challenge is that of loneliness! In the many (professional) development projects that I have worked, talking to colleagues has been a great way of confirming your thoughts for a solution or helping to find a way through a problem. When you are on your own, this isn't possible, or is it? However the use of technology can certainly help (think of IM, email, VOIP (with video)) and it can certainly be seen as 'virtual' pair programming. This isn't a new idea (see here); however, there is still the problem of finding a suitable 'pair' to link up with.

The traditional problems of planning don't go away when working on your own (as I said at the session 'Fail to plan, plan to fail'). Working on your own necessitates an efficient process for planning and managing the plan (a bit more than the 'back of an envelope' note). There are a number of Eclipse plugins (e.g. mylyn) which offer the right balance in terms of monitoring the work activity without distracting from the task-at-hand.

Some interesting ideas were discussed, including the observation that many team collaboration software packages often includes a free version for a small team (where small can be up to 5). This can overcome a often heard comment which was that I don't have the time/experience/etc to set up a 'professional' development environment (the minimum being a source control system, an IDE and a bug tracking system) from scratch.

Peter has now started a Google group to further the debate. I will watch with interest.

SPA2009

Having attended last year's conference and thoroughly enjoyed it, I was disappointed that my work commitments meant that I could only attend a single day of this year's conference. Attending a single day meant that I was keen to catch as much of the buzz as possible as well as catching up with some friends. I managed to attend 2 sessions, on AJAX web testing using Selenium and Solo Scrum, together with 2 BoF sessions on Web security (or lack of it) and Lean/Kanban in a stimulating day. I will summarise my notes over the next few days.

Case Studies in Enterprise Architecture

A few nights ago, I attended a fascinating evening with Wayne Horkan, the CTO of Sun Microsystems for UK and Ireland, who presented his views on Enterprise Architecture. The event was jointly organised by IET, the BCS Manchester branch with a pitch for the recently formed BCS Enterprise Architecture Specialist Group forming the warm act before Wayne's presentation.

Wayne presented his experience in 3 (anonymous) organisations who experienced difficulties in their approach to delivering an enterprise architecture. The problems could easily be addressed by recognising the key and important role of enterprise architects who maintain an understanding of the business context, provides domain knowledge and ensure that the architecture and supporting IT remains aligned with the business. 

In assessing any programme, a simple 4 stage approach is followed:

1. Gather the facts by a series of interviews and information gathering. This includes understanding the current 'as is' enterprise architecture, the current IT/IS setup and the 'vision' for the future.
   
2. Analyse the data
   
3. Determine the key conclusions/recommendations and the way foward. This includes establishing the order of change to ensure that the changes deliver value to the organisation without destablising the exsiting organisation.
   
4. Assess the project against the five key areas of likely project failure:
   

• Strategy. Is it really needed? Why is it needed now? Will it deliver a realistic ROI?
  
• Contractual. Is the contract, including the financial elements, appropriate defined and achievable?
  
• Process/procedures. Are we in control or out of control?
  
• Politics and personnel. Is there buy-in from all concerned/affected?
  
• Technical and architectural. The least likely cause of project failure (apparently) but need to ensure that the right resources are available in order to deliver the right solution.
  

Wayne concluded with a number of best practices based on his experiences with these (and other) organisations:

1. Ignore the religious arguments about which framework to use. Just pick a single framework and stick with it. Most organisations use a blend of Zachman (artefact focused) and TOGAF (procedurally focused). None of the frameworks provide a magic bullet with which success is guaranteed.

2. Obtain sponsorship from key stakeholders. The stakeholders must have a strategy for the business, which may change over time, that is communicated to the EA team.

3. The EA team must continually:

• Remain delivery focused, providing value to the organisation

• Involve the whole team (consisting of Enterprise, Infrastructure and Application architects) in all decisions, ensuring that cross-team communication is integral to the team (no silos)

• Ensure that there is the right mix of technical skills to deliver the right architecture

• Obtain and foster Sponsorship, Sponsorship, Sponsorship without which a programme is likely to fail

• Keep aligned (and checking) with the business strategy

As a reality check, Wayne highlighted that the current economic climate now requires that the investment in enterprise architecture must be seen to deliver value far more quickly (i.e. in the matter of months rather than years) to the business than has previously been tolerated.

The session was well attended and left the audience with much to consider in future projects.

The slides, and Wayne's personal views on the session are contained here.

Going forward with PHP

I run a number of websites which provide a daily update of a shareprice together with the value of the shareprice converted to a loical currency (e.g. £, $, Aus$). When I started, I selected Perl as my scripting language which provided me the necessary facilities (the LWP package) to retrieve a shareprice from the Yahoo finance site. I used the URL

http://finance.yahoo.com/d/quotes.csv?f=l1d2c1p2&s='.$stock_symbol

as my data page which returned the share's closing price and other useful information.

I also generated a graph which showed the historical share price, relative to one of the major indices (e.g. FTSE, CAC40).

As I couldn't host my scripts on my web site, I was required to run a number of Perl scripts to generate updated files and then upload the files to the webhost. I wrapped the scripts in a DOS batch file and set up a scheduled task (under Windows) to run the scripts every weekday after the markets had closed.

This worked very well, most of the time. However it suffered from a number of problems. If I wasn't around, and didn't turn my PC on, then the scripts didn't run and the shareprice wasn't updated. I also suffered from some problems with my web connection which meant that the scripts didn't run as expected and I obtained a zero shareprice.

I had been wondering how to make the system better and less dependent on my PC being turned on. Most ISPs that I have experienced don't allow the LWP package to be installed as part of the standard Perl installation, presumably based on security concerns. I therefore considered alternative scripting languages. I had already seen Python at SPA2008 and was keen to try and see Python was a suitable replacement. However, my web host didn't offer Python in its standard configuration, but did offer the other P language, PHP. 

PHP treats a URL as a filename and does not appear (in the standard configurations installed on webhosts) to suffer from the security constaints that Perl installations suffer. A simple function retrieves the values for a stock symbol returning the parameters in an array.

function get_share_price($stock_symbol)

{

$url="http://finance.yahoo.com/d/quotes.csv?f=l1t1d1c1p&s=".$stock_symbol;

$filesize = 2000;

$handle = fopen($url, "r");

$arr = fgetcsv($handle , $filesize , ',');

fclose($handle);

return $arr;

}

This resolved the problem of updating a daily share price, and also offered the opportunity for offering live share price updates (subject to the market delays which free services such as Yahoo suffer - at  least 15 minutes delay).

To resolve the historic share price required PHP again but this time with a different Yahoo feed which I had only recently discovered, the ichart interface which returns the data as a stream of  CSV separated values. 

function plot($numdays, $stock_symbol)

{

// Get current date

list($e,$d,$f)=split('-',date("j-n-Y"));

$d--; // Months are 0-11

$yahoostr="http://ichart.yahoo.com/table.csv?s=".$stock_symbol."&a=0&b=1&c=2003&d=".$d."&e=".$e."&f=".$f."&g=d&ignore=.csv";

$arrResult = array();

$min=1000;

$max=0;

$handle = fopen($yahoostr, "r");

if( $handle ) 

{

$n=0;

while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) 

{

if ($n == 0)

{

$n=1; // Ignore header row

}

else

{

$arrResult[] = $data;

}

}

fclose($handle);

// Select data subset

$arr1=array_slice($arrResult,0,$numdays);

// Determine chart scale

foreach ($arr1 as $data ) 

{ 

$min = min($data[4], $min); 

$max = max($data[4], $max); 

} 

// Data is currently stored Newest-> Oldest.

$arr = array_reverse($arr1);

// Now plot data (date in element 0, value in element 4).

}

}

In processing the data, I discovered a number of additional PHP features, including

• array_reverse -to reverse an array (required because the Yahoo data is returned with the most recent data first)
• array_slice -to take a subset of an array
• min, max - to determine the minimum and maximum values (how many times has code for this suimple function been written!)
  

This meant that I had the data in a format which I could then process to produce a graph (dynamically). Now the next question was to select a suitable graphing package which offered a good PHP interface. But that is another story.

The Power of the Door

I have just read an interesting presentation on the rules of productivity. It presents 8 rules to determine the most productive number of hours per week (40), the type of office environment (team rooms) and how a team should be located (non-siloed - multi-disciplined). In most cases the results are possible counter-intuitive to most management but each of the results are backed up by sound evidence.

It made me think about my experiences on numerous development projects and I would tend to agree with the recommendations outlined in the presentation. I remember when I was working long hours AND studying for a post-graduate degree trying to write some very simple Eiffel code and failing to get it work after 2 hours of staring at a simple logic problem. I simply couldn't solve it at the end of a 15 hour working day. I went home and came back a few days later, refreshed. I solved the problem in 5 minutes. The lesson was clear to me then - you need sleep, not heroes. This was admirably demonstrated in an overnight session to get a demo working and after 8 hours through the night of being in a no better situation than we were when we started!.

As organisations have changed over the years, office space has become at a premium. The 'power of the door' was demonstrated when I was a young engineer doing a major retargetting exercise. I remember we had to convince our manager of the benefit of having our own server to work on this exercise - he agreed provided we could reduce the schedule by 6 months. This we did easily, not just because of having our own server but because we had a large wooden door on our small team office. If it was shut (which it was normally) people just walked past so you didn't get disturbed. I would guess that if we reverted back to small offices, the software industry would be much more productive than it is today. I wonder how many project  managers have the power or insight to challenge the office environment and make the necessary changes that will increase the chance of project success and increase team morale. 

The role of Project Managers in Agile Projects

I recently attended Allan Kelly's presentation on 'Why and How to Become Agile', an event organised by the BCS Project Management Specialist Group. As you would expect from an experienced agile practitioner, Allan provided a good overview of what agile was and why he considered agile to be better.

However, given that the audience was mainly project managers of one sort or another, I was interested in his statements on the role of project managers in agile developments. Mainstream agile methods such as XP, SCRUM or Crystal are very silent about the role of the Project Manager. This doesn't mean that they can be dispensed with, it is just that the various agile developments concentrate on the approach to improving business value with a development rather than the associated management tasks. I know agile promotes self-organising teams (in my experience this is removing a hierarchy of developers, architects and testers) but I know of few organisations that don't allocate a project manager to a development project however small. I therefore advocate that the project manager role remains as critical as ever although the scope of some of his tasks may change.

Regardless of the development approach, all projects need to manage risk, budgets, communications and resources (physical and people). The classic management approach is for these tasks to be allocated to a (often dedicated) project manager so that the workers can get on with serious development work. I don't think there is any need for this approach to significantly change. One task which Allan indicated would change for the project manager was planning as the emphaisis on planning changes in agile developments to be much less formal. This may be appropriate for (small) purely software development projects but the vast majority of projects are multi-disciplined in which the dependencies between the various activities need to be accepted and understood by all parties regardless of the formality of capturing this information. The project manager's role still remains an important factor in the eventual success of the project and the choice of project manager is probably more important than ever. In my experience the best project managers for developments adopting some agile practices are those who are hands-on, are developers who have experienced agile first-hand, understand the project and are empowered to make decisions. It is the last point which mustn't be under-estimated; delays in decision making processes are classic signs of a development that is struggling and agile developments can't afford unnecessary delays.

I note that the latest DSDM version (Atern) now explicitly includes the role of a Project Leader (you can even get a qualification). Does this now recognise that the classic project manager now needs to more formally recognised in agile developments? I would probably say no; it is probably more a reflection on the type of organisations using DSDM who feel comfortable with an explicit role being defined rather a comment on the management of agile developments in general.

Now that agile development approaches have become more accepted as a 'normal' way of performing software developments, it is probably true to say that the role of the project manager has survived relatively intact. However the project manager now has a key role in being much more involved with the development rather than performing a purely managerial overview role; but this is what the best project managers have always done.