Software, Software Development and other Atrocities

I’ve been reading about structuration theory from Giddens (The constitution of society, 1984). This theory tries to describe and explain the nature of social phenomena, like human actions and interactions. I came across two interesting concepts: practical and discursive consciousness. Practical Consciousness is the ability to act according to one’s stock of knowledge (to act in a knowledgeable way) and Discursive Consciousness is the capacity of articulating one’s knowledge (expressing what we know). I find these ideas very informative at explaining some communication issues in software development. Being able to perform actions does not mean that one can describe them. The ability to articulate ideas is a higher level skill that needs reflection from users and developers. I can relate this concept to two aspects of software development.

First, from my readings in online forums I see that developers do not follow formal methodologies. They use what they call “common sense” and perform the activities the way they think is best. That is, they can act in a knowledgeable way (Practical consciousness). However, few of them will admit that they follow any methodology at all. I say that whatever you are doing, if your actions are based on your knowledge about the situation and software tools and if you have organised these actions in a sensible fashion, then you are following a methodology. The problem as I see it is that (most) developers are incapable of expressing and more importantly justifying their selection of practices in a rational way other than “it is common sense”. Applying some reflection on their jobs could help developers to explain why and where they think the practices work better and thus help their colleagues and justify their actions with management.

I can also relate these concepts to Ethnography (a method developed by anthropologists used to observe people performing their everyday activities). The acknowledgement of a difference between practical and discursive consciousness could justify developers working with users and the use of ethnographic methods in ISD. Asking users about what they do would not be enough (as we are not sure about their discursive consciousness). Rather observing people performing their activities as they normally do would add more value to the gathering of requirements.

It is interesting to see how the news about the london attacks are being spread in weblogs, wikis, flickr, etc. Many victims and witnesses have been posting their pictures and personal accounts on the web. I am amazed by the speed and variety of information that we can get now. Appart from the official sources, BBC, Reuters, etc. which are more cautious at releasing information, now we can have a look at more personal and maybe realistic accounts from people who were there.

10 years ago, this wouldn't have happened like this. I think that the web and social software, as instatiations of the globalization phenomenon, are not only helping us to communicate better but to create awareness in the civilized world. We are more united by the web now, and I hope we can do better and stop war and terrorism.

Have a look at:
http://flickr.com/groups/74918957@N00/pool/
http://www.flickr.com/photos/verseguru/24243212/

George G. was a trainee who was brought into a diagnostic prgramming group with all signs pointing to an outstanding programming career. He had just received a Master's degree in Mathematics with honors, and only two of the other seven team members had even gone to a university. The team was working on a real-time diagnistics for a military computer, and George was assigned to write the diagnostics for the high-speed drum, which was a critical part of the system. Although this was his first real program, he developed an ingenious scheme in which the program wrote all ones on the drum and then, by analysing the patterns of zeros read back, could indicate the exact circuit that was failing.

George received high praise for this work, praise that he didn't need because he was fully aware of how superior it was to the unimaginative work of his fellows. Indeed, George lost no opportunity of telling them about their shortcommings, as reflected in the clumsiness of their program when compared with the grace of this. In a very short time, a serious crisis was brewing in the team, and several members began to look for a way to push George out - while several others, of a different bent, began to seek new positions for themselves. Then fate intervened in the form of a visit from some military brass to inspect the new hardware.

Great preparations were made for the inspection, but when the brass arrived, the computer could not be made to function. The diagnostics had all been run successfully, but the other software would not work at all. The diagnostics were rerun, but again they worked and the regular programs did not. At this point, the military was getting rather petulant, and management was growing furious. Then, somebody chanced to walk behind the main frame and noticed a cable lying loose. When he casually inquired what it was, the problem was solved. The drum had not been connected to the system!

The drum was hooked on, and the military was soothed with a fine demonstration and a lunch well oiled with marinis. Long after they were gone, however, the programmers on the team were still getting theirs back at George. He had failed to take into acount the behavior of the drum interface when the drum was not on-line. The system was designed to give an interrupt if an unattached device was selected, but George had masked that off. If the interrupt was masked, a write was sumply ignored and a read produced a string for all ones. Thus, George's marvelous program had a simply indicated that all was well with the drum, even though it was not even attached.

It was a blunder that any programmer could make, and it was easily patched up. Not so George's ego, for the other programmers took every possible opportunity to rib him. Perhaps if he hadn't claimed such infallibility, his fall from the heights might not have hurt so much; but as it was, George couldn't take it. After two weeks, he simply failed to show up on Monday morning, and that was the last time anyone ever heard of George G."

From "The psychology of computer programming" (1971) by Weinberg, G. M

I have met some Georges at work ... what about you?

Most developers state that it is very difficult to communicate and work with users who do not know what they need. We always complain about users who specify mutually exclusive requirements, change their minds everyday or argue among themselves. We would prefer to work with people who can present clear and detailed requirements... oh yes, that would be heaven. mmmh, However, I think I'm not being fair with users. Some of them really do know their business and are aware of the technical implications of software design. Sometimes the problem is not the user but the developer. Sometimes we underestimate our users and assume that they know nothing and simply need to accept what we are going to develop for them. The problem is that we *don't listen* to them. And if users are not listening to us, meetings become conversations between deaf. This is the worst way to start a software development project.

I was wondering if there were some guidelines on how to deal with/understand users. So far I've read ideas about putting yourself on user's shoes or learning user's language. But, how do you do that? Maybe, if we sorted this communication problem out, we could reduce the level of complexity of software development. Any ideas?

I found this definition of complex in the dictionary: " ...having parts so interconnected as to make the whole perplexing." Complex is related to complicated, which "stresses elaborate relationship of parts." Anyway, for me, a situation (a problem, a business setting, users, etc.) is complex (in the context of software development) when it is not that clear to the developers' eyes and when it cannot be modeled and transformed into unambiguous programming code. There are many things that can make a business situation complex for developers, and I am of the opinion that most of these things are human and social in nature. The technical aspects of businesses are in a way less complex as there is help from formal methods (e.g. software engineering) that have been proved in the field many times.

So, what are the things that make developers rant? Well, I have a short list of things may be familiar to you:

- Excessive and unnecessary control of information. As developers we build a mental picture of the business processes the software is going to serve. This picture is often different from the one users who like to control everything have. For developers that extra control is not part of the "real process" and only adds more complexity to it.
- Users working in different ways and expecting different things from the same system.
- In social processes many outputs can come from one input.
- Free social interactions, users who do not follow formal procedures but who use their brain to decide what to do on each situation.
- Covert channels, informal activities --> things that users do in real life but which are very different from what's in business documentation (i.e. business rules)

I am not saying that if we changed these things, organisations will be clearer to developers' eyes (mmmh, possibly but that isn't my point). I think that there are reasons for them to exist, otherwise the organisations themselves wouldn't exist. What I am saying is that we need to find a way to deal with complexity in organisations, and so far, we are not doing that well.

How do you design social software?
Do you interview users, draw uml diagrams and then create prototypes?
As the user of Social Software is the group and not the individual, How do you account for group interactions and group goals?

So far what I've read in forums is that someone has an idea, any idea... takes the risk and develops it. The software is thrown out to the market and people wait to see what happens. Somehow the software is successful, thousands of people are using it and are sending feedback to the creators. Here we have a decisive point in which the creators decide on the future of their yet successful software. If they want to continue in the market they have to listen to user feedback and alter the software to improve the most wanted features and fix or remove the least wanted. At the end, the software is being used by millions of people, it allows them to communicate as they wish and to create a social network that accounts for their goals as individuals and as a group.

Seems nice and easy, but yet.... this looks like an accident, a fluke, really good luck.

If I wanted to create a business application, something that allows people to work and interact... I would really want to use some ideas from social software. (For me social software is every software used by people.) The problem is that, as this is a relatively new field, there is very little on this yet. ... and I am still wondering about how to adapt software development practices so they take into account the social aspects of the business settings in which the software is going to be used?

Social software is software that enables and/or encourages social behaviour of certain type. It is both, an old and a new concept. Social software has been around the software development field for decades but known by different names, like groupware or CSCW. (You can learn more about the history of Social Software in Christopher Allen's excellent article Tracing the evolution of social software.) Social software is new in the sense that it is an attempt (the first?) to recognise the social aspects of software. So far software and software development have been guided purely by mechanistic perspectives. I don't think this is completely wrong because software is also 0s and 1s in computer chips. However, there is also another side of software which has been neglected, the human and social one. Software is part of our daily lifes, we read our e-mail every day, we read the newspaper online everymorning, we buy online, cellular phones, play station, etc. ... and in business settings it is part of our jobs. Software plays an important role at supporting the social interactions in organisations. Regardless of the type of application, payroll, ERP, workflow, groupware, e-mail or online forum, there are always *people* using the software and possibly interacting through them.

For me, when software developers develop business applications they need to consider two aspects of their users, the user as an individual and the user as a group. So far the user as an individual has been addressed in the guidelines for usability. There are very well known and proven practices that allow developers to create userfriendly, intuitive and attractive applications, that users will be willing to use. However, I haven't seen guidelines for *group-sability* so far. I think that the emergence of social software as a new field would bring more development in this area.

Off the top of my head--> I am not sure if prototyping would be useful for *groups-ability* as it is for usability design. It is easy to sit with a user and design a screen in front of him/her and see whether s/he thinks it is easy to use or not. However, how could we *see* the social interactions happening when a group of people is using a software? I think Ethnography could give us some answers in this regard... (but off course who's gonna pay for an ethnographer to be observing people for weeks or even months while you could be delivering software and charging for it?) Probably an adaptation of ethnographic practices could help, some sort of ethnography which is led by feedback from users.

Adaptability in the Software Development process and in Software itself is one of the main concerns of my research. From my readings about theories of organisations I gather that as time passes modern organisations look for more inventive ways to compete in the market. As this happens, the software that are being used in the organisations become outdated because the organisation needs are changing constantly. So my question is, what could we do to keep the software development machine aligned with the organisation's evolution? I think that an exploration of the concept of adaptation within the context of software development would be useful to find the answer. From my readings in forums I gathered the following ideas:

In relation to adaptability of developers and the development process:

1. The process of evolution in software development practices follows this pattern: Experience --> Reflection --> Adaptation
2. Iterative and incremental processes are designed to allow adaptation.
3. Involving clients/users in developers work helps the process of alignment.
4. There is no single methodology that works in every situation. Developers have to adapt known methods to the problem situation and their clients' working styles.
5. Developers need to change their "attitude" towards development. Accepting that change is inevitable and that requirements will change many times during a project may help developers to adapt easily.
6. If developers thought about software as a service provided by people and not as a product, it could be easier for them to think about adapting software (i.e. their service) to evolving organisations.

In relation to adaptability of business settings and clients/users:

1. Users can adapt better (to technology) when they have already experienced change. After one successful experience further changes are easier to take as people are more receptive to them.
2.The level of adaptability of users to software and of software to the business situation could be improved if users had the facility of maintaining/changing/adapting their software to their needs. (An utopian idea at the moment but may be possible in the future.)
3. Organisation's culture can allow or impede technology change. New, agile methods for developing software are more difficult to be adopted in companies with endure strict and heavy processes.

In relation to the adaptability of the software itself:

1. Software should not last forever.
2. Software adaptability depends on the level of dependence of the organisation on the software. If the software is a core part of the business processes then it is more likely that it will be evolve with the organisation. However if the software is just a support tool then it is less likely that it will be adapted to changes.

I've been marking assignments for a week now. This is so mindnumbing... I will never be a lecturer!

In recent years many agile methodologies have been created to respond to the weaknesses of the traditional or waterfall life cycle methodologies.

The main philosophical underpining of the agile methodologies is to walk beside organisation's evolution. As modern organizations evolve and adapt to their environments rapidly so should software development.

Traditional development methodologies (waterfall lifecycle) were created upon the assumption that requirements never change. Software were created based on a snapshot of the organization's situation. Projects (price, time and design) were planned in advance and any deviation from them were considered a mistake which had to be corrected.

Agile methodologies are based on the assumption that organisations, users, and their requirements change. Agile methodologies are designed to adapt to changes: they are selfadaptive. To achieve that, users and developers work together and expect change all the time. Requirements, specs and code are reviewed constatly and changed when necessary. The aim is for a good quality software that meets the needs of the clients and not for meeting the deadlines and not running over the budget.

Today I've been reading blogs and forums about agile methodologies. So far this is what I've been able to gather:

1. Agile methodolgies prefer "Responding to custumer feedback" strategies reather than "Planning ahead" ones.
2. Agile methodologies are suitable for Incrementalists/realists kinds of developers. Incrementalists like to split software in bits and develop the most important bits first. They have an idea about what is achievable in real life in a certain period of time.
Traditional methodologies prefer Completionists or idealists because they can see the "big picture" (but fail at implementing because they have to account for every single detail in the design). In traditional methodologies every step has to be finished before attempting the next one.
3. Agile methodologies require more flexible customers capable of dealing with "uncertainty". Customers will receive the software in bits and will not have an idea about how the final will look like until the end of the project. They will give feedback to developers constantly and will work with them to decide on the course of actions.
On the other hand some developers say that agile methodologies are more "predictable". This is because it is more probable with agile that the final result will meet the customer needs. With waterfall one cannot predict if the final product will meet customer expectations, especially because the specifications are written so much in advance. One learns new and important things as one progresses in the programming.
4. Agile (XP), is not a cheaper alternative to waterfall. It was created to provide 100% customer satisfaction, that is, the customers gets what s/he wants. The final product is "good enough" to satisfy users.
5. Applications developed with agile approaches are more flexible and less resistant to change because of the constant additions and refactions during development time.
On the other hand because of constant change there is code in the applications that is no longer being used but rarely removed.
6. Agile works best in corporate solutions or business applications with big databases, where users can be consulted as opposed to shrinkwrap applications which can be more generic and where there are no users to talk to.
7. Agile are better fitted to ill-defined problems.
8. In agile methodologies design and coding are merged.
9. Most agile methodologies don't address maintenance which according to some (Boehm) is 80% of the SDLC.
10. There are two types of documentation, documentation for now (first development) and documentation for later (maintenance). IMO some agile methodologies like XP consider only the first type and neglet the other.
11. Most used techniques in agile methodologies -> pair programming, refactoring, test driven development, nunit testing... By the way, pair programming is the most mentioned by advocates and retractors. Some developers feel their productivity increases when they work in pairs, some others find it difficult to work with other people.

Hace un rato escribi un mini ensayo a cerca de algo que me pasó hoy en la mañana. Resumiendo, algunos estudiantes me agotaron la paciencia + terminé cansadísima después de la clase e incapaz de pensar hoy en la tarde. Tengo mucho trabajo que hacer en mi doctorado y me da colera no poder avanzar. Justo en este momento estoy analizando datos, pero mi cerebro no me da. Decidi borrar el ensayo, siguiendo el consejo de un viejo.... amigo. (mas sabe el diablo por viejo que por diablo). Parece que el ensayo estaba un poco... fuerte. Ejem.... la vaca no se acuerda cuando fue ternera. Cuando era estudiante de pregrado o maestría me parecia muy fácil quejarme de los profesores y de los tabajos (que en realidad son un chancay de a 20 comparado con el doctorado). Ahora que me toca enseñar ya se lo que se sufre para hacer que un estudiante aprenda, sobre todo los que no estan motivados (y tambien los brutos).
Ya se que la audiencia de mi blog es 1 o 2 personas, pero aún asi decidi borrar la entrada anterior... no vaya a ser que en algunos años me haga famosa y alguien me lo saque en cara. :p

Me voy, creo que ya me esta entrando la inspiración.

I started my PhD thinking on Decision Support Systems (DSS). I’d been working on one in my previous job. Having found so many difficulties while gathering information (interviewing people and reading documentation), and designing the system, I thought that it was a good idea to investigate better development practices to do this. My initial work was on describing the kind of settings I had found during my years as a developer. To do this I first identified the problematic factors that made analysis and design so complex. This is what I found:

• There was a great variety of activities. Those varied from academic related issues, like module planning and student complaints to administrative issues such as budget preparation and marketing promotions. There were a number of people who combined their academic and administrative functions in an unclear fashion.

• Users performed similar processes in different ways among the university areas. The personal styles that dominated shaped the development of the processes.

• Processes changed through time specially when the authorities of the university were replaced.

• A decision making process may be initiated by any member of the administrative or academic staff in the university or by students.

• The nature of the problem and the working styles influenced the way people would carry out the decision making.

• Most of the processes involved flow of a huge variety of documentation like reports with information extracted from the databases (tables, charts) but also of documents containing unstructured information such as texts with decision makers’ opinions and decisions.

• Users worked alone, in groups or in multiple groups depending on the convenience and availability of the interested.

• The university board and other committees meetings were, in some cases, the last stage in the process. However, most of the hard work was performed before they met.

• There was a requirement of recording all the data resulting from the decision making processes that were carried out at the highest levels of the university, as it was specified in the university regulations. However, this need was also perceived in most of the medium level staff as they believed that this information may facilitate their work in the future.

• Information was required in most of the stages of a process.

well well well I got bored..... zzzzzzzzzzzzzz

To be continued…

I’ve done software development most of my professional life. Now I am doing a PhD in the same topic. It is a challenge, but it is also rewarding. When I started I had no clue about how to develop an idea worthy for a PhD nor did I have a clue about how to investigate it. Thankfully, I found my way through it. I decided to study the social aspects of business organisations that affect the process of software development. To do this I designed a qualitative research. As I am very fond of internet and computers (and not so much of libraries, books and essays), I planned an online ethnography on software developers, where I could chat with peers about their problems at work. Now, I have finished my fieldwork and I am doing data analysis. I guess that it will take me two months or so to finish it. After that I will start to write my thesis.

I will use this blog to tell you about my life as a PhD student, to comment on my findings about complexities in software development and to share my experiences at doing online ethnography.

Hoy, 04 de abril de 2005, día memorable, en el cual inicio un blog memorable, (ejem, ...eso espero)
Today, 4th of April 2005, I start this (I hope) memorable blog.