You’re sprinting over the agile waterfall, wondering why you’re sitting down at your stand-up, retrospectively defining acceptance criteria with your scrum team while they burndown the backlog of critical blockers.
Like any industry, web development has evolved and matured over time. It’s only natural that web developers have come up with buzzwords to describe the concepts and processes particular to our field.
You might find this daunting, but if you’re investing in a new website, it’s important to understand what to expect, and your role in the process. This knowledge helps you get a new website delivered on-budget, on-time, and to your expectations.
With this in mind, this article gives you a bird’s-eye view of the essential web development processes. It focuses on clarifying the purpose and value of each step, as well as the day-to-day real-world activities that you, as a business executive, marketing manager, CEO or owner, can expect to be involved in along the way.
Table of Contents
Objectives Of This Guide
This guide will help you understand:
- the purpose and value of a well-defined web development process
- the key phases and activities in a well-defined web development process
- the different web development methodologies, how they affect workflow, and how they can be combined for the ultimate process.
Understanding these basic principles will provide you with the knowledge to:
- fully brief and effectively communicate your requirements to a web development agency so they can deliver a proposal and end-product in line with your vision
- benchmark your expectations of the web development processes
- understand your role in the project
- fairly evaluate an agency’s response to your brief, and understand the value of the activities that support the development work.
What Are The Issues?
There is a saying in project management that a project can be good, fast, or cheap – pick two! It is often said with tongue firmly planted in cheek and a knowing wink, but there is some truth in it.
The reality is a fair bit more involved than this. It is more like a sliding scale between the three optimal outcomes. But the glib saying neatly captures what you want to know when you engage a web development and/or web design agency:
- How long will it take?
- How much will it cost?
- What will be delivered?
The answers allow you to make informed decisions across a range of concerns, such as budgeting, return on investment, feasibility, risk management, staffing, scheduling, and marketing.
Sometimes, however, the simplest questions to ask can be the most difficult to answer!
At their core, websites are software and, unless your requirements are extremely basic, software development can be complicated and risky. As a result, the answers you want are not always immediately apparent even to skilled and experienced developers.
Without a clearly defined development workflow, you risk ending up with missing features, or features that don’t work like you thought they would. You might also see scope creep and insufficient stakeholder involvement – “No one mentioned the CEO wanted <feature> and now it’s the end of the project and we’ve run out of budget!”.
This often leads to late-stage expensive and time-consuming revisions and remediation work, which are stressful and grating to any client-agency relationship.
There is also the counter-intuitive risk of “over-baking” the approach. Workflows always introduce some overheads, and are only justifiable where the benefits of risk reduction outweigh the additional processes.
If you’re after something common and off-the-shelf, a full architecture and design phase might be overkill. On the other hand, diving headfirst into a complex and highly customised project without a structured approach is usually a one-way ticket to disaster.
What’s The Solution?
A structured web development workflow is essential for creating a website that meets all of your requirements and provides effective answers to the questions of cost, duration, and quality.
There are about as many workflows as there are web development agencies. Everyone will be quick to sell you on the benefits of their particular blend, so it can be difficult at first to separate the good advice from the fluff.
Experienced developers will be more open to tailoring a methodology and workflow from a wide range of industry best practices to suit your specific needs. They will also be able to explain the rationale and value of their approach as it applies to you.
To help you understand the value of some of the typical processes, this article starts with a primer on two of the more popular methodologies. It then expands on some of the common processes you may be pitched in your next website project.
Waterfall Vs Agile Methodologies
Before we dive into the specific steps of a web development workflow, it’s useful to have a basic understanding of development methodologies.
In the world of web development, a “methodology” is simply a philosophy or school of thought about the processes and workflows that best capture your requirements, reduce risks, and deliver a high-quality completed product.
It is important to understand, at least in principle, why these schools of thought exist and what they represent. Engaging a team whose philosophy does not mesh with your own will generate some level of friction and can often undermine the success of you project.
Waterfall and Agile are two of the more common methodologies.
The Waterfall Model
You may have come across the traditional “waterfall” methodology, so-called because of how the diagram looks if you map each phase flowing down into the next.
Waterfall is quite mature, and while the term itself was defined in the 1970s, the actual approach was common some decades earlier. Some may see the approach as a bit of a dinosaur due to its age, but it is still heavily favoured by many organisations due to its benefits.
In a waterfall project, each phase is completed before moving on to the next. It is very much like you would build a house in a particular order – you don’t build the roof until you’ve built the walls.
The waterfall model has clearly documented processes and toolsets and values structured organisation over individuals and interactions. This mitigates risks against shifting teams, which is especially valuable in large organisations where staff members may come and go over the lifetime of a long project.
It also values:
- early design and architecture definition, which sets effort and budget expectations early on, limits the risks of scope and feature creep, and mitigates the impact of hidden or unforeseen requirements on the success of a project.
- clearly planned development milestones, which allows organisations to effectively plan future staff allocations, communicate clearly with customers and management, and objectively forecast budgeting requirements.
In a waterfall project, you can expect to spend quite a bit of time upfront in discovering and documenting requirements as well as designing the solution before the developer even writes a single line of code. The web team takes a step-by-step approach, and clear deadlines and costs are set at the start and adhered to throughout.
The Agile Method
Due to its modern popularity, you may have already heard about the “agile” methodology. The term itself, however, seems to be a buzzword for many organisations who claim to be “agile” without understanding or implementing the method. At surface level, “agile” is a philosophy that seems almost directly opposed to the waterfall approach. As you will see later, however, there can be some value in a hybrid approach that combines some of the positive aspects of both approaches.
Agile processes follow similar steps to waterfall. Features are still analysed, designed, built, tested and maintained. But rather than tackling the entire scope of the project in each phase, the agile workflow operates on a smaller handful of features at a time.
In each stage of agile – the “Scrum” – features are arranged into mini-projects called “sprints”, usually lasting two weeks. The planning, development, testing, and delivery all occur within this sprint, before moving on to the next mini-project, which goes through the same process again.
The goal of each sprint is to deliver an incremental improvement over the previous iteration, ideally a working version of the software, for rapid testing and feedback before progressing to the next round of changes.
For example, one sprint for a website could be for a certain section of an overall website build, such as a customer portal. The web team implements the learn, plan, create, test, deploy stages within the sprint period, then moves to the next section of the website to repeat these steps. In the waterfall method, the customer portal would be planned, created, tested, and deployed along with the entire website.
As outlined in the Agile Manifesto, the agile methodology tends to value:
- individuals and interactions over processes and tools – the focus is on using the strengths and expertise of team members to drive development forward, increasing the opportunity to think outside the box and develop better solutions more quickly instead of forcing everyone to march along to the same beat
- working software over comprehensive documentation – allowing developers and other contributors to focus on building, adding features and delivering the product itself, rather than getting bogged down in endless planning
- customer collaboration over contract negotiation – which acknowledges the fact that you may not really know what all your requirements are at the start of a project, and aims to avoid friction by providing flexibility around specific deliverables and schedules.
- responding to change over following a plan – because even the best-planned projects will often have unexpected changes, and the focus is on delivering fast, incremental working versions of the product to allow feedback and changes to be captured earlier in the development process.
What you will find when working with an agile team is that there is much less upfront design and engineering than a waterfall approach. Instead, the focus is on hitting the ground running and building a working copy of the website as quickly as possible in smaller increments.
A Hybrid Methodology – The Best of Both Worlds?
Hybrid methodologies are a combination of two or more project methodologies. When designing a hybrid approach, you need to take a great deal of care to understand the specific drawbacks that you’re aiming to avoid. This way you end up with the best of both worlds rather than a mutated beast.
Waterfall and agile methodologies seem very much at odds with each other, so how could they possibly work in harmony? While both have clear strengths, these strengths can also be their undoing.
- Agile scrum deals with projects a single sprint at a time, great for rapid delivery of early working prototypes, but can struggle to capture critical “bigger picture” issues that could sink the whole project financially. Due to the reduced emphasis on upfront investigation and discovery, it’s very risky for developers to provide firm commitments on what they will end up delivering at the end of the project.
- Waterfall may seem superior in that the requirements are analysed in an objective and structured fashion before costly development begins. Unlike agile, however, you won’t receive a working version of your website until toward the end of the development lifecycle, which can be months. Once you actually have a chance to test and use the product, it may be that you need to change tack or rework a concept. Stepping back to an earlier phase to redesign, re-architect, and re-develop significant portions of the project can be a costly and time-consuming affair.
Waterfall is great at dealing with “known unknowns”, the unknowns that you can reasonably identify and solve during early planning phases. Agile, on the other hand is more suited to dealing with “unknown unknowns”, the surprise issues or blockers no ordinary human could have reasonably predicted during early planning phases.
A hybrid approach can combine the flexibility and dynamic nature of agile with the assurances and objectivity of waterfall. You gain the benefit of some foresight and planning, while still being able to take delivery of early iterations of the product as it is being built.
Ultimately, hybrid methodologies are not some “dirty” compromise on a “pure” approach. A well-designed hybrid approach is a valid methodology in its own right.
How to Choose a Website Project Methodology?
There are no hard rules other than to pick an approach that best suits the principles and structure of your business.
- Agile scrum works well on smaller projects with limited risk and scope. Delivery is expected within a few weeks or a couple of months, and you or your organisation is also operating in a flexible and agile manner when it comes to budgets and objectives.
- Waterfall suits larger, more complex, riskier projects that span many months or years, and generally is more favourable for organisations with strict procurement processes, auditability requirements, and change management policies.
- A hybrid approach can provide a good “best of both worlds” for both types of organisations. It gives some assurances around scope and deliverables while also offering flexibility and rapid feedback in the day-to-day website build.
If you are still wondering what might suit you the best, ask your web development agency to tailor a workflow to your needs. Have them explain to you why they have chosen their particular approach.
Fundamental Phases In A Web Development Workflow
Now that we have established a basic understanding of two of the more common methodologies and how they can work in unison together, we can talk about the fundamental phases in a web development workflow.
Despite differences in each methodology, the phases and associated activities tend to be consistent. You still need to go through the motions of understanding, planning, building, releasing, and maintaining code no matter how you shuffle the steps.
At a high level, the phases you should expect to encounter in this order are:
- Phase 1 – Gathering requirements
- Phase 2 – Website design & architecture
- Phase 3 – Website development
- Phase 4 – Testing and deployment
- Phase 5 – Support and maintenance
Some common activities that appear in these phases are outlined below.
Web Development Phase Activities
Phase 1 – Gathering Requirements
Whether it’s a formal Request For Quotation (RFQ), a statement of requirements, or just a detailed email, everything starts with the initial brief you give to your web agency. At a minimum, you should describe what you want, the “feature requirements”; why you want it, the “business requirements”; and when you need it to be done by.
Typically, the more detail you provide, the better informed your developer or agency is, which results in more accurate estimates and improved project outcomes. If your requirements are common, you should only need a short brief. If you’re unsure what detail you need to provide, most web development agencies are happy to offer an initial consultation to gain a broad feel for your needs.
After the initial discussion of the brief, it’s fairly typical for the web agency to prepare follow-up questions and presentations designed to clarify their understanding of your needs. This is an important step in projects of any size. It is always best to discover any missed or misinterpreted requirements as early as possible. Simpler projects may be confirmed via a basic phone call, while more complex projects may require a number of workshops involving key personnel to fully capture all requirements.
Regardless of the size of your project, the more input you can provide at this stage, the more effective your return on investment will be for the overall project, and the lower the overall risk of a hidden obstacle throwing your project into disarray.
Scope definition and proposal
Having come to understand your requirements, the agency will provide a contract – Statement of Work or some other document – that clearly outlines what they will deliver in exchange for your money. In a waterfall context, you should expect to see a list of deliverables that address each of your requirements to be included “in scope”. In an agile context, your developer may define these at the start of each sprint with the overall “scope” of the project much more open-ended.
The true mark of a professional web agency is the inclusion of documentation regarding:
- “exclusions”, things the developer explicitly won’t deliver
- “assumptions”, which show why they came up with their approach
- “risks”, which provide transparency around known risk factors.If you don’t get these assessments, ask for them so you can closely review and provide corrections or raise concerns if anything does not line up with your vision for the project.
Estimating and scheduling
By this stage, you’ll no doubt be quite keen to understand how much your project will cost, and how long it will take. The preceding activities will give your developer enough context to provide estimates on each feature. In an agile project, this may occur at the start of each sprint, while a waterfall approach will result in a broader project-level estimate alongside a complete statement of work and vendor agreement.
Depending on how complex your requirements are, it is not uncommon for these estimates to be further refined throughout the design and development phases of the project, as the understanding of the project grows or if requirements change.
Phase 2 – Website Design & Architecture
Sometimes referred to as digital design, graphic design, or user interface design, this is the process of laying out the user interface in a way that is visually appealing, accessible to all users, and cross-functional on mobile and desktop devices. The visual design can be limited by the technical capabilities of the underlying system, and likewise the visual design can inform the design of the technical solution.
A well-structured process will bring together the technical and design team at various touchpoints to ensure everyone is on the same page throughout the project. For more details on the design workflow, check out our article “Web Design Process: Unpacking The 5 Stages Of Optimal Design Workflow & Methodology”.
Use cases and user story definition
A use case is how your website visitors will perform set tasks on your website from start to finish. User stories are descriptions of website features from a visitor’s point of view, showcasing how it will meet its objective. Detailed use cases and user stories help to refine the understanding of what is required and what will be delivered. They focus on how each type of end-user will interact with the website.
This level of detail gives your developer information to more accurately estimate the effort and delivery schedule of your project. It also provides framework for User Acceptance Testing in Phase 4. This does overlap somewhat with certain website design activities, but is also immensely useful for technical design.
At the nuts-and-bolts level, websites are information systems that have some kind of database to store information in, and some kind of data processing to manipulate and present that information to end-users. Information architecture is a process that a developer does to design the “database stuff” in a way that most closely represents the information needed to meet your requirements.
Developers have a range of tools to visually map information and the relationships between pieces of information. These include entity-relationship diagrams, information architecture diagrams, and content hierarchy documents. These can help you understand the purpose behind each feature or element, and how this point can be reached.
Integration architecture and data flows
Due to your broader business processes, it’s quite likely that some of your data will not be managed directly in the website itself. It may reside elsewhere, such as in your Customer Relationship Management (CRM) system – Salesforce, Hubspot, or similar. If this is the case, your web developer is likely able to map out the integration of these third parties with your new website.
Well-designed integrations ultimately save you time and money as it means you and your staff don’t have to manually double-handle information between your different information systems and your website.
System architecture and hosting
You’ve probably already heard a lot of buzz around “the cloud”, and there’s plenty of marketing materials that try to convince you that web hosting systems are a universally “solved” problem that can be spun up at the drop of a hat. While there is some truth to this, websites are still computer programs, and as such they need some kind of computer to run on, no matter how it’s packaged up for sale. It may seem odd to start thinking about the kind of hosting your website will need right at the beginning of the project. You may even think that “hosting” is a non-issue or solved problem in web development these days.
Different websites and businesses will have different use-cases and need different hosting systems. Whatever your case may be, knowing what kind of systems a website will ultimately need to work on is essential to developing a finished product that works, and allows your business to function as needed online.
Revised estimating and scheduling
Armed with comprehensive designs and blueprints from this phase, your developer will have a much clearer concept of the effort involved in developing each component of your website. Now is a great time to ask for an updated estimate and schedule. For more complex projects, you should also get a revised set of risks, assumptions, inclusions, and exclusions.
Phase 3 – Website Development
This phase encompasses the nitty-gritty of developing the website. It’s where the programmers write the code, the database engineers build the data models, systems engineers set up the servers, and so on.
By this stage, if the above phases have been followed closely, the website developers should have all of the information they need to build, test, and deliver each feature.
Often website developers will specialise in frontend or backend development. At some agencies, one skilled person will be responsible for both.
Frontend Website Development
Frontend developers are responsible for building the interactive parts of your site. In other words, they take the visual design and makes it a fully functioning reality. They produce what the website will look like to normal visitors arriving on your site. This is a specialised discipline, which requires in-depth knowledge and understanding of Web Accessibility standards such as WCAG, ARIA, CSS and HTML, CSS and HTML, device and browser capabilities, and website profiling and performance tools. Websites that don’t have complex integrations or requirements may only need the services of a frontend developer.
Backend Website Development
Backend developers specialise in building custom data models, process automation, perform integrations with other information systems, and any other business logic that supports the operations of the website and the services it offers. Basically everything you DON’T see when you look at a website, but that should tick along effortlessly behind the scenes .Typically, a backend developer is well versed in:
- Scripting languages including PHP or NET
- Remote systems integrations protocols such as REST, SOAP, and GraphQL [https://graphql.org/]
- Database query languages like SQL
- Content Management Systems (CMS’s) such as WordPress or Drupal.
The list of acronyms could be as long as this article. Suffice it to say that to build a modern website that makes the final product look easy and seamless often takes a ton of work behind the scenes.
This may not be your first rodeo. If you have an existing website or web services, then you’re probably going to need all of the existing information transferred to the new and improved website. Data migration can sometimes be as easy as having someone draw the short straw and spend a few hours, or days, copying and pasting content, images, and resources from the old site to the new site.
If you have far too much information for this to be feasible, perhaps an eCommerce site with thousands of products, then you can use automated migration to streamline this migration. This process can take anywhere from a few hours to several weeks, depending on how similar the new website’s data models are to your original website, how much information has to be migrated, and whether it’s done by your staff or the web development agency.
Content production and entry
A new website with a new design tends to require new content, or a rethinking of how old content is structured.It is quite common for some web development agencies to take a hands-off approach to content. Essentially they will build your website but leave the content management up to you, which can be quite daunting for many companies. Other web development companies, including us here at Living Online, offer a range of content production and copywriting services if required.
Developing new content for your site gives you the chance to make sure this content is written with two end goals: to improve your website’s SEO, and to convince your website visitors to take action and become customers. A full-service digital marketing and web development agency like Living Online is a prime choice to ensure the content you are uploading to your new website contributes towards your overall ROI. It’s always worth checking whether or not your developer’s estimates include any content production.
DevOps, short for “development operations”, refers to a wide range of process automation, systems administration tasks, and other activities that overall serve to make the development process run more smoothly. For example, a DevOps engineer may put in place systems that more effectively manage how developers access and update your website’s source code, or configure automatic deployment of new website features to your testing, staging, and production (live) servers. A dedicated DevOps track may not be required on small and simple projects. On larger projects with multiple developers, however, the benefits quickly stack up.
Phase 4 – Testing And Deployment
Before a developer passes a new feature over to you for testing, they will often have another member of their team perform a dry run of the user acceptance tests. This process is called quality assurance, or QA, testing. For the frontend of your website, this can include everything you would see, such as making sure links and buttons click through to the right area; that all page elements look great across all devices and browsers; and that all forms and eCommerce functionality work as expected.
In the backend, this could mean a code review to catch any mistakes and potential issues before they end up being published on your live site. On smaller projects with smaller budgets, you might combine QA testing with user acceptance testing (see below), and take on the testing responsibility yourself.
User acceptance testing (UAT) and remediation
You and your team play the pivotal role in this step of the process, as you are essentially signing off on the successful delivery of the features you have paid for. During the requirements gathering phase, when a project is broken down into use-cases, each of the desired features will include “acceptance criteria”. Acceptance criteria is just a fancy name for a checklist of steps or conditions a feature must pass in order to be considered complete. Once everything is checked off the list successfully, the feature is ready to go.
If any of the UAT tests fail, your feedback is essential so that the developers can rectify the issues and resubmit for testing. Note that larger projects typically have a budget for remediation.
Performance testing and improvements
Before your new website goes live, you should ensure there is allocation in the project to stress test it. This step should push a website to its limits, allowing you to determine if it’s able to meet the expected demands of your customer base. This is mainly relevant to websites that expect large volumes of traffic. It will make sure that your prospective customers can still access it and complete their required actions no matter how many other people are on the site.
Performance improvements can be time-consuming and expensive if the end goal is unclear, so rather than just saying “make it faster”, be precise and data-driven in your requirements from the start. For example, “It must serve an average of 100 users per minute but accommodate a spike of 10,000 users an hour once per month. Average response time on a typical landing page should be 500ms even under maximum load.” If you’re unsure about these figures, talk to your website development agency.
Software like Google Analytics, which shows minimum and maximum website traffic numbers over time, can help provide insights into what your performance levels should be. The clearer your expectations around performance, the easier it is for web developers to engineer an overall solution that will hit your targets.
Your features have been built, tested, and accepted. Your marketing team is ready to pull the trigger on a huge campaign to advertise your new website to your audience. Before you give the go-ahead to push the big red button, make sure that your developer has provided a launch plan prior to and immediately after going live.
A launch plan should include steps that mitigate the risk of data loss, reduce the overall time your website needs to be offline, and an exit strategy that lets you get your original website back in place, if you have one, should anything go off the rails. It should also test key conversion points such as forms and eCommerce abilities, as well as SEO performance to prevent any loss to existing online performance.
Unfortunately, one of the main reasons we receive requests for help at Living Online is when a company has just had their new website launched by another agency, and all of a sudden, their SEO has tanked and they have dropped off the face of Google. This is incredibly costly, and is easily prevented by with a pre- and post-launch SEO audit to checks redirect, canonicals, meta directives, and other technical elements that come naturally for an agency with SEO on the mind.
Phase 5 – Support And maintenance
Your new site has successfully launched, but that’s not the end of the road. There are some follow-on activities you need to consider:
- Core upgrades and patching – it is important in today’s world of privacy and security concerns to always keep your website’s codebase fully updated.
- Incremental improvements – often in the rush to meet deadlines, certain requirements don’t make the cut for launch. These can always be introduced more gradually after your website has launched.
- Hotfixes and troubleshooting – while a solid QA and testing process greatly reduces the risk of post-launch issues, websites are complex beasts and you should always make sure your developer has processes in place for managing and prioritising critical issues as they arise.
The amount of post-launch support and maintenance is generally difficult to quantify in advance. Your developer should propose an approach to dealing with these factors, whether by way of a support retainer, warranty period, service subscription, or something else.
Where To Next?
The best place to start is to consider your ideal workflow and figure out what methodology suits your organisation. Then work out which activities in each phase would bring the most value to your project.
Are you more of an agile business, or do you require the structure, documentation, and formality of waterfall? Or something in between?
Knowing where you stand ahead of time will greatly help you narrow down a shortlist of web development agencies. It will also help get buy-in from your team and reduce the potential for friction with your hired developers.
If all of this seems a bit much, after all you’re almost certainly slammed with your own pile of stuff to do already, then you may want to consider delegating to a digital project manager well versed in these processes.
Even if your web development agency provides their own project management services, having your own in-house asset provides a clear conduit through which all project-related information is managed, and who can filter the important business-critical information to you as needed. Having one contact for your web agency also helps reduce any scope blowouts that may come from having multiple members of your team all requesting different requirements in conflict with each other. This ultimately compromises the end result.
Although this can seem to be a lot to take in all at once, we would be happy to run through all of this with you in easy-to-understand language over a coffee or on a call.
- Different methodologies suit different needs. Are you agile, waterfall, or a mix of both? Success is a team sport and you need to be playing by the same rules as your developers.
- Underdone processes are just as harmful as over-engineered ones, so make sure your web development agency’s process scales to your requirements. You wouldn’t cook toast with a flamethrower, or slow-cook a meal with a lighter.
- Great developers are flexible and can tailor a web development process to your specific circumstances. Be wary of methodology puritans – just because they have a hammer doesn’t mean every problem is a nail.
- Your developers should be able to answer “How did you come up with this estimate”? If they have followed a structured process their answer should be more detailed and evidence-based than “we guessed, from experience.” Ask to see past website projects that they’ve delivered, and if possible, get information around the costs and time frames for those.
- Don’t just take our word for it. The more you learn about web development processes, the better armed you are to see “behind the curtain” and judge a web development agency and their process on their merits.
Last but not least, we here at Living Online are more than happy to catch up and have a chat about your project and our processes, and trust that we can tailor an approach to your specific needs.