6 Keys to Implementing Salesforce B2C at Scale
The story is not new. The project team has designed, implemented, tested, and deployed a custom solution to production that meets a crucial business need. Business stakeholders are happy as the time-to-market was good and sales are increasing every day. The marketing team plans to advertise on TV during a highly rated event.
But when the event comes…
The whole solution becomes unstable. You lose sales. Customer service is overwhelmed. Consumers complain on social media. Business stakeholders want to know who or what is responsible for the failure. What went wrong? Well, probably many things. But how could you, the Salesforce architect, have avoided these issues?
One way to avoid scenarios like this is to apply the lessons learned on similar projects by your own team or other teams who have gone before you on large-scale B2C projects. By learning these lessons now and applying them on your next project, you can avoid the problems faced by teams that continue to learn these lessons the hard way.
The customer story
The lessons outlined in this post are based on my experiences at a continental scale media company that uses Salesforce to sell streaming services subscriptions to their consumers. Consumers can subscribe using a web browser or mobile app; the first sales flow uses Customer Community and the second uses custom Apex web services.
The solution was apparently stable, until a sales campaign launched and we started to see “Concurrent requests limit exceeded” “TotalRequests Limit exceeded.” “AsyncApexExecutions Limit exceeded.”
The media company worked with Salesforce and the integration partner to solve the situation. A success manager started an Expert Coaching session for an Org Health assessment, and engaged developer support to help analyze and debug error messages and provide recommendations. A program architect (me) mapped quick wins and the necessary mid- and long-term enhancements, while the integration partner started implementing my findings.
The required enhancements impacted code, architecture, and the development lifecycle process. They took months to implement, and in fact, some of them are still in the backlog. Hitting most Salesforce limits is no longer a concern, but some still are and those are being monitored. When advertising a new sales campaign, the company now knows which metrics to look at and what to do in case performance is affected.
The customer is now in the process of implementing a headless architecture, gaining development flexibility, enhancing reliability, and enabling rich user experiences. They are also building a Center of Excellence with the end goal of standardizing best practices, enhancing quality, and improving time-to-market. Here are the lessons I learned during this project.
1. Mind what’s not functional
When in a hurry to deliver new business features, it is common to focus on the functional aspects of the user story. For large-scale scenarios, though, performance, scalability, and other non-functional requirements are equally important. You must ask questions to figure out what’s behind the project’s functional requirements. And, as my fifth-grade teacher would say, there are no dumb questions. I’ve found two questions that are key for assessing performance and B2C scalability requirements:
- How many consumers might access the platform simultaneously to perform an action? Consider all business process steps, for example, browsing offers, adding items to the cart, and checking out.
- What transactions are triggered by these processes? Examples here include synchronizing pricing data, applying discounts, sending emails, and calculating next best actions.
Aside from these two fundamental questions, there are many others that may be specific to your business or architecture. Could there be background processes running during the peak times of concurrent access? Do different business processes have different peak times, or do they run at the the same time? Are consumers starting the transactions through the same interface or through different interfaces?
Finding these numbers to answers these questions can be a challenge for IT teams. Ask the business teams for data from previous campaigns or for an estimate of the target market. In many cases, no single source has the answer. You might need help from business and technical teams, and you may need to dig through lots of data to find the answers.
2. Know your limits
Salesforce architects must learn the platform limits and consider them during solution design. Make sure you understand what limits your solution could hit, both today and in the future. From daily API calls to concurrent synchronous long-running transactions to maximum number of community users, there are many angles to cover. Estimate the usage of these limited resources and track how much of each is currently in use.
When a resource is not available, consider the pros and cons of alternate approaches. If asynchronous transactions are an option, they can be a good fit as limits are higher. If what you need is an increase in allocations, you can try buying add-on licenses. This is often the case, for instance, for API requests, platform events, and platform cache. For exceptional cases, some limits can be temporarily increased by Salesforce Customer Support.
Keep in mind that Salesforce lets you exceed some limits without being blocked. Other limits block the related resource until usage decreases. The documentation on each limit describes whether Salesforce allows exceeding it by some margin or not. Add to your solution monitoring the limits, alarms in case of issues, and a recovery plan.
3. Know all your tools, and use the right one for each task
The Salesforce platform offers many capabilities to support your design, and more are released every quarter. Be sure to take to time research what’s available before deciding on a design.
- Avoid mixing automation techniques in the same object. If that’s not possible, mind the order of execution. For instance, a field update action in a Process Builder will cause all of an object’s triggers to re-execute.
- When processing speed is key, Apex triggers are preferable to Process Builder. Triggers also make it easier to avoid recursion by referencing a global static variable in the Apex context. The Architect’s Guide to Record Triggered Automation has the latest best practices published by the Salesforce Platform Team.
- Optimize your data design and how you use it. Make use of the Query Plan tool, Platform Cache, Skinny Tables, and Indexes.
- Understand the limits of the many different integration capabilities Salesforce offers. These capabilities act on different levels (user interface, application logic, or data model) and they have different timing (synchronous or asynchronous as well as real time, near-real time, or batch). They also have (guess what) different limits.
4. Ensure quality from the beginning
Quality begins with you. And the first quality assurance step is reviewing the architectural solution. Ask other experienced architects, developers, or quality assurance engineers to review it. Even better: ask all of them.
Then, move on to the coding. Bad code can break even the most scalable architecture. (I know you’ve seen SOQL queries inside loops in production). Look for bad coding practices from the beginning by implementing a peer review process. This not only helps uncover issues earlier, it also helps spread coding best practices throughout the team.
Use the non-functional requirements you identified to plan and execute performance testing. You have to plan in advance as Salesforce needs to approve your test plan. Create scenarios with both average and peak load (for example, the largest expected size of input or number of records). Create scenarios for peak concurrent access (for example, maximum expected simultaneous transactions). This will help identify not only runtime limits that can be hit, but also uncover errors that are not easily found. In particular, “Unable To Lock Row — record currently unavailable” errors require attention. Combined with a high volume of transactions, record locking issues can cause excessive concurrent long-running processes. These are both bad for customer experience and limited by the platform. At worst, transactions waiting for a lock to be released can overload the instance’s database, and a throttle can be applied to your organization.
5. Keep the platform healthy
As you deploy new features to your organization and gain more customers, your solution changes. The amount of data increases. Processes get more complex. More concurrent consumers use your system.
As a result, you are using more of the available resources. Transactions that were instantaneous now make your customer wait longer.
To ensure your system is not a victim of your own success, you should regularly assess its health. Tools such as Salesforce Optimizer helps in this regard, as do other diagnostics and monitoring tools for Salesforce.
Some of the resources you’ll want to govern are:
- Data volume. A data archival strategy helps avoid performance deterioration and data storage-related costs. Old information about abandoned shopping carts, custom logs, chat transcripts, and so on should be considered for archival or deletion. Keep in Salesforce only the data that is used in daily transactions, and store “cold” data off-platform.
- Performance. As the complexity of your code and volume of data increases, the performance of your solution may suffer. You should collect performance data during tests (or use a monitoring tool) and take action to remediate performance issues. Event Monitoring can help in this regard providing information about execution of Apex triggers, APIs, and user activities. It also has a friendly Analytics app powered by Tableau CRM, which enables you to view these KPIs easily.
- Exceptions. In a headless architecture, exceptions can be hidden by inappropriate exception handling. Configure email addresses to receive emails when your Apex code encounters unhandled exceptions, or when a process or flow interview fails.
- Release updates. Salesforce periodically releases updates to improve performance, security, and usability. Take time to review and activate release updates to gain these benefits and make sure your customization is not affected.
These governance processes are usually under responsibility of the Salesforce administrator. On larger implementations, a “Center of Excellence” can be in place to manage different aspects of the platform health.
What characteristics and processes should you govern? The answer varies by implementation, and there is no one-size-fits-all list. Every new feature deployed could bring a new dimension to the table. Pay attention to transactions executed on a large scale as they might reach platform limits. An increase of 100ms in runtime might not mean much for a single customer, but it could impact your solution for 10,000 simultaneous customers.
6. Maintain organizational alignment
You might be saying (or hearing from your manager), “Applying these lessons will take a lot of effort”. That’s probably right. But consider how many sales you’ll lose if your customers can’t access your platform after that ad runs on TV? And, even worse, how it will impact your brand image? Do the math, and you’ll see this effort pays for itself. You’ll be able to convince yourself and your manager.
Further, these recommendations can bring long-term value in different ways. Begin estimating the volume of accesses during a campaign and you cannot only test load, but also calculate return on investment. Put in place monitoring capabilities and you can increase mean time between failures. Build a Center of Excellence and it can increase quality, reuse, innovation, and more.
It is also common for Salesforce customers to dislike limits. You may have heard, “I need Salesforce to give me more memory/CPU/database/other infrastructure capacity”. As an architect, you must know that adding resources is not a sustainable way to fix a solution with exponential complexity. This is true in Salesforce and any other platform. Thinking about limits — no matter the deployment platform — helps us architect better solutions.
Finally, organizational alignment and sponsor buy-in is key for success. A complex project is more likely to succeed when all stakeholders understand the need of quality processes and governance. Many quality issues found in code and design are due to the company not placing a high priority on quality from the start.
Conclusion
Large scale back-end B2C implementations require attention to dimensions that sometimes go overlooked. This post covered hard-earned lessons that you can apply when architecting these solutions for Salesforce.
To learn more about scale on Salesforce, check out the following Trailhead modules:
Have questions? Want to share other experiences? Feel free to comment in this post or reach me on LinkedIn.
